Pooja Suvarna

I’m a Data Engineer with 6 years of experience in Big Data Engineering, ETL/ELT pipeline development, and cloud data platform design. I focus on building scalable data pipelines, data warehousing solutions, and data migration workflows that perform well under real-world load.

In my recent work at Carelon Global Solutions, I designed and developed batch processing data pipelines on Amazon EMR using Apache Spark and Python. I built AWS Glue ETL jobs with PySpark to move data from Amazon S3 into curated datasets, leveraging the AWS Glue Data Catalog for metadata and schema discovery.

I’ve also strengthened reliability and speed by optimizing distributed processing—tuning Spark configurations, partitioning strategies, shuffle behavior, and memory allocation to resolve bottlenecks. I’ve implemented Snowflake bulk-loading with data validation, access control, and performance tuning, and automated ingestion and orchestration with robust error handling, logging, and reusable modules.

Earlier, at LKQ India Private Limited and Bullfinch Software, I worked across end-to-end integration and migration, using Sqoop for large-scale migrations to Hadoop clusters and Hive/Spark for efficient querying and warehousing. I’ve consistently applied performance tuning techniques and workflow orchestration (including AWS Step Functions and Airflow) to reduce overhead and deliver measurable cost savings.