Open to opportunities

Shivam Surya

@shivamsurya

Message

Senior Data Engineer who builds cost-efficient batch data pipelines, lakes, and warehouses on AWS.

India

Message

What I'm looking for

I’m looking to build and optimize batch data pipelines on AWS—focusing on scalable PySpark architectures, reliable Airflow orchestration, strong data quality, and measurable cost/runtime improvements in production environments.

I’m a Senior Data Engineer with 4+ years designing end-to-end batch data pipeline architectures, data lakes, and data warehouses across fintech, e-commerce, and healthcare. I specialize in PySpark, Apache Airflow, SQL, and AWS services, delivering production systems processing 15M+ records/day across 40+ global markets.

In my current role, I designed a sources-to-warehouse pipeline (10+ APIs → S3 Raw → PySpark/EMR → Hive/Redshift → BI) using a medallion pattern, migrated legacy Talend ETL to modular PySpark, and achieved 40% pipeline runtime reduction and 60% infrastructure cost savings. I also manage 50+ Airflow DAGs with SLA monitoring and failure alerting, and I implement automated data quality and schema drift detection for reliable operations.

Experience

Work history, roles, and key accomplishments

Current

Senior Data Engineer

Current

Nielsen

Sep 2024 - Present (1 year 10 months)

Designed an end-to-end batch pipeline architecture using the medallion pattern (Bronze/Silver/Gold) to process 15M+ records daily across 40+ global markets. Migrated legacy Talend ETL to PySpark, managed 50+ Airflow DAGs, and optimized Spark jobs to reduce pipeline runtime by 40% and infrastructure costs by 60%.

PySpark Apache Spark Amazon EMR Airflow Hive Amazon Redshift AWS Glue SQL ETL ELT Batch Processing Medallion Architecture Data Quality Data Governance Schema Drift Detection Observability

Data Engineer

InvoLead Services Pvt Ltd

Dec 2023 - Sep 2024 (9 months)

Built a healthcare analytics platform on AWS from scratch, processing 5,000+ clinical documents daily. Implemented Textract + Lambda + S3 extraction and PySpark/AWS Glue ETL into Redshift, reducing manual entry by 80% while maintaining HIPAA/GDPR compliance with encryption, RBAC, and audit logging.

AWS AWS Lambda S3 PySpark Apache Spark AWS Glue Amazon Redshift ETL Standardization Data Loading HIPAA GDPR RBAC Audit Logging

Data Engineer

Publicis Sapient

Jan 2022 - Dec 2023 (1 year 11 months)

Developed batch ETL pipelines for US fintech and global e-commerce, processing 500GB+ daily transaction data using PySpark and Airflow. Optimized Amazon Redshift performance (distribution keys, sort keys, vacuum) to reduce query time by 60% and containerized deployments using Docker.