Anupam Srivastava

I’m a Data Engineer with 4+ years of experience building scalable batch and near real-time data pipelines using PySpark, SQL, AWS, and Snowflake. I focus on ETL optimization, dimensional modeling, and analytical dataset design that helps teams make data-driven decisions with confidence.

At Piramal Finance, I architected the migration of 10+ production ETL pipelines from Python to PySpark on AWS Glue, improving fault tolerance and reducing processing latency by 83% (from 2 hours to 20 minutes). I also built near real-time streaming pipelines leveraging Apache Kafka on AWS EMR, ingesting 2M+ events daily into Snowflake and feeding 5+ dashboards.

I’ve consistently improved reliability and efficiency—delivering $36K+ in annual cloud cost savings by profiling and optimizing Spark execution plans, refining partitioning strategies, and right-sizing AWS Glue clusters. I developed an LLM-powered explainability workflow using Amazon Bedrock (Claude 3.5 Sonnet) to automatically identify and explain incentive payout factors for 15,000+ employees.

I design pipelines with lineage, quality, and performance in mind, including a multi-stage incentive computation pipeline using Snowflake stored procedures with daily snapshot versioning and AWS Glue delivery workflow. I’ve mentored 2 engineers and reduced production incidents by 30% through validation checks, performance tuning, and close collaboration with business stakeholders.