shashank shukla

I’m a Data Engineer with 4 years of experience designing scalable ETL/ELT pipelines and ensuring high-quality, validated data at scale. I focus on optimizing large-scale data systems and maintaining data integrity across 2TB+ daily pipelines.

In freelance work, I executed end-to-end migration of 10TB+ data from SQL Server to Azure Synapse using Azure Data Factory and Azure Databricks across multiple operational domains. I built CDC-based incremental pipelines that land raw data as Parquet in the bronze layer, reducing data processing volume by ~60–70%.

I implement Medallion Architecture (Bronze/Silver/Gold) with scalable PySpark transformations, including deduplication, schema enforcement, null handling, and cross-domain joins—cutting downstream query time by ~25–40%. I also apply business rule validations and data quality frameworks across pipeline stages to ensure accuracy, completeness, and consistency.

Earlier, as a Junior Analyst (Data Engineer), I built production ETL pipelines on Azure Data Factory and Databricks for 2TB+ daily data and supported hybrid banking migrations to Azure Synapse with governance compliance. I designed incremental partition-based processing to reduce daily processing time by 30–35%, and created event-driven ELT pipelines on GCP using GCS notifications to reduce latency by ~60–70%.