At Capgemini, I designed and deployed near real-time ingestion pipelines that processed 150M+ semi-structured records from REST APIs and blob storage into ADLS Gen2.
I developed data validation checks that reduced manual data quality issues by 45% and improved data accuracy to 98%. I also built weekly REST API ingestion pipelines with a 99.9% data delivery SLA.
I implemented a Delta Lake Lakehouse with Bronze, Silver, and Gold layers, including SCD Type 1 and Type 2 dimension merges and incremental fact-table upserts for a Gold-layer star schema.
I built real-time ingestion with Azure Event Hubs and Databricks Structured Streaming, using checkpoint-based exactly-once processing. I also optimized Spark jobs and ADF pipelines, reducing processing time by 25%.
