At Tech Mahindra, I design and implement scalable Azure Data Factory pipelines that ingest data from multiple source systems into ADLS Gen2. I build Bronze-Silver-Gold Medallion architectures with Azure Databricks and Delta Lake, delivering structured, analytics-ready datasets.
I develop PySpark transformations, implement SCD Type 1 and Type 2 logic through Delta Lake MERGE operations, and improve Spark performance using partitioning, caching, OPTIMIZE, and Z-ORDER. I also build dependency-driven execution and data-quality checks covering row counts, nulls, schemas, and rejected-record tracking.
