parth Shah

I’m an Azure Data Engineer with 4+ years of experience building scalable data solutions on Azure Cloud. At Tredence Analytics, I led migration of ADF and Airflow pipelines to Databricks using Unity Catalog, strengthening governance and access management while improving lineage and reliability with monitoring, logging, and validation mechanisms.

I design end-to-end ETL/ELT flows: I built Azure Data Factory pipelines to ingest from REST APIs, Azure Blob Storage, and on-premise databases into the Bronze layer, then used Databricks and PySpark for cleansing and transformation into the Silver layer. I also built optimized Gold layer datasets for analytics/reporting and reduced processing time by 30% using Delta Lake optimization techniques like partitioning and Z-Ordering. Previously at TCS, I developed re-runnable, fault-tolerant ADF pipelines with alerting, optimized PySpark/Spark SQL jobs for large-scale processing, maintained fact/analytics tables with less than 0.1% variance from legacy systems, and delivered production support through data profiling, cleansing, and validation.