At Celebal Technologies, I architected and deployed production ETL pipelines on Azure Databricks and Delta Lake, processing 10M+ daily records and improving analytics data availability by 20%.
I integrated Azure Data Factory, Azure Data Lake, and Azure SQL Database to create dynamic, event-driven pipelines, eliminating 8+ hours per week of manual data movement. I also developed data validation and reconciliation frameworks that reduced pipeline error rates by 30%.
On my Metadata-Driven ELT Pipeline project, I created a framework using Azure Data Factory and PySpark to migrate 25+ Oracle data sources to Databricks Enterprise Data Lake. Its metadata control table enabled fully parameterized pipeline execution, while incremental loading reduced data transfer volumes by 60%.
I also implemented SCD Type 2 logic with ADF Mapping Data Flows and Delta Lake, and added reconciliation checks, integration testing, and audit logging. My work includes a publication on predictive maintenance and real-time monitoring of electric vehicles, published in IEEE TENCON 2024.
