ESWAR REDDY User

I’m an Azure Data Engineer with 5+ years of experience designing, developing, and optimizing large-scale data platforms on Microsoft Azure. I focus on building robust batch and near real-time pipelines that keep analytics trustworthy and fast.

In my recent role, I developed 20+ Azure Data Factory and Databricks pipelines to ingest and transform 50M+ records daily from SQL Server, CSV, JSON, and flat files into ADLS Gen2 and Synapse Analytics. Using Medallion Architecture with Delta Lake, I supported scalable, ACID-compliant data pipelines and improved data freshness from daily to hourly.

I consistently improve performance and reliability: I optimized PySpark using AQE, broadcast joins, partition pruning, caching, and skew mitigation—reducing Databricks compute costs by 30% and cutting ETL runtimes by up to 40%. I also implemented incremental loading with CDC and SCD Type 1/Type 2 logic to maintain historical datasets and dimensional models.

I care deeply about governance, data quality, and smooth delivery. I built data quality frameworks (schema validation, reconciliation checks, null handling, duplicate detection) to reach 99.8% accuracy, secured credentials with Azure Key Vault and role-based access controls, and used Unity Catalog for centralized governance and lineage. I’ve also automated deployments with CI/CD and orchestrated workflows using Apache Airflow and ADF, with monitoring and alerting to proactively reduce incidents.