Rohit Dubey

I am a Data Engineer with 2+ years of hands-on experience delivering faster data processing via PySpark and SparkSQL on Databricks, with a strong focus on ETL design, performance tuning, and building scalable, highly available pipelines.

At my current role I orchestrated an incremental ingestion framework in Azure Data Factory into a Databricks Medallion Architecture, using PySpark, Unity Catalog, Azure Key Vault, and Azure SQL Database, achieving a 40% reduction in runtime and complexity.

Previously, I migrated a 50TB enterprise data warehouse to Databricks, ensured regulatory compliance using Delta Lake and Unity Catalog, and boosted query performance by 35% through advanced Spark optimization techniques while automating end-to-end workflows with Airflow and Databricks Workflows.

I implement CI/CD with Azure DevOps, perform unit and functional testing on Databricks notebooks, integrate external APIs, and build KPI-driven Gold datasets for BI and ML consumption. I thrive in Agile teams where I can optimize pipelines, improve data quality, and support analytics and ML initiatives.