Open to opportunities

Anurag Aileni

@anuragaileni

Data Engineer at Mckinsey & Company building Databricks pipelines processing over 5 TB of data daily.

At Mckinsey & Company, I design and operate Spark and Python pipelines in Databricks, including production pipelines that process over 5 TB daily.

I’ve developed and maintained more than 15 SparkSQL jobs using medallion architectures and incremental data modeling. I also created dimensional models and schema designs that improved query performance by 20%.

To strengthen data quality and traceability, I implemented frameworks and lineage instrumentation using Monte Carlo, reducing data issues by 30%. I’ve also optimized Spark and SQL pipelines to reduce runtime by 30% and cloud compute costs by 25%.

At Progrite System, I designed reporting data models and supported production pipelines monitored with Airflow and Monte Carlo. I also supported Kafka and Kinesis streaming ingestion for event-driven reporting.

Experience

Work history, roles, and key accomplishments

Current

Data Engineer

Current

Mckinsey & Company

May 2024 - Present (2 years 4 months)

Designed and operated end-to-end Spark and Python data pipelines in Databricks, ensuring high reliability and maintainability. Developed over 15 SparkSQL jobs using medallion architectures and incremental data modeling, and implemented data quality frameworks with Monte Carlo to reduce data issues by 30%.

Spark Python SQL Databricks SparkSQL Medallion Data Architecture Incremental Modeling Dimensional Data Models Schema Design Data Quality Framework Data Lineage Monte Carlo Airflow Data Lake Data Marts KPIs

Data Engineer

Progrite System

Aug 2023 - May 2024 (9 months)

Extracted and optimized complex SQL queries using Python automation, improving data retrieval performance by 30%. Designed dimensional data models and schemas to standardize reporting datasets, and implemented incremental data modeling and Airflow orchestration for automated daily data refreshes.