Open to opportunities

Muskan Yadav

@muskanyadav1

Message

I’m a Senior Data Engineer building reliable, real-time data pipelines with Python, SQL, PySpark, and Azure.

India

Message

What I'm looking for

I’m looking for a team where I can own scalable Azure-based data pipelines—especially real-time streaming and reliable ETL—with strong engineering practices, data quality, and performance optimization as core priorities.

I’m an experienced Data Engineer (3+ years) designing, developing, and optimizing data pipelines end to end. I work across batch and streaming using Python, SQL, PySpark, and Azure services, with a strong focus on ETL development, data modeling, query performance, and data quality.

In my most recent role, I designed scalable SCM data pipelines with PySpark for 1 TB+ daily processing, added Apache Kafka for low-latency streaming ingestion, and orchestrated workflows using Apache Airflow DAGs. Previously, I improved Spark job performance by 40% through caching and join/partition optimizations, supported Delta Lake incremental loads and merges, and implemented data quality rules and audit logs to reach 90% data reliability—while also maintaining 99% SLA compliance through production monitoring.

Experience

Work history, roles, and key accomplishments

Current

Sr Data Engineer

Current

Clover Infotech Private Limited

Jan 2026 - Present (7 months)

Designed and developed scalable SCM data pipelines with PySpark, processing 1 TB+ of data daily with reliability and optimized performance. Implemented Kafka-based real-time ingestion and used Airflow to orchestrate batch and streaming ETL workflows with monitoring, retries, and alerting.

PySpark Kafka Airflow ETL Caching

Data Engineer

Datamatics Global Services Limited

Apr 2023 - May 2025 (2 years 1 month)

Built batch ETL pipelines in Azure using PySpark on Azure Databricks and transformations on data stored in ADLS via ADF and Spark SQL. Optimized Spark jobs and supported Delta Lake incremental loads/merge logic while implementing data quality rules, audit logs, alerting, and data validation checks.

PySpark Azure Databricks Caching Broadcast Joins Delta Lake Data Quality Audit Logs Alerting Data Validation

System Engineer

QodeNext India Private Limited

Jul 2022 - Apr 2023 (9 months)

Developed data ingestion pipelines using Azure Data Factory to load data from CSV files and REST APIs into ADLS and SQL Server. Created and optimized stored procedures, triggers, and UDFs, and monitored production pipelines to resolve failures while maintaining 99% SLA compliance.

Azure Data Factory SQL Server Stored Procedures Triggers Spark Job Troubleshooting