Open to opportunities

Anurag Palanki

@anuragpalanki

Data Engineer at Informatik Data Solutions building Spark pipelines and optimizing Databricks workloads.

At Informatik Data Solutions, I engineered Spark-based production data pipelines in Databricks and improved transactional data accuracy. I also implemented SparkSQL medallion architecture with Airflow orchestration and lineage instrumentation.

At Byjus, I developed SQL queries and dimensional data models for recurring reports, and helped migrate production pipelines to a cloud data lake on Databricks. I hold Databricks Data Engineer Associate and Microsoft Fabric Analytics Engineer Associate certifications.

Experience

Work history, roles, and key accomplishments

Current

Data Engineer

Current

Informatik Data Solutions

Aug 2023 - Present (3 years 1 month)

Engineered over 50 Spark-based production data pipelines in Databricks, improving data accuracy and reducing processing errors by 30%. Implemented medallion architecture, Airflow orchestration, and monitoring dashboards to enhance reliability and reduce compute costs.

Spark SparkSQL Python SQL Databricks Airflow Data Lineage Data Quality Monitoring Dashboarding Kafka Spark Structured Streaming

Jr Data Engineer

Byjus

Jun 2020 - Jul 2021 (1 year 1 month)

Developed complex SQL queries and dimensional data models, reducing report runtime by 35% and supporting over 15 recurring reports. Migrated pipelines to cloud data lake architecture on Databricks, reducing batch job failures by 20%.

SQL Schema Design Dimensional Modeling Databricks Data Pipelines Data Quality Monitoring Troubleshooting