Open to opportunities

Aditya Narayan Padhi

@adityanarayanpadhi

Message

Lead Data Engineer building cloud-scale, observability-driven data platforms and GenAI-enabled pipelines.

India

Message

What I'm looking for

I seek senior roles building resilient, scalable data platforms and GenAI solutions where I can lead teams, improve observability, and deliver production-grade ML/AI pipelines.

I am a Lead Data Engineer with 10+ years designing and delivering cloud-scale streaming and batch data platforms that enable analysts, data scientists, and ML/AI teams to ship experiments to production quickly and reliably.

I have led and mentored teams (recently a team of 5), built metadata-driven reusable platforms, migrated large systems to Azure with zero data loss, and implemented Medallion architectures and observability stacks that raised pipeline success rates to 98% while cutting pipeline build time by 80%.

Hands-on with Spark, Kafka, Airflow, Kubernetes, MLflow, vector stores and GenAI/RAG solutions, I focus on resilient, cost-efficient architectures, robust data quality, and product-aligned delivery that accelerates business outcomes.

Experience

Work history, roles, and key accomplishments

Lead Data Engineer

HTC Global Services

May 2023 - Aug 2025 (2 years 3 months)

Architected a metadata-driven, reusable data platform and led a team of 5 engineers to cut new-pipeline build time by 80%, raise pipeline success rate to 98%, and compress release cycles by ~70% while running 700+ production pipelines under SLAs.

Pyspark Kafka Airflow Kubernetes Delta Lake Azure MLFlow Prometheus Grafana

Senior Data Engineer

WalkingTree

Mar 2022 - Apr 2023 (1 year 1 month)

Architected and built large-scale real-time and batch data pipelines (700+) using PySpark, Kafka and Airflow to support low-latency power trading analytics and renewable energy optimization.

Databricks Pyspark Kafka Airflow Kubernetes Azure Delta Lake Python SQL

Senior Platform Engineer

Monsanto

May 2019 - Oct 2021 (2 years 5 months)

Developed and automated ML pipelines and containerized model deployments using PySpark, Kafka and Airflow, enabling scalable model execution and cross-platform SQL migrations to Redshift/BigQuery.

Pyspark Kafka Docker Airflow Jenkins Redshift BigQuery SQL Kubernetes

Software Engineer

Accenture

Mar 2015 - Apr 2018 (3 years 1 month)

Provided production support and automation using Python, SQL and Splunk, improving operational efficiency and reducing manual effort across multiple client applications.

Python SQL Splunk Automation Shell Scripting Support Monitoring

Education

Degrees, certifications, and relevant coursework

Liverpool John Moores University

Master of Science, Machine Learning & AI

2021 - 2024

Activities and societies: Master's thesis: AutoTrader — predictive trading models with end-to-end data pipeline.

Master of Science in Machine Learning & AI completed remotely with coursework and a thesis project focused on predictive trading and end-to-end data pipelines.