Open to opportunities

Shiv Kumar

@shivkumar2

Message

Data engineer specializing in PySpark, Azure Databricks, and cost-efficient, high-performance pipelines.

India

Message

What I'm looking for

I seek roles where I can design scalable, cost-efficient data platforms, optimize Spark performance, and collaborate with business stakeholders to deliver reliable analytics and measurable cost savings.

I am a Data Engineer with 5+ years of experience building and optimizing enterprise-grade data platforms using PySpark, SQL, and Azure Databricks.

I design Medallion architectures, implement Delta Lake best practices, and tune Spark (memory, joins, AQE) to deliver reliable analytics for global stakeholders.

My work has reduced job completion times by 40% through skew mitigation, cut cloud costs via minimal-scan strategies, and ensured 100% data accuracy with reconciliation and monitoring.

I collaborate across business and engineering, translating finance and marketing needs into scalable data models while maintaining strong observability, modular ETL pipelines, and cost-efficient cluster sizing.

Experience

Work history, roles, and key accomplishments

Current

Data Engineer

Current

Adidas

Apr 2023 - Present (2 years 8 months)

Designed Medallion Architecture and end-to-end Azure Databricks pipelines, resolving extreme data skew and improving job completion times by 40% while enforcing Minimal-Scan strategies to reduce cloud costs.

Pyspark Python SQL Databricks Azure Data Factory Delta Lake Medallion Architecture

Data Engineer

Tata Consultancy Services

Sep 2020 - Apr 2023 (2 years 7 months)

Developed modular PySpark ETL and Databricks Jobs to process diverse sources, applying broadcast hints and partition tuning to improve query execution time by ~20% and increase pipeline observability.