Looking for a job

Shubham gawande

@shubhamgawande

Message

I build scalable cloud data pipelines and Lakehouse architectures.

India

Message

What I'm looking for

I'm looking to build reliable, scalable cloud data platforms and Lakehouse solutions, collaborating with analytics and data science teams on governed, high-quality datasets and automated delivery pipelines.

At NeenOpal, I engineer production cloud data pipelines for multi-terabyte datasets using PySpark, SQL, Azure Data Factory, and Azure Databricks.

I improved reliability by building monitoring, data-quality checks, and alerts that reduced pipeline failures by about 40%. I also optimized Delta Lake storage and query performance, reducing cloud storage costs by 25%.

Previously at Infinity Data Technologies, I built Spark-based data lakes handling 50M+ daily records, developed batch ETL pipelines, and delivered data cleansing and QA workflows for retail, e-commerce, and insurance reporting.

Experience

Work history, roles, and key accomplishments

Current

Data Engineer

Current

NeenOpal Inc.

Feb 2025 - Present (1 year 7 months)

Engineered and maintained scalable production cloud data pipelines processing multi-terabyte datasets using PySpark, SQL, Azure Data Factory, and Azure Databricks. Built automated monitoring, data quality validation checks, and alert notifications using Databricks Workflows, reducing pipeline job failure rates by ~40% and improving operational reliability.

PySpark SQL Azure Data Factory Azure Databricks Databricks Workflows Delta Lake Unity Catalog Microsoft Fabric Data Factory OneLake Azure DevOps CI CD

Data Engineer

Infinity Data Technologies Private Limited

Jul 2021 - Feb 2025 (3 years 7 months)

Architected a Spark-based Data Lake integrated with Azure ADLS Gen2, handling over 50M+ daily records to support scalable, low-latency analytics. Developed and optimized batch ETL pipelines in Python/PySpark, standardizing raw unstructured/semi-structured data into clean, versioned Delta Lake tables.