Open to opportunities

Shubham Shah

@shubhamshah1

Message

Senior Data Engineer specializing in cloud-native, scalable ETL/ELT and large-scale healthcare data platforms.

United States

Message

What I'm looking for

I seek a senior data engineering role building cloud-native, secure, and scalable ETL/streaming platforms, with emphasis on analytics enablement and compliance.

I am a Senior Data Engineer with 7+ years designing and deploying cloud-native data solutions across AWS, Azure, and GCP for healthcare, financial services, and retail. I build scalable ETL/ELT pipelines, enterprise data warehouses, and streaming architectures that support mission-critical analytics and regulatory compliance.

My work includes engineering 100+ ETL pipelines processing 10TB+ daily, implementing Delta Lake and Medallion architectures, and orchestrating reliable 24/7 workflows with strong observability and security controls (HIPAA, SOC2, PCI-DSS). I have improved data quality, reduced latency from hours to seconds, and scaled processing while cutting infrastructure costs.

I bring hands-on expertise with Databricks, PySpark, Spark Streaming, Airflow, Kafka, Snowflake, Redshift, Azure Synapse, Terraform, and containerized deployments, and I focus on enabling self-service analytics, robust data governance, and production-grade ML/AI integrations.

Experience

Work history, roles, and key accomplishments

Current

Senior Data Engineer

Current

Change Healthcare

Jun 2023 - Present (2 years 7 months)

Architected and optimized 100+ ETL pipelines processing 10TB+ healthcare data daily, improving data quality 40% and reducing analytics latency from hours to seconds via hybrid batch/real-time ingestion and streaming technologies.

Pyspark Scala Databricks Kafka AWS Glue EMR Delta Lake Apache Airflow Terraform

Data Engineer

Capital One

Aug 2021 - May 2023 (1 year 9 months)

Designed Azure-based ETL workflows and Medallion architecture on ADLS Gen2, improving data reuse 30% and delivering 99.95% pipeline reliability with sub-second streaming ingestion for live risk dashboards.

Azure Data Factory Pyspark Azure ADLS Gen2 Azure Event Hubs Synapse Terraform

ETL Developer

Signify Health

Mar 2020 - Jul 2021 (1 year 4 months)

Built Spark and Hadoop ETL workflows ingesting 50+ healthcare feeds, accelerating daily refreshes from 4 hours to 45 minutes and achieving 98%+ processing reliability through incremental loads and data contracts.

Hadoop Hive Pyspark Sqoop Oozie Kafka SQL Python Data Quality

Python Developer

Best Buy

Jul 2018 - Dec 2019 (1 year 5 months)

Developed Python automation and ETL scripts for inventory and POS data, achieving 100% accuracy in quarterly audits and reducing manual reporting effort by 70% across 500+ locations.