Open to opportunities

Shuo Zheng

@shuozheng

Message

Senior Data Engineer specializing in scalable data platforms, lakehouses, and real-time ML pipelines.

United States

Message

What I'm looking for

I seek a senior data engineering role building scalable data platforms and lakehouses, collaborating with ML and product teams, emphasizing data quality, cost optimization, and mentorship.

I am a Senior Data Engineer with 9+ years designing and scaling modern data platforms at top tech companies including Pinterest, DiDi, Google, and Amazon.

I build high-performance pipelines and lakehouse architectures using Databricks, dbt, Apache Spark, Kafka, and Airflow on AWS and GCP, focusing on ML-ready datasets and real-time analytics.

My work has reduced compute costs, cut latency, and accelerated ML cycles—examples include a 30% compute cost reduction, 40% latency reduction with medallion pipelines, and automated data quality across 300+ pipelines.

I mentor engineers, drive data governance and cost optimization, and partner with product and data science teams to enable reliable, self-serve analytics at scale.

Experience

Work history, roles, and key accomplishments

Current

Senior Data Engineer

Current

Apr 2020 - Present (6 years 2 months)

Led migration to a Spark platform on AWS EKS, cutting compute costs by 30% and architected a Databricks Delta Lake lakehouse that reduced latency by 40% and enabled self-serve analytics for 200+ users.

Databricks Delta Lake Spark Kafka DBT Great Expectations Airflow AWS Streaming

Senior Data Engineer

DiDi

Mar 2018 - Apr 2020 (2 years 1 month)

Built real-time trip analytics pipelines ingesting 100M+ events/day with Kafka and Spark, improved ETL performance 2× and reduced production incidents by 35% through automated quality checks.

Kafka Spark DBT Hive Presto HDFS S3 Airflow Streaming

Data Engineer

Google

Jul 2016 - Mar 2018 (1 year 8 months)

Built unified analytics pipelines using Dataflow (Beam) and BigQuery processing billions of daily events, improving query speed by 40% and saving ~$500K annually via optimization.

Dataflow Apache Beam BigQuery Pub Sub Cloud Composer GCP)ETL Partitioning GCP

Data Analyst

Amazon

Feb 2015 - Jul 2016 (1 year 5 months)

Automated reporting pipelines with Python, Lambda, and Redshift, cutting manual effort by 80% and delivering analyses that contributed to ~$2M annual savings and a 10% retention lift.

Python AWS Lambda Redshift SQL ETL Dashboards Reporting Root Cause Analysis