Open to opportunities

Michael Chen

@michaelchen6

Message

Senior Data Engineer building real-time, low-latency data platforms on Kafka, Spark, Flink, and Snowflake.

United States

Message

What I'm looking for

I’m looking to build and optimize real-time, low-latency data platforms—partnering with product teams to improve reliability, cost, and analytics speed with strong streaming resilience and governance.

I’m a Senior Data Engineer focused on designing fault-tolerant, event-driven data systems that turn massive streams into reliable business signals. At DoorDash, I architected a real-time event platform processing 50B+ events/day.

I enabled near-real-time pipelines for order lifecycle, delivery tracking, and pricing, helping teams make operational decisions with data latency reduced from 6–12 hours to under 5 minutes. I also designed and scaled ELT workflows in Snowflake using dbt across 100+ datasets to standardize transformations and reduce ad-hoc dependencies.

I engineer streaming pipelines with DLQ, replay, and checkpointing mechanisms, reducing production data incidents by ~35% and improving recovery time. I built a self-serve data platform (API + reusable templates) adopted by 20+ engineering teams, cutting onboarding time from days to under 2 hours.

I’ve also strengthened analytics and governance foundations, including a centralized metrics layer (dbt + Snowflake) and a data validation framework to improve audit readiness. Earlier, I delivered the HealthLake Analytics Platform using Spark and Databricks under HIPAA compliance, accelerating data delivery speed by 3–4x.

Experience

Work history, roles, and key accomplishments

Current

Senior Data Engineer

Current

DoorDash

Mar 2022 - Present (4 years 4 months)

Architected a real-time event platform using Kafka and Spark/Flink, processing 50B+ events/day and reducing end-to-end data latency from 6–12 hours to under 5 minutes. Built and scaled Snowflake (dbt) ELT pipelines and streaming infrastructure (DLQ/replay) to power order lifecycle, delivery tracking, and pricing, improving reliability and reducing incidents by ~35%.

Kafka Apache Spark Flink Python SQL Scala Snowflake DBT ELT Airflow Event Driven Architecture CDC Streaming Pipelines S3 EC2 IAM Azure Data Factory Databricks Docker Kubernetes Terraform GitHub Actions Clustering

Data Engineer

Optum

Aug 2018 - Mar 2022 (3 years 7 months)

Delivered the HealthLake Analytics Platform using Spark and Databricks to support large-scale ETL for healthcare claims and clinical data under HIPAA compliance. Improved data delivery speed by ~3–4x through incremental ETL and Spark optimizations, enabling near real-time ingestion and strengthening data governance with validation, lineage, and audit logging.

Spark Databricks Delta Lake HIPAA ETL Data Quality Data Validation Data Lineage Audit Logging Contracts Kafka Near Real Time Ingestion

Data Analyst

Databricks

Oct 2016 - Aug 2018 (1 year 10 months)

Created a SparkInsight telemetry platform using PySpark and Spark SQL, implementing batch ETL pipelines for product usage analytics. Improved processing efficiency through incremental aggregation and partition optimization and reduced compute usage via Spark execution plan analysis and performance tuning.

Pyspark Telemetry Incremental Aggregation Partitioning Spark Execution Plan Analysis Performance Tuning

Education

Degrees, certifications, and relevant coursework

Databricks

Databricks Data Analyst, Data Analytics

2016 - 2018

Completed the Databricks Data Analyst program (10/2016–08/2018), creating a SparkInsight telemetry platform with PySpark and Spark SQL to build batch ETL pipelines for product usage analytics.