Scott Chen
@scottchen4
I’m a Senior Data Engineer building scalable batch and streaming lakehouse pipelines for trusted analytics.
United StatesWhat I'm looking for
I’m a Senior Data Engineer who has engineered scalable batch and streaming pipelines on Uber’s Apache Hudi Data Lake using Spark, Kafka, and Airflow—supporting rider demand, driver supply, pricing, and operational analytics. I designed reusable ETL workflows that ingest trip, dispatch, payment, incentive, and driver activity data into curated Hudi datasets consumed across Product Analytics, Finance, Marketplace Engineering, and Machine Learning teams.
Across my roles, I’ve implemented incremental data processing with Change Data Capture (CDC), schema evolution, partition management, and automated backfill strategies—reducing pipeline execution time by 40% and compute costs by 25%. I also built an automated data quality framework for freshness, completeness, reconciliation, anomaly detection, and SLA monitoring, and earlier supported analytics and adoption reporting using Spark and SQL at the Unified Analytics Platform (UAP) and during Azure Databricks launches.
Experience
Work history, roles, and key accomplishments
Engineered scalable batch and streaming pipelines on Uber’s Apache Hudi data lake using Spark, Kafka, and Airflow to support marketplace analytics. Designed reusable ETL workflows into curated Hudi datasets consumed by Product Analytics, Finance, Marketplace Engineering, and Machine Learning teams.
Delivered Azure Databricks pipelines supporting Eli Lilly’s Enterprise Serialization and Manufacturing Digital Thread, integrating ERP, MES, laboratory, and quality systems into a governed analytics platform. Built metadata-driven ingestion to Delta Lake and implemented data quality, reconciliation, audit validation, and governance controls aligned with GMP/FDA/serialization compliance requirement
Education
Degrees, certifications, and relevant coursework
Texas State University
Bachelor's degree in Computer Science, Computer Science
2012 - 2016
Earned a Bachelor's degree in Computer Science at Texas State University from 2012 to 2016.
Tech stack
Software and tools used professionally
Availability
Location
United StatesAuthorized to work in
Job categories
Skills
Interested in hiring Scott?
You can contact Scott and 90k+ other talented remote workers on Himalayas.
Message ScottGet matched with your dream remote job
Sign up now and join over 250,000+ remote workers who receive personalized job alerts, curated job matches, and more for free!
