Joy Miao

I’m a hands-on Senior Data Engineer with 6+ years designing, building, and operating production-grade cloud data platforms. I focus on distributed batch and streaming pipelines that deliver analytics-ready data products. My strength is turning complex source data into governed, reusable datasets that scale across teams.

At Bell Mobility, I architected a reusable, configuration-driven Source-to-Raw ingestion framework on GCP, standardizing production ingestion from RDBMS and Salesforce Bulk API sources using Dataflow and Serverless Dataproc/PySpark. I also delivered a reusable GCS-to-BigQuery Raw-to-Standardized framework that standardized column names, data types, null handling, and timestamps, reducing tenant onboarding from multiple weeks to approximately one week.

I led a versioned data contract capability used by 10+ tenant teams, with CI-based static validation of tenant-authored YAML contracts, Cloud Functions registration in Firestore, and runtime enforcement within the Raw-to-Standardized pipeline. For high-volume Salesforce and RDBMS sources, I evaluated and delivered CDC patterns covering incremental ingestion, watermark management, deduplication, recovery, and full-reload paths—balancing latency, scalability, cloud cost, and operational resilience. I raised team engineering standards through mentoring and technical design/code reviews across Spark and SQL performance, reusable component design, failure handling, and production readiness.

Earlier, I migrated 5+ years of telecom data from Hadoop and Amazon S3-based environments to GCS and BigQuery, automating backfills supporting approximately 2 TB/day of raw CDR binary data and 3.5 TB/day of decoded JSON, and reducing GCS storage costs by 89%. I designed and operated event-driven streaming pipelines with Spark Structured Streaming and Confluent Kafka for near-real-time telecom telemetry, plus a Dataplex-based metadata and governance solution that reduced configuration by approximately two-thirds. I also built centralized observability with Logstash/Elasticsearch/Kibana and delivered Terraform changes through CI/CD workflows to keep production releases dependable.