JOSEPH TANG

I’m a Staff Data Engineer at Databricks, focused on enterprise-grade lakehouse and streaming platforms that power petabyte-scale analytics and ML workloads. I architected the Enterprise Lakehouse Platform (ELP), enabling multi-tenant data access across 50+ enterprise customers while reducing data latency by 45%.

I designed real-time ingestion and feature pipelines with my “StreamHub” system, using Spark Structured Streaming and Kafka to support sub-second availability for downstream ML inference and analytics by 20%. I also orchestrated Delta Lake + Spark lakehouse pipelines across massive datasets, improving pipeline throughput by 3x and reducing compute cost by 35%.

I drive governance, reliability, and operational excellence through Unity Catalog-based controls, including fine-grained RBAC and data lineage, plus a data quality and observability framework with anomaly detection and alerting that reduced data incidents by 60%. Across multi-cloud environments, I implement scalable medallion architectures, optimize PySpark performance, and build reproducible infrastructure with Terraform, CI/CD, and orchestration to support production ML/LLM workflows.