Shyam Patidar

I’m a Module Lead Data Engineer with 6.5+ years of end-to-end experience architecting and scaling enterprise Lakehouse platforms on Azure Databricks. I progressed from Data Engineer to Lead within 2.5 years, leading 8+ member cross-functional teams across architecture, pipeline delivery, governance, and production ML/GenAI integration.

In my current role (GFIP), I built an enterprise Lakehouse using Medallion Architecture (Bronze/Silver/Gold), unifying IoT telemetry and logistics feeds to cut reporting latency from 24 hours to 1 hour. I delivered a 40% query performance gain through Spark tuning (Z-Ordering, broadcast joins, AQE), built DLT ELT pipelines with automated quality checks and SCD Type 2, and reduced pipeline maintenance by 30%.

I’m deeply hands-on in streaming and governance—event-driven ingestion with Azure Event Grid and Databricks Autoloader, plus enterprise governance via Unity Catalog (RBAC, PII masking, row-level security, audit logging). I also deploy production ML/GenAI solutions: XGBoost anomaly detection (85% accuracy) and RAG using Azure OpenAI GPT-4 on governed data with compliance controls, achieving 3–4% operational savings and reducing analyst investigation time by 30%.