Lalit Namdeo

I’m a Senior Data Engineer with 7+ years specializing in Databricks, PySpark, and SQL, building scalable, cloud-native data platforms using Databricks and Azure. I design metadata- driven architectures, optimize distributed data processing, and deliver enterprise-grade solutions across large-scale datasets. At Deloitte, I led end-to-end delivery, including a metadata-driven framework for dynamic onboarding of multiple data sources that significantly reduced manual pipeline effort.

I’ve built and optimized large-scale PySpark pipelines processing millions of enterprise datasets daily, and improved pipeline performance by more than 70% using Spark optimization techniques like partition tuning, broadcast joins, and caching. I implement production-grade orchestration with Databricks Workflows, enforce governance and fine-grained access control with Unity Catalog, and deliver ETL/ELT solutions using Matillion, Azure Data Factory, and Azure Synapse—while using AI-assisted engineering tools to accelerate development, debugging, and reviews.