Vivek Shah

I am a Senior Data Engineer with 7+ years of experience focused on Azure Databricks, Azure Data Factory, and cloud data engineering. I build scalable ELT/ETL pipelines and optimize Spark performance to deliver reliable, high-performance data platforms.

At Nokia, I led an enterprise-wide migration to Databricks Unity Catalog and implemented role-based access control across workspaces, improving compliance and audit readiness. I architected Delta Lake lakehouse solutions with medallion patterns and engineered SQL Warehouse infrastructure to accelerate analytics and reporting.

I drive cost and performance improvements through cluster optimization, auto-scaling, memory tuning, partition strategies, and CI/CD automation using Azure DevOps and Git. My work has reduced infrastructure and cloud spend through proactive pausing, reserved capacity planning, and compute optimizations.

I have a track record of large-scale data onboarding, legacy migrations (IBM Netezza to Databricks), implementing SCD Type 2, CDC, and building reusable PySpark libraries and data validation frameworks. I mentor teams, establish best practices, and focus on delivering production-grade, observable data pipelines.