Open to opportunities

Vivek Shah

@vivek_shah

Message

Senior Data Engineer specializing in Azure Databricks, scalable ELT pipelines and cost optimization.

India

Message

What I'm looking for

I seek senior data engineering roles where I can architect cloud-based lakehouse solutions, optimize Spark/Databricks performance, lead migrations, and drive automation and cost-efficiency while mentoring teams.

I am a Senior Data Engineer with 7+ years of experience focused on Azure Databricks, Azure Data Factory, and cloud data engineering. I build scalable ELT/ETL pipelines and optimize Spark performance to deliver reliable, high-performance data platforms.

At Nokia, I led an enterprise-wide migration to Databricks Unity Catalog and implemented role-based access control across workspaces, improving compliance and audit readiness. I architected Delta Lake lakehouse solutions with medallion patterns and engineered SQL Warehouse infrastructure to accelerate analytics and reporting.

I drive cost and performance improvements through cluster optimization, auto-scaling, memory tuning, partition strategies, and CI/CD automation using Azure DevOps and Git. My work has reduced infrastructure and cloud spend through proactive pausing, reserved capacity planning, and compute optimizations.

I have a track record of large-scale data onboarding, legacy migrations (IBM Netezza to Databricks), implementing SCD Type 2, CDC, and building reusable PySpark libraries and data validation frameworks. I mentor teams, establish best practices, and focus on delivering production-grade, observable data pipelines.

Experience

Work history, roles, and key accomplishments

Current

Senior Data Engineer

Current

Nokia

Mar 2022 - Present (3 years 10 months)

Led enterprise migration to Databricks Unity Catalog and implemented Delta Lake medallion architecture, improving governance and analytics performance while reducing compute and infrastructure costs by ~25%.

Databricks Delta Lake Unity Catalog Pyspark Azure Data Factory Spark Optimization CI CD Azure DevOps

Data Engineer

Fragma Data Systems

Jul 2021 - Mar 2022 (8 months)

Orchestrated Netezza-to-Databricks migration and built ELT pipelines with PySpark and Delta tables enabling incremental loads and SCD Type 2 history for time-series analytics.

Pyspark Databricks Netezza Delta Lake Azure Data Factory ELT Data Validation

Software Engineer II

MAQ Software

May 2018 - Jul 2021 (3 years 2 months)

Developed end-to-end analytics solutions using Azure Data Factory, Databricks, SQL Server and Power BI, migrated legacy ETL to cloud and optimized analytical queries reducing execution time by 20-40%.

Azure Data Factory Databricks SQL Server Power BI ETL Python

Education

Degrees, certifications, and relevant coursework

V.V.P. Engineering College

Bachelor of Engineering, Information Technology

Grade: CGPA: 8.03

Bachelor of Engineering in Information Technology from V.V.P. Engineering College under Gujarat Technological University, graduated June 2017 with a CGPA of 8.03.