Neha Pandey
@nehapandey1
Senior Data Engineer specializing in Azure Databricks lakehouse delivery, streaming pipelines, and Delta Lake optimization.
IndiaWhat I'm looking for
I’m a results-driven Data Engineer with 10+ years architecting and delivering scalable Databricks lakehouse solutions on Azure. I own the full data engineering lifecycle—from ingestion to governed analytics—bringing deep expertise in PySpark, Spark Structured Streaming, and Delta Lake architecture.
At MetLife, I architected high-performance ETL on Databricks using PySpark and Azure Data Factory, building Delta Lake bronze/silver/gold layers on ADLS with partitioning, Z-ORDER indexing, and schema evolution. I engineered real-time ingestion from Azure Event Hub into Delta tables and authored Databricks notebooks to process 20+ heterogeneous sources reliably. I also built rule-driven data-quality validations, operationalized observability with Azure Monitor/Log Analytics, and reduced data ingestion time by 50% through Spark performance tuning and ADF integration.
Earlier at Accenture, I implemented Databricks Autoloader ingestion with schema inference/evolution and built streaming pipelines consuming Confluent Kafka with checkpointing and watermarking. I automated the full Databricks job lifecycle and CI/CD releases (Azure DevOps + Databricks REST API) across Dev/QA/UAT/Prod for zero-downtime deployments. I’m energized by Agile collaboration and translating complex requirements into maintainable, production-grade lakehouse pipelines.
Experience
Work history, roles, and key accomplishments
Architected and implemented high-performance ETL pipelines on Databricks using PySpark and Azure Data Factory, building a governed Delta Lake bronze/silver/gold architecture on ADLS. Engineered real-time ingestion from Azure Event Hub into Delta, adding data quality validations, observability with Azure Monitor/Log Analytics, and Spark performance optimizations.
Developed Databricks Autoloader ingestion pipelines and Spark Structured Streaming solutions, integrating data from ADLS and Confluent Kafka into Delta Lake with checkpointing and watermarking. Automated Databricks job lifecycle management and CI/CD releases using Databricks REST API and Azure DevOps across Dev/QA/UAT/Prod environments.
Built Azure Data Factory pipelines to ingest data from Reporter, SSO DW, SharePoint, and Azure Blob Storage into a centralized data lake, using incremental load strategies based on watermark and change tracking. Designed Star Schema models with SCD Type 2 dimensions, implemented transformations with SSIS and Azure SQL stored procedures, and managed consistent DEV-to-PROD deployment workflows.
Engineered Azure Data Factory pipelines for large-scale data migration from on-premise Netezza to Azure, ensuring data integrity through row-count and checksum reconciliation. Implemented transformations using Azure Data Lake Analytics with U-SQL and automated production dataset and pipeline deployments using PowerShell scripts.
Education
Degrees, certifications, and relevant coursework
Indira Gandhi Government Engineering College
Bachelor of Engineering (B.E.)
2011 - 2015
Completed a Bachelor of Engineering (B.E.) at Indira Gandhi Government Engineering College in Madhya Pradesh from July 2011 to May 2015.
Tech stack
Software and tools used professionally
Availability
Location
IndiaAuthorized to work in
Job categories
Skills
Interested in hiring Neha?
You can contact Neha and 90k+ other talented remote workers on Himalayas.
Message NehaGet matched with your dream remote job
Sign up now and join over 250,000+ remote workers who receive personalized job alerts, curated job matches, and more for free!
