Open to opportunities

Deepak Pal

@deepakpal3

Databricks-certified data engineer building metadata-driven lakehouse ingestion and governed ETL/ELT pipelines with PySpark and Delta Lake.

India

Message

What I'm looking for

I’m looking to build production-grade, governed lakehouse pipelines—metadata-driven ingestion, data quality frameworks, and performant Databricks workflows—where I can deliver reliable ETL/ELT at scale.

I build lakehouse solutions at Deloitte, designing config-driven, metadata-driven data object ingestion on Databricks that automates pipeline generation. I’ve optimized framework performance to cut runtime by 75%, deployed governed pipelines using Unity Catalog, and delivered 25 reusable objects via Databricks Workflows with zero open defects in SIT Round 1.

Before that, at Tredence, I developed 30+ batch and streaming workflows in Databricks with PySpark, Spark SQL, Autoloader, and Structured Streaming, processing 50+ TB across S3, Snowflake, and GCP. I also built a config-driven data quality framework (10+ validation checks), shipped dbt- and Snowflake-based transformations, and automated audit reporting to save 4–6 hours of manual effort weekly.

Experience

Work history, roles, and key accomplishments

Current

Data Engineer Consultant

Current

Deloitte

May 2025 - Present (1 year 3 months)

Designed and built a config-driven metadata-driven data object ingestion framework on Databricks, achieving 75% runtime reduction and delivering 25 common data objects with zero defects.

PySpark Delta Lake Delta Live Tables Unity Catalog Databricks Workflows Medallion Architecture CI CD Python SQL

Data Engineer

Tredence

Nov 2022 - May 2025 (2 years 6 months)

Developed 30+ batch and streaming workflows in Databricks using PySpark and medallion architecture, processing 50+ TB across cloud platforms. Built a config-driven Data Quality Framework improving data integrity by 30%.

PySpark Databricks Delta Lake Snowflake GCP DBT Data Quality Python

Application Development Analyst

Accenture

Oct 2019 - Oct 2022 (3 years)

Designed end-to-end data solutions for a healthcare client using Python, PySpark, Databricks, and GCP, improving data-processing efficiency by 20%. Migrated legacy COBOL workflows to PySpark, cutting processing times by 30%.

Python PySpark Databricks GCP BigQuery COBOL