Open to opportunities

Deepak kumar

@deepakkumar26

Message

Senior Data Engineer with expertise in ETL and cloud solutions.

India

Message

What I'm looking for

I am looking for a role that challenges my skills in data engineering and offers opportunities for professional growth in a collaborative environment.

As a Senior Data Engineer with over 5 years of experience, I specialize in data engineering, ETL pipelines, and cloud-based solutions. My proficiency in technologies like Snowflake, Apache Spark, and Python has enabled me to deliver impactful results in various projects. At Illumina, I led the migration from HVR-based replication to Spark-driven ETL pipelines, achieving a remarkable 35% cost reduction for onboarding data from SQL Server to Snowflake.

Previously, at EY, I developed a custom DAG orchestration framework that significantly improved task execution efficiency, resulting in annual cost savings of $200k. My experience at Tata Consultancy Services further honed my skills in creating cloud-agnostic frameworks and optimizing data ingestion processes. I am passionate about leveraging my technical expertise to drive data-driven decision-making and enhance operational efficiencies.

Experience

Work history, roles, and key accomplishments

Current

Senior Data Engineer

Current

Illumina

Jan 2025 - Present (6 months)

Migrated from HVR-based replication to Spark-driven ETL pipelines with Apache Iceberg tables on Amazon S3 and Polaris catalog, leveraging Dbt and Snowflake SQL for business transformations directly on Iceberg tables. Achieved 35% per-GB cost reduction for onboarding data from SQL Server to Snowflake by optimizing data pipeline architecture and storage.

Apache Spark Apache Iceberg Amazon S3 DBT ETL

Data Engineer (Senior Consultant)

Jul 2022 - Present (3 years)

Developed a Python-based custom DAG orchestration framework, enabling execution of complex tasks with runtime DAG restructuring and custom JSON encoder/decoder for efficient state management. Achieved $200k annual cost savings by improving task execution efficiency and engineered a centralized financial fact table with SCD Type 2 for NCCI reporting, optimizing data processing and historical tracki

Python fastAPI Snowflake

Data Engineer

Tata Consultancy Services

Aug 2020 - Present (4 years 11 months)

Created a Python-based, cloud-agnostic ELT framework for Snowflake, integrating data from 12 source systems (S3, SQL Server, Profisee) into modeled DataVault tables. Reduced data ingestion time by 40% through Connection Pool implementation, minimizing parallel database connections.

Python Snowflake AWS Serverless AWS Batch S3 Cloudwatch AWS CloudFormation

Education

Degrees, certifications, and relevant coursework

Institute of Engineering & Management

B.Tech., Computer Science

2016 - 2020

Completed a Bachelor of Technology in Computer Science. The curriculum covered core computer science principles and practical applications relevant to the field.