Looking for a job

Dipesh Chaudhary

@dipeshchaudhary1

Message

Data Engineer at Celebal Technologies building real-time data lakes and pipelines for analytics.

India

Message

At Celebal Technologies Pvt. Ltd., I designed a real-time ingestion pipeline with PySpark Structured Streaming on Databricks, bringing API and Kafka data into Delta Lake. I also developed incremental data solutions for more than 15 TB to support analytics and Data Science model training and fine-tuning.

At CloudGritz Technologies Pvt. Ltd., I developed ETL pipelines to migrate more than 200 SQL Server tables to Amazon Redshift, processing over 2 TB of daily incremental data. I also worked on a Databricks real-time data lake, optimizing query performance and strengthening pipeline reliability.

Experience

Work history, roles, and key accomplishments

Current

Data Engineer

Current

Celebal Technologies Pvt. Ltd.

Mar 2025 - Present (1 year 6 months)

Designed and implemented a scalable real-time data ingestion pipeline using PySpark Structured Streaming on Databricks, integrating data from APIs and Kafka into Delta Lake. Developed robust API ingestion logic with retry handling, pagination, and error management, and implemented Databricks Asset Bundles for automated CI/CD deployment.

Azure Data Factory Kafka Databricks PySpark Spark Structured Streaming Spark SQL Delta Lake CI CD Azure DevOps

Data Engineer

CloudGritz Technologies Pvt. Ltd.

May 2022 - Mar 2025 (2 years 10 months)

Designed and developed ETL pipelines using AWS Glue and PySpark to migrate and transform 200+ SQL Server tables to Amazon Redshift, processing 2TB+ of daily incremental data. Optimized Spark configurations and Redshift table structures, reducing ETL runtime by 40% and improving query performance by 40%.