# Dipesh Chaudhary

Dipesh Chaudhary is a remote worker on Himalayas, Data Engineer at Celebal Technologies Pvt. Ltd., based in India.

Source: https://himalayas.app/@dipeshchaudhary1

```text
Data Engineer at Celebal Technologies building real-time data lakes and pipelines for analytics.
```

| Detail | Value |
| --- | --- |
| Availability | Looking for a job |
| Location | India |
| Authorized to work in | India |
| Salary expectations | 1490k+ INR |
| Job categories | Data Engineer |
| Skills | Data Engineering, ETL, Real time Data Processing, Data Lake Architecture, Data Warehouse, Data Migration, Apache Spark, PySpark, Spark Structured Streaming, Spark SQL, Databricks, Delta Lake, Kafka, Azure Data Factory, AWS Glue |
| LinkedIn | https://linkedin.com/in/dipesh-chaudhary-a89596136 |

## About

```text
At Celebal Technologies Pvt. Ltd., I designed a real-time ingestion pipeline with PySpark Structured Streaming on Databricks, bringing API and Kafka data into Delta Lake. I also developed incremental data solutions for more than 15 TB to support analytics and Data Science model training and fine-tuning.

At CloudGritz Technologies Pvt. Ltd., I developed ETL pipelines to migrate more than 200 SQL Server tables to Amazon Redshift, processing over 2 TB of daily incremental data. I also worked on a Databricks real-time data lake, optimizing query performance and strengthening pipeline reliability.
```

## Experience

### Data Engineer (current)

Celebal Technologies Pvt. Ltd.. Mar 2025 - Present

```text
Designed and implemented a scalable real-time data ingestion pipeline using PySpark Structured Streaming on Databricks, integrating data from APIs and Kafka into Delta Lake. Developed robust API ingestion logic with retry handling, pagination, and error management, and implemented Databricks Asset Bundles for automated CI/CD deployment.
```

Skills: Azure-Data-Factory, Kafka, Databricks, PySpark, Spark-Structured-Streaming, Spark-SQL, Delta-Lake, CI-CD, Azure-DevOps

### Data Engineer

CloudGritz Technologies Pvt. Ltd.. May 2022 - Mar 2025

```text
Designed and developed ETL pipelines using AWS Glue and PySpark to migrate and transform 200+ SQL Server tables to Amazon Redshift, processing 2TB+ of daily incremental data. Optimized Spark configurations and Redshift table structures, reducing ETL runtime by 40% and improving query performance by 40%.
```

Skills: AWS-Glue, S3, Athena, DynamoDB, IAM, Cloudwatch, CloudFormation, Redshift, PySpark, Python, SQL, GitHub, Spinnaker, Datadog, Kafka, Databricks, Jenkins

## Education

### Amity School of Engineering and Technology

Bachelor of Technology, Electronics and Communication Engineering. 2014 - 2018

```text
Pursued a Bachelor of Technology in Electronics and Communication Engineering from 2014 to 2018.
```

## Tech stack

- Amazon Redshift
- Apache Spark
- AWS Glue
- AWS IAM
- Microsoft Azure
- Amazon S3
- GitHub
- Jenkins
- GitHub Actions
- PySpark
- MySQL
- Databricks
- AWS CloudFormation
- Azure DevOps
- Python
- Kafka
- Amazon DynamoDB
- Linux
- Datadog
- Amazon Athena
- SQL
- Spinnaker
- Delta Lake

To contact Dipesh, message them from this profile with a Himalayas employer account: [Hire on Himalayas](https://himalayas.app/hire).

## People also viewed

- [Aryan Singh](https://himalayas.app/@aryansingh5): Data Engineer, Impetus
- [Kalash Jindal](https://himalayas.app/@kalashjindal): Senior Data Engineer, NASDAQ
- [HARSHAVARDHAN M](https://himalayas.app/@harshavardhanm): Data Engineer, Celebal Technologies
- [rajin User](https://himalayas.app/@rajinuser): Specialist Data Engineer, LTIMindtree
- [Gatik Misra](https://himalayas.app/@gatikmisra): Data Engineer, Accenture Technologies
- [Madhan Lingareddy](https://himalayas.app/@madhanreddy575): Senior Data Engineer, Procore Technologies
- [Satyansh User](https://himalayas.app/@satyanshuser): Data Engineer, Memorae.ai
- [Aditi Vaidya](https://himalayas.app/@aditivaidya): Senior Data Engineer, Konverge.AI
- [Nayan Goyal](https://himalayas.app/@nayangoyal): Data Engineer, Celebal Technologies
- [satyansh shrivastava](https://himalayas.app/@satyanshshrivastava): Data Engineer, Memorae.ai
