# Nayan Goyal

Nayan Goyal is a remote worker on Himalayas, Data Engineer at Celebal Technologies, based in India.

Source: https://himalayas.app/@nayangoyal

```text
Data Engineer at Celebal Technologies who cut Databricks pipeline runtimes to 8–10 minutes and automated setup by 70%.
```

| Detail | Value |
| --- | --- |
| Availability | Looking for a job |
| Location | India |
| Authorized to work in | India |
| Job categories | Data Engineer |
| Skills | Data Engineering, Databricks, Apache Spark, PySpark, SparkSQL, DBT, ETL, ELT, REST API Integration, Databricks Workflows, Lakebridge, Airflow, Informatica, BigQuery |

## About

```text
At Celebal Technologies, I build data pipelines and integrations with Databricks, PySpark, and DBT. I automated multi-workflow creation from Lakebridge JSON configs, cutting manual pipeline setup effort by 70%.

I reduced pipeline runtimes from 30–60 minutes to 8–10 minutes by rewriting SQL patterns, applying broadcast joins, and using table persistence strategies. I also migrated 15+ Informatica workflows to Spark-native Databricks pipelines and moved Airflow-orchestrated DAGs to Databricks Workflows.

On my Databricks Historical Load & Reconciliation Utility project, I built orchestration tools for seven connector types and added live job-log streaming with Server-Sent Events. The utility automates source-to-target validation across five dimensions, and I tested it with 47 automated tests.
```

## Experience

### Data Engineer (current)

Celebal Technologies. Apr 2025 - Present

```text
Built a Databricks REST API orchestration utility that automated multi-workflow creation, cutting manual pipeline setup effort by 70%. Migrated 15+ legacy Informatica workflows to Spark-native Databricks pipelines and reduced pipeline runtime from 30-60 minutes to 8-10 minutes.
```

Skills: Databricks, PySpark, SparkSQL, REST-APIs, ETL, DBT, BigQuery, dataform, Airflow, SQL-Server, Snowflake, MongoDB, Python, SQL, Scala, JavaScript, TypeScript, C++, NodeJS, ExpressJS, Git, GitHub, Postman

## Education

### The LNM Institute of Information Technology

Bachelor of Technology, Computer Science Engineering. 2021 - 2025. Grade: 7.18 CGPA

```text
Pursued a Bachelor of Technology in Computer Science Engineering, achieving a CGPA of 7.18.
```

## Tech stack

- Postman
- Snowflake
- Apache Spark
- Dataform
- GitHub
- PySpark
- dbt
- MongoDB
- Node.js
- Databricks
- JavaScript
- Python
- C++
- Scala
- TypeScript
- Git
- Airflow
- SQL
- AWS

To contact Nayan, message them from this profile with a Himalayas employer account: [Hire on Himalayas](https://himalayas.app/hire).

## People also viewed

- [Abhishek Jha](https://himalayas.app/@abhishekjha2): Data Engineer I, Rakuten India
- [Hemant Bawane](https://himalayas.app/@hemantbawane1): Associate Data Engineer, Celebal Technologies
- [Riyanka Pal](https://himalayas.app/@riyankapal): Data Engineer, ZS Associates
- [Abhimanyu Gabhrani](https://himalayas.app/@abhimanyugabhrani): Data Engineer, Celebal Technologies
- [Satyansh User](https://himalayas.app/@satyanshuser): Data Engineer, Memorae.ai
- [Debjit Nandan](https://himalayas.app/@debjitnandan): Associate Data Engineer, Celebal Technologies
- [Lekhana Donthula](https://himalayas.app/@lekhanadonthula): Data Engineering Intern, Celebal Technologies
- [Dipesh Chaudhary](https://himalayas.app/@dipeshchaudhary1): Data Engineer, Celebal Technologies Pvt. Ltd.
- [Prateek Singh](https://himalayas.app/@prateeksingh10): Technical Lead, Celebal Technologies
- [satyansh shrivastava](https://himalayas.app/@satyanshshrivastava): Data Engineer, Memorae.ai
