# Gunj Desai

Gunj Desai is a remote worker on Himalayas, Engineering Manager at HackerRank, based in India.

Source: https://himalayas.app/@gunjdesai

```text
Data engineering leader specializing in real-time pipelines, Spark, Kafka and cost optimization.
```

| Detail | Value |
| --- | --- |
| Availability | Open to opportunities |
| Location | India |
| Authorized to work in | India |
| Job categories | Data Engineer, Data Engineering Manager, Staff Engineer, Principal Engineer, Solutions Architect, Backend Engineer, Big Data Engineer, Live Streaming Engineer, Data Platform Engineer, Data Engineering, Lead Data Engineer |
| Skills | React, Data Engineering, Apache Spark, Apache Hudi, Debezium, S3, Embeddings, Scala, Python, JavaScript, NodeJS, Django, MySQL, TiDB, Elasticsearch, Trino, Presto, Hadoop, Hive, Airflow, Apache Druid, Docker, Ansible, Akka, Bloom Filter Implementation, Click Stream Data, OLAP, Blockchain, Druid, Ethereum, GitHub, GitHub Actions, Kafka, SQL, Go |

## What I'm looking for

```text
I am looking for senior/data‑platform roles where I can lead engineering teams to build real‑time, cost‑optimized data systems, focusing on scalability, low latency and measurable business impact.
```

## About

```text
I am a data engineering leader with 12+ years building full‑stack and data‑centric products, focused on scalable, real‑time systems. I specialize in Kafka, Apache Hudi, Spark, vector embeddings and building ACID capabilities on S3.

I have designed and led platforms that ingest billions of events weekly and support multi‑million daily throughput with low latency, including a streaming platform averaging 3s latency and pipelines handling 2k messages/sec across 100+ tables.

My achievements include moving a warehouse off Redshift to an OpenLakehouse saving ~40% cost, building a near real‑time export delivering 50+ table joins in under 5 seconds, and a Text‑to‑SQL solution using vector embeddings across 50+ tables.

I manage and mentor engineering teams, drive cost‑optimized architectures, and build production systems for analytics, clickstream, blockchain event parsing and recommendation platforms to deliver high‑performance, reliable data products.
```

## Experience

### Engineering Manager (current)

[HackerRank](https://himalayas.app/companies/hackerrank). Dec 2023 - Present

```text
Core contributor and Engineering Manager overhauled the warehouse and redefined big data pipelines, moving Warehouse away from Redshift to an OpenLakehouse and reducing warehouse costs by ~40% while enabling near-real-time exports across 50+ tables in under 5 seconds.
```

Skills: Data-Engineering, Apache-Hudi, Spark, Debezium, S3, ETL, Kafka

### Staff Engineer

ngram. Feb 2023 - Dec 2023

```text
Built pipelines to parse Ethereum events, transactions and traces at scale (≈3.9 TB) and designed ABI deduplication to reduce storage duplication, plus export services for automated client data delivery.
```

Skills: Apache-Spark, ABI-Parsing, ETL, Scala, Python

### Solutions Architect

Doubtnut. Apr 2021 - Feb 2023

```text
Core contributor for a near-real-time video recommendation and analytics platform; built ACID capabilities on S3 using Debezium, Kafka and Hudi to support 100+ table pipelines with 45s latency and improved engagement via smart aggregates.
```

Skills: Kafka, Apache-Hudi, Debezium, Recommendations, Spark, Bloom-Filters, S3

### Senior Data Engineer

[Razorpay](https://himalayas.app/companies/razorpay). Dec 2019 - Apr 2021

```text
Built a near-real-time aggregation platform returning personalized payment options under 100ms, revamped events platform (≈1TB/day) reducing errors by 40%, and set up an OLAP store for petabyte-scale low-latency queries.
```

Skills: Real-Time-Aggregation, OLAP, Event-Stitching, Data-Engineering, Spark, Kafka, Performance-Optimization

### Assistant Manager

BookMyShow. Mar 2016 - Dec 2019

```text
Led Big Data and Clickstream platform and PWA engineering; built a clickstream pipeline ingesting 2M+ events per 5 minutes, increased mobile web conversions by 80% and improved initial 2G load times to 3.1s.
```

Skills: PWA, Big-Data, Kafka, Akka, Performance-Optimization, Web-Analytics, Spark

### Software Engineer

Shipler. Jan 2015 - Feb 2016

```text
Developed website and CMS features supporting e-commerce and platform operations as part of core engineering team.
```

Skills: Web-Development, CMS, NodeJS, JavaScript, Backend-Development

### Software Engineer

Plancess. Apr 2014 - Jan 2015

```text
Implemented e-commerce and edtech platform features contributing to product development and release cycles.
```

Skills: Edtech, Backend-Development, CMS, JavaScript

### Software Engineer

Mofirst Solutions. Nov 2013 - Apr 2014

```text
Worked on CMS development and integration for client projects, delivering web features and content management capabilities.
```

Skills: CMS, Web-Development, PHP, Frontend, Backend

## Education

### Mithibai College, Mumbai

Bachelor of Science, Science. Grade: Magna Cum Laude

```text
Bachelor of Science degree awarded with Magna Cum Laude distinction from Mithibai College, Mumbai.
```

## Tech stack

- Apache Spark
- Druid
- Blockchain
- Ethereum
- GitHub
- GitHub Actions
- Debezium
- DB
- MySQL
- Hadoop
- Django
- JavaScript
- Kafka
- Elasticsearch
- Ansible
- SQL
- Akka
- Trino
- Apache Hudi
- TiDB

To contact Gunj, message them from this profile with a Himalayas employer account: [Hire on Himalayas](https://himalayas.app/hire).

## People also viewed

- [Ambuj Kumar](https://himalayas.app/@ambujkumar1): Senior Data Engineer, British Petroleum
- [Arun K C](https://himalayas.app/@arunkc1): Open Source Contributor, naas.ai
- [Pritish Kamath](https://himalayas.app/@pritishkamath): Technical Lead, Bridgenext Inc
- [Raj Gandhi](https://himalayas.app/@rajgandhi): Data Engineer I, Amazon
- [Abin Bijo](https://himalayas.app/@abinbijo): Member of Technical Staff III, SMC Global Securities Ltd
- [Madhan Lingareddy](https://himalayas.app/@madhanreddy575): Senior Data Engineer, Procore Technologies
- [Jitesh Mogre](https://himalayas.app/@jiteshmogre): Staff Data Platform Engineer, GumGum
- [Wameen Ahmad](https://himalayas.app/@wameenahmad): Data Engineer Lead, Cisco
- [Abhijit Patra](https://himalayas.app/@abhijitpatra1): Lead Data Engineer, PayNearby
- [Arpan Samariya](https://himalayas.app/@arpansamariya): Data Engineer III, Zepto
