Open to opportunities

Gunj Desai

@gunjdesai

Message

Data engineering leader specializing in real-time pipelines, Spark, Kafka and cost optimization.

India

Message

What I'm looking for

I am looking for senior/data‑platform roles where I can lead engineering teams to build real‑time, cost‑optimized data systems, focusing on scalability, low latency and measurable business impact.

I am a data engineering leader with 12+ years building full‑stack and data‑centric products, focused on scalable, real‑time systems. I specialize in Kafka, Apache Hudi, Spark, vector embeddings and building ACID capabilities on S3.

I have designed and led platforms that ingest billions of events weekly and support multi‑million daily throughput with low latency, including a streaming platform averaging 3s latency and pipelines handling 2k messages/sec across 100+ tables.

My achievements include moving a warehouse off Redshift to an OpenLakehouse saving ~40% cost, building a near real‑time export delivering 50+ table joins in under 5 seconds, and a Text‑to‑SQL solution using vector embeddings across 50+ tables.

I manage and mentor engineering teams, drive cost‑optimized architectures, and build production systems for analytics, clickstream, blockchain event parsing and recommendation platforms to deliver high‑performance, reliable data products.

Experience

Work history, roles, and key accomplishments

Current

Engineering Manager

Current

HackerRank

Dec 2023 - Present (1 year 11 months)

Core contributor and Engineering Manager overhauled the warehouse and redefined big data pipelines, moving Warehouse away from Redshift to an OpenLakehouse and reducing warehouse costs by ~40% while enabling near-real-time exports across 50+ tables in under 5 seconds.

Data Engineering Apache Kafka Apache Hudi Spark Debezium S3 ETL

Staff Engineer

ngram

Feb 2023 - Dec 2023 (10 months)

Built pipelines to parse Ethereum events, transactions and traces at scale (≈3.9 TB) and designed ABI deduplication to reduce storage duplication, plus export services for automated client data delivery.

Apache Spark ABI Parsing ETL Scala Python

Solutions Architect

Doubtnut

Apr 2021 - Feb 2023 (1 year 10 months)

Core contributor for a near-real-time video recommendation and analytics platform; built ACID capabilities on S3 using Debezium, Kafka and Hudi to support 100+ table pipelines with 45s latency and improved engagement via smart aggregates.

Apache Kafka Apache Hudi Debezium Recommendations Spark Bloom Filters S3

Senior Data Engineer

Razorpay

Dec 2019 - Apr 2021 (1 year 4 months)

Built a near-real-time aggregation platform returning personalized payment options under 100ms, revamped events platform (≈1TB/day) reducing errors by 40%, and set up an OLAP store for petabyte-scale low-latency queries.

Real Time Aggregation OLAP Event Stitching Data Engineering Spark Kafka Performance Optimization

Assistant Manager

BookMyShow

Mar 2016 - Dec 2019 (3 years 9 months)

Led Big Data and Clickstream platform and PWA engineering; built a clickstream pipeline ingesting 2M+ events per 5 minutes, increased mobile web conversions by 80% and improved initial 2G load times to 3.1s.