Raj Gandhi

I’m a Data Engineer with 4 years of experience designing and operating scalable batch and real-time data pipelines. I build solutions end-to-end—modeling, enrichment, and reliable data delivery—so analytics teams can trust their numbers.

In my current role, I designed and implemented a low-latency real-time sessionization system using Redis, Lua scripts, and Java, achieving ~300 ms P99 latency. I also provisioned and operated infrastructure with CloudFormation, ensured consistency through atomic Redis updates, and validated correctness using data parity checks and monitoring/alarms.

I’ve led data privacy and compliance work by removing sensitive data from datasets owned across multiple teams. I collaborated with privacy partners to analyze sources and deliver a centralized sanitized dataset for downstream consumption, ensuring policy compliance at scale.

Earlier, I built large-scale Spark ETL pipelines and optimized terabyte-scale processing by tuning joins and shuffle/memory behavior to improve performance and reliability. I also migrated data from GCP to AWS using PySpark (200–300 GB/day) and automated workflows with Python, AWS Lambda, and AWS Glue—plus delivered stakeholder-facing QuickSight KPI dashboards and a Streamlit reporting web app.