rahul lohia

I’m a Senior Data Engineer with 5+ years designing and optimizing production-scale data pipelines and big data platforms across travel and telecom domains. I focus on ETL/ELT engineering that improves reliability, performance, and operational clarity.

In my current role, I reduced a data platform’s monthly infrastructure cost from $85K to $4K (~95%, ~$972K/year saved) by rewriting Spark execution plans, right-sizing EMR clusters, and automating S3 storage lifecycle management. I’ve also owned production RCA and delivered permanent fixes, cutting Spark job compute and runtime by up to 80% through broadcast join tuning, partition optimization, and cluster-level configuration.

I build end-to-end pipelines with Apache Spark (Scala/PySpark) for both batch and real-time workloads. I orchestrate workflows via Apache Airflow and connect observability and alerting through Datadog, including SLA breach detection and anomaly monitoring to keep pipelines dependable.

My technical depth spans Kafka, Spark Structured Streaming, Hive, HBase, and AWS, plus data governance and quality through Collibra Data Quality (CDQ). I’ve delivered outcomes like sub-minute end-to-end latency for high-volume event streams and automated Kafka-vs-Hive reconciliation with HTML audit reporting to proactively detect data loss incidents.