jairaj jagarapu

I’m a results-driven Data Engineer with 4+ years of experience designing and delivering enterprise-grade data platforms on AWS, Databricks, and Snowflake. I focus on building pipelines that are reliable in production, measurable in performance, and ready for analytics at scale.

In my current role at Capgemini, I engineered a fault-tolerant AWS ingestion platform integrating 10+ databases, REST APIs, and third-party sources into a centralized cloud data lake. I redesigned Apache Airflow DAG architecture with SLA alerts, retry logic, and dead-letter queue handling—reducing pipeline failures by 50% and improving reliability for daily workflows.

I optimize for both speed and cost: I tuned PySpark executor memory allocation, shuffle partitions, and broadcast join thresholds on Databricks for multi-terabyte workloads within strict SLA windows. I also implemented watermark-based CDC incremental ingestion to eliminate costly full-table scans, improving processing turnaround by 40% while reducing AWS Glue compute costs.

I also build AI-ready data infrastructure, including RAG pipelines, vector database workflows, and LLM embedding orchestration using LangChain and OpenAI API. I’ve contributed semantic search initiatives using LangChain and Pinecone embedding workflows to improve contextual retrieval quality across internal knowledge platforms.