Rajas daryapurkar

I’m a Senior Software Engineer (Data) who builds end-to-end data platforms on Databricks, turning complex, multi-source data into reliable analytics. I specialize in ETL automation, data quality, and performance tuning with a strong focus on downstream usability.

At Persistent Systems, I built and automated an ETL pipeline on Databricks to process 10M+ medical claims records daily from 15+ hospital systems and insurance providers into a centralized Delta Lake. I implemented Medallion Architecture (Bronze → Silver → Gold) with schema enforcement and data validation, reducing data inconsistencies.

I also optimized Delta Lake tables using Z-ordering, vacuuming, and liquid clustering—improving query performance by 68% for downstream reporting and reducing storage costs by 35%. Further, I improved pipeline performance by 40% using Spark SQL optimization techniques, caching, and partitioning, while automating workflows with Databricks Jobs and Databricks Workflows.

I’m recognized with a Top Talent award for FY25 and have delivered analytics experiences like interactive dashboards for 300+ clinical and business analysts, reducing report generation time by 55%. On the side, I’ve built a GenAI-powered data analysis app with Streamlit and PandasAI for conversational insights from uploaded files.