Aarsh User

I’m a Data Engineer specializing in end-to-end ETL/ELT pipeline development and large-scale pipeline optimization across AWS and Azure. I translate business requirements into robust data solutions that keep downstream analytics fast and reliable.

At Innovaccer, I designed and optimized Python and PySpark ETL/ELT pipelines that load client EHR and claims data into a Databricks-backed Snowflake warehouse, processing 50M+ records weekly at multi-terabyte scale. I engineered SQL-based transformation layers in PostgreSQL and Snowflake—using indexing, partitioning, materialized views, and stored procedures—to reduce query latency for BI consumers.

I also built Change Data Capture (CDC) pipelines using AWS DMS for real-time incremental ingestion, eliminating full-load dependencies to ensure low-latency data availability. Performance improvements have been measurable, including reducing execution time from 9 hours to 2 hours for 30M records and cutting overall processing time by 30% through multithreading, parallel processing, and Spark partition tuning.

To improve data trust and operational efficiency, I authored Python-based validation and alerting frameworks that reduced manual QA effort by 40%, and I built AI-assisted automation tools and agentic workflows that converted multi-step validation into one-click executions—improving efficiency by 70%. My projects reinforce this focus on streaming reliability, fault-tolerant storage design, and analytics-ready modeling.