Karmanya Verma

I’m a Data Engineer with 4 years of experience building and optimizing production-grade data pipelines, orchestration systems, and cloud infrastructure in the healthcare domain. I focus on ELT pipeline development, performance optimization, and platform engineering, with a strong bias toward measurable outcomes.

At UST Global, I led migration of an on-premise member identity resolution system to Databricks by designing a Master Member Index (MMI) to generate unique UUIDs for cross-system deduplication across fragmented datasets. I built a scalable state compliance pipeline using conditional masking based on record-level flags and state-specific rules, and I engineered high-volume transformations to generate structured JSON bundles (8–9M records) for API-ready downstream consumption. I also improved processing performance by 30% through optimized state lookup logic, and resolved critical Spark OOM failures by tuning executor configuration to stabilize pipelines without adding infrastructure cost.

I also drive platform and orchestration improvements across AWS and Databricks. I migrated 15+ Airflow DAGs to Databricks Workflows using Databricks Asset Bundles (DABs), reducing orchestration costs by 69% while keeping Airflow only for manual intervention steps. In parallel, I provisioned AWS and Databricks environments using CDK and Terraform IaC, handled AWS MWAA upgrades across 15 environments, ran performance benchmarking under 30+ concurrent workloads, and implemented a Databricks cost governance framework with compute dashboards and a three-tier SQL warehouse model mapped to Okta user groups.