Gautam Kumar Gupta User

I’m a Data Engineer with 3.3 years of experience in enterprise data warehousing for the insurance (BFSI) domain. I specialize in designing and building scalable batch and real-time data pipelines using Azure Databricks, PySpark, and Azure Data Factory.

In my current role, I develop end-to-end ETL pipelines that ingest and process insurance data across 7+ lines of business into a unified enterprise data platform. I implement CDC with SCD Type 2 for historical tracking and efficient incremental loads, and I optimize Spark jobs using partition pruning, caching, and broadcast joins to improve performance by 30–40%.

I architect lakehouse pipelines with Medallion Architecture (Bronze → Silver → Gold) using Azure Databricks and Delta Lake, and I migrate legacy batch workflows to Delta Live Tables to reduce manual intervention in incremental processing by ~20–30%. I also lead Duck Creek Claims and Policy migration, ensuring data quality, integrity, and compliance with insurance industry standards.

I strengthen data reliability through Unity Catalog governance across 50+ tables and datasets, and I apply data quality practices such as Great Expectations. Before specializing in engineering, I delivered remote freelance data engineering/data analysis work with Python- and SQL-based ETL pipelines and validation checks—backed by an operations background that shaped my focus on accuracy and disciplined processes.