Harsh Kadam

I’m a Data Engineer with almost 3+ years of experience designing, developing, and maintaining robust ETL pipelines. I focus on optimizing data workflows to ensure seamless data integration across platforms and delivering high-performance solutions.

At LTIMindtree, I designed end-to-end ETL pipelines using Azure Data Factory to ingest data from CSV, SQL Server, and REST APIs into ADLS Gen2 and Azure Databricks. I build dynamic, reusable ADF pipelines with parameterization for multi-environment support, and I develop scalable transformations using PySpark following the Medallion Architecture (Bronze, Silver, Gold).

I also create and manage Delta Lake tables with support for ACID transactions, schema enforcement/evolution, and time travel. To handle historical and current state tracking, I implemented SCD Type 1 & 2 logic using Delta Merge operations, while using Unity Catalog for centralized governance, access control, and lineage tracking.

I tune and automate workloads for performance and reliability—optimizing Spark jobs (partition pruning, broadcast joins, AQE, caching, and more), applying Delta Lake optimizations (OPTIMIZE with ZORDER, VACUUM), and scheduling notebooks and JAR-based executions with retry policies. I’m also committed to data security and compliance through RBAC, encryption, and Unity Catalog policies.