Kumar Sonal

I’m a Data Engineer with 2 years of experience building scalable data pipelines and processing large datasets. I focus on ETL/ELT workflows, data ingestion pipelines, and pipeline automation that support BI, reporting, and enterprise decision-making.

In my current role, I designed and implemented Azure Data Factory and Databricks pipelines to integrate multiple structured and semi-structured sources into Azure Data Lake Storage, supporting a centralized Lakehouse architecture. I established Medallion Architecture (Bronze–Silver–Gold) with Delta Lake to enable incremental processing, schema evolution, and ACID transactions, delivering curated datasets for downstream analytics.

I also build distributed PySpark transformation pipelines for data cleansing and enrichment, and optimize Spark workloads using partitioning, broadcast joins, caching, and query tuning. I implement data quality validation, schema enforcement, logging, and monitoring, along with governance and lineage practices using Unity Catalog and related frameworks.