Amit Singh Negi

I’m a Data Engineer with 3.5+ years’ experience building scalable data pipelines using Azure (Databricks, Data Factory, Fabric) and PySpark. I focus on ETL/ELT design, data modelling, and reliable big data processing.

In my recent work on Microsoft Fabric, I designed and optimized end-to-end workflows (ingestion, transformation, view creation, orchestration) processing 100 GB/day and reducing latency by 30% for hospital and clinical datasets. I’ve also built config-driven dynamic pipelines, cutting development effort by 40%, and implemented automated data quality checks to reduce inconsistencies by 50%.

Previously, I migrated an on-premises SQL project to Azure Data Factory, Databricks, and ADLS, using PySpark optimization and incremental processing to reduce refresh time by 30%. I also built automated data validation in Databricks to cut monthly-report error rates by 40%, leveraging Fabric Dataflows Gen2 and OneLake to centralize access and eliminate duplication.

I pair engineering rigor with modern AI work—building a Retrieval-Augmented Generation (RAG) system and a multi-agent AI research assistant using LCEL. I ensure pipeline reliability with structured outputs, schema validation via Pydantic, and unit tests using Pytest, while staying grounded in strong fundamentals like Delta Lake, governance, and performance tuning.