At Brillioz Infotech Pvt. Ltd., I develop Azure Data Factory pipelines that move data from Oracle databases and APIs into Azure Data Lake Storage Gen2. I use parameterization, schedules, and triggers to automate ingestion.
I build Silver and Gold pipelines in Azure Databricks with PySpark and Spark SQL, transforming data into curated Delta tables. My work includes cleansing, standardization, deduplication, joins, and business rules.
I validate data across sources and processing layers, and help keep production pipelines stable through monitoring, troubleshooting, error handling, and reruns. I’ve also optimized Spark workloads by improving joins and partitioning and reducing unnecessary shuffling.
At CDAC Pune, I worked on an EPF data processing initiative to make employee and financial datasets available through an Azure-based platform. I supported ingestion and prepared datasets for downstream use with SQL workflows, cleansing, and validation.
