At Tata Consultancy Services, I developed end-to-end ETL pipelines for a banking data migration, moving data from on-premise SQL Server into Azure Data Lake Storage Gen2. I implemented Bronze, Silver, and Gold layers with Azure Databricks and Delta Lake.
I built PySpark notebooks for cleansing, transformation, validation, and aggregation, and configured Databricks Jobs to run through Azure Data Factory pipelines. I also implemented incremental loading and Spark optimization techniques.
Earlier, at Narayani Engineering & Software Technologies, I assisted with Azure Data Factory pipelines, PySpark transformations, and Delta Lake tables. I supported pipeline monitoring, testing, deployment, and production activities.
