At Tata Consultancy Services, I migrated legacy ETL workflows to Azure Databricks, rebuilding data layers with PySpark and SQL as part of a data warehousing modernization effort.
I rebuilt Bronze notebooks to ingest CSV and Parquet data from ADLS Gen2 and Azure SQL Server, and created Standard Views for column standardization and datatype changes.
I migrated Silver transformations using Delta Lake MERGE for incremental updates and full overwrites for batch refreshes. I also implemented Auto Loader for incremental ingestion and to prevent duplicate file processing.
In my System Engineer (Azure Data Engineer) role, I engineered pipelines from ADLS Gen2 and Azure SQL Server to Power BI-ready Gold tables. I managed Azure DevOps CI/CD releases and used Databricks Workflows, Azure Key Vault, and centralized logging and email alerts to support execution and monitoring.
