At MILLIMAN, I architected the migration of a SQL Server warehouse to an Azure Databricks lakehouse, reducing costs and processing time. I used medallion architecture with Delta Lake and Delta Sharing.
I developed production pipelines with automated monitoring and SLA compliance, and optimized Spark jobs with caching and partitioning to improve ETL performance by 40%+.
I built a Python converter for T-SQL to Spark SQL that accelerated migration and reduced errors. In a project, the converter reduced manual effort by 70% and accelerated the migration timeline by six months.
I also implemented validation frameworks, established CI/CD pipelines and branching practices, and mentored junior engineers. My work includes secure data exchange through Delta Sharing and cloud cost optimization through Spark job refactoring and storage lifecycle policies.
