At Deloitte, I built PySpark pipelines on Azure Databricks processing 20M+ records for IFRS 17 financial reporting. I also optimized Azure Data Factory copy activities moving data from PostgreSQL to Oracle, tuning parallelism and batch settings to address throughput bottlenecks.
I focus on Spark performance tuning, including data skew, shuffle optimization, and cluster memory configuration. At Deloitte, I also implemented PostgreSQL range partitioning and analyzed Azure DBU costs to identify excess spend.
