At Wells Fargo, I design Spark jobs and Python production pipelines that process over 5 TB of daily transactional data. I improved pipeline throughput by 40% while maintaining communication with stakeholders.
I implemented data quality frameworks and data lineage instrumentation using SparkSQL, Airflow, and Monte Carlo monitoring, reducing production data errors by 30%. I also developed schema designs and dimensional data models for a cloud data lake.
At American Express, I engineered Spark and Python pipelines for financial transactions and developed streaming pipelines with Kafka, Spark Structured Streaming, and Kinesis for near-real-time transaction monitoring. Earlier, at HSBC Bank, I worked on ETL pipelines, data quality frameworks, and a migration of production pipelines to Databricks.
