At Shell, I design end-to-end ETL pipelines on Azure Databricks using Medallion Architecture, PySpark, Delta Lake, and ADLS Gen2. These pipelines process more than 5 million records per run.
I built a reusable black-box ETL testing framework with PySpark and SQL, including synthetic test data generation and automated output validation. The team adopted it as its standard pipeline testing blueprint.
I migrated a source system pipeline from initial load to delta mode, reducing execution time from 9 hours to 2 hours. I also built a GitHub Copilot agent in VS Code to automate converting more than 80 Scala notebooks to Python during a Unity Catalog migration.
Before moving into data engineering, I worked at Shell as a Security & Compliance Analyst, assessing SAP PaPM risks and implementing risk-mitigation controls. I hold an ISTQB Certified Tester – Foundation Level (CTFL) certification.
