At Applab, I designed a Lakehouse data platform for QF_TMS using Azure Databricks, Delta Lake, and Bronze, Silver, and Gold layers. I developed pipelines from Azure Database for MySQL into ADLS Gen2 and applied Unity Catalog controls, including column-level security and dynamic data masking.
For Applab’s IoT use case, I built a real-time time-series pipeline with Kafka and PySpark Structured Streaming, handling late-arriving data and storing processed data in Delta Lake. I’ve also developed Azure analytics pipelines for QMRA and Python ETL workflows syncing REST API and PostgreSQL data, including delivery to AWS S3.
