Arshia Shahidiany

Designed and implemented robust alerting mechanisms for data pipelines using Azure Monitor and Logic Apps, enabling proactive issue identification and faster resolution. Performed detailed root cause analysis on pipeline failures, reducing downtime and improving operational reliability. Streamlined and optimized Databricks notebook code, leading to significant cost savings in cloud compute resourc

Successfully migrated current systems from Hive Metastore to Unity Catalog, ensuring reproducible code across enterprise workspace environments. Developed a CI/CD framework to streamline workflows, introduce automated testing, and establish testing and staging environments. Championed the adoption of proper software development practices to eliminate manual code transfers.

Created a data Lakehouse architecture within the Databricks platform to identify suitable tech roles for consultants in key markets. Processed and cleaned data from various third-party sources using SparkSQL and PySpark to derive meaningful insights. Developed a sophisticated analysis system to pinpoint in-demand tech roles, guiding business strategy and client satisfaction.

Created data pipelines to facilitate insightful dashboards and analytics tools for logistics trade and supply, enhancing distribution operations. Utilized Azure Databricks (PySpark, Spark SQL) for data processing and reverse engineering, storing processed data in a data warehouse. Developed and implemented conceptual and logical data models, ensuring accurate representation of business requirement

Addressed data quality concerns related to duplication by automating the identification and rectification process. Devised a meticulous solution involving advanced data processing, regular expressions, and Python code. Implemented sophisticated fuzzy matching algorithms, drastically reducing manual checks and optimizing resource utilization.