sham avaradi

I’m an Azure Data Engineer with about 3 years of experience delivering analytics-ready data through scalable, governed platforms. I focus on building robust ETL/ELT pipelines and ensuring data reliability from ingestion to trusted outputs for BI and reporting.

In my recent role, I developed scalable data transformation frameworks with Azure Databricks and PySpark to process healthcare claims at large volume. I implemented Medallion Architecture (Bronze, Silver, and Gold) with Delta Lake, built reusable parameterized Databricks notebooks, and created dimensional models for Fact Claims, Fact Premiums, and supporting dimensions.

I also strengthened performance and quality end-to-end. I wrote complex Spark SQL/SQL using joins, aggregations, CTEs, and window functions, then optimized Databricks workloads using partitioning, caching, Delta optimizations, broadcast joins, and cluster tuning. To keep data trustworthy, I implemented data quality validation and reconciliation using PySpark and SQL across all layers.

Previously, in ML Data Operations, I supported retail analytics by performing data validation, anomaly detection, and root cause analysis on operational datasets. I collaborated with quality auditors and operations teams, helped resolve data discrepancies, and supported reporting needs using Excel and SQL-based tools in an Agile environment.