Open to opportunities

Harshitha Shetty

@harshithashetty

Message

Senior Data Engineer building compliant, scalable Azure data platforms with governance and zero-SLA breaches.

Netherlands

Message

What I'm looking for

I’m looking for a role where I can build governed, audit-ready data platforms in regulated domains—using Azure Databricks, PySpark/Delta, and strong testing/CI/CD. I want ownership of reliable pipelines and opportunities to deepen real-time and data product delivery.

I’m a Senior Data Engineer with 10+ years of experience building compliant, scalable data platforms across Azure, AWS, and GCP. I specialize in Azure Databricks, PySpark, Delta Lake, and Unity Catalog, delivering audit-ready pipelines in regulated insurance and trading environments with zero SLA breaches.

At ARAG, I designed GDPR-compliant PySpark pipelines (500GB+ daily), implemented Delta Lake reliability features, and enforced governance and lineage using Unity Catalog. I also built automated pytest frameworks that reduced pipeline failures by 40%, orchestrated ETL with Apache Airflow, and supported CI/CD releases with Azure DevOps—bringing 60% faster data delivery to analytical teams. I’m currently building real-time Kafka/Databricks portfolio projects and pursuing the Databricks Certified Data Engineer Associate certification (expected April 2026).

Experience

Work history, roles, and key accomplishments

Senior Data Engineer

ARAG

Sep 2023 - Aug 2025 (1 year 11 months)

Designed and delivered GDPR-compliant PySpark pipelines on Azure Databricks processing 500GB+ of regulated insurance data daily, maintaining zero SLA breaches over 12 months. Implemented Delta Lake and Unity Catalog governance, reducing pipeline failures by 40% and cutting data preparation time by 60% for downstream analytics.

Python Pyspark Databricks Delta Lake Unity Catalog Apache Airflow Azure Data Factory Azure Key Vault Azure DevOps

Data Engineer

All Options

Jul 2022 - Jul 2023 (1 year)

Built near real-time PySpark pipelines on Azure Databricks and Azure Event Hub processing 5M+ trading records daily with sub-minute latency for live P&L and risk dashboards. Delivered governed analytics pipelines using Azure Synapse, ADF, Snowflake/dbt, and Airflow, reducing manual deployment effort by 70%.

Pyspark Databricks Azure Event Hubs Azure Synapse Azure Data Factory Snowflake DBT Apache Airflow Azure DevOps SQL

Data Engineer

Ugam Solutions

Dec 2021 - May 2022 (5 months)

Built Python and SQL ingestion pipelines with multi-stage schema validation and integrity checks, achieving 100% data accuracy across pharma and retail datasets. Developed FastAPI services to trigger and manage web scraping workflows and implemented AWS S3/Glue/Lambda ETL patterns, plus delivered Adobe Experience Platform onboarding and training for a team of 8.

Python SQL fastAPI Apache Airflow AWS S3 AWS Glue AWS Lambda ETL Data Validation

Data Consultant

Deloitte

Dec 2020 - Dec 2021 (1 year)

Built a production Customer Data Platform using Matillion ELT, Python, and Snowflake, consolidating customer data from 10+ sources with schema governance and identity resolution for enterprise banking clients. Developed AWS S3/Glue ingestion into Snowflake with dbt transformation models and implemented Adobe Experience Platform data models to enable personalized customer journey analytics.

Matillion Python Snowflake DBT AWS S3 AWS Glue AWS RedShift Data Modeling SQL

Data Analyst

Accenture

Jan 2014 - Sep 2020 (6 years 8 months)

Delivered end-to-end ETL solutions using DataStage and Informatica BDM across 6+ enterprise clients in banking, retail, telecoms, and logistics. Improved batch processing performance by 35% through SQL optimization and parallelization, and mentored production and testing teams of 10+ members on data quality practices.