Open to opportunities

Harshit Agrawal

@harshitagrawal

Message

Big Data Engineer scaling high-performance data platforms on Azure and Databricks for measurable efficiency gains.

India

Message

What I'm looking for

I’m looking for a role where I can build governed, high-performance data platforms on Azure/Databricks—optimizing cost and performance, improving reliability with CI/CD, and partnering with teams to deliver measurable efficiency gains.

I’m a Big Data Engineer with 12 years of experience scaling high-performance data platforms in the Azure and Databricks ecosystems. I focus on turning complex pipelines into governed, reliable systems—collaborating with cross-functional teams to deliver outcomes like a 20% efficiency boost and faster, cleaner decision-making.

Recently, I architected a Platinum KPI Layer that became a governed single source of truth, reducing metric reporting discrepancies by 100% across business units. I’ve also driven major performance and cost improvements (30% compute cost reduction and 8x query performance) through Liquid Clustering, automated operational efficiencies with Predictive Optimization, and standardized deployments with Databricks Asset Bundles for smoother CI/CD. I enjoy mentoring engineers on hard SQL and PySpark logic, and I bring that same execution mindset from prior work like real-time streaming with Kafka and PySpark Structured Streaming, automated monitoring with ADF, and experimentation with retrieval-augmented generation.

Experience

Work history, roles, and key accomplishments

Current

Senior Big Data Engineer

Current

EnableData

Apr 2025 - Present (1 year 2 months)

Architected a Platinum KPI layer as a governed single source of truth, eliminating 100% of metric reporting discrepancies across business units. Optimized cloud compute costs by 30% and improved query performance by 8x using Liquid Clustering, while standardizing deployments with Databricks Asset Bundles to cut environment-related errors by 40%.

Databricks Pyspark Delta Live Tables Unity Catalog Azure Data Factory CI CD

Senior Big Data Engineer

Nagarro

Aug 2024 - Mar 2025 (7 months)

Architected a metadata-based ingestion pipeline to ingest data from multiple sources and schedules via a single Azure Data Factory pipeline. Built a proof of concept demonstrating Lakehouse Declarative Pipelines (DLT) in Databricks for streaming use cases.

Azure Data Factory Databricks Lakehouse Declarative Pipelines (DLT)Data Ingestion Streaming Data Spark

Big Data Engineer

Concentrix Software Solutions

Dec 2021 - Aug 2024 (2 years 8 months)

Automated financial data processing on Azure Databricks using PySpark (P&L, sales, purchases, reconciliation) while guaranteeing 100% accuracy. Reduced execution time up to 35% through Spark optimization (partitioning, Z-order, join/aggregation, caching) and implemented real-time cleansing via PySpark Structured Streaming consuming Kafka with data mismatch monitoring via ADF (15% team efficiency l

Azure Databricks Pyspark Spark Structured Streaming Apache Kafka Azure Data Factory Vector Search

Data Engineer

Simulytics Services

Aug 2018 - Dec 2021 (3 years 4 months)

Engineered Spark jobs to automate monthly warehouse management tasks (Cycle Count and warehouse defragmentation). Implemented a Spark-based data archiving strategy that improved OLTP performance by 70% and built an ETL pipeline for SFTP transfers with 0% manual intervention.

Apache Spark Pyspark ETL Data Archiving SFTP Data Warehousing Warehouse Management Systems (WMS)

Senior Data Analyst

Magicpin.in

May 2016 - Jul 2018 (2 years 2 months)

Revamped the reporting framework using MySQL, Python, and BI tools, improving analyst efficiency by 25%. Extracted data from OLTP/CRM/payables sources to build Python cashback optimization models that reduced spillage by 80% and implemented fraud detection, while customizing SuiteCRM to improve sales manager funnel efficiency by 40%.

MySQL SQL Python BI Reporting Data Extraction predictive modeling Fraud Detection SuiteCRM

Data Analyst

Wizikey

May 2013 - Nov 2015 (2 years 6 months)

Led a Python and SQL-driven data cleaning initiative across multiple sources (news websites, Twitter, Google Forms), improving data quality by 40%. Analyzed weekly/monthly/quarterly performance metrics to support targets and enabled marketing optimization using Facebook/Google Ads analytics to save 15% in marketing spend via CPC/CPA calibration.

Python SQL Data Cleaning Data Quality Assurance Marketing Analytics Facebook Ads Google Ads KPI Reporting