Mahesh Patil

I’m a Data Engineer with 3.1+ years of experience building scalable ETL/ELT pipelines and real-time data processing across GCP and Azure. I focus on making data reliable, performant, and analytics-ready for teams that need fast, trustworthy insights.

In my recent work, I architected and scaled high-performance ETL pipelines using Python, Apache Spark, and GCP Dataflow, improving throughput and ensuring reliable processing. I also optimized BigQuery using a Medallion Architecture (Bronze–Silver–Gold), reducing query execution and streaming pipeline time by ~40% and lowering costs.

My approach combines strong data modeling with automation and governance. I implemented serverless ingestion pipelines, reusable transformation logic (including BigQuery stored procedures), and automated infrastructure and data quality checks using Terraform—improving data reliability and reducing manual effort.

Before that, I built batch processing and large-scale structured/semi-structured pipelines with Azure Databricks, PySpark, and Azure Data Factory. I designed Medallion-style solutions with Delta Lake, tuned Spark jobs with partitioning/caching/join strategies, and supported reliable workflow orchestration and incremental ingestion.