Nayan Bhagawati
@nayanbhagawati
Sr Data Eng | Databricks Cert | PySpark, Delta, ADF, Airflow, Unity Catalog | 6+yr Exp | $14M Savings | Remote to UK/EU/ANZ from India | Lead Ready
IndiaWhat I'm looking for
As a Senior Data Engineer, I specialize in building high-performance, cost-efficient data architectures using Azure Databricks, Spark, and Delta Lake. I recently spearheaded a large-scale optimization initiative that delivered $14M in cost avoidance by re-engineering ETL pipelines, implementing Z-Ordering, and right-sizing compute resources for global workloads. I am passionate about transforming complex raw data into scalable, production-grade assets that drive significant bottom-line value for global organizations.
Experience
Work history, roles, and key accomplishments
Architected a global Azure Databricks IDoc platform delivering $14M cost avoidance. Modernized legacy batch to Medallion Architecture (Delta Lake), cutting latency by 37% for 5TB+ datasets. Reduced pipeline failures from 9% to 1.3% via PySpark & Spark SQL. Optimized compute using Z-Ordering & broadcast joins. Integrated SAP HRV & led async global teams with 24/7 SLA ownership.
Contributed to a $2M Azure Databricks modernization program, delivering an automated enterprise IDoc error monitoring platform using Python and Spark that generated ~$14M in cost avoidance. Reduced pipeline failure rates from 9% to 1.3%, and optimized Cassandra storage from 120TB to 80TB to reduce licensing costs from $150K to $90K.
Managed 80TB+ IoT datasets and improved system reliability by enhancing SQL query performance and observability across data pipelines. Supported scalable data processing for production-grade analytics use cases.
Education
Degrees, certifications, and relevant coursework
Nayan hasn't added their education
Don't worry, there are 90k+ talented remote workers on Himalayas
Tech stack
Software and tools used professionally
Availability
Location
IndiaAuthorized to work in
Social media
Job categories
Interested in hiring Nayan?
You can contact Nayan and 90k+ other talented remote workers on Himalayas.
Message NayanFind your dream job
Sign up now and join over 250,000+ remote workers who receive personalized job alerts, curated job matches, and more for free!
