Open to opportunities

shailendra Rathore

@shailendrarathore

Message

Senior Big Data Engineer specializing in Spark/ETL optimization and real-time data platforms.

India

Message

What I'm looking for

I’m looking to lead end-to-end data engineering—building high-performance Spark pipelines, streaming ETL, and reliable dashboards—while coaching teams in Agile environments and delivering measurable runtime and data-quality improvements.

I’m a Senior Big Data Engineer with 7.4 years of experience designing and delivering large-scale data platforms across pharma, automotive, insurance, and eCommerce. I’ve reduced pipeline runtimes by up to 40% through advanced Spark optimization, while staying equally strong as an individual contributor and as a technical lead.

I’ve worked extensively across the Apache Hadoop ecosystem—MapReduce, HDFS, Hive, and HBase—and built ETL and real-time processing using Spark (PySpark and Scala). In Databricks, I’ve used Delta Live Tables and Spark Streaming to deliver production-grade pipelines with strong SQL-backed data analysis.

I also lead teams in Agile/Scrum delivery, mentoring developers and coordinating with owners and SMEs to ship measurable outcomes. My work includes end-to-end pipeline implementation, NiFi-based ingestion prototypes, data quality validation (including Great Expectations), and cloud solutions across AWS and Azure—plus contributions like building KPI dashboards and scalable data models.

Experience

Work history, roles, and key accomplishments

Lead Data Engineer

Syren Technologies Pvt Ltd

Jul 2024 - Mar 2026 (1 year 8 months)

Implemented an end-to-end ETL pipeline and a real-time KPI dashboard for a pharma client, updating metrics across countries, sites, and categories. Reduced Spark pipeline runtime from 55 minutes to 20 minutes using predicate pushdown and column pruning, and delivered data validation with standardized before/after capture.

Azure Data Factory Azure Databricks Delta Live Tables Pyspark Scala Data Validation Predicate Pushdown

Lead Data Engineer 2

Codvo Pvt Ltd

Oct 2023 - Mar 2024 (5 months)

Built ETL and streaming pipelines using Mage.ai and Azure Databricks Delta Live Tables, ingesting from Airbyte into governed raw/silver/gold layers. Improved data quality and integrity using pandas transformations and Great Expectations, and exposed RESTful Python services backed by database validation and U.S. phone verification.

Mage.Ai Airbyte Azure Databricks Delta Live Tables Pyspark Pandas Great Expectations REST APIs NumPy Apache NiFi

Senior Data Engineer

IBM India Pvt Ltd

Nov 2021 - Sep 2023 (1 year 10 months)

Led requirements gathering and team execution (10–15 developers) to design data workflows for risk calculation and UI-facing analytics. Implemented Databricks POCs to materialize processed results into Delta tables, added Git-based CI/CD for notebook updates, and created pseudo-anonymization using RSA (Java and Python) distributed as a .whl via Azure Artifacts.

Azure Databricks Delta Lake Python Pyspark Anonymization RSA Encryption Data Modeling Scrum

Senior Software Engineer

Celebal Tech

Sep 2020 - Sep 2021 (1 year)

Managed multiple concurrent projects and tracked daily delivery using Scrum with a team of 10–15 developers. Optimized Apache NiFi ingestion for large structured/unstructured datasets, integrated PySpark-based validation/profiling/transformation rules, and delivered Django-based microservices APIs to trigger NiFi jobs from a UI.

Apache NiFi Pyspark Apache Spark Django REST APIs Data Validation Data Profiler Azure DevOps

Software Developer

Qualtech Consultants

Aug 2019 - Sep 2020 (1 year 1 month)

Developed a Windows service to restrict attendance marking to users logged in via RDP within the organization network, using WinAPI hooks and registry-driven SOA API calls. Built a multithreaded image compression desktop utility that converts common image formats (including PDF) and detects corrupted inputs during processing.

.NET C++Windows Services Windows APIs Multithreading SOA Image Processing

Software Developer

Unikove Technologies Pvt Ltd

Jan 2018 - May 2019 (1 year 4 months)

Built an eCommerce platform using .NET MVC with CMS capabilities and promotion engines (e.g., buy-one-get-one and percentage discounts) based on product binding types. Implemented cruise notification functionality with push notifications (APNS) plus a scheduled delivery system using Web APIs and Windows services.

MVC C++REST APIs SQL Server Push Notifications JavaScript jQuery Windows Services