Shiva Kumar

I’m a Senior Data Engineer with 5+ years of experience architecting, building, and operating cloud-native, distributed data platforms in banking and enterprise environments. I bring deep, hands-on expertise in Python, SQL, Apache Spark, Databricks, Azure, AWS, and Snowflake—covering source ingestion, CDC, batch and real-time streaming pipelines, and optimized storage layers.

At Molina Healthcare, I designed and implemented a healthcare data platform ingesting patient, provider, claims, and eligibility data to support HEDIS, RAF risk adjustment analytics, and operational dashboards. I built end-to-end batch ingestion with Azure Data Factory to ADLS Gen2, used Databricks/PySpark for transformations and validations, and implemented Delta-based incremental loads, watermarking, and CDC to avoid full reloads. I also integrated Kafka for near real-time ingestion, enforced data reliability with automated data quality checks and monitoring/alerting, and improved pipeline execution time by over 30% through performance tuning.

Previously at HDFCBank, I designed a real-time data processing platform for fraud detection and risk monitoring, using Kafka for streaming ingestion and Snowflake for analytics. I automated orchestration with Apache Airflow, added retry logic and dead-letter queues for resilience, and tuned SQL queries and indexes for high-volume performance. I enabled near real-time dashboards in Tableau and reduced end-to-end data latency by over 40% through streaming optimizations.

I’m equally comfortable across data engineering and analytics delivery—building ELT/EDW pipelines on AWS/Azure, modeling dimensional and flattened schemas, and applying Python-driven analytics and ML (e.g., logistic regression and forecasting) when the problem demands it. I prioritize governance and security with RBAC/RLS and data masking, set up CI/CD for repeatable deployments, and mentor engineers while partnering closely with product, data science, and business stakeholders to ship production-grade solutions.