Shiva Muppa

Designed and built scalable ETL/ELT pipelines using AWS Glue, Athena, and Python, handling over 10 billion records monthly across payroll and benefits domains. Developed real-time ingestion pipelines using Apache Kafka, integrating with AWS Lambda and Step Functions, reducing batch latency by 70%. Built serverless microservices for preprocessing and cleansing data using Lambda, with robust error h

Developed modular, scalable pipelines using Azure Data Factory, Databricks, and PySpark, standardizing logic across ingestion flows. Built metadata-driven architecture leveraging parameterized datasets, triggers, and Data Lake Gen2, enabling dynamic, reusable pipeline configurations. Optimized Spark job performance by adjusting executor memory, partition counts, and cache strategies, improving loa

Re-architected legacy ETL pipelines to distributed Apache Spark jobs using Scala and PySpark, reducing end-to-end job time by 60%. Built custom RDD transformations to process complex XML/CSV inputs into Parquet, achieving schema evolution and format standardization. Implemented SCD Type 2 logic in AWS Redshift, preserving historical snapshots of evolving customer data.

Developed batch ETL pipelines using Spark and Scala, processing large-scale insurance claims and customer demographics data. Used Kafka with Spark Streaming to stream underwriting events in real-time, enabling timely alerting and reporting. Built Hive-based preprocessing and cleansing layers, applying filters, validations, and deduplication prior to ingestion.

Completed coursework in Applied Machine Learning, Data Analysis, and Data Modeling. Studied Python Programming, Data Harvesting and Storage, and Natural Language Processing.