At Wipro Technologies, I design and develop PySpark batch pipelines for Citi Bank's mutual fund model outputs, processing millions of transaction records daily. I also build ingestion workflows that transform structured and semi-structured data into optimized Hive tables.
I improved Spark job execution time by 20% through partitioning strategies, executor memory tuning, parallelism, and shuffle partition optimization. I also improved Hive query performance with partitioning, bucketing, and partition pruning.
I automated 10+ Oozie workflows and coordinator jobs, and implemented a data validation framework with schema enforcement. I also support CI/CD pipelines for PySpark and Hive code using Jenkins and Git.
