At Kfintech, I modernized an on-premise data platform into a scalable GCP Data Lake using GCS, Dataproc, Spark (Scala), Apache Hudi, and BigQuery. I implemented end-to-end CDC, ETL, and data-processing pipelines while reducing infrastructure costs by 30%.
I optimized Dataproc clusters and Apache Spark workloads through resource and parallel-processing tuning, reducing Mirror processing time from two hours to one hour. I also reduced overall pipeline execution from 5.5 hours to four hours.
I migrated Cloud Composer (Apache Airflow) to Cloud Workflows and Cloud Functions, reducing orchestration costs from Rs. 1 lakh to Rs. 1K per month. My batch ETL and incremental CDC work reduced reporting turnaround from three days to six hours and improved Looker dashboard load time for 300+ users.
Previously as a Data Engineering Intern at Kfintech, I developed and supported Spark, Scala, and SQL ETL pipelines processing 8M+ financial transaction records. I also implemented Pub/Sub-triggered Cloud Functions to maintain SCD Type 2 master tables for historical tracking.

