At Tata Consultancy Services, I build and maintain production batch ETL pipelines for banking data, translating functional specifications into reliable Python, PySpark, Spark SQL, and SQL processing logic. I reduced batch processing time by 33% through Spark tuning across partitioning, caching, broadcast joins, predicate pushdown, and checkpoint placement.
I also trace production failures through source-to-target validation, own releases across UAT, STG, and PROD, and improved GitLab CI and Jenkins deployments. In my BankLake project, I built incremental pipelines, data-quality checks, Airflow orchestration, Kafka and Avro integration, and AI-assisted monitoring for failed runs.
