Souptik Bose

Senior Data Engineer with 5+ years of experience designing, optimizing, and modernizing large-scale distributed data platforms on AWS cloud environments. I specialize in turning enterprise needs into high-performance analytics infrastructure, including processing enterprise datasets exceeding 150TB.

I bring deep hands-on expertise across PySpark, Spark SQL, AWS EMR, AWS Glue, Redshift, and Databricks. I architected and optimized scalable AWS-based distributed ETL pipelines, built cloud data warehousing using Amazon Redshift and S3, and led modernization efforts to migrate legacy systems to cloud-native architectures.

I automate ingestion and ETL using PySpark, Spark SQL, and AWS Glue, reducing manual processing and improving reliability. I also strengthen end-to-end operations with workflow orchestration (Azkaban, Cron Jobs), CI/CD-driven deployment, and Spark performance tuning through partitioning, shuffle optimization, caching, and query/cluster optimization.

On the data layer, I design dimensional models and implement Slowly Changing Dimensions (SCD) to support scalable business intelligence. I focus on stakeholder collaboration, governance, privacy, and compliance—earning recognition for outstanding contribution, improved ETL efficiency, and enterprise-scale operational excellence.