Rio Deb

I’m a Data Engineer focused on designing, building, and optimizing large-scale analytics platforms using Databricks, Apache Spark, and Delta Lake. I deliver high-performance, reliable, and scalable data solutions—especially in enterprise financial analytics environments.

In my current role, I manage enterprise analytics pipelines handling massive volumes, including ~1500 tables, 60+ TB of structured data, and datasets up to 500+ GB with 300B+ rows. I build standardized Bronze-to-Silver ETL frameworks with incremental ingestion, overwrite/merge workflows, and SCD Type 1/2 implementations, including Delta Live Tables for Silver and Gold layers.

I also prioritize automation and maintainability: I created a reusable metadata-driven transformation framework that dynamically generates Databricks SQL and PySpark logic, reducing manual query writing and accelerating onboarding. I enabled automated schema evolution across pipelines and standardized modular job templates so large teams can build consistently.

Performance and data quality are core to how I work. I optimize Spark workloads through shuffle reduction, join/aggregation re-architecture, partition pruning, predicate pushdown, and deep analysis with Spark UI and execution plans—improving stability, runtime, and cost efficiency. I back this with automated validation frameworks across Bronze, Silver, and Gold, plus Terraform-based Databricks job automation and parameterized reusable workflows.