At Celebal Technologies, I build data pipelines and integrations with Databricks, PySpark, and DBT. I automated multi-workflow creation from Lakebridge JSON configs, cutting manual pipeline setup effort by 70%.
I reduced pipeline runtimes from 30–60 minutes to 8–10 minutes by rewriting SQL patterns, applying broadcast joins, and using table persistence strategies. I also migrated 15+ Informatica workflows to Spark-native Databricks pipelines and moved Airflow-orchestrated DAGs to Databricks Workflows.
On my Databricks Historical Load & Reconciliation Utility project, I built orchestration tools for seven connector types and added live job-log streaming with Server-Sent Events. The utility automates source-to-target validation across five dimensions, and I tested it with 47 automated tests.
