Skip to main content
vishal guptaVG
Open to opportunities

vishal gupta

@vishalgupta8

I build scalable cloud data pipelines and orchestration frameworks across GCP, Databricks, and Apache Spark.

India
Message

I've built cloud data engineering solutions at IBM, including a Python and Jinja framework that dynamically generates 600+ Airflow DAGs from BigQuery metadata. I also created an end-to-end observability framework for pipeline monitoring.

Across Publicis Sapient, Verizon, and Infosys, I've developed and optimized ETL pipelines using Databricks, Apache Spark, PySpark, BigQuery, Google Cloud Composer, and Delta Lake. My work includes performance tuning, cost optimization, data lake design, Teradata migration, and automated workflow orchestration.

I've also applied Python, Pandas, and Matplotlib to churn and attrition analysis projects, translating data into targeted retention recommendations. I'm certified as a Google Professional Data Engineer, Google Professional Cloud Architect, Databricks Associate Data Engineer, and AWS Machine Learning Specialty professional.

Experience

Work history, roles, and key accomplishments

IBM logoIB
Current

Senior Data Engineer

Jun 2024 - Present (2 years 2 months)

Designed and implemented a Python and Jinja-based framework to dynamically generate Airflow DAGs using metadata from BigQuery, resulting in a scalable solution to generate 600+ DAGs dynamically. Built robust observability framework for end-to-end pipeline monitoring and optimized scalable data pipelines using Apache Spark on Databricks.

Education

Degrees, certifications, and relevant coursework

CK

CSJMU University Kanpur

Bachelor of Computer Applications, Computer Application

Studied Computer Application in BCA.

Get matched with your dream remote job

Sign up now and join over 250,000+ remote workers who receive personalized job alerts, curated job matches, and more for free!

Sign up
Himalayas profile for an example user named Frankie Sullivan