Open to opportunities

Shoaeb Eqbal

@shoaebeqbal

I’m a Senior Data Engineer specializing in scalable lakehouse platforms, HIPAA/FHIR data engineering, and real-time streaming systems.

United States

Message

What I'm looking for

I want to lead end-to-end data platform work—lakehouse architecture, real-time streaming, ELT/CDC, and governance for regulated domains—while mentoring teams and driving measurable outcomes like lower latency, better data quality, and cost efficiency.

I’m a Senior Data Engineer with 10+ years of experience architecting enterprise-scale data pipelines, lakehouse platforms, and real-time streaming systems across healthcare and e-commerce. I translate complex data challenges into scalable, production-ready infrastructure that supports millions of events and terabytes of data daily.

In my current role as Lead Data Engineer, I architected and scaled lakehouse and data warehouse platforms using Databricks, Snowflake, and BigQuery, processing 8–15 TB of data daily. I deployed Medallion Architecture, adopted Delta Lake and Apache Iceberg for efficient versioning and time-travel queries, and engineered ELT pipelines that reduced latency from hours to minutes.

I build compliant, interoperable healthcare data platforms by applying HL7 and FHIR standards and enforcing HIPAA-aligned governance with RBAC controls. I’ve improved ingestion reliability by reducing data ingestion errors by 30% and enabled seamless interoperability through standards-driven design.

I also drive measurable outcomes in performance, cost, and delivery speed—reducing compute costs by 35%, cutting development time by 40% with metadata-driven pipelines, and improving decision-making via Power BI and Tableau dashboards. I lead and mentor teams (including a team of 7 engineers), strengthening CI/CD practices and automating engineering workflows with Azure DevOps and GitHub.

Experience

Work history, roles, and key accomplishments

Current

Lead Data Engineer

Current

VDart

Feb 2024 - Present (2 years 4 months)

Architected and scaled enterprise lakehouse and data warehouse platforms using Databricks, Snowflake, and BigQuery, processing 8–15 TB of data daily. Improved data quality and governance with Medallion architecture, cut development time 40%, reduced data latency from hours to minutes, and saved 35% in annual compute costs while leading and mentoring 7 engineers.

Databricks Snowflake BigQuery DBT Apache Spark Delta Lake Apache Iceberg Apache Kafka Google Pub Sub Power BI

Senior Data Engineer

Osprey

Nov 2019 - Feb 2024 (4 years 3 months)

Built and scaled a HIPAA-compliant Snowflake platform for 5+ TB/day of Medicaid and Medicare claims data. Automated ELT ingestion with Fivetran (50% less manual work), delivered sub-second real-time analytics with ClickHouse/Elasticsearch, and enforced RBAC and HIPAA governance for secure, auditable access.

Snowflake Fivetran Clickhouse Elasticsearch AWS Glue AWS Lambda S3 RBAC HIPAA compliance ML Pipelines

Data Engineer

Cummins

May 2018 - Nov 2019 (1 year 6 months)

Built high-throughput Kafka and Apache Spark streaming pipelines processing 200M+ events/day with low latency for real-time e-commerce data. Improved Snowflake query performance by 45%, implemented Debezium-based CDC to reduce latency from hours to minutes, and automated ETL workflows with Python and Apache Airflow to cut manual effort by 50%.

Apache Kafka Apache Spark Debezium Snowflake MongoDB Cassandra Elasticsearch Apache Airflow Change Data Capture Python

Data Engineer

Waffle House

Jan 2016 - May 2018 (2 years 4 months)

Designed and built Python/SQL ETL pipelines to ensure accurate reports were available when needed. Improved reporting quality with Star Schema and SCD Type 2 models and boosted SQL performance by 60% while reducing ETL processing time by 40%.

Python SQL ETL Data Modeling Star Schemas SQL Optimization Performance Tuning Report Modeling