Open to opportunities

Mandeep Sidhu

@mandeepsidhu

Message

Staff Data Engineer crafting reliable lakehouse and streaming data platforms.

Canada

Message

What I'm looking for

I’m looking for a team where I can own data platform reliability and lakehouse/streaming architecture end-to-end, build governed activation pipelines, and partner cross-functionally to ship trusted, revenue-critical data products.

I’m a Senior/Staff Data Engineer with 8+ years of end-to-end ownership across data platform architecture and reliability—from ingestion and governance to activation. I bring an ML-aware mindset to data quality, monitoring, and data product design, so teams can trust what’s driving analytics and revenue.

I’ve led major reliability and architecture turnarounds, including fixing Redshift issues caused by DMS CDC replication lag and duplicate/stale base tables. From there, I drove repeatable dedupe/backfill and validation patterns, and migrated from Redshift/Spectrum-heavy patterns to a decoupled lakehouse on S3 with Iceberg and Trino.

I build both batch and real-time activation systems that align definitions across GTM systems. I’ve implemented CDC patterns (Postgres CDC → S3 via AWS DMS) to prevent duplication, and designed Kafka-based event pipelines to publish business events to platforms like HubSpot, Salesforce, and Intercom—while also supporting continuity when third-party ingestion breaks.

In parallel, I’ve delivered production applied ML systems and worked as an ML & Data Engineering leader/consultant, improving insight quality through validation and business-rule enforcement. I also lead platform operational posture—standardizing orchestration/IaC/observability, mentoring engineers, and setting roadmaps that reduce cost and reporting latency.

Experience

Work history, roles, and key accomplishments

Current

Senior Data Engineer

Current

Jobber

May 2025 - Present (1 year 1 month)

Owned data platform reliability, ingestion standards, and activation pipelines across Product Analytics, RevOps, Marketing, Sales Engineering, and Fintech domains, including PII and revenue-critical datasets. Led a Redshift reliability turnaround tied to DMS CDC replication lag and duplicate/stale tables, then migrated from Redshift/Spectrum patterns to an S3 Iceberg + Trino lakehouse while buildi

AWS RedShift DMS PostgreSQL CDC Apache Iceberg Trino DBT Apache Kafka Amazon S3 HubSpot Salesforce

AI & Data Engineering Consultant

Acclogic Strategic Consulting Inc.

Oct 2024 - Apr 2025 (6 months)

Delivered real-time analytics using a multi-agent architecture over structured and unstructured data, improving insight quality with iterative validation and business-rule enforcement. Built and extended a thermal-image ML pipeline for livestock weight prediction and derived thermal efficiency/health metrics using Python-based image analysis.

Multi Agent Systems Real Time Analytics Python Data Validation Image Analysis Production ML Systems

Staff Data Engineer & Team Lead

Ecoation Innovative Solutions Inc.

Feb 2019 - Sep 2024 (5 years 7 months)

Led a team building scalable batch and real-time pipelines, ingesting and processing 105 TB/month, and operated an EKS platform (on-demand + spot) that reduced compute costs by 65% and improved in-memory workflows to lower disk I/O by 30%. Migrated from Data Lake to Lakehouse (Snowflake + S3) to improve query performance by 50% and reduce storage/API costs by $75K annually, and redesigned ELT with

Kubernetes Snowflake Amazon S3 DBT Apache Airflow Machine Learning Pipelines Jetson Xavier CUDA Containers

Machine Learning Engineer

Oasis Technologies Inc.

Oct 2017 - Jan 2019 (1 year 3 months)

Developed ML algorithms and deep neural networks for computer vision and business intelligence, improving surveillance and leak detection in mechanical equipment. Built object and motion detection techniques that filtered 30% more unwanted images and delivered predictive maintenance programs that saved 25% on equipment costs and reduced labor by 20%.

Computer Vision Object Detection Predictive Maintenance Neural Networks Filtering Model Development

Machine Learning Engineer

Cantest Solutions Inc.

Jun 2017 - Sep 2017 (3 months)

Deployed genetic algorithms and deep neural networks to optimize technician and sales workflows, building an automatic scheduling system that improved technician efficiency by 35%. Developed predictive and classification models to improve targeting accuracy by 25% and documentation accuracy by 50%.

Genetic Algorithms Neural Networks Scheduling Optimization Workflow Automation Classification predictive modeling Documentation Accuracy Modeling