Merry Shah

I’m a Lead Data Engineer with 9+ years of experience designing, developing, and optimizing data pipelines, cloud architectures, and analytics solutions. I focus on scalable ETL workflows, cloud data warehouses, and real-time processing that turn events into actionable insight.

In my current role, I architected high-performance real-time pipelines using Apache Kafka, Apache Flink, and Apache Spark Streaming—handling 100M+ daily events with sub-second latency. I’ve migrated on-prem warehouses to AWS Redshift and Snowflake, improving query performance by 40% and reducing infrastructure costs by 25%, while also cutting processing latency and pipeline time with incremental loads, partitioning, and tuning.

I’m especially strong in building reliable data platforms with orchestration, validation, and governance. I use Apache Airflow, AWS Glue, and automated data validation (including EvidentlyAI and Prometheus) to improve data quality and achieve 99.9% pipeline uptime, alongside HIPAA-compliant healthcare pipeline work.

I also lead with a product mindset—integrating machine learning and predictive analytics (Python, Scikit-learn, XGBoost, Spark MLlib) into data pipelines and delivering interactive BI dashboards with Tableau and Power BI. I enjoy mentoring teams and creating maintainable systems that support data maturity, governance lineage, and strategic growth.