Open to opportunities

Andrew Martin

@andrewmartin3

Message

Senior AI Data Engineer building scalable lakehouse and RAG platforms for real-time enterprise AI.

United States

Message

What I'm looking for

I’m looking to lead GenAI data platform and RAG/LLM evaluation pipelines on a modern lakehouse, partnering with researchers and product to ship secure, observable AI systems at scale.

I’m a Senior AI Data Engineer with 10+ years of experience building large-scale AI, data, and analytics platforms across Generative AI, Healthcare, SaaS, and Retail. I specialize in Databricks Lakehouse architectures, real-time data pipelines, AI-ready datasets, RAG platforms, and ML infrastructure—processing 25TB+ of data and billions of events daily.

At Scale AI, I led the engineering and evolution of a GenAI Data Engine Platform, processing 25TB+ of enterprise documents and AI telemetry daily using Databricks Lakehouse, Delta Lake, and AWS. I designed a Medallion Lakehouse (Bronze/Silver/Gold) with Delta Live Tables, PySpark, dbt, and Unity Catalog, built high-throughput Kafka and Spark Structured Streaming pipelines, and implemented automated LLM evaluation/benchmarking and an Agent Observability and AI Telemetry Platform. Earlier, I built HIPAA-compliant governance and high-reliability healthcare pipelines at Cohere Health, and developed large-scale workflow analytics and lakehouse migrations at Smartsheet, plus demand forecasting platform work at Nike.

Experience

Work history, roles, and key accomplishments

Current

Senior AI Data Engineer

Current

Scale AI

Apr 2023 - Present (3 years 3 months)

Led the engineering and evolution of Scale AI’s GenAI Data Engine Platform, processing 25TB+ of documents and AI telemetry daily to support enterprise AI training, RAG, model evaluation, and agentic AI workloads. Implemented a Delta Lake medallion architecture with orchestration and built high-throughput streaming, RAG pipelines, LLM evaluation/benchmarking, and AI telemetry/observability using Da

Delta Lake Pyspark DBT Unity Catalog Apache Airflow Apache Kafka Spark Structured Streaming Snowflake Pgvector Pinecone LangGraph MCP MLFlow Prometheus Grafana Great Expectations Data Lineage Audit Logging Tableau

Senior Data Engineer

Cohere Health

Apr 2021 - Apr 2023 (2 years)

Built and maintained an AI-powered prior authorization data platform, including a Databricks Lakehouse and integrated healthcare data from EHRs, claims, provider portals, and HL7/FHIR APIs. Developed real-time and batch pipelines with Kafka, Spark Structured Streaming, Airflow, and Databricks, and implemented HIPAA-compliant governance, lineage, and data quality frameworks.

Delta Lake Pyspark AWS S3 Snowflake Apache Kafka Spark Structured Streaming Apache Airflow FHIR HL7 Unity Catalog Great Expectations IAM HIPAA Spark Partitioning SQL Tableau Power BI

Senior Data Engineer

Smartsheet

Aug 2018 - Mar 2021 (2 years 7 months)

Developed Smartsheet’s Enterprise Workflow Intelligence Platform, using Kafka, AWS, Spark, Airflow, Snowflake, and Databricks to enable product analytics, customer health scoring, and executive KPI reporting. Led migration to a modern Snowflake + Databricks lakehouse architecture and established data governance, observability, and quality frameworks to reduce reporting incidents.

Apache Kafka AWS S3 Pyspark Apache Airflow EMR Snowflake Databricks CDC Data Governance Great Expectations Cloudwatch Tableau SQL Python

Software Engineer

Nike

Apr 2014 - Jul 2018 (4 years 3 months)

Developed Nike’s Digital Demand Sensing & Forecasting Platform, integrating sales, inventory, e-commerce, promotion, and supply chain data to support SKU-level demand forecasting and inventory planning. Built scalable batch and near real-time ETL pipelines on Spark/Kafka/Hadoop/Hive and implemented centralized data ingestion into a Hadoop data lake for downstream BI and forecasting.