Open to opportunities

Stephen Koo

@stephenkoo1

Message

Senior Software Engineer specializing in scalable AI and LLM infrastructure.

United States

Message

What I'm looking for

I seek roles building and scaling AI/LLM infrastructure where I can drive low-latency, reliable ML serving, mentor engineers, and collaborate cross-functionally to deliver production generative-AI features.

I am a Senior Software Engineer with 8+ years building scalable backend systems, AI inference platforms, and production LLM services across multi-region cloud environments. I design and operate Kubernetes-based ML serving, optimize inference pipelines for low latency and high availability, and integrate generative AI using Vertex AI and Gemini to deliver user-facing AI features.

I have led architecture and implementation of inference orchestration services, improved P95 latencies through batching and caching, and scaled platforms with Cloud Spanner, Pub/Sub, and GKE to sustain 99.99% availability. I collaborate cross-functionally, mentor engineers, and produce tooling and control-plane UIs that support experimentation, monitoring, and reliable rollouts of AI capabilities.

Experience

Work history, roles, and key accomplishments

Current

Senior Software Engineer

Current

Google

Oct 2017 - Present (8 years 5 months)

Architected and built AI inference orchestration and ML inference platforms for Gmail and Google Chat, reducing p95 latency by up to 30%, increasing throughput 3×, and sustaining 99.99% multi-region availability for production LLM services.

Golang Vertex AI RAG Distributed Systems CI CD Cloud Spanner Pub Sub React Typescript

Education

Degrees, certifications, and relevant coursework

Stanford University

Master of Science, Computer Science (Artificial Intelligence)

2015 - 2017

Completed a Master of Science in Computer Science with a focus on Artificial Intelligence, covering advanced AI topics and research-oriented coursework.

Stanford University

Bachelor of Science, Computer Science (Artificial Intelligence)

2011 - 2015

Completed a Bachelor of Science in Computer Science with a focus on Artificial Intelligence, including foundational coursework in algorithms, systems, and AI.