Looking for a job

Carlton Johnson

@carltonjohnson3

Message

Senior AI Software Engineer at Coval who built agent evaluation gates that cut release cycles from weeks to days.

United States

Message

What I'm looking for

I’m looking for a Senior Software Engineering role focused on AI Agents, Evaluation Systems, or AI Reliability. Ideal opportunities involve building evaluation frameworks, agent platforms, RAG systems, and production-grade AI infrastructure, preferably in enterprise or regulated environments where quality, governance, and reliability are critical.

At Coval, I built an evidence program for AI agent releases, with golden conversations, calibrated rubrics, and evaluation gates that no model or prompt release could bypass. I also created simulated agent scenarios to test tool use, interruptions, and recovery.

I wired regression gates into GitHub Actions CI and added trace sampling and replayable production traces, cutting the evaluation cycle from weeks to days. I built FastAPI and Node.js services and Next.js surfaces for the platform's multi-tenant API layer.

At Constellation Energy, I built production AI services and retrieval-grounded assistants over tariff sheets, procedures, and regulatory filings. I also created an outage-triage evaluation suite that became the primary evidence base for model selection during internal reliability review.

At Microsoft, I built forecasting and risk models for energy and retail partners and developed sequential testing and variance-reduction features for the Microsoft Research experimentation platform. Earlier at AT&T, I built subscriber and network-performance reporting and automated Python workflows for analyst data.

Experience

Work history, roles, and key accomplishments

Senior Software Engineer (AI Evaluation & Agent Reliability)

Coval

Apr 2025 - Oct 2026 (1 year 6 months)

Built the evidence program in Python and TypeScript, designing golden conversations, calibrated rubrics, and inter-rater agreement gates. Developed simulated agent scenarios and adversarial task suites, and integrated evaluation into CI with trace sampling and replayable production traces.

Python TypeScript fastAPI NodeJS Next.js LangGraph Lang Chain MCP Ragas DeepEval Promptfoo OpenAI Anthropic Gemini VLLM PostgreSQL Pgvector GitHub Actions Langfuse Langsmith OpenTelemetry GenAI Kubernetes Docker AWS GCP

Senior Software Engineer, AI

Constellation Energy

Aug 2023 - Mar 2025 (1 year 7 months)

Built production services in Python and TypeScript under FERC- and NERC-regulated constraints. Designed retrieval-grounded assistants (RAG) over tariff sheets and procedures, and established the model governance framework and risk tiering.

Python TypeScript RAG PostgreSQL Pgvector Kubernetes Langfuse Model Governance Multi tenant Isolation

Software Engineer / Data Scientist

Microsoft

Jul 2018 - Jul 2023 (5 years)

Built production forecasting and risk models for energy and retail partners on Azure ML, improving forecast accuracy across tens of thousands of series. Developed sequential testing and variance-reduction features on the Microsoft Research experimentation platform (ExP).

Python SQL Azure ML Azure Pipelines Experimentation Forecasting Risk Modeling Explainability

Software / Data Analyst

AT&T

Jul 2016 - Jun 2018 (1 year 11 months)

Built subscriber and network-performance reporting, took ownership of SQL warehouse extracts and reconciliation checks, and reduced exception queues. Prototyped a churn-signal report that became a retention pilot and automated recurring Python workflows.

SQL Python Data Analysis Reporting Data Warehouse Automation