At Qualtrics, I build an automated platform for evaluating AI coding tasks in a real SaaS codebase. I turn repository changes into challenging, reproducible tasks and validate them with Python, Git, and Pytest.
I also built SQL and Python ETL dashboards to analyze task pass/fail rates and identify model performance bottlenecks. GitHub Actions pipelines automate task validation, while static analysis, linters, and type checkers help maintain quality.
At Suzy, I built an automated evaluation suite and standardized templates for coding cases sourced from production repositories. I added regression and edge-case testing, and established secure sandboxes for running model-generated code.
Earlier, at Zappi, I built backend test case generators and pipelines that mined project diffs for coding problems. I also documented benchmarking scripts and contributed peer-review standards for task design.
