United States onlyYou will be a strategic partner to product, engineering, and trust and safety teams, responsible for defining evaluation frameworks, leading experiments (A/B, quasi-experiments, etc.), and turning offline and live model performance into product improvements. This role requires a strong track record in startup-style experimentation (moving quickly with scrappy but rigorous methods) and product experimentation at scale. The ideal candidate will also bring proven experience in leading and managing teams to deliver high-impact data science work.
100% remote
Salary Range $120,000 - $260,000
Essential Job Functions
●Lead end-to-end experimentation: hypothesis generation, metric design, experiment design (A/B, multivariate, sequential, etc.), analysis, and interpretation.
●Build and maintain evaluation frameworks for LLMs: correctness, consistency, safety, hallucination detection, bias/fairness, etc.
●Develop predictive models, classification/ranking systems, and heuristics to improve product features related to AI/language generation.
●Collaborate with prompt engineers & model builders to test prompt strategies, fine-tuning, or model selection; work on failure modes/error analysis.
●Automate experiment pipelines: dashboards, monitoring, alerting, instrumentation. Ensure data quality & measurement integrity.
●Use causal inference / observational studies when randomized experiments are not feasible.
●Present findings and recommendations to both technical and non-technical leadership; influence roadmap decisions.
●Drive experimentation in startup-like environments: rapid iteration, learning from limited data, and balancing speed with rigor.
●Shape large-scale product experimentation: define frameworks for experimentation at scale and integrate results into product strategy.
●Lead and mentor teams of data scientists, analysts, and engineers; set best practices for experiment design and AI product evaluation.
Requirements
- ~8-12+ years of experience in data science / ML roles, ideally with experiment design/product analytics.
- Proven track record in both startup-style and large-scale product experimentation.
- Experience leading teams, setting strategy, and driving execution in cross-functional environments.
- Strong background with statistical methods, causal inference, and rigorous measurement.
- Experience using LLMs / NLP / AI / prompt engineering or a closely related field.
- Excellent coding skills in Python (or similar), strong SQL, experience building and deploying models or analytic pipelines.
- Ability to work in cross-functional teams, translate technical results into business or product changes.
- Strong communication skills; ability to explain complex analyses to non-technical stakeholders.
Preferred Qualifications
- Experience fine-tuning or working with multiple LLM providers / APIs.
- Experience with experiment platforms or building internal tooling for experimentation & model evaluation.
- Experience in voice / ASR or other multi-modal data.
Benefits
- Health Care Plan (Medical, Dental & Vision)
- Retirement Plan (401k)
- Life Insurance (Basic, Voluntary & AD&D)
- Flexible Paid Time Off
- Family Leave (Maternity, Paternity)
- Short Term & Long Term Disability
- Training & Development
- Work From Home
- Stock Option Plan
India only
