India onlyAbout the Company:
About the Role
We are looking for a Manager – AI Performance & Analytics to lead a team responsible for analyzing conversational AI performance and driving continuous improvements to Netomi’s AI platform.
This role sits at the intersection of data analytics, conversational AI, and LLM evaluation. The team analyzes customer conversations at scale, identifies quality gaps and automation opportunities, and partners with AI and product teams to improve model performance.
You will help define how we measure and improve the performance of AI systems that power customer support automation across millions of interactions.
Responsibilities
Analyze large volumes of chatbot and AI conversation data to identify performance gaps and improvement opportunities.
Define and track key AI performance metrics such as resolution rate, containment rate, response quality, and customer satisfaction.
Build dashboards and reporting frameworks using SQL, Python, and BI tools such as Tableau.
Design frameworks to evaluate the response quality of LLM responses across dimensions such as:
factuality and groundedness
Relevance to user intent
Instruction Adherence
tone and customer experience
safety and compliance
Build scalable processes for LLM-based automated evaluation and human review workflows.
Identify systematic failure patterns in AI responses and recommend improvements.
Partner with AI and engineering teams to evaluate model upgrades, prompt changes, and new AI capabilities.
Design and analyze experiments to measure the impact of model improvements.
Generate insights that inform model training data, prompts, and product features.
Identify trends in customer interactions to uncover:
new AI automation opportunities
gaps in intent coverage
common customer pain points
Translate analytics insights into actionable recommendations for product and AI teams.
Lead and mentor a team of data analysts focused on AI performance and conversational analytics.
Establish best practices for analytics, experimentation, and AI evaluation.
Collaborate closely with product, AI research, engineering, and customer success teams.
Team Leadership
Model Improvement & Experimentation
Conversational Insights
AI Performance Analytics
LLM Evaluation & Quality Insights
Requirements
7–10 years of experience in data analytics, data science, or AI analytics.
Experience managing or mentoring analytics teams.
Experience analyzing conversational data, customer support data, or AI-driven products is a strong plus.
Strong SQL and data analysis skills.
Experience with BI tools such as Tableau or similar platforms.
Experience using Python for data analysis or experimentation.
Familiarity with machine learning or large language models (LLMs).
Experience evaluating AI or machine learning model performance.
Experience working with conversational AI platforms or chatbots.
Experience with experimentation frameworks and A/B testing.
United States