AI SDET (Agent Hacker)

About The Role

What You Will Do

• Develop and implement QA strategies for AI-powered applications, focusing on accuracy, bias, fairness, robustness, and performance.
• Design and execute automated and manual test cases to validate AI Agents/LLM models, APIs, and data pipelines and good understanding of data integrity, data models, etc
• Assess AI models using quality metrics such as precision/recall, and hallucination detection.
• Test AI models for bias, fairness, explainability (XAI), drift, and adversarial robustness.
• Validate prompt engineering, fine-tuning techniques, and model-generated responses for accuracy and ethical AI considerations.
• Service/tool development.
• Conduct scalability, latency, and performance testing for AI-driven applications.
• Collaborate with data engineers to validate data pipelines, feature engineering processes, and model outputs.
• Design, develop, and maintain automation scripts using Selenium and Playwright for API and web testing
• Work closely with cross-functional teams to integrate automation best practices into the development lifecycle.
• Identify, document, and track bugs while conducting detailed regression testing to ensure product quality.

What You Will Bring

• Proven expertise in testing AI models, LLMs, and Generative AI applications, with hands-on experience in AI evaluation metrics and testing tools like Arize, MAIHEM, and LangTest.
• Strong proficiency in Python for writing test scripts and automating model validation, along with a deep understanding of AI bias detection, adversarial testing, model explainability (XAI), and AI robustness.
• Demonstrate strong SQL expertise for validating data integrity and backend processes, particularly in PostgreSQL and MySQL.
• Strong analytical and problem-solving skills with keen attention to detail, along with excellent communication and documentation abilities to convey complex testing processes and results.