Sr QA Automation Engineer - AI

About Anaconda

Be at the center of AI

With more than 47 million users, Anaconda is the most popular operating system for AI providing access to the foundational open-source Python packages used in modern AI, data science, and machine learning through a seamless platform. We pioneered the use of Python for data science, championed its vibrant community, and continue to steward open-source projects that make tomorrow’s innovations possible. Our enterprise-grade solutions enable corporate, research, and academic institutions around the world to harness the power of open source for competitive advantage, groundbreaking research, and a better world. To learn more visit https://www.anaconda.com.

Here is why people love most about working here: We’re not just a company, we’re part of a movement. Our dedicated employees and user community are democratizing data science and creating and promoting open-source technologies for a better world, and our commercial offerings make it possible for enterprise users to leverage the most innovative output from open source in a secure, governed way.

Summary:

Anaconda is seeking a talented Senior AI Engineer with QA expertise to join our rapidly growing company on our Anaconda AI team. In this role, you’ll have the exciting opportunity to work across multiple platforms, focusing on building and testing AI applications and tools, maintaining test frameworks, and ensuring the reliability of our AI core components. This position offers an excellent opportunity to leverage your QA expertise within the AI and machine learning ecosystem while implementing automation solutions that enhance testing efficiency and ensure the quality and reliability of AI models, pipelines, and systems across our offerings.

What You’ll Do:

• Design, develop, and maintain automated test frameworks and test suites for AI/ML-based applications, including APIs, models, and end-to-end workflows
• Design and develop AI-driven agents for quality assurance (QA) to boost overall productivity. Our objective is to build intelligent agents capable of generating code and tests, creating QA metrics dashboards, and automatically logging defects from pipeline failures, along with other efficiency-enhancing tasks
• Collaborate with cross-functional teams, including AI/ML engineers, data scientists, and product managers, to define test strategies and ensure high-quality deliverables
• Develop and execute test plans, test cases, and scenarios tailored to AI applications, focusing on model accuracy, performance, and reliability
• Perform in-depth testing of machine learning models, including validation, inference testing, and edge-case analysis
• Implement continuous testing processes integrated with CI/CD pipelines to enable rapid and reliable releases
• Integrate responsible AI practices into QA workflows by monitoring for potential biases, fairness concerns, and unintended outcomes
• Advocate for best practices in AI testing and validation by creating and maintaining testing guidelines tailored to AI systems and large language models (LLMs)
• Drive continuous improvement of the QA automation stack, including Web, CLI, and API automation, to support the diverse needs of our team
• Analyze test results, troubleshoot flaky tests, and raise defects in Jira with clear, actionable information
• Stay updated on advancements in AI/ML technologies and apply best practices for testing AI systems
• Mentor junior QA engineers and provide guidance on testing strategies for complex AI systems

What You Need:

• 8+ years of experience in developing robust and reliable automation scripts for applications and APIs, including experience in QA automation, particularly in AI/ML projects and related domains
• Experience in building AI-driven agents and developing custom GPTs to enhance productivity through automation, intelligent testing, and process optimization
• Strong background in Python development and familiarity with AI/ML concepts and testing methodologies
• Experience with testing models for performance, accuracy, reliability, and identifying biases
• Experience with ML model testing strategies, including validation, verification, and data quality assessment techniques
• Hands-on experience with test automation tools (e.g., Playwright, Cypress)
• Proven expertise in CI/CD pipelines (we use GitHub Actions, ArgoCD); familiarity with other tools (Jenkins, Gitlab, AWS CodePipelines) is also valuable
• Solid understanding of version control (Git) and best practices for branching and merging
• Experience with Linux command line and scripting languages for test automation and build processes
• Demonstrated experience in delivering product features within Agile or Scrum teams
• Willingness to learn new technologies, frameworks, and workflows to improve the QA landscape

What Will Make You Stand Out:

• Expert level experience with various AI tools - ChatGPT, Claude, GitHub Copilot, Cursor, etc.
• Deep familiarity and experience with AI and LLMs testing strategies
• Proficiency in CLI (Command Line Interface) automation across Windows, Linux, and Mac
• Good to know containerization technologies like Docker, Kubernetes, etc.
• Contributions to the open-source community (bonus if you maintain an open-source project)

Why You’ll Like Working Here:

• Unique opportunity to translate strong open-source adoption and user enthusiasm into commercial product growth
• Dynamic company that rewards high-performers
• On the cutting edge of enterprise application of data science, machine learning, and AI
• Collaborative team environment that values multiple perspectives and clear thinking
• Employees-first culture
• Flexible working hours
• 100% remote
• Competitive benefits

*FTE employees based on your region

The US base salary range for this role is $135,150-$201,000 + bonus + equity + benefits. Our salary ranges are determined by role, level, and location. The range displayed in each job posting reflects the minimum and maximum target salaries for the position across all US locations.

Within the range, individual compensation is determined by various factors, including work location, job-related skills, experience, and relevant education or training. Your recruiter will provide more specific details on the salary range for your preferred location during the hiring process.

In addition to base salary, we offer a comprehensive benefits package that includes:

• Flexible Vacation Policy
• Medical, Dental, and Vision Insurance
• Short Term and Long Term Disability
• Paid Parental Leave
• Annual Remote Work and Wellness Stipend
• Employee Assistance Program and Mental Health Resources

Application deadline: There is no deadline; we accept applications on an ongoing basis.

An Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, or protected veteran status and will not be discriminated against on the basis of disability.

Anaconda, Inc. (“We”, “Us”) are committed to protecting and respecting your privacy. This Privacy Notice sets out the basis on which the personal data collected from you, or that you provide to Us, will be processed by Us in connection with Our recruitment processes. By clicking “Submit Application”, you acknowledge you have read our Privacy Policy and that Anaconda can retain your application data for up to 1-year, unless otherwise stated. For the purpose of the General Data Protection Regulation (“GDPR”) ”) and the version of the GDPR retained in UK law (the “UK GDPR”) the Data Controller is Vanessa Macilwaine.

Anaconda is an EEO/AA employer M/F/V/D.