Japanese AI Model Rater

AI Model Rater, Japanese (Coding knowledge required, with focus on APIs)

Join our team of Japanese speakers with coding knowledge! Apply now to help improve the next generation of AI models' tool-use capabilities.

Location: Remote, global (not US-based)

Status: Project-based

Pay: USD $28.80 per hour

Benefits: Flexible schedule

To ensure eligibility for this project, applicants must be international freelancers. Unfortunately, we are unable to hire U.S. citizens, Green Card holders, or candidates currently residing in the U.S.

Project Requirements:

• Advanced/Expert fluency in Japanese language, as spoken in the Japan region (ISO: ja_jp)
• Strong comprehension of English (read and write)
• Strong conceptual knowledge of APIs and digital services (ability to understand how AI tools interact with external apps like search, maps, or calendar, without needing to write code).
• Ability to follow complex technical guidelines precisely.

The Project:

We are seeking detail-oriented and motivated Japanese speakers to join our AI LLM evaluation project. This project focuses on assessing the accuracy and quality of AI-generated responses that rely on integrated applications and tools (APIs).

The work involves evaluating a variety of AI-generated outputs, specifically paying attention to the correct and appropriate use of tool outputs (e.g., verifying that a proposed calendar event is correct, or a search result is relevant). This task evolves reading and judging the quality of API code. You must be able to adapt to various API standards by efficiently reading the provided API documentation.

The sample conversations are displayed in Japanese (as spoken in Japan). The entire evaluation must be written inEnglish.

All work must be completed in accordance with the detailed guidelines provided upon acceptance. Please note that this work is project-based, and the workload may vary. There is no guarantee of a minimum or maximum number of hours.

Duties:

• Analyze and rate AI-generated responses and their associated tool outputs against a detailed set of quality guidelines.
• Verify that the AI successfully integrates information from different digital services (APIs) as required by the prompt.
• Evaluate the accuracy, completeness, and overall quality of the AI's response and tool execution.
• Follow provided instructions to achieve task goals, with all evaluations recorded in English.

Engagement Requirements:

• All participants must have, or be willing to create, an Upwork account. The project will be managed via the Upwork platform.
• While participating in this project, adherence to the confidentiality terms outlined in the Upwork User Agreement is required. Any information accessed or received related to this project is confidential and may not be shared or disclosed to third parties.
• Applicants must pass a quick skill verification assessment (MC, duration ~10 minutes).
• Work must be performed on participant’s own devices.
• You must use your own laptop, and smartphone (if required).
• A Gmail account may be required to access certain tools. If you do not have a Gmail account, you must be willing to create one.
• Using AI during the assessment and work is STRICTLY forbidden, and results/inputs will be monitored to detect AI usage. Any detections will result in immediate ban from any future engagements.

About Us:

As a global data company, Productive Playhouse “PPH”, is pioneering our approach to language and data services, while incorporating their roots as a production company. Originally creating content to support children's language acquisition, our commitment to excellence, forward-thinking strategies, and world-wide cultural experience has proven key for delivering exceptional service.

Originally founded as an educational production company, Productive Playhouse made a mark with our award winning children’s series, which taught fundamental subjects through engaging and effective programming. This early success paved the way for our evolution into a comprehensive data services provider.

Since 2011, Productive Playhouse has expanded rapidly to offer an extensive suite of data services. Our current offerings include transcription, translation, linguistic analysis, rating, systems testing, localization, field and studio recording, language skill verification, and specialized data handling with a focus on sensitivity and diversity. Our commitment to innovation means we continually enhance our service portfolio to meet the evolving needs of our clients.

At Productive Playhouse, we are proud of our reputation for addressing complex challenges with agility and delivering premium secure data solutions across diverse environments. Our dynamic team is dedicated to maintaining the highest standards and ensuring exceptional service every time.

Disclaimer

The job description provided is designed to convey information essential to understanding the scope of the position and the general nature of the work performed. It is not intended to be an exhaustive list of duties, responsibilities, or qualifications associated with the job. Productive Playhouse reserves the right to modify or revise the job description as necessary

Productive Playhouse is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive work environment for all employees. Employment with Productive Playhouse is at-will, meaning that either the employee or the employer can terminate the employment relationship at any time, with or without cause or notice.

All offers of employment at Productive Playhouse are contingent upon the candidate’s ability to provide valid documentation of identity and eligibility to work in the United States or relevant hiring location. Productive Playhouse participates in E-Verify.

Productive Playhouse provides reasonable accommodation for qualified individuals with disabilities. If you need assistance or accommodation due to a disability, please contact the Human Resources Department.