information technology (IT) analyst
Title posted on Talent.com - AI Evaluation Expert - Rubric Specialist
Posted on November 17, 2026 by Employer details Mercor
Job details
<p><strong>About the job</strong></p> <p><strong>Mercor</strong> connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include <strong>Benchmark</strong>, <strong>General Catalyst</strong>, <strong>Peter Thiel</strong>, <strong>Adam D'Angelo</strong>, <strong>Larry Summers</strong>, and <strong>Jack Dorsey</strong>.</p> <p><strong>Position:</strong> AI Evaluation Rubric Reviewer<br/> <strong>Type:</strong> <strong>Contract</strong><br/> <strong>Compensation:</strong> <strong>$40?$75/hour</strong><br/> <strong>Location:</strong> <strong>Remote</strong></p> <p><strong>Role Responsibilities</strong></p> <ul> <li>Review <strong>AI-drafted tasks</strong> and evaluate <strong>model attempts</strong> for fairness and accuracy.</li> <li>Assess if the <strong>grading rubric</strong> aligns with the prompt and correct any unfair criteria.</li> <li>Flag tasks that cannot be saved and provide concise feedback on each evaluation.</li> <li>Analyze long agent transcripts to identify errors and improve <strong>AI model performance</strong>.</li> <li>Ensure rubric structures are correct per guidelines and enhance <strong>training data quality</strong>.</li> </ul> <p><strong>Qualifications</strong></p> <p><strong>Must-Have</strong></p> <ul> <li>Substantial experience writing or reviewing rubrics for <strong>LLM evaluations</strong>.</li> <li>Ability to read long agent transcripts and identify errors.</li> <li>Precise writing skills and a commitment to data verification.</li> </ul> <p><strong>Application Process (Takes 20?30 mins to complete)</strong></p> <ul> <li>Upload resume</li> <li>AI interview based on your resume</li> <li>Submit form</li> </ul> <p><strong>Resources & Support</strong></p> <ul> <li>For details about the interview process and platform information, please check: https://talent.docs.mercor.com/welcome</li> <li>For any help or support, reach out to: support@mercor.com</li> </ul> <p><em>PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.</em></p>- Location Toronto, ON
- Work location On site
- Salary$40.00 to $75.00HOUR hourly
- Terms of employment Part time
- Starts as soon as possible
- vacancies 1 vacancy
- Source Talent.com #646460838536028168
Important notice: This job posting has been provided by a partner site. Job Bank is not responsible for this content.
Report a problem with this job posting
Page details
- Date modified: