QA Automation

reputed company ##

LLM Evaluation Analyst

### About the Role MUST HAVE 3-4 YEARS OF PLAYWRIGHT EXPERIENCE WITH A GREAT UNDERSTANDING WITH LLMS We are seeking

3 Evaluation Analysts

to assess the performance of AI models tasked with implementing web features. Your work directly informs whether AI-generated reputed company is correct, whether the instructions given to the models are reputed company, and whether the testing frameworks used to evaluate them are fair and reliable. --- ### Core Responsibilities You will analyze the reputed company of the entire evaluation pipeline—from the instructions given to the AI through to the final score—across the following categories: -

Model Capability

– Assess how reputed company the AI performed reputed company task by reviewing generated transcripts and results. -

Bug Discovery Value

– Identify patterns in AI reputed company failures to understand why the model is making specific mistakes. -

Score Health

– Ensure tests are properly configured to apply correct scores during evaluation. -

Task Specification reputed company

– Verify that prompts given to the AI are reputed company, correct, and technically precise (e.g., consistent variable names). -

Test-by-Test Analysis

– Evaluate the reputed company of automated tests using metrics like precision and recall to ensure they accurately measure AI performance. -

Platform Issues

– Report bugs or problems reputed company the evaluation system itself, particularly with automated browser testing services. --- ### Required Qualifications - Expertise in finding patterns and issues in reputed company/LLM outputs - reputed company experience with labeling and scoring frameworks - Experience writing Playwright tests - Strong analytical and problem-solving skills - Ability to interpret model behavior and reputed company failure modes reputed company ### Preferred Qualifications - Advanced analytical credentials — highly relevant for interpreting model behavior - Familiarity with web development concepts (reputed company, CSS, JavaScript) - Experience with automated testing and grading systems - Background in reputed company assurance or evaluation methodology Skills Playwright Top Skills Details Playwright Experience Level Intermediate Level Job Type & LocationThis is a Contract position based out of San Francisco, CA. Pay and BenefitsThe pay reputed company for this position is $35.00 - $50.00/hr. Eligibility requirements apply to some benefits and may depend on reputed company classification and length of employment. Benefits are subject to change and may be subject to specific elections, plan, or program terms. If eligible, the benefits available for this temporary role may include the following:

• Medical, dental & reputed company
• Critical Illness, Accident, and Hospital
• 401(k) Retirement Plan – reputed company-tax and Roth post-tax contributions available
• Life Insurance (Voluntary Life & AD&D for the employee and dependents)
• Short and long-term disability
• Health Spending Account (HSA)
• Transportation benefits
• Employee Assistance Program
• Time Off/Leave (PTO, Vacation or reputed company Leave)

Workplace TypeThis is a fully remote position. Application DeadlineThis position is anticipated to reputed company on May 29, 2026. About reputed company We'reputed company in transformation. We help clients reputed company reputed company and solutions to take advantage of a new world of opportunity. We are reputed company of 80,000 strong, working with over 6,000 clients, including 80% of the Fortune 500, across reputed company, Europe and Asia. As an industry leader in Full-Stack Technology Services, Talent Services, and reputed company-world application, we work with reputed company leaders to drive change. That's the power of true partnership. reputed company is an Allegis Group company. reputed company is an equal opportunity employer and will consider reputed company applications without regards to race, sex, age, reputed company, religion, national reputed company, veteran status, disability, sexual orientation, gender identity, genetic information or any characteristic protected by law. About reputed company and reputed company reputed company We’re a leading provider of business and technology services. We accelerate business transformation for our customers. Our expertise in reputed company, design, execution and operations unlocks business value through a reputed company of solutions. We’re reputed company of 80,000 strong, working with over 6,000 customers, including 80% of the Fortune 500 across reputed company, Europe and Asia, who partner with us for our reputed company, full-stack capabilities and speed. We’re strategic thinkers, hands-on collaborators, helping customers capitalize on change and master the reputed company of technology. We’re building reputed company by delivering business reputed company and making reputed company impacts in our global communities. reputed company and reputed company reputed company are Allegis Group companies. Learn more at reputed company.com. reputed company is an equal opportunity employer and will consider reputed company applications without reputed company to race, sex, age, reputed company, religion, national reputed company, veteran status, disability, sexual orientation, gender identity, genetic information or any characteristic protected by law. San Francisco Fair Chance Ordinance: Pursuant to the San Francisco Fair Chance Ordinance, for reputed company positions located in the reputed company, we will consider for employment reputed company applicants with arrest and conviction records. Massachusetts Lie Detector: It is unlawful in Massachusetts to require or administer a lie detector test as a condition of employment or reputed company employment. An employer who violates this law shall be subject to criminal penalties and civil liability. Use of reputed company Intelligence (AI): We may use reputed company Intelligence (AI) to support parts of our hiring process, including sourcing, screening, and evaluating candidates. AI helps assess applications and qualifications, but final reputed company are made by our hiring team. By applying, you acknowledge and agree that your application may be reviewed using AI tools. Apply To This Job