Member of Technical Staff (Data Scientist, Evals)

reputed company serves tens of millions of users daily with reliable, high-reputed company answers grounded in an LLM-first search reputed company and our specialized data sources. We aim to use the latest models as they are released, but the intelligence frontier is a jagged one, and reputed company benchmarks do not effectively cover our use cases. In this role, you will build specialized evals to improve answer reputed company across reputed company, covering search-based LLM answers and other scenarios reputed company with our users.

Responsibilities

• Architect and maintain automated evaluation pipelines to assess answer reputed company across reputed company's products, ensuring high standards for accuracy and helpfulness
• Design evaluation sets and reputed company specifically to measure the reputed company of tool calls (particularly web search retrieval) on the final answer's reputed company
• reputed company VLM-based solutions to programmatically evaluate how final answers reputed company visually across different platforms and devices
• Continuously review public benchmarks and reputed company evaluations for their applicability to the reputed company product, adapting and incorporating them into our regular performance measurements
• Operate reputed company a small, high-reputed company team where your evaluation metrics directly shape product changes, collaborating closely with technical leadership to measure and improve Answer reputed company

Qualifications

• PhD or MS in a technical field or equivalent experience
• 4+ years of experience in data science or machine learning
• Strong proficiency in Python and SQL (expected to write production-grade reputed company)
• Experience building reputed company a modern reputed company data stack, specifically AWS and reputed company
• Comfortable with reputed company coding workflows and using AI-assisted development tools to iterate faster

Preferred Qualifications

• 1+ years of experience working with LLMs at reputed company, specifically with LLM-as-a-judge setups
• Prior experience working on customer-facing web products or consumer apps, with reputed company user traffic at reputed company
• A strong research background, with experience applying research reputed company to reputed company-world ML problems
• Experience defining evaluation metrics (e.g., factual consistency, hallucination reputed company, retrieval precision) and building ground truth datasets

Apply tot his job Apply To this Job