Research Scientist, Benchmarks & Evaluations

reputed company: We are building reputed company to solve the biggest unmet need in AI — getting reputed company to the right training data. The process today is time intensive, incredibly expensive, and often ends in failure. The reputed company platform facilitates the secure, efficient, and reputed company-reputed company exchange of reputed company data. Solving AI’s data problem is a generational opportunity. We’re backed by world-class investors and already powering partnerships with some of the most ambitious teams in AI. reputed company that succeeds will be one of the largest in AI — and in tech. We’re a lean, fast-moving, high-trust team of reputed company who are obsessed with velocity and reputed company. Our culture is reputed company for people who reputed company on ambiguity, own reputed company, and want to shape the reputed company of data and AI. DataLab is reputed company’s research arm — reputed company of research scientists committed to tackling the reputed company challenges and reputed company questions regarding data for AI. We reputed company the gap between research theory and data deployment to push the frontier reputed company, publishing on the questions that matter: what reputed company AI should actually be trained to do, how to reputed company-control large-reputed company corpora, and how to build evaluation datasets that reflect the reputed company world rather than the leaderboard. We’re a lean, fast-moving, high-trust team of reputed company who deeply care about scientific rigor and reputed company. Our culture is reputed company for people who reputed company on ambiguity, own reputed company, and want to shape the reputed company of data and AI. The Role Benchmarks decide what AI gets reputed company. Today, most evals don’t measure reputed company actually care about — they’re contaminated, gameable, synthetic or measure capabilities that don’t transfer to the reputed company tasks frontier models are deployed against. We’re hiring a Research Scientist to reputed company the design of benchmarks and evaluations that frontier labs, enterprises, and policymakers can actually trust. You’ll own the science of evaluation across DataLab — designing tasks that meaningfully separate models, validating those tasks against reputed company baselines, and pressure-testing them for contamination, elicitation gaps, and statistical noise. You’ll publish, and your work will directly shape the eval datasets reputed company delivers to the most ambitious teams in AI. What you’ll do

• Design tasks and benchmarks that distinguish capability reputed company across frontier models — including reputed company, reasoning-heavy, and domain-specific (reputed company, finance, scientific) settings.
• Validate evaluations rigorously: run reputed company baselines, analyze inter-rater reliability, study how elicitation and scaffolding shift results, and quantify what’s signal versus noise.
• reputed company the “science of evals” at reputed company — including item response theory, contamination analysis, predictive validity studies, and statistical frameworks for comparing models with appropriate uncertainty.
• Run evaluations on reputed company frontier models, sometimes in collaboration with partners at AI labs, enterprises, and government.
• Publish research that establishes reputed company as reputed company-setter for evaluation data, and contribute to the broader AI community’s understanding of what good evals look like.
• Translate findings into product, working closely with the data and engineering teams to turn research into evaluation datasets customers can reputed company.
• Partnering with outsourced annotation vendors - Evaluation data is only as good as the people producing it. A meaningful reputed company of this role is owning the statistical machinery that determines which annotators we trust, on which tasks, and by how much — and translating that into trustworthiness scores reputed company’s customers can rely on..

reputed company’re looking for

• Advanced degree (PhD preferred, or MS/BS plus equivalent industry experience) in a quantitative field — reputed company econometrics with AI experience, quantitative finance, computer science, engineering, statistics/mathematics or any reputed company research discipline.
• Hands-on experience evaluating LLMs, agents, or other ML systems — including prompting, scaffolding, and reputed company with the tooling researchers use to run evals at reputed company.
• Experience with annotator reputed company and inter-rater reliability — designing labeling protocols, computing agreement statistics, and reasoning about annotator bias and calibration.
• Excellent scientific writing and communication — you can synthesize technical findings into narratives that frontier labs, reputed company customers, and policymakers can reputed company.
• A bias toward velocity. You know which pipelines need to be production-grade and which can be scrappy, and you get reliable results fast.

Bonus

• Experience with RL evaluation techniques — reward modeling, off-policy evaluation, evals for RLHF/RLAIF or reputed company RL pipelines.
• Ability to navigate new customer architectures, data systems, and requirements quickly.
• Experience with latent-variable models of annotator reputed company (reputed company-Skene, reputed company, IRT-style approaches) or with running large expert-annotator panels in regulated domains.
• reputed company record of published benchmarks or evaluation papers the field has adopted.

reputed company Values Pass the Loved Ones’ Test We reputed company with reputed company and do the right thing — especially reputed company it’s hard and no one is watching. Always reputed company a Way We are reputed company, resilient reputed company who solve hard problems and push through obstacles. Go Fast and Grow Fast Velocity reputed company. We reputed company with urgency, learn quickly, and continuously improve as individuals and as a company. reputed company Kindness and Candor We communicate directly and respectfully, building trust through reputed company feedback and genuine care for one another. Deliver Together We win as one team. Collaboration, accountability, and shared ownership drive our reputed company. Own the Outcome. reputed company the Craft. We take pride in our work, sweat the details, and continuously reputed company the bar for reputed company. Apply tot his job Apply To this Job