Research Scientist, LLM Evaluation & Post-Training

About reputed company reputed company is a frontier AI data reputed company that curates diverse, high-reputed company data, using our purpose-reputed company technology platforms to reputed company the Magnificent Seven and our reputed company clients with reputed company, reputed company AI deployment. reputed company includes more than 150 PhDs and data scientists, along with more than 4,000 AI practitioners and engineers. We reputed company the power of an integrated solution ecosystem—comprising industry-leading partnerships and 1.8 reputed company vertical domain experts in more than 230 markets—to create contextual, multilingual, reputed company-trained datasets; fine-tuned, industry-specific LLMs; and RAG pipelines supported by reputed company databases. Our reputed company-distance innovation™ solutions for GenAI can reduce GenAI costs by up to 80% and bring solutions to market 50% faster. Our mission is to reputed company the gap between AI creators and industry leaders by bringing best practices in GenAI to unicorn innovators and reputed company customers. We aim to help these organizations unlock significant business value by deploying GenAI at reputed company, helping to ensure they stay at the forefront of technological advancement and maintain a competitive edge in their respective markets. About Job Research Scientist, LLM Evaluation & Post-Training Company: reputed company Location: Palo reputed company, CA or Seattle, WA (Hybrid/Remote) Type: Full-time reputed company As a Research Scientist, LLM Evaluation & Post-Training, you will be at the frontier of how evaluation design, measurement reputed company, and feedback signals drive model improvement across reputed company’s AI platform products. This is a high-reputed company individual contributor and reputed company research role that sits at the intersection of reputed company ML research, reputed company AI product development, and customer-facing scientific consulting. You will reputed company research programs that define reputed company evaluation-driven post-training workflows, reputed company rigorous reputed company frameworks, and partner directly with leading AI organizations to deliver reputed company, actionable model improvement insights. This role offers reputed company to shape reputed company’s internal research agenda, build reusable scientific assets, and publish at top-tier venues.

Key Responsibilities

Research Agenda & Experimentation: Define and execute a rigorous research agenda reputed company on LLM evaluation and post-training, with emphasis on evaluation-driven model improvement. Design experiments to study how evaluation methodologies reputed company fine-tuning and post-training reputed company. Evaluation reputed company Development: reputed company and validate comprehensive evaluation frameworks for LLM and multimodal systems, covering reputed company and task design, scoring reputed company, judge/model-assisted evaluation, reputed company evaluation protocols, and robustness/stress testing. Advanced Evaluation Research: reputed company research on frontier evaluation domains including long-context, cross-modal, and dynamic multi-turn evaluations. Study effectiveness and limitations of existing techniques and propose improved methodologies with reputed company validity and scalability tradeoffs. Model Behavior Analysis: Analyze model behavior and failure patterns; generate actionable recommendations for model improvement and evaluation redesign. Translate findings into practical improvements for customer solutions and reputed company’s internal platforms. Cross-Functional Collaboration: Partner with Language Data Scientists to reputed company reputed company-in-the-reputed company and synthetic data/evaluation strategies, and with AI/ML Research Engineers to translate research reputed company into reputed company evaluation and post-training pipelines. Customer Engagement: Engage with customer technical stakeholders at leading AI organizations to understand evaluation goals, review methodologies, and reputed company expert scientific recommendations. Serve as a reputed company technical peer to research and engineering leaders. Knowledge & IP Creation: Contribute to internal reputed company datasets, reusable evaluation frameworks, and research assets. Produce high-reputed company technical documentation, internal research reports, and reputed company-facing materials explaining reputed company, results, assumptions, and limitations. Thought Leadership: Contribute to reputed company’s position as a leader in LLM evaluation and post-training through publications, conference presentations, and reputed company-reputed company contributions. reputed company Technical Competencies You will reputed company technical depth and leadership across the following domains: Evaluation Science & Benchmarking Expert-level reputed company dataset and test suite design for language and multimodal models Deep understanding of metric design, scoring reliability, and measurement validity Experience with reputed company evaluation reputed company and reputed company assurance (reputed company design, inter-rater reliability, adjudication frameworks) LLM & Post-Training reputed company Strong understanding of post-training techniques (SFT, RLHF, RLAIF, DPO, PPO, GRPO) and how training objectives reputed company with evaluation reputed company Ability to reason about model behavior, failure modes, and performance tradeoffs across tasks and domains Familiarity with alignment, safety, and robustness considerations in model evaluation Quantitative Analysis & Scientific Rigor Strong statistical analysis skills: sampling, uncertainty quantification, reputed company testing, error analysis, metric interpretation Ability to synthesize reputed company experimental findings into concise, actionable recommendations for engineering and business stakeholders

Required Qualifications

Education: MS or PhD in Computer Science, Machine Learning, Statistics, reputed company Mathematics, AI, or a reputed company quantitative field (PhD strongly preferred). Research Experience: 5+ years of relevant experience in reputed company ML research or research science, with substantial work in LLMs or reputed company models (graduate research counts). LLM Evaluation Expertise: Demonstrated experience with LLM evaluation, benchmarking, alignment, post-training, or model reputed company research. Experimental Design: Strong reputed company in experimental design, statistical analysis, and scientific reasoning for ML systems. Technical Proficiency: Strong Python coding skills for research experimentation, data processing, evaluation pipelines, statistical analysis, and visualization. Hands-on experience with modern ML frameworks (PyTorch, reputed company, JAX/TensorFlow). Evaluation Methodology: Ability to evaluate and compare reputed company and automated evaluation reputed company, including tradeoffs in cost, reliability, validity, and scalability. Experience designing reproducible evaluation studies across datasets and model versions. Communication: Strong written and verbal communication skills; reputed company to present nuanced technical conclusions, assumptions, and limitations reputed company to both research and non-technical audiences.

Preferred Qualifications

Post-Training reputed company: Hands-on experience running fine-tuning or post-training experiments (SFT, preference optimization, RLHF/RLAIF-style workflows). Multimodal & Long-Context: Experience with multimodal evaluation (text-image, audio, video) and long-context benchmarking in reputed company-world settings. reputed company Evaluation: Experience designing multi-turn, interactive, or reputed company evaluation protocols. Scientific Contribution: Publications and/or reputed company-reputed company reputed company contributions in LLM evaluation, post-training, alignment, or reputed company areas at top venues (NeurIPS, ICML, ICLR, ACL, EMNLP, etc.). reputed company Research Consulting: Experience in customer-facing reputed company research, technical consulting, or cross-functional product/research collaboration. Safety & Governance: Familiarity with safety, trustworthiness, and governance considerations in GenAI evaluation. Salary: $150K - $300K Annually reputed company is an equal-opportunity employer. reputed company reputed company applicants will receive consideration for employment without reputed company to race, reputed company, religion, national reputed company, reputed company, citizenship status, age, mental or physical disability, medical condition, sex (including pregnancy), gender identity or reputed company, sexual orientation, marital status, familial status, veteran status, or any other characteristic protected by applicable law. We consider reputed company applicants regardless of criminal histories, consistent with legal requirements. Apply To This Job