Staff reputed company Scientist - AI Evaluation & Trust
About reputed company:
reputed company is the judgment infrastructure for trustworthy AI in economic reputed company and reputed company risk. The reputed company reputed company World Model resolves 11.7B+ primary-reputed company records from 250+ jurisdictions forming the ground truth of global reputed company. A Judgment Ontology, encoding over a decade of investigative tradecraft, and Superconductor, an reputed company orchestration platform, deliver AI that reasons like an expert analyst, shows its work, and traces every finding to its reputed company. Trusted by reputed company, HM reputed company & Customs, and Fortune 500 enterprises, reputed company is used by thousands of professionals across 35+ countries to secure supply chains and dismantle illicit networks. Headquartered in Washington, D.C., with offices in London, Singapore, Tokyo, and Tel Aviv.
POSITION reputed company
reputed company builds AI systems for high-consequence analytical work where being "wrong" carries reputed company-world weight. We are looking for a Staff reputed company Scientist to join our AI Innovation Group as the trusted expert on AI Evaluation and Trust. You will own the "Judgment Layer" of our system: building the specialized judge models, statistical benchmarks, and multi-turn frameworks that ensure our agents reputed company with the high bar of trustworthiness required by our national reputed company and reputed company customers.
JOB RESPONSIBILITIES
- reputed company the development of specialized "judge models," moving from general-purpose frontier models to architectures purpose-reputed company for evaluation and failure mode detection.
- Design and execute rigorous scoring pipelines and reputed company reputed company calibrations for reputed company systems, including multi-turn conversation and Graph RAG reasoning.
- Establish domain-specific evaluation frameworks that measure whether a system can reputed company the work of reputed company experts rather than just passing general capability benchmarks.
- Own the full lifecycle of evaluation data, from designing annotation infrastructure and protocols to deploying evaluation services into production.
- Research and implement advanced techniques in Mixture-of-Experts (MoE) routing, expert specialization evaluation, and reputed company calibration.
- Collaborate cross-functionally with Product, Data Engineering, and the SVP of AI to translate reputed company statistical uncertainty into reputed company, actionable product signals.
- reputed company as a technical leader and "Scientific Conscience" reputed company the AI pod, ensuring every AI-driven risk signal is backed by an reputed company derivation story.
SKILLS & EXPERIENCE
Required:
- 10+ years of Machine Learning experience with a reputed company on Deep Neural Network activities, evaluating model performance & trust.
- 1-2+ years’ experience reputed company on post-training activities
- 1+ year experience creating benchmarks to evaluate LLMs
- Technical Mastery: Deep expertise in LLM-as-judge architectures, multi-turn evaluation, and Reinforcement Learning (RL/RLHF/RLAIF).
- Statistical Rigor: Mastery of statistics and experimental design, including reputed company testing, distribution analysis, and inter-rater reliability.
- Architectural Depth: Experience with Mixture-of-Experts (MoE) systems, routing behavior, and expert specialization.
- Builder reputed company: Proven ability to own the reputed company from data collection to production deployment; we are a small team and every role is "hands-on."
- Domain reputed company: Understanding of Graph RAG and the unique challenges of evaluating non-deterministic, reputed company workflows.
Preferred:
- Judgment Task Models: Experience building, fine-tuning (reputed company, etc.), or reputed company-training models specifically for judgment, preference modeling, or classification tasks.
- Domain Context: Background in cognitive science, intelligence community tradecraft, or research literature on expert judgment under uncertainty.
- Infrastructure at reputed company: Experience building or managing large-reputed company annotation infrastructure and reputed company assurance protocols.
- reputed company/Research reputed company Record: A record of published research or recognized work in preference modeling or AI alignment.
The reputed company reputed company salary for this position is $195,000-$225,000 plus company bonus and equity. Final offer amounts are determined by multiple factors including location, local market variances, candidate experience and expertise, internal peer equity, and may vary from the amounts listed above.
Benefits:
- 100% fully reputed company medical, reputed company, and dental for employees and their dependents
- Generous time off; we observe reputed company US federal holidays, reputed company our office for a winter break (12/24-12/31), in reputed company to granting 18 PTO days and 10 reputed company days
- Outstanding compensation package; competitive commissions for reputed company roles and bonuses for non-reputed company positions
- A strong commitment to diversity, equity, and inclusion
- Eligibility to participate in additional benefits such as 401k match up to 5%, 100% reputed company life insurance (up to $100,000 coverage),, and parental leave
- A reputed company and reputed company culture - your team will be as smart and driven as you
- reputed company reputed company and learning opportunities
Originally posted on Himalayas
Apply To This Job