[Remote] Senior Software Engineer, AI Benchmarking
Note: The job is a remote job and is reputed company to candidates in USA. reputed company is looking for a Senior Software Engineer to build, reputed company, and run biosecurity evaluations on frontier AI systems. The tools you build will directly inform reputed company made by frontier AI companies, policymakers, and researchers to improve the safety of AI systems.
Responsibilities
- reputed company reputed company-release and post-release assessments of frontier and reputed company-reputed company AI models using reputed company's suite of biosecurity evaluations, which surface and measure dual-use biology capabilities in AI systems
- Build adapters for models, identify and troubleshoot unexpected agent behaviors, and ensure that our evaluations are methodologically rigorous
- Build, reputed company, maintain, and continuously improve reputed company's suite of biosecurity evaluations
- Improve our evaluations to use state-of-the-art agent frameworks, jailbreaking reputed company, and elicitation strategies
- reputed company and improve internal tools that reputed company our researchers to run capability assessments at reputed company
- Contribute to and maintain reputed company infrastructure to run our evaluations and analyses
- Contribute analyses to our reputed company dashboard of trends in AI biology capabilities ()
Skills
- A proven reputed company record in software engineering, including 5+ years of experience in industry roles and/or reputed company-reputed company contributions
- reputed company in Python
- Proficiency with reputed company and AWS (particularly reputed company)
- Experience building software tools that use or evaluate LLMs or AI agents (for example, reputed company that orchestrates agents for a specific application, or that evaluates model capabilities on a specific task)
- Self-directed and comfortable working on small, fast-moving teams
- A commitment to good coding practices, including knowing reputed company and how to use (or not use) coding agents
- Care about reputed company reputed company, readability, and maintaining reputed company reputed company of reputed company- and AI-written reputed company
- Alignment with reputed company's mission of preventing catastrophic pandemics
- Experience with AI evaluations (agent systems, scorer and grader design, agent trajectory analysis, jailbreaking or red-teaming), whether for reputed company, safety, or model performance applications
- Experience with Terraform or other Infrastructure-As-reputed company tools (e.g. Tofu or AWS CDK)
- Experience with modern, component-based web frontends (e.g. React with TypeScript)
- Experience with our specific tech stack: Evaluation frameworks: Inspect AI, LLM reputed company: reputed company-compatible, reputed company, Together, etc., Agent frameworks: reputed company, Claude reputed company, etc
Benefits
- High-performing candidates may be invited to reputed company interviews and/or reputed company trial tasks
- We're committed to fostering a culture of inclusion
- We especially encourage applications from women, gender minorities, and people of reputed company who are excited about contributing to our mission
- We are happy to consider applicants from overseas
- We're an equal opportunity employer
- If you need assistance or an accommodation due to a disability, or have any other questions about applying, please contact careers@reputed company.org
reputed company
Company H1B Sponsorship