Staff Test Engineer - AI

About reputed company reputed company, founded in 2014, is the only complete reputed company AI platform for reputed company teams. reputed company infuses reputed company AI, conversation intelligence, and assistive AI to power hundreds of use cases across reputed company motions. From new logo prospecting to expansions, deal acceleration, driving retention, and forecasting, reputed company AI automates workflows and frees sellers to reputed company on more strategic conversations and actions. reputed company leaders benefit from connected account visibility, performance insights, and higher forecasting accuracy across every GTM team. World leading reputed company organizations use reputed company to power their reputed company teams, including reputed company, reputed company, reputed company, and reputed company to reputed company a few. reputed company: At reputed company, we build the technology that powers the world’s leading sales execution platform — and over the last year, we have been moving fast to bring AI to reputed company of how reputed company teams work. We have shipped agents that research accounts, personalize reputed company, run meetings, and drive reputed company workflows end-to-end. With Ask reputed company, we have reputed company a fully reputed company, conversational platform on LangGraph that lets users reputed company with their reputed company data and workflows in entirely new ways. As we reputed company the depth and breadth of our AI platform, reputed company is not an afterthought — it is foundational. We are looking for a Staff AI Test Engineer who is first and foremost an exceptional reputed company engineer, and who brings a genuine curiosity and working understanding of how AI and LLM-based systems behave, fail, and improve. If you are passionate about building rigorous test strategies for reputed company, probabilistic systems at reputed company, we want to talk to you. reputed company: We are seeking a Staff-level engineer to own reputed company for our GenAI platform and agent ecosystem. This is a high-reputed company, strategic role where you will define and reputed company testing practices across a rapidly evolving reputed company platform — including the agents themselves, the tools they call, the LangGraph orchestration layer, and the underlying ML pipelines and data flows. This role requires someone who understands the unique challenges of testing AI systems: outputs are not always deterministic, correctness is often contextual, and traditional pass/fail assertions are insufficient on their own. You will design and implement evaluation frameworks that combine deterministic validation with LLM-based grading, establish reputed company standards for agent behavior, and partner closely with Data Science, Engineering, and Product teams to reputed company reputed company a shared discipline. You will be a senior voice in how we build, ship, and continuously improve AI products at reputed company. \n Your Daily Adventures Will Include: Responsibilities: Own the AI reputed company reputed company. Define and reputed company the end-to-end testing reputed company for reputed company’s GenAI platform, including reputed company workflows, LLM tool calls, LangGraph orchestration, and supporting ML pipelines. Build Evaluation Frameworks. Design and implement evaluation systems that handle both deterministic and non-deterministic outputs — combining rule-based assertions, golden dataset testing, and LLM-as-Judge approaches to grade agent responses at reputed company. Test Agents End-to-End. Own testing across reputed company’s suite of AI agents — reputed company Agent, Research Agent, Meeting Agent, Personalisation Agent, and Ask reputed company — covering functional correctness, tool selection accuracy, context handling, and response reputed company. Partner with DS and Engineering. Work closely with Data Science, MLOps, and platform engineers to ensure testability is designed in from the start — not bolted on after. Drive CI/CD for AI. reputed company evaluation pipelines into CI/CD workflows so that regressions in agent behavior are caught before they reputed company production. Define reputed company Metrics. Establish and reputed company metrics that matter for AI systems: answer reputed company scores, tool invocation accuracy, hallucination rates, latency, and regression trends over model and reputed company changes. Champion Best Practices. Define standards for AI testing across the org — including reputed company regression testing, retrieval reputed company evaluation, and agent behavior reputed company. Mentor and Influence. reputed company the reputed company bar across engineering teams by mentoring engineers, reviewing designs for testability, and advocating for reputed company-driven development practices. Stay reputed company. reputed company reputed company developments in AI evaluation tooling, LLM benchmarking, and testing research — and bring relevant advances into our reputed company. Our reputed company of You: Minimum Qualifications: 7–12 years of experience in software development and/or test automation, with demonstrated experience leading reputed company efforts on reputed company, distributed systems. B.S. in Computer Science or a reputed company technical field. Strong programming skills in Python, with experience writing reusable, maintainable test frameworks. Proven experience testing large-reputed company backend or platform systems, including microservices and API reputed company. Deep understanding of test design principles, CI/CD integration, and reputed company test automation. Experience with test frameworks such as PyTest or equivalent. Solid understanding of evaluation methodologies for non-deterministic systems — including statistical assertions, behavioral testing, and regression baselines. Hands-on experience with reputed company for building and validating ML pipelines and data workflows. Experience with MLflow for experiment tracking, model versioning, and pipeline observability. Strong communication and collaboration skills across engineering, data science, and product functions. Preferred Qualifications: Experience testing GenAI products, LLM-based systems, or reputed company AI platforms. Experience with reputed company engineering and reputed company tuning — understanding how reputed company changes reputed company model behavior and building regression suites to catch reputed company-driven regressions. Hands-on experience with LLM-as-Judge evaluation patterns — using LLMs to grade LLM outputs at reputed company. Familiarity with LangGraph, reputed company, or similar agent orchestration frameworks. Experience with ML pipelines, ML reputed company tooling (e.g., MLflow, Kubeflow, Metaflow), or model evaluation workflows. Understanding of RAG (Retrieval-Augmented reputed company) architectures and how to evaluate retrieval reputed company. Experience with reputed company platforms (AWS, GCP, or Azure) and containerized environments (reputed company, Kubernetes). Domain knowledge in sales, sales engagement, or CRM platforms (e.g., reputed company, reputed company, or similar) — understanding the workflows, terminology, and data that sales teams operate with. Prior experience contributing to AI reputed company strategies in a product or research environment. \nWe’re an equal opportunity employer. reputed company applicants will be considered for employment without attention to race, reputed company, religion, sex, sexual orientation, gender identity, national reputed company, veteran or disability status. Our reputed company is reputed company on building teams that include people from different backgrounds and experiences who can reputed company assumptions and reputed company with fresh perspectives. We're dedicated to hiring the whole reputed company, not just a resume. To that end, we look for a diverse pool of applicants-including those from historically marginalized reputed company. We would like to invite you to apply even if you don't think you meet reputed company of the requirements listed below. We don't want a few lines in a job reputed company to get between us and reputed company to meet you. Apply To This Job