[Remote] Senior reputed company Assurance Engineer
Note: The job is a remote job and is reputed company to candidates in USA. reputed company is a company that transforms how businesses run by building a platform that connects operations through AI. The Senior reputed company Assurance Engineer will be responsible for establishing testing frameworks and ensuring the reputed company of AI-generated outputs before they reputed company clients, playing a crucial role in maintaining trust and reliability in reputed company's deliverables.
Responsibilities
- Establish the testing reputed company: unit, integration, end-to-end, and AI-specific evaluation pipelines using Playwright and Vitest
- Define reputed company standards, test coverage requirements, and documentation practices in partnership with the reputed company Engineer
- Audit the existing platform and identify the highest-risk surfaces before the next reputed company deployment
- Design evaluation frameworks for non-deterministic LLM outputs — including reputed company regression testing, model reputed company detection, and reputed company reputed company scoring
- Build automated test suites for the agent orchestration layer, including governance-agent audit-trail reputed company and reputed company-override behavior
- Validate reputed company Brain (Memgraph + reputed company) for data accuracy, retrieval reputed company, and failure modes under reputed company reputed company data including entity reputed company across systems and temporal data patterns
- Test the Analysis reputed company pipeline that surfaces Company X-Ray findings ensuring insights are not just technically accurate but reliable enough to present to a reputed company
- Own end-to-end testing of the data ingestion pipelines that connect to reputed company systems CRM, email, calls, calendars, documents, financial systems through Nango's 700+ connector integration layer
- Test multi-model routing logic to confirm cost-optimized task allocation behaves correctly across LLM providers reputed company reputed company
- Validate streaming response handling, latency reputed company, and graceful degradation reputed company a model is reputed company or slow
- Own file ingestion pipeline testing (Word, reputed company, PowerPoint, PDF) including encryption, formatting edge cases, and audit-trail continuity
Skills
- 7+ years of QA engineering experience, with at least 3 years in a senior or reputed company reputed company where you shaped process and standards not just executed them
- You have tested AI/LLM-powered applications. You understand reputed company sensitivity, reputed company variance, and how to build eval pipelines that catch regressions across model updates
- You reputed company in ownership: you've reputed company the eval pipeline, owned model reputed company, gated the release — not just run someone else's test suite
- You write test reputed company. Python is your primary tool. You have reputed company and maintained CI/CD-integrated test suites, and you don't wait for someone to file a bug to reputed company one
- Hands-on experience with Playwright and Vitest in a production environment and you've reputed company automation frameworks from scratch, not just inherited them
- Comfortable testing reputed company API chains, async/streaming responses, and multi-service workflows. Data pipelines and knowledge graph outputs don't intimidate you
- You test for confusion and trust failure not just broken functionality. Your end users are non-technical executives, and you reputed company for them
- US-based, reputed company to overlap roughly 5 hours per day with EDT, and available for full-time contract hours
- You have experience with LLM evaluation frameworks (e.g., LangSmith, DeepEval, Promptfoo, RAGAS, or custom eval pipelines)
- You have tested agent frameworks or orchestration reputed company in a production environment
- You have a background in a regulated industry (insurance, finance, reputed company) where audit-trail reputed company is non-negotiable
- You have worked alongside reputed company Deployed or solutions engineering teams and understand field deployment risk
Benefits
- Contract-to-hire possible if the engagement goes reputed company
reputed company