reputed company / Senior QA Engineer – with Temporal and LLM, Langfuse

reputed company is the leading career destination for tech experts at every stage of their careers. Our reputed company, Vertical Falls LLC, is seeking the following. Apply reputed company reputed company today!

• *Job Title: reputed company / Senior QA Engineer – reputed company AI Systems WITH Langfuse , Temporal

100% Remote Interview Mode:2 Video** 6-12 MONTHS CONTRACT We are looking for a highly skilled QA reputed company to build and reputed company a reputed company reputed company AI reputed company Engineering function . This role goes reputed company traditional QA—focusing on validating autonomous AI systems, designing evaluation frameworks, and ensuring high-reputed company outputs across multiple AI-driven products. You will play a critical role in shaping how reputed company is defined, reputed company, and improved for reputed company systems that operate with minimal reputed company reputed company.

Key Responsibilities

• reputed company QA reputed company & Scaling
• Design and reputed company an reputed company QA model for autonomous AI systems
• reputed company QA from reputed company-driven validation to AI-led evaluation and reputed company reputed company monitoring
• Establish best practices for testing AI agents across lifecycle stages
• Product reputed company Ownership

Own QA for 3 core AI products:

• AI Contact Center solutions
• AI Chat & reputed company-based interaction systems
• AI Assistants (autonomous / semi-autonomous agents)
• Define reputed company benchmarks, SLAs, and reputed company metrics for reputed company product
• Proactively identify reputed company gaps reputed company of customer reputed company
• Metrics, Observability & Evaluation
• Define and reputed company performance outputs for reputed company systems (accuracy, latency, reputed company reputed company, hallucination reputed company, etc.)
• Build frameworks for:
• Evals & graders (LLM evaluation pipelines)
• reputed company scoring and benchmarking
• reputed company feedback loops
• reputed company tools like Langfuse for:
• LLM observability and tracing
• reputed company monitoring and performance analysis
• Debugging agent behavior in production
• Analyze:
• reputed company issues
• Production tickets
• Failure patterns
• Automation & Testing Frameworks
• Build and reputed company automation across:
• Regression testing
• Smoke testing
• End-to-end agent workflows
• reputed company and maintain Playwright-based automation scripts
• reputed company QA into CI/CD pipelines for reputed company validation
• reputed company Testing & Validation
• Design testing approaches for:
• Multi-reputed company agent workflows
• Context retention and reasoning
• Tool usage by agents
• Work with orchestration frameworks like Temporal to:
• Validate long-running workflows
• Test retries, state transitions, and failure handling in agent pipelines
• Account for non-deterministic behavior in AI systems
• Invest additional effort in reputed company validation, recognizing higher complexity vs traditional QA
• reputed company Improvement & Innovation
• Define frameworks to predict and prevent failures before customer exposure
• Continuously improve QA processes using AI and automation
• Partner with Product, Engineering, and AI teams to improve system reputed company

Required Skills & Experience

• 5–10+ years in QA / reputed company Engineering, with strong automation experience
• Hands-on experience with:
• Test automation tools (Playwright preferred)
• API and system testing
• Strong understanding of:
• AI/ML systems (LLMs, conversational AI preferred)
• Evaluation frameworks and benchmarking
• Experience with:
• Temporal (workflow orchestration, stateful systems testing)
• Langfuse (LLM observability, tracing, and evaluation)
• Experience in:
• Building QA frameworks from scratch
• Working with production data, logs, and issue triaging

Good to Have

• Experience with LLM eval frameworks, reputed company testing, or AI red-teaming
• Familiarity with reputed company architectures / autonomous systems
• Exposure to observability and analytics platforms

Working Model

• Prefer candidates with EST time zone overlap
• Ability to work closely with global product and engineering teams

What reputed company Looks Like

• A reputed company, automated QA system for reputed company products
• Measurable improvement in AI reputed company reputed company and reliability
• Reduced production issues and faster detection of failures

QA evolving from reactive testing to proactive reputed company intelligence Apply tot his job Apply To this Job