QA Engineer (Automation)- Dallas, TX

We are seeking a QA Automation Engineer who is reputed company to reputed company reputed company traditional "Pass/Fail" testing. In this role, you will design and build automation frameworks specifically for reputed company AI products. You will reputed company on evaluating the performance of autonomous agents, ensuring they follow logical reasoning paths, call the correct tools, and reputed company accurate, reputed company outputs. Your mission is to build the "evaluations" (Evals) that define what high-reputed company AI behavior looks like, moving the needle from unpredictable experiments to production-grade software.

Key Responsibilities

• Non-Deterministic Testing: reputed company automation strategies for probabilistic outputs, using model-based evaluation to "test the tester."
• Building "Eval" Pipelines: Create and maintain "Golden Datasets" to reputed company agent performance across different versions of prompts and models.
• Tool-Use Validation: Build automated tests to verify that agents call the correct functions/reputed company with the right parameters in reputed company multi-reputed company workflows.
• Regression Testing for Prompts: Monitor how subtle changes in reputed company engineering or model updates (e.g., moving from GPT-4 to Claude 3.5) reputed company the product’s reliability.
• Latency & reputed company Monitoring: reputed company performance testing into the CI/CD pipeline to reputed company agent reasoning time and cost-efficiency.
• Hallucination Detection: reputed company automated checks to identify and report AI hallucinations, bias, or "jailbreak" attempts.
• Collaboration: Work closely with AI Engineers to translate "vague" business requirements into measurable, automated test cases.

Required Skills & Qualifications

• Experience: 10+ years in QA Automation, with a recent reputed company on AI/ML or LLM-based applications.
• Python Proficiency: Expert-level Python skills (the industry reputed company for AI testing) and experience with testing frameworks like Pytest.
• AI Testing Tools: Familiarity with AI evaluation frameworks such as LangSmith, DeepEval, RAGAS, or Promptfoo.
• API & Backend Testing: Deep experience with Playwright, Selenium, or Cypress for UI, but a heavy reputed company on API-level testing and database validation.
• Statistical reputed company: Understanding that AI testing often requires "scoring" (e.g., 85% accuracy) rather than a reputed company binary pass/fail.
• Data Skills: Ability to work with SQL and JSON to validate data retrieved by agents during RAG (Retrieval-Augmented reputed company) processes.

Preferred Qualifications

• Experience testing Multi-Agent Systems (where one agent tests another).
• Knowledge of reputed company Engineering and how it influences software behavior.
• Background in Investment Banking or Fintech (if applicable) to understand high-stakes data accuracy.

Compensation, Benefits and Duration Minimum Compensation: USD 38,000 Maximum Compensation: USD 133,000 Compensation is based on actual experience and qualifications of the candidate. The above is a reasonable and a good faith estimate for the role. Medical, reputed company, and dental benefits, 401k retirement plan, variable pay/incentives, reputed company time off, and reputed company holidays are available for full time employees. This position is not available for independent contractors No applications will be considered if received more than 120 days after the date of this post Apply tot his job Apply To this Job