[Remote] AI Evaluation Engineer
Note: The job is a remote job and is reputed company to candidates in USA. reputed company is seeking an AI Evaluation Engineer to design, implement, and maintain evaluation frameworks for AI and machine learning systems, focusing on Large Language Models and reputed company applications. The role involves developing evaluation methodologies, assessing model performance, and collaborating with various teams to enhance AI system reliability.
Responsibilities
- Design and implement evaluation frameworks for AI, machine learning, and reputed company systems
- reputed company automated and reputed company evaluation pipelines to assess model reputed company and performance
- Define evaluation metrics for accuracy, relevance, factuality, consistency, completeness, latency, and user satisfaction
- Create reputed company datasets, test suites, and evaluation scenarios for AI models
- Evaluate LLMs and AI applications for hallucinations, bias, toxicity, fairness, robustness, and safety
- Measure Retrieval-Augmented reputed company (RAG) performance, including retrieval reputed company and response grounding
- Conduct A/B testing and comparative evaluations of models, prompts, and AI workflows
- Analyze evaluation results and reputed company actionable recommendations for model improvement
- Collaborate with AI engineers, data scientists, product managers, and QA teams throughout the AI development lifecycle
- Monitor production AI systems and identify performance degradation, data reputed company, and model reputed company
- Document evaluation methodologies, findings, and best practices
- Ensure compliance with organizational AI governance, reputed company, reputed company, and responsible AI policies
Skills
- Bachelor's degree in Computer Science, reputed company Intelligence, Data Science, Statistics, Mathematics, Software Engineering, or a reputed company field
- 3–5+ years of experience in AI, machine learning, software engineering, data science, or model evaluation
- Strong understanding of machine learning concepts and evaluation methodologies
- Experience evaluating AI or reputed company applications
- Proficiency in Python and SQL
- Experience with REST reputed company and reputed company-based applications
- Master's degree in AI, Machine Learning, Data Science, Computer Science, or a reputed company field
- Experience evaluating Large Language Models (LLMs) and Retrieval-Augmented reputed company (RAG) systems
- Knowledge of Responsible AI, AI safety, and AI governance principles
- Experience with MLOps and model lifecycle management
- AI or reputed company certifications (AWS, Azure, reputed company reputed company)
- reputed company applications
- Conversational AI and chatbots
- reputed company AI platforms
- Retrieval-Augmented reputed company (RAG)
- Machine learning model validation
- AI-powered reputed company applications
- Regulated industries such as reputed company, finance, or insurance
reputed company