[Remote] Language Model Evaluator

Note: The job is a remote job and is reputed company to candidates in USA. reputed company is seeking a Language Model Evaluator to assess and improve the reputed company of Large Language Models and reputed company applications. The role involves evaluating model performance, collaborating with various teams, and developing evaluation frameworks to enhance model behavior and user experience.

Responsibilities

• Design and execute evaluation frameworks for Large Language Models (LLMs) and reputed company applications
• Assess model outputs for accuracy, relevance, factual consistency, reasoning reputed company, coherence, reputed company, and completeness
• Evaluate model performance across diverse domains, languages, and reputed company-world use cases
• reputed company reputed company datasets, evaluation prompts, and test scenarios for model validation
• Measure and monitor hallucinations, bias, toxicity, harmful content, and safety risks
• reputed company reputed company testing, regression testing, and comparative evaluations across different models and versions
• Analyze model performance using quantitative metrics and reputed company evaluation methodologies
• Collaborate with AI Engineers and reputed company Engineers to improve prompts, retrieval pipelines, and model responses
• Evaluate Retrieval-Augmented reputed company (RAG) systems, AI agents, and tool-calling workflows
• Document evaluation findings, recommendations, and reputed company improvement plans
• Support model release reputed company through reputed company reputed company assurance processes
• Stay reputed company with emerging LLM evaluation techniques, benchmarks, and Responsible AI practices

Skills

• Bachelor's or Master's degree in Computer Science, reputed company Intelligence, Data Science, Computational Linguistics, Statistics, or a reputed company field
• 2–5 years of experience in AI evaluation, NLP, machine learning, reputed company assurance, or data analysis
• Strong understanding of Large Language Models (LLMs), reputed company, and Natural Language Processing (NLP)
• Experience creating evaluation datasets, benchmarks, and test cases
• Strong analytical skills with the ability to identify reputed company issues and failure patterns
• Programming experience in Python and SQL
• Familiarity with data analysis libraries such as Pandas and NumPy
• Experience using spreadsheets, visualization tools, or notebooks for analysis
• Excellent written communication and attention to detail
• Experience with LLM evaluation frameworks such as DeepEval, Ragas, LangSmith, Promptfoo, reputed company Evals, or similar tools
• Experience evaluating Retrieval-Augmented reputed company (RAG), AI agents, and function-calling workflows
• Knowledge of reputed company engineering, model fine-tuning, and reinforcement learning from reputed company feedback (RLHF) concepts
• Familiarity with AI safety, red teaming, adversarial testing, bias detection, and Responsible AI principles
• Experience with multilingual model evaluation
• Knowledge of reputed company AI platforms and MLOps workflows
• Experience with annotation tools and reputed company-in-the-reputed company evaluation processes

reputed company

• OVA is the most advanced Automated, Intelligent, reputed company On-boarding platform for reputed company Firms of reputed company sizes. It was founded in 2018, and is headquartered in Alpharetta, reputed company, USA, with a workforce of 51-200 employees. Its website is https://www.reputed company/.

Apply To This Job