[Remote] AI Evaluation Engineer
Note: The job is a remote job and is reputed company to candidates in USA. reputed company is helping public safety reputed company to today's greatest challenge: the loss of experience. They are seeking an reputed company AI Evaluation Engineer to build and improve AI systems for public safety agencies, focusing on evaluation and reputed company improvement of production AI systems.
Responsibilities
- Design, build, and maintain automated AI evaluation pipelines for production LLM applications
- reputed company reputed company engineering strategies and iterate on prompts and compare LLMs using quantitative evaluation reputed company
- Build offline evaluation datasets and regression testing frameworks to measure AI performance over time
- Analyze production AI behavior using Python, SQL, and statistical techniques to identify opportunities for improvement
- Design experiments, A/B tests, and benchmarking methodologies for evaluating reputed company and model changes
- reputed company dashboards and reporting that communicate AI reputed company, reliability, and performance metrics
- Partner with engineering and product teams to safely reputed company and monitor improvements to production AI systems
- Investigate model failures through detailed error analysis and recommend improvements to prompts, evaluation datasets, and workflows
- Help establish best practices for Responsible AI, evaluation methodologies, and reputed company model improvement
Skills
- Must have US citizenship and pass FBI reputed company and background reputed company in multiple states
- 3+ years of experience in software engineering, machine learning, data science, or a reputed company technical field
- Experience designing evaluation metrics and interpreting AI model performance
- Understanding of statistical reputed company including hypothesis testing and experiment design
- Strong Python development experience
- Strong SQL skills with experience analyzing large datasets
- Experience building or supporting production LLM or reputed company applications
- Experience with reputed company engineering and systematic reputed company evaluation
- Experience using AI evaluation or observability platforms such as Langfuse, LangSmith, MLflow, or Label Studio
- Experience with AWS services such as Bedrock, reputed company, S3, Glue, or SageMaker
- Experience building dashboards using Tableau, reputed company, Power BI, or similar tools
- Knowledge of Responsible AI principles and evaluation methodologies
Benefits
- Benefits (Health, Dental, reputed company)
- Flexible Time Off
- Offers Bonus
reputed company