20. AI Evaluation Engineer
Evidence label: Role family with unstable naming. Current titles include Research Engineer, Model Evaluations; Backend Software Engineer (Evals); evaluation-infrastructure leadership; and product-specific AI evaluation engineering.
What they do
Build and operate the measurement layer that tells an organization whether model or system behavior meets explicit criteria over time. Evaluation can target model capabilities, safety, a product workflow, an agent trajectory, or the infrastructure that runs those tests.
Why the title varies
Anthropic currently lists Research Engineer, Model Evaluations and Evals Infrastructure leadership. OpenAI's Backend Software Engineer (Evals) builds reproducible pipelines, golden datasets, drift monitoring, and evaluation services for a product domain. These postings confirm a role family but also show that research, infrastructure, and product evaluation require different profiles.
Responsibilities
- Design evaluation frameworks with explicit metrics (task completion rate, tool correctness rate, hallucination rate)
- Build reproducible datasets, runners, and comparison pipelines
- Combine deterministic checks, human review, and model-based graders with documented limitations
- Detect regressions across model, prompt, context, tool, or traffic changes
- Maintain benchmark provenance and test for contamination or overfitting
- Connect evaluation results to release gates, incident analysis, and product decisions
Required skills
Statistical experiment design, Python, task and failure-mode analysis, dataset curation, grader validation, CI/CD integration, and reproducible reporting.
Tools
Tool choice depends on the boundary: test runners for deterministic checks, experiment stores for datasets and versions, tracing for agent trajectories, human-review interfaces, and model-based graders whose error rates are measured against labeled samples.