How to Test-Drive the Shortlist
Test two or three candidates on 8 to 12 representative tasks from the same repositories. Use the same model, repository instructions, tool permissions, and resource budget where the products allow it. Repeat critical tasks because one successful run does not establish reliability. Give each run an isolated worktree and write pass criteria before execution.
Record the following measurements for every task:
| Measurement | What to record |
|---|---|
| Human verdict | Accepted, accepted after correction, or rejected |
| Interventions | Approval prompts, redirects, manual edits, and restarts |
| Plan drift | Work outside the requested files, contract, or acceptance criteria |
| Wall time | Time from task start to reviewed result |
| Accepted-task cost | Total model and platform cost divided by accepted tasks |
| Recovery | Whether the run resumes after a controlled interruption without repeating or losing work |
| Setup friction | Time and specialist work needed to reproduce the environment |
| Turns to accepted completion | Model or agent turns required before the reviewed result passes; Patrick Debois proposes turn count as an enablement signal |
| Requirement coverage | Passed, failed, and unresolved acceptance criteria, each linked to its evidence |
| Proof artifacts | Commands, outputs, tests, screenshots, traces, or runtime captures needed to reproduce acceptance |
| Tail latency | Per-run distribution and slow cases, not only the mean; see Amit Kushwaha's distribution-aware benchmark argument |
| Model-harness pair | Exact model, version, reasoning mode, harness version, and configuration used for the result |
| Repeated-run reliability | Success consistency, perturbation tolerance, predictable resource use, and bounded failure severity; see Towards a Science of AI Agent Reliability |
Review the produced diff and tests, not the agent's self-report. A green test suite is necessary but may not cover every requirement. Simon Willison's captured command-and-output workflow is one way to preserve a human-reviewable proof artifact, while Shachar Azriel's executable-spec pattern maps verification to individual requirements. Keep consequential actions behind a human or policy gate during the trial. Use the Agent Evaluation framework to define acceptance evidence, Observability to capture traces and interventions, and Security Hardening to test the execution boundary rather than trusting a product label. Apply the Agentic Benchmark Checklist before treating a score change as a product result: task setup and grader defects can exceed the claimed improvement.