Model Benchmark Sample Report

This is a sanitized example of the real-model benchmark evidence format. Raw benchmark runs live under benchmark-runs/ and are ignored by git because they can contain provider output, local paths, timing data, and prompts.

What The Benchmark Checks

python3 scripts/model_benchmark.py --providers deepseek openai-chat

The suite creates isolated temp homes and runs:

Provider	Cases
deepseek	mini harness, full tour, and s01-s24 chapter eval traces.
openai-chat	mini harness, full tour, and s01-s24 chapter eval traces through the provider adapter.

Each chapter has its own interactive teaching CLI and a shared --eval entrypoint. The benchmark uses --eval so every provider writes a comparable model/tool trajectory under benchmark-runs/<run>/traces/.

Required Pass Markers

Case Type	Evidence Required
mini	Audit verified: True and Transcript events:
full	RESULT: OK, provider_probe: true, and audit verification
lesson	non-empty JSONL trace with model_response, tool_call, tool_result, and case_end events

Sanitized Example Summary

total cases: 52
passed: 52
failed: 0
providers: deepseek, openai-chat
raw stdout: stored locally under benchmark-runs/<run>/stdout/
raw traces: stored locally under benchmark-runs/<run>/traces/

Publication Rule

Do not commit raw benchmark output. If maintainers want to publish a result, copy only the aggregate counts, provider families, case IDs, and remediation notes. Remove:

• API keys or authorization headers
• model raw responses that may contain private prompts or paths
• local temp directories
• provider request IDs
• user-specific environment values

The benchmark is an engineering gate, not a claim about a proprietary product.