11. Harness Optimizers and Meta-Harnesses
A runtime harness improves one agent run. A harness optimizer improves the code and configuration that govern future runs. The optimizer can edit prompts, context policies, tools, middleware, memory, control flow, verification, or routing, then use an external evaluator to decide which candidate survives.
This section uses meta-harness in that research sense. Databricks and Omnigent use the same term for a unified developer surface that dispatches a task to one of several existing coding harnesses. That dispatcher can reduce switching friction and preserve tool-policy consistency, but it belongs to the orchestrator or control-plane layer unless it also searches, changes, and evaluates the target harness. Source and terminology review: Databricks cost-management resource evaluation.
| Work | Optimization surface | Reported evidence | Boundary |
|---|---|---|---|
| Automated Design of Agentic Systems, ICLR 2025 | Code-defined agent systems | Meta Agent Search discovers prompts, tools, and workflows that transfer across domains and models | System-level search; not every result isolates a fixed-model harness effect |
| AFlow, ICLR 2025 | Code-represented workflows | 5.7% average improvement across six benchmarks | Heterogeneous model-plus-workflow baselines |
| ACE, ICLR 2026 | Context and memory playbooks | +10.6% on agent benchmarks and +8.6% on finance tasks | Context optimization, not whole-harness search |
| GEPA, ICLR 2026 Oral | Prompts from reflected trajectories | 6% average gain over GRPO across six tasks, with up to 35 times fewer rollouts | Prompt optimizer; model control varies by experiment |
| Meta-Harness, 2026 preprint | End-to-end harness code | +7.7 points with four times fewer context tokens; +4.7 points across five held-out models on 200 math problems | TerminalBench search and final evaluation reuse the same 89 tasks; total search compute is not reduced to one comparable cost figure |
| Agentic Harness Engineering, 2026 preprint | Prompt, tools, middleware, skills, subagents, and memory | Terminal-Bench 2 pass@1 rises from 69.7% to 77.0% over ten iterations | Coding-focused preprint; external replication remains limited |
| HarnessOpt-Bench, 2026 preprint | The optimizer itself | Five optimizer models, four downstream tasks, 111 scored runs | Benchmark protocol, not a production optimizer |
The evidence is promising but young. Do not promote a candidate because its development score improved. A credible optimization run needs:
- Frozen invariants: target model, tool permissions, task contract, grader, and resource limits remain fixed unless the experiment declares otherwise.
- Separated data: detailed development traces, bounded validation feedback, and a hidden test partition that the optimizer cannot inspect.
- Explicit budgets: evaluation calls, task attempts, tokens, wall time, and human interventions are metered.
- Versioned candidates: every change has a diff, rationale, predicted benefit, regression risk, score, and rollback path.
- Multi-metric promotion: success, accepted-task cost, latency distribution, repeated-run reliability, and failure severity all participate in the decision.
- External evaluation: the optimizer cannot alter the evaluator, reveal hidden cases, or award its own final score.
HarnessOpt-Bench is the clearest current protocol: a trusted execution environment hides the test partition, enforces evaluation budgets, meters resources, and preserves candidate versions. Establishing Best Practices for Building Rigorous Agentic Benchmarks explains why those controls matter: task and reward defects can distort reported agent performance by up to 100% in relative terms, and its checklist reduced CVE-Bench overestimation by 33%.
The practical conclusion is not “self-improving agents solve harness engineering.” It is that harness engineering can become an auditable experiment when the search space, evidence, budget, and promotion boundary are explicit.