7. Test Distribution and Component-Stacking Anti-patterns

An empirical study across 39 open-source agent frameworks and 439 agentic applications (arXiv 2509.19185) found that more than 70% of testing effort in agentic systems targets the deterministic components (tools, APIs, workflow logic), while less than 5% targets the Plan Body, the LLM reasoning core. Adoption of dedicated LLM evaluation tools (DeepEval) was below 1% despite those tools' high marketing visibility.

This is structurally backwards. The deterministic components are the most testable with standard unit tests and will fail loudly when broken. The LLM reasoning core is where the non-obvious, non-deterministic failures live: the ones that produce plausible-looking but wrong outputs, miss edge cases in tool selection, or hallucinate capability claims.

Recommended rebalancing: give the Plan Body its own behavioral and repeated-run tests instead of assigning an arbitrary percentage target. The pass^k pattern addresses non-determinism: run a critical test several times and report how many executions pass. Promptfoo's --repeat flag implements this. LangWatch Scenario SDK does it at the multi-turn simulation level.

For deterministic components: standard unit tests, schema validation, explicit tool call checks. For LLM reasoning: behavioral simulation (LangWatch Scenario), LLM-as-judge with bias awareness, regression suites on known-good examples from production.

Tests are necessary, but a green suite is not a complete proof bundle. Simon Willison's 2026 engineering-practices talk pairs tests with captured commands and outputs, then identifies sandboxing as the main execution boundary. For an agentic coding task, retain the requirement map, relevant commands, outputs, test results, review verdict, and any visual or runtime evidence needed to reproduce acceptance.

Do not assume that stacking planning, tools, memory, reflection, and retrieval is monotonic. Cross-Component Interference tested all 32 subsets of five components on HotpotQA and GSM8K. A single-tool configuration exceeded the all-in system by 32% on HotpotQA; a three-component subset exceeded it by 79% on GSM8K. This is a preprint over two benchmarks, but its design supports a strong engineering rule: start with the smallest sufficient harness and require each added component to pass a paired ablation and regression test.