Macro Evals for Agentic Systems

For the complete documentation index, see llms.txt. Markdown versions of documentation pages are available by appending .md to the page URL.

When an agentic system fails, the problem is often larger than a single bad response. A handoff may happen too late, a specialist agent may miss the same signal across many runs, or a review process may trigger for the wrong class of cases. To improve the system, teams need to see recurring behavior across the whole population of traces.

This cookbook walks through a macro-eval workflow for a multi-agent system. We use a synthetic EV order workflow where specialist agents handle pricing, compliance, supply, factory routing, scheduling, and release decisions while market and operational conditions change.

The notebook uses precomputed synthetic traces and saved lower-level eval labels, so you can run the full workflow without an OpenAI API key.

You will learn how to:

1. Generate or collect many traced agent runs;
2. Run lower-level evals on each completed run;
3. Turn each trace into a compact document;
4. Discover recurring behavior patterns across the population; and
5. Drill into one high-impact pattern to find where a human should inspect the system next.

The goal is not to build a perfect taxonomy of every trace. The goal is to show how an AI engineering team can move from thousands of agent events to a small number of patterns that are understandable by both technical and business stakeholders.

本篇目录

• End-to-End Agentic System Map
• 1. Why Macro Evals?
• Setup and Data Materials
• 2. The Simulation: Automotive Orders in a Changing World
• 3. Lower-Level Agent Evals with Promptfoo
• 4. Build the Analysis Dataset
• 5. BERTopic-Style Discovery
• 6. AgentTrace-Style Diagnosis
• 7. What We Learned and What to Do Next
• Contributors