Build an Agent Improvement Loop with Traces, Evals, and Codex

For the complete documentation index, see llms.txt. Markdown versions of documentation pages are available by appending .md to the page URL.

This notebook builds an improvement flywheel for an agent. We start with real traces, add human and model feedback, turn that feedback into evals, and use the resulting evidence to propose the next harness changes for Codex to implement.

You will:

• Create an OpenAI Agents SDK-backed financial analyst
• Run it on synthetic company data and capture traces
• Add example human feedback and LLM-generated feedback from those runs
• Turn that feedback into Promptfoo evals that can be rerun later
• Use HALO to rank the next harness changes and write a Codex-ready handoff

In this notebook, the harness is the full contract around the model, including instructions, tools, routing, output requirements, and validation checks.

The flywheel preserves what you learn from each run. Traces show what happened, feedback explains what mattered, evals make those expectations reusable, and Codex can act on the resulting change set.

本篇目录

• What you will build
• Prerequisites
• Step 1. Create synthetic company data
• Step 2. Define the Agents SDK-backed analyst
• Step 3. Generate traced runs
• Step 4. Generate example human feedback and model insights
• Step 5. Generate Promptfoo evals from traces and feedback
• Step 6. Validate the current harness with Promptfoo
• Step 7. Run HALO and write the handoff
• Step 8. Hand the full report to Codex
• Top 3 changes to implement first
• Insights by feedback source
• Step 9. Close the loop
• Conclusion
• Next steps