What you will build
By the end, you will have:
- An OpenAI Agents SDK-backed financial analyst that reviews a fictional company's diligence materials across five traced runs
- Human and LLM-generated feedback over those same traces
- An automatically generated Promptfoo eval suite
- A Promptfoo validation gate over the current agent behavior
- A HALO optimization pass over the traces, feedback, and eval results
- A developer-facing handoff to Codex so it can implement the recommended harness changes
The agent supports acquisition diligence for a fictional company. It reviews financial exports, customer data, contracts, security notes, board materials, and management narratives, then answers diligence questions with citations and reviewable artifacts.
The loop writes one file that carries the work forward: the generated codex_handoff.md file under ARTIFACT_DIR. It contains the full HALO diagnosis, the ranked recommendations, the evidence behind them, and the implementation guidance Codex needs for the next harness update.
The degree of automation is up to the developer. You can use the loop to propose a reviewed change set, or connect it to a workflow that opens, merges, and deploys pull requests automatically. A common starting point is a reviewed loop, where the system proposes the change set and a developer approves the diff before merge. As the eval gate becomes more trusted, the same handoff can support deeper automation. The core workflow is the same in either case: traces plus human and model feedback become concrete harness changes instead of remaining disconnected comments.
Compared with examples that stop at traces or evals, this notebook keeps traces, reviewer judgment, generated evals, optimization, and implementation handoff inside one runnable improvement loop.