Prove the Outcome in the Real Environment

Verification belongs in the job. A change is complete when the evidence shows the promised result in the environment where a user, operator, or dependent system will rely on it.

Define success where it will be experienced

Start by naming the product features and the journeys through them. Those journeys give implementation, review, and release a shared target that an agent can execute and observe.

In the Aakash Gupta interview, a new engineer who had previously worked as a product manager noticed that the repository did not document the product's features. Codex crawled the codebase and drafted a feature inventory; people on the team reviewed it against their product knowledge. The reviewed inventory made the critical user journeys explicit, and QA agents could then boot the app, exercise those journeys, and assert that they still worked. Deployment required less manual smoke testing because one teammate's observation had become a repeatable acceptance suite.

Define the acceptance boundary before implementation:

• who or what experiences the outcome;
• which starting state and inputs matter;
• which visible behavior and side effects must occur;
• which invariants must remain true;
• how the observer will distinguish success from a plausible imitation; and
• what remains outside the claim.

This specification can stay small. Its job is to identify the evidence the worker must be able to produce.

Match evidence to the claim

Map each green check to the assertion it actually makes. Then gather evidence at the affected boundary.

Claim	Evidence at the claim boundary
browser behavior	a real browser journey with semantic and rendered state
generated content	complete source-to-output comparison and freshness
compatibility migration	target-corpus parity, edge cases, and explicit accepted differences
security impact	a reproducer, bounded exploitability, and regression coverage
deployment	the validated artifact running with post-deploy health
consequential remote change	staged canary, cutover, recovery, and post-cutover checks
spreadsheet calculation	formulas, units, source cells, and rendered workbook state
dataset transformation	schema, provenance, and source-to-output reconciliation
document or presentation	required structure and rendered pages; citations or accessibility when promised
analytical conclusion	reproducible transformation and conclusions supported by evidence
business-state mutation	required approval, receipt, and observed postcondition

Unit tests, type checks, lints, and builds establish important internal properties. Standards compatibility needs conformance evidence. Semantic completeness needs source-to-output comparison. A successful upload needs a healthy deployment before it supports a release claim.

When intermediate code stays inside an agent trajectory, the proof packet speaks in the user’s domain. A spreadsheet formula, rendered slide, cited finding, or completed state transition carries the claim. Passing tests for the hidden script establish only internal properties. The result is established at the user’s domain boundary.

Give the agent access to the real system

The agent should be able to launch the application, drive the browser or UI, read logs, query metrics and traces, inspect persistent side effects, compare generated corpora, and observe CI and deployment. A person should not have to relay evidence the agent can inspect directly.

The canonical harness-engineering essay describes per-worktree application instances and ephemeral observability stacks that let Codex reproduce bugs, query logs and metrics, drive the UI, and validate its own changes. The Production Function Changed carries the same requirement into review: attach test results, a QA plan, staging logs, screenshots, or video that demonstrate the result before asking for review.

Use semantic and rendered evidence together. DOM snapshots and structured state are efficient for roles, values, and relationships. Screenshots, video, or computer use expose layout, clipping, focus, rendering, and interaction defects that the semantic view omits. Logs and traces establish runtime behavior that a visual demonstration cannot.

Ryan Lopopolo's OpenAI team exposed one such surface inside its Electron application. The Electron component workbench was a launchable development window containing the application's native rendering canvas and full design-system component library. Codex could compose new screens from the production components, render them in the surface the team would ship, and return screenshots to a designer for review.

The team also let Codex start a production-like observability stack for local metrics and logs. Its local browser shell mounted the UI with Chrome DevTools already connected, giving the worker the box model, console, and other browser-native observations. Ryan described built-in computer use as the approach he would use today to inspect the application, click through a journey, and observe whether the intended side effects occur.

Another complementary technique pairs semantic and rasterized views. A semantic representation helps the model identify interface objects and their roles. Rasterizing the interface exposes spatial relationships such as layout, clipping, and occupied space. Together they give the worker both the named objects and the rendered geometry it is manipulating.

Direct access also lets the agent choose the smallest useful inspection tool. The agent may query a trace archive directly when a human-facing dashboard would only add an intermediate artifact and another relay. The Latent Space interview recounts a teammate spending an afternoon on a polished trace viewer before discovering that Codex could answer the question from the trace tarball itself in about five minutes.

Compress the trajectory for review

Reviewers need enough evidence to assess the outcome without replaying an entire session. The same interview describes PR video as a compression of the work: a teammate presents the evidence needed to make the merge decision instead of asking the reviewer to shoulder-surf the implementation.

A useful proof packet contains:

• the intended outcome and affected boundary;
• material design and risk decisions;
• the exact tests and journeys that ran;
• screenshots, video, logs, traces, diffs, or reproducers that carry the claim;
• known limits and unproved behavior; and
• the identity of the artifact proposed for delivery.

Plans may coordinate the trajectory. Because plans are not shipped, the proof packet covers the behavior, side effects, and exact artifact promised to the user.

The packet should be concise because reviewer attention is scarce. More evidence helps only when it changes confidence in the claim.

Preserve artifact identity through delivery

Release proof follows one immutable artifact from validation through cutover. Rebuilding in a more privileged job breaks that chain: the deployed bytes are no longer the bytes the earlier evidence covered.

Release Integrity develops the complete path: build once, record provenance, promote the same artifact, verify the running system, and preserve rollback. A post-deploy check belongs to the proof because successful delivery is an operational claim about the running boundary.

The homelab canary applies this principle to a remote-access upgrade. A successful build could not establish that the upgraded machine would remain reachable. The proof proceeds through an isolated canary identity, an independently verified access path, a backed-up production cutover, and post-cutover access and log checks.

Give the agent ground truth it can test against

Objective graders make iteration cheap when the domain supplies them. Miri is an interpreter for Rust's mid-level intermediate representation that can run a binary or test suite and report classes of undefined behavior, including out-of-bounds access, use-after-free, invalid use of uninitialized data, and some aliasing violations. In his Codex-and-Miri work, Ryan has used Miri as executable ground truth while Codex searches popular Rust crates for unsound executions. A clean Miri run is limited to the executions it explored; a reported violation gives the worker a concrete failure to reproduce and repair.

Accessibility standards and assistive technology provide concrete quality checks. In a CTF-style UI exercise—“CTF” means capture the flag—the team plants markers behind real application workflows. A skill succeeds by driving the interface and capturing the marker, giving repeated improvement runs an executable win condition. Compatibility vectors, canary health, and complete corpus comparisons can likewise turn vague goals into observable outcomes.

The hard part is building a validator that rewards the real outcome and resists gaming. Ryan describes this as the need for an unhackable grader. Practitioners also carry hard cases from their own work. Encoding those cases as repeatable local trials complements broad public benchmarks with the team's own artifacts and risks.

Nonfunctional requirements become proof obligations at the boundary they govern. Representative evidence can include compatibility against a relevant corpus, version matrix, or specification; performance against a named workload and threshold; accessibility conformance checks plus assistive-technology or user journeys; and security controls evaluated against a threat model. Qualitative requirements such as tone, taste, risk tolerance, polish, and acceptable shortcuts become more reliable when a team names its judgment and gives reviewers a convergence policy. What Does It Mean to Do a Good Job? develops that part of verification.

Evaluate the Harness separates outcome, proof, architecture, and trajectory cost so a convenient proxy cannot silently replace the job being measured.

Say what the evidence did not establish

Security evidence should state the conditions under which a reproducer works and the conclusions it supports. In Prove a Security Claim, corruption required a custom hasher and recovery after panic; the reproducer did not establish memory unsafety on the default path. Codex developed the report, reproducer, patch, tests, and responses to substantive implementation review. Ryan reviewed the implementation, approved it, and separately authorized the merge, point release, and RustSec report. The operative instruction was to prove impact or exploitability for every reported issue; speculative security-shaped output did not satisfy the job.

The proof burden scales with consequence. A typo, a parser migration, a remote upgrade, and a production release need different evidence and authority. The Repository Review Playbook follows those claims through representative repository journeys and records any boundary the available evidence does not reach.