Drive plane selection for agent verification

Traceability

• Spec ID: drive-plane-selection
• Status: Implemented

Intent

Add a companion reference for references/project-harness/agent-verify-loop.md that owns the missing axis in the current verification model: where the agent injects control into the system under verification, and what that choice costs in observability, isolation, and judging expense.

The existing model has two axes and neither answers the question:

• agent-verify-loop.md Scenario Families answers what is verified (service chain, web UI, mini-program, mobile app, CLI, pipeline, model output, IaC).
• verification-environment.md Fidelity Ladder answers how real the dependencies are (L0 preflight .. L5 real-boundary calibration).

Neither distinguishes a browser driven through an in-browser automation protocol from the same browser driven through operating-system input events with screenshot-based judging. Both are the Web UI family, and both can sit at Fidelity Ladder L3, yet their trigger shape, evidence set, isolation model, cleanup obligations, parallelism limits, and failure modes differ enough that an agent cannot pick between them from the current references. Coding agents now reach for browser-automation drivers, attach-mode browser agents, and computer-use desktop loops interchangeably; without a stated axis the choice is made by tool availability rather than by the verification claim.

The reference must also record the concrete technology classes behind each plane, because the recurring failure is not choosing the wrong family — it is assuming that a driver which can read the DOM and a driver which can only see pixels produce evidence of the same strength.

Recorded Field Observations

Provenance: a field write-up contrasting a Playwright-based verification chain with a Windows desktop automation chain, reviewed on 2026-08-04, plus repository observations from AGENTS.md and scripts/harness-analysis/validate-canvas.mjs. No new services or tests were run for this spec; these are design inputs, not runtime validation.

A second evidence source was collected on 2026-08-04 by scanning local Codex session transcripts (~/.codex/sessions rollout JSONL): 70 sessions containing UI-verification evidence, 57,879 extracted commands plus all tool calls. The scan scripts are local one-offs (not repository assets); headline counts are recorded here so the reference does not depend on private transcripts:

• Attach-mode browser driving (agent-browser --cdp against a running Electron/IDE instance) appeared in 25/70 sessions (1,041 commands) — the D2 plane below — while OS accessibility driving (computer_use.get_app_state/click by element index) accounted for ~490 tool calls — the D3 plane.
• Observation split in the wild: accessibility-tree snapshots (667) and DOM eval state dumps (299) served as structural oracles; screenshots (151 captures, 240 view_image model-judged reads) served as the visual channel.
• Pixel-diff tooling was used 0 times in 70 sessions; masks 3 times; viewport pinning in 4/70 sessions. The de-facto visual judge was the driving model's own multimodal read — exactly the same-model drive/judge hazard DPS-AC-4 addresses.
• 166 of 236 distinct screenshot paths were overwritten and re-viewed under the same name, so no visual baseline survived a session; naming idioms were before/after (41) and numbered step sequences (40) rather than baseline/expected (4).
• Fixed waits (wait <ms>/sleep) appeared in 29/70 sessions (999 commands) versus bounded polling loops in 17/70 — terminal conditions are the exception, not the rule, at D2/D3.
• The observed D3 loop shape was list_apps → get_app_state → click(element index) → get_app_state (state re-read as judge; element indexes never cached), and the observed locate fallback ladder was CSS selector → text/aria match → raw coordinates from a previous DOM eval.

The compared chains

Playwright chain:
  test code -> Playwright client -> browser automation protocol (CDP)
    -> BrowserContext/Page -> DOM locators, protocol-level input, page events
    -> target page

Desktop chain:
  model visual decision -> desktop screenshot -> visible controls and state
    -> step driver -> window focus (accessibility/automation API)
    -> OS mouse/keyboard events -> existing browser profile and session
    -> target page -> before/after screenshots plus a JSON step record

The first keeps the control plane inside the browser or its automation protocol, so the driver reads the DOM, manages contexts, and usually runs a throwaway profile. The second keeps the control plane outside the browser: it does not read the target page DOM, does not create a browser context, does not inject scripts, and does not alter browser-exposed automation properties, so the page runs inside the user's ordinary session, tab, and window lifecycle. The cost is that structural evidence disappears and judging falls back to pixels plus side effects.

Decomposition into three independent axes

The two chains differ on three axes that are freely composable, not a binary choice. Recording them separately is the point of this change.

Axis	Playwright chain	Desktop chain
Decision source	script-predefined steps	model decision at run time
Control-plane location	in-browser automation protocol	out-of-browser OS input plane
Session origin	fresh throwaway profile	existing real profile and login state

Other combinations are valid and already in field use: script-driven OS automation with a real profile, and model-driven decisions over an automation-protocol driver attached to an existing browser. Collapsing the three axes into "Playwright versus computer use" hides that the decision source determines the judging mode, while the control-plane location determines the available evidence.

Drive planes

Plane	Control-plane location	Structural evidence	Session realism	Judging cost	Parallel-safe
D0	In-process render tree	full	none (no real browser)	lowest	yes
D1	In-browser automation protocol	DOM, accessibility tree, network, console	moderate (throwaway profile)	low	yes
D2	In-browser agent attached to a running browser	mostly available	high (real profile and login state)	medium	no
D3	OS input and accessibility plane	accessibility tree only, or pixels only	high (ordinary user session)	high	no
D4	Physical or device plane	pixels only	highest	highest	device-bound

Observability decreases and session realism increases monotonically along the ladder; isolation moves the opposite way from Fidelity Ladder intuition, which is the concrete gap this reference closes.

Technology classes per plane

Named tools are examples of a class, in the style of the verification-environment.md cross-stack table, not endorsements or a closed list.

Plane	Technology class	Examples
D0	Component render harness	Testing-library style renderers, jsdom, in-browser component runners
D1	Automation-protocol driver	Playwright, Puppeteer, Selenium/WebDriver, WebDriver BiDi, raw Chrome DevTools Protocol
D2	Attach-mode browser agent	Extension or DevTools-protocol agents attached to an already-running browser, browser-control MCP servers
D3	OS input driver	Windows UI Automation and pywinauto, cross-platform input synthesis such as PyAutoGUI, macOS Accessibility API and System Events scripting, X11/Wayland input tools
D3	Computer-use agent loop	Screenshot-in, action-out model loops that emit click/type/scroll steps
D3	Terminal/TUI driver	Pseudo-terminal drivers, tmux control mode, expect-style scripts
D3/D4	Mobile driver	ADB and uiautomator, Appium, XCUITest, Maestro, device farms
D1'	Vendor devtools driver	Mini-program devtools automation, platform emulators and simulators
Observation only	Network observation and virtualization	Recording/forwarding proxies, service virtualizers
Observation only	Vision judge	Vision-model screenshot judging against a rubric
Observation only	State oracle	Direct state, table, log, or trace queries

Drive plane and observation plane must be named separately

An automation-protocol driver is both driver and observer, which is why the two roles are usually conflated. The desktop chain separates them: it drives through the OS and observes through screenshots plus a JSON step record. The transferable rule is that a drive adapter (an Exercise-plane asset) and an observation adapter (a Judge-plane asset) are chosen independently, and a low drive plane must be compensated with stronger side-effect evidence — persisted state, emitted events, network calls, backend records — rather than with model confidence.

When the same model both drives and judges, the loop has no independent oracle and lands directly on the existing Self-reported success anti-pattern.

Shared six-step pattern

Every plane runs the same skeleton, and the differences concentrate in three steps.

acquire session -> reset state -> locate target -> apply action
  -> wait for terminal condition -> collect evidence and judge -> clean up

Step	D1	D2	D3
Acquire session	create a fresh context	attach to a running browser, require window focus	focus a real window, require foreground exclusivity
Reset state	context isolation is the reset	explicit sign-out or storage reset; no inherent isolation	scripted desktop and profile state reset
Locate target	selector or role query	selector or role query	accessibility tree, or template/text recognition on pixels
Wait	render-idle and network-idle signals	same as D1	pixel stability plus a mandatory deadline
Collect evidence	DOM snapshot, screenshot, network, console	same as D1	before/after screenshots, JSON step record, independent state oracle
Clean up	close the context	restore the user profile state	close windows, restore focus; must run after failure and timeout

The real risk at D2 and D3 is not driving but reset and cleanup: reusing a real profile means a run mutates user state, and an OS input plane is exclusive, so concurrent runs corrupt each other. A missing terminal condition at D3 turns a settling page into either a hung loop or a flaky verdict.

Acceptance Scenarios

• DPS-AC-1 (axis decomposition): The reference separates decision source, control-plane location, and session origin as independent axes, and states that the popular two-way framing of automation-protocol versus computer-use driving conflates them.
• DPS-AC-2 (drive plane ladder): The reference defines drive planes D0 through D4 by control-plane location, each with its available structural evidence, session realism, judging cost, and parallel-safety, and states the monotonic trade-off between observability and session realism.
• DPS-AC-3 (plane and observation split): The reference requires a drive adapter and an observation adapter to be selected and recorded separately, and requires a lower drive plane to compensate with side-effect evidence rather than model confidence.
• DPS-AC-4 (independent judge at low planes): When the driving model is also the judging model, the reference requires an independent oracle plus a known-bad control before a pass verdict is admissible, routing to the existing oracle-independence guidance.
• DPS-AC-5 (technology map): The reference records the technology classes behind each plane with concrete examples, labelled as class examples rather than a closed or endorsed list, including automation-protocol drivers, attach-mode browser agents, OS input drivers, computer-use loops, terminal drivers, mobile drivers, vendor devtools drivers, and observation-only adapters.
• DPS-AC-6 (six-step pattern with per-plane deltas): The reference gives one shared six-step execution skeleton and a per-plane delta table covering session acquisition, state reset, locating, terminal condition, evidence, and cleanup, and names reset, cleanup, and exclusivity as the dominant D2/D3 risks.
• DPS-AC-7 (isolation inversion recorded): The reference states that a real-profile browser has higher session realism but lower isolation than the Fidelity Ladder L3 isolated real browser, and that this inversion must be recorded in the case constraints instead of being read as a higher rung.
• DPS-AC-8 (self-execution wiring): The reference states how a drive adapter becomes something an agent runs without prompting: a single named entrypoint command, the drive adapter and correlation handle recorded in the case trigger field, cost-tiered escalation from the cheapest sufficient plane, a machine-parseable verdict-first output using the four-valued verdict domain, an enforcement point that fires when the agent claims completion, and repaired failures promoted into the regression skeleton.
• DPS-AC-9 (ownership and routing): The reference declares its ownership boundary against agent-verify-loop.md (planes, verdicts, scenario families, skeleton shape) and verification-environment.md (dependency fidelity, environment contract, oracle independence). The Exercise plane in agent-verify-loop.md routes to it, verification-environment.md routes to it from its browser fidelity row or ladder note, and references/project-harness/README.md registers it under Owns and Read Next.
• DPS-AC-10 (link integrity): All relative Markdown links to and from the reference resolve, and docs/better-harness-doc-links.mmd is regenerated so the routing graph is not stale.

Non-goals

• Ship an executable driver, adapter registry, screenshot-diff runner, or any runtime tooling; the reference stays prose.
• Provide guidance on evading bot detection, anti-automation controls, or risk systems. Session realism is scoped to verification fidelity on systems the team is authorized to exercise; verifying payment, login, or other high-authority flows keeps the authorization, read-only scope, redaction, and environment-boundary requirements already owned by verification-environment.md.
• Redefine or extend the Fidelity Ladder rungs, the four planes, the verdict domain, or the Scenario Families table. This change adds an axis and routes to it; it does not renumber existing calibrated models.
• Add rows to the Scenario Families table. That table answers what is verified; mixing in how the system is driven would blur both axes.
• Endorse a vendor, or claim that any named tool preserves a given behavior on a given platform.
• Change scoring models, report schemas, skills, scripts, templates, hooks, or host adapters.

Plan and Tasks

1. Add references/project-harness/ui-and-system-drivers.md with the ownership boundary, a Load When block, the three-axis decomposition, the D0..D4 drive plane table, the technology-class map, the drive/observation split rule, the shared six-step pattern with per-plane deltas, the self-execution wiring, and an anti-pattern list.
2. Route the Exercise plane in references/project-harness/agent-verify-loop.md to the new reference with one sentence, without moving verdict, family, or skeleton ownership.
3. Route references/project-harness/verification-environment.md to the new reference where browser fidelity is selected, and record the realism-versus-isolation inversion as a pointer rather than a new rung.
4. Register the reference in references/project-harness/README.md under Owns and Read Next.
5. Regenerate the routing graph with node scripts/doc-link-graph/cli.mjs skills/better-harness, then run the focused link and model tests followed by the full suite.
6. Run a Review Readiness Check over the local diff before review.

Test and Review Evidence

Implemented on 2026-08-04:

• node scripts/doc-link-graph/cli.mjs skills/better-harness — routing graph regenerated (36 files, 52 links).
• node --test test/doc-link-graph.test.mjs — 6/6 pass; relative link resolution and routing-graph freshness (DPS-AC-10).
• node --test test/maturity-models.test.mjs — 3/3 pass; consumer integrity for the routed references (DPS-AC-9).
• npm test — full suite, 1149/1149 pass.
• git diff --check — clean.
• DPS-AC-1..DPS-AC-8: manual contract review against the reference headings, tables, and anti-pattern list in references/project-harness/ui-and-system-drivers.md.

Risks

• Axis proliferation: a third axis on top of scenario families and the fidelity ladder can make the model harder to apply than the problem it solves. Mitigate by keeping the new reference selection-only and routing rather than restating the other two models.
• Tool-list decay: named drivers and agent tools change faster than the framework. Mitigate by presenting them as class examples with a provenance date, matching the existing cross-stack table convention.
• Realism read as fidelity: a real-profile session can be mistaken for a higher fidelity rung, hiding lost isolation and unsafe parallelism. Mitigate with DPS-AC-7 and explicit reset/cleanup obligations.
• Model-judged pixels: at D3 and D4 the driving model can also judge the screenshots and produce consistently confident false passes. Mitigate with DPS-AC-4, requiring an independent oracle and a known-bad control.
• Authorization drift: discussion of real login state invites exercising production accounts. Mitigate by keeping the authorization boundary owned by verification-environment.md and restating it as a constraint, not guidance.

Open Questions

• [NEEDS CLARIFICATION: whether a future change should add a Fidelity Ladder rung, or a per-rung isolation column, for real-profile and OS-driven sessions. This spec deliberately routes instead of renumbering, because the ladder is a calibrated consumer surface; changing it needs a maintainer decision.]
• [NEEDS CLARIFICATION: whether the repository's own visual verification contract in AGENTS.md should name its drive plane explicitly (currently a D1 automation-protocol driver against the preview URL) once the reference lands.]