UI and System Drivers

Use this reference to choose where the agent injects control into the system under verification — in-process, through a browser automation protocol, by attaching to a running browser, through the operating system's input and accessibility plane, or against a physical device — and to record what that choice costs in observability, isolation, judging expense, and parallelism.

Agent Verify Loop Scenario Families answer what is verified. The Verification Environment Fidelity Ladder answers how real the dependencies are. Neither distinguishes a browser driven through an automation protocol from the same browser driven through OS input events with screenshot judging: both are the Web UI family and both can sit at Fidelity L3, yet their trigger shape, evidence set, isolation model, cleanup obligations, and failure modes differ enough that an agent cannot pick between them from those two axes alone. Without a stated axis the choice is made by tool availability, not by the verification claim.

Ownership Boundary

This reference owns:

• the three-axis decomposition of driving decisions (decision source, control-plane location, session origin);
• the D0..D4 drive plane ladder and its trade-off table;
• the technology-class map behind each plane;
• the within-plane filter on what failure evidence a driver emits for an agent;
• the degradation path when a tool boundary makes the chosen plane unreachable;
• the rule that drive adapters and observation adapters are selected and recorded separately; and
• the shared six-step execution skeleton with per-plane deltas.

It does not own the four verify-loop planes, the verdict domain, the scenario families, or the regression-skeleton shape (Agent Verify Loop); nor dependency fidelity, the environment contract, oracle independence, or authorization boundaries (Verification Environment).

Load When

• A UI, desktop, mobile, or terminal case needs a trigger and more than one driver class could exercise it.
• A verify loop is being designed around an attach-mode browser agent or a computer-use loop, and the evidence it can produce must be stated before the judging mode is chosen.
• A review finds screenshot-judged acceptance where structural evidence (DOM, accessibility tree, state store) was available one plane lower.
• A tool boundary blocks the driver the claim asked for, and the case needs a degradation path instead of an unstable automation.
• A case reuses a real logged-in profile or a running application window, and isolation, reset, or parallelism questions surface.

Three Independent Axes

The popular two-way framing — "Playwright versus computer use" — conflates three axes that are freely composable. Record them separately per case:

Axis	One end	Other end
Decision source	script-predefined steps	model decision at run time
Control-plane location	in-browser automation protocol	out-of-browser OS input plane
Session origin	fresh throwaway profile	existing real profile and login state

Script-driven OS automation with a real profile, and model-driven decisions over an automation-protocol driver attached to an existing browser, are both valid and both in field use. The decision source determines the judging mode; the control-plane location determines the available evidence; the session origin determines isolation and reset obligations.

Drive Plane Ladder

Planes are ordered by control-plane location. Observability decreases and session realism increases monotonically along the ladder — isolation moves the opposite way from Fidelity Ladder intuition, which is the gap this reference closes.

Plane	Control-plane location	Structural evidence	Session realism	Judging cost	Parallel-safe
D0	In-process render tree	full	none (no real browser)	lowest	yes
D1	In-browser automation protocol	DOM, accessibility tree, network, console	moderate (throwaway profile)	low	yes
D2	In-browser agent attached to a running browser	mostly available	high (real profile and login state)	medium	no
D3	OS input and accessibility plane	accessibility tree only, or pixels only	high (ordinary user session)	high	no
D4	Physical or device plane	pixels only	highest	highest	device-bound

A real-profile browser (D2) or an ordinary user session (D3) has higher session realism but lower isolation than the Fidelity Ladder L3 isolated real browser. Record this inversion in the case constraints; do not read a real profile as a higher fidelity rung.

Select the cheapest plane whose evidence satisfies the claim, and escalate only on fail, unobserved, or a claim that genuinely lives higher (window focus, OS chrome, real login state, device hardware).

Technology Classes per Plane

Named tools are examples of a class (provenance: 2026-08), not endorsements or a closed list.

Plane	Technology class	Examples
D0	Component render harness	Testing-library style renderers, jsdom, in-browser component runners
D1	Automation-protocol driver	Playwright, Puppeteer, Selenium/WebDriver, WebDriver BiDi, raw Chrome DevTools Protocol
D2	Attach-mode browser agent	Extension or DevTools-protocol agents attached to an already-running browser, browser-control MCP servers
D3	OS input driver	Windows UI Automation and pywinauto, cross-platform input synthesis such as PyAutoGUI, macOS Accessibility API and System Events scripting, X11/Wayland input tools
D3	Computer-use agent loop	Screenshot-in, action-out model loops that emit click/type/scroll steps
D3	Terminal/TUI driver	Pseudo-terminal drivers, tmux control mode, expect-style scripts
D3/D4	Mobile driver	ADB and uiautomator, Appium, XCUITest, Maestro, device farms
D1'	Vendor devtools driver	Mini-program devtools automation, platform emulators and simulators
Observation only	Network observation and virtualization	Recording/forwarding proxies, service virtualizers
Observation only	Vision judge	Vision-model screenshot judging against a rubric
Observation only	State oracle	Direct state, table, log, or trace queries

Diagnostic Artifacts Are a Second Filter

Drivers inside one plane are not interchangeable for an agent. The plane determines what evidence exists while the case runs; what the driver emits determines whether the agent can localize a break after it fails. Filter within the plane on that, because it sets the cost of the diagnose step in the Agent Verify Loop post-verdict sequence.

Failure artifact the driver emits	What the agent can do after a fail
Structured step trace: each action with before/after snapshot, network, console	Localize offline from one run; no rerun needed
Recorded replay or video	Useful to a human; an agent can use it only as far as it can address frames or steps programmatically
Machine-readable result report (structured or JUnit-style)	Learn which assertion broke, not why; usually needs one more run
Logs only	Rerun with added instrumentation, repeatedly, and hope the failure reproduces

Two consequences follow. First, a driver one plane lower that emits a structured trace can be the better choice over a higher-plane driver that emits only logs, when the claim allows either. Second, an agent should drive a task-oriented wrapper command rather than a raw driver API: the wrapper is where the trace, the result schema, and the cleanup guarantee live. See Agent-Friendly CLI for that contract.

When the Chosen Plane Cannot Be Reached

Tool boundaries are real: vendor devtools may not expose element-level automation, signing, account, or store review can gate a device plane, and custom-rendered UI may publish no accessibility tree at all. Forcing an unstable driver at such a boundary produces oscillating verdicts, which is worse than admitting the boundary — a flaky judge teaches the loop to ignore the judge.

Degrade by changing evidence type, not by dropping to a lower drive plane that shares the same wall. The usual move is to replace element-level UI assertions with API and persisted-state assertions, and to hand the remaining visual or experiential detail to scheduled human sampling. Record four things with the case:

• the evidence class forfeited (for example, element-level interaction proof);
• the substitute evidence and what it does cover;
• the owner and cadence of the human sampling that covers the rest; and
• the decision itself, in whatever decision record the project uses, because a successor will otherwise read the gap as an oversight.

Such a case is not wholesale unobserved. The substitute evidence carries its own verdict for what it checks, and the forfeited part is a named residual gap in constraints. Recording the whole case as unobserved throws away evidence that was actually collected; recording it as pass without naming the gap is a false green.

Drive and Observation Are Separate Choices

An automation-protocol driver is both driver and observer, which is why the two roles are usually conflated. They must be chosen and recorded independently: the drive adapter is an Exercise-plane asset, the observation adapter is a Judge-plane asset. A low drive plane must be compensated with stronger side-effect evidence — persisted state, emitted events, network calls, backend records — never with model confidence.

Field transcripts confirm both the practice and the hazard. Sessions driving through an OS accessibility plane used the accessibility tree itself as the oracle (re-reading application state after each action), and sessions driving an attached browser dumped DOM state as structured JSON for exact assertions — both are correct plane/observation splits. But across 70 UI-verification sessions, pixel-diff tooling appeared zero times while the driving model visually judged its own screenshots hundreds of times, and most screenshot paths were overwritten in place so no baseline survived the session.

When the same model both drives and judges, the loop has no independent oracle and lands on the Self-reported success anti-pattern in Agent Verify Loop. Before such a run may report pass, it needs an independent oracle plus a known-bad control, per the oracle-independence requirements in Verification Environment. A model-judged screenshot read alone is unobserved, not pass.

Shared Six-Step Pattern

Every plane runs the same skeleton; the differences concentrate in session acquisition, state reset, and cleanup.

acquire session -> reset state -> locate target -> apply action
  -> wait for terminal condition -> collect evidence and judge -> clean up

Step	D1	D2	D3
Acquire session	create a fresh context	attach to a running browser, require window focus	focus a real window, require foreground exclusivity
Reset state	context isolation is the reset	explicit sign-out or storage reset; no inherent isolation	scripted desktop and profile state reset
Locate target	selector or role query	selector or role query	accessibility tree, or template/text recognition on pixels
Wait	render-idle and network-idle signals	same as D1	pixel stability plus a mandatory deadline
Collect evidence	DOM snapshot, screenshot, network, console	same as D1	before/after screenshots, JSON step record, independent state oracle
Clean up	close the context	restore the user profile state	close windows, restore focus; must run after failure and timeout

The real risk at D2 and D3 is not driving but reset and cleanup: reusing a real profile means a run mutates user state, and an OS input plane is exclusive, so concurrent runs corrupt each other. A missing terminal condition at D3 turns a settling page into either a hung loop or a flaky verdict — and fixed sleep/wait <ms> delays dominate field transcripts where bounded polling with a deadline was needed.

Two locate-step rules from the same transcripts:

• Never cache positional handles (element indexes, coordinates) across actions; re-read the accessibility tree or DOM before every interaction, because any re-render invalidates them.
• Degrade deliberately: semantic selector → text/aria match → raw coordinates, and record which level the case ended on — a coordinate-level locate is a determinism liability that belongs in the case constraints.

Self-Execution Wiring

A drive adapter becomes something an agent runs without prompting when:

• one named entrypoint command exercises the case end to end;
• the case trigger field records the drive adapter and the correlation handle, per the Agent Verify Loop skeleton shape;
• plans start from the cheapest sufficient plane and escalate on fail or unobserved, never by default;
• output is machine-parseable verdict-first, using the four-valued verdict domain;
• an enforcement point fires when the agent claims completion — a claim without a matching drive-adapter run is non-passing evidence; and
• every diagnosed-and-repaired failure is promoted into the regression skeleton through the existing human gate.

Anti-Patterns

• Plane by availability: choosing computer-use or an attach-mode agent because it is installed, when the claim needed only D0/D1 structural evidence.
• Forcing an unstable plane: insisting on element-level automation at a known tool boundary and living with oscillating verdicts, instead of degrading to another evidence type and recording the forfeited class, the substitute, the human-sampling owner, and the decision.
• Realism read as fidelity: treating a real-profile session as a higher fidelity rung instead of recording the isolation loss in constraints.
• Same-model drive and judge: the driving model visually approves its own screenshots; without an independent oracle and a known-bad control this is unobserved, not pass.
• Baseline overwritten in place: re-capturing to the same screenshot path destroys the before/after evidence chain; keep captures append-only within a run.
• Cached positional handles: reusing element indexes or coordinates across re-renders, producing clicks on the wrong control.
• Fixed waits as terminal conditions: sleep/wait <ms> instead of a settled-state probe with a deadline; eventual consistency becomes flaky verdicts.
• Cleanup only on success: D2/D3 cleanup that does not run after failure or timeout leaves mutated profiles and stolen window focus for the next run.