Verification Intent Anchors and the Post-Failure Repair Loop

Traceability

• Spec ID: verify-intent-and-repair-loop
• Status: Implemented

Intent

Close three gaps in the current verification references that a review against field material on AI-driven cross-platform E2E verification made visible. All three are conceptual gaps, not wording gaps: each one lets a loop report a passing or clean result while a real class of defect is never checked.

1. Coverage has no forward anchor. The Discover plane in references/project-harness/agent-verify-loop.md crosses three machine sources — runtime telemetry, code contracts, real usage samples — and every one of them reads the system as it already exists. Behavior that a spec, acceptance criterion, prototype, or interface contract promised but nobody implemented produces no telemetry, no code contract, and no usage sample, so it never enters the case inventory and can never be judged fail. A new feature with no traffic history has no anchor at all.
2. fail has no defined continuation. The reference ends at the verdict and the aggregate rules. models/agent-work-loop.md scores failure-repair and validate-again as evaluation checks, but no reference states the execution sequence: reproduce, localize, repair at the smallest owner, rerun the same case, record the round, and stop at a convergence bound. Without a stated bound and a same-case rerun requirement, a loop can loosen the assertion each round until the run turns green, or retry indefinitely.
3. Drivers inside one plane are treated as interchangeable.references/project-harness/ui-and-system-drivers.md groups every automation-protocol driver into one D1 technology class. For an agent they are not equivalent: what a driver emits after a failure — a structured step trace, a replay, a machine-readable report, or only logs — determines whether the agent can localize the break at all, and therefore the cost of gap 2. The same reference also lacks the case where the chosen plane is simply unreachable because of a tool boundary, and the degradation is a change of evidence type rather than a lower drive plane.

Recorded Field Observations

Provenance: field material on AI-assisted cross-platform E2E verification (traditional QA regression versus an agent-driven verify chain, a per-platform tool matrix for Web/Android/iOS/mini-program/H5, a unified testing tool surface, and a stated advantages-and-limits list), reviewed on 2026-08-04, plus the Codex session scan already recorded in docs/specs/2026-08-04-drive-plane-selection.md. No new services or tests were run for this spec; these are design inputs, not runtime validation.

Observations that drive the changes below:

• The reviewed material anchors its assertion path on Spec / AC / prototype / interface docs and states the limit plainly: verification quality is bounded by the quality of the spec and the check plan, and behavior absent from the spec will not be verified for you. Our Discover plane has no equivalent source.
• Its evidence chain names a repair record holding root cause, diff, and round number as a first-class output, i.e. the repair loop is expected to be multi-round and recorded, not implicit.
• Within one plane the material separates drivers by what the agent can consume: an execution trace viewer plus an official tool-surface integration, a cloud failure replay with flaky management, or a legacy grid where the stated guidance is that the agent should drive a wrapped domain command rather than raw WebDriver.
• Its mini-program case is an explicit tool-boundary degradation: element-level UI automation was limited by the vendor devtools, so that part was degraded to API plus database assertions, the degradation decision was recorded, and visual detail moved to human sampling. The stated conclusion — admitting the tool boundary and recording it beats forcing an unstable automation — matches the flaky-verdict hazard we already record.
• Its H5/WebView row verifies that the core path still completes under degraded runtime capability (constrained network, older engine, missing API, failing bridge) and explicitly does not require pixel equality. Our Fidelity Ladder answers how real the dependencies are and has no row for how degraded the target runtime is.
• Its stated limits include that aesthetic and experience judgment stays human, which is the residual-gap owner that a degraded case must name.

Deliberately not adopted, with reasons recorded so the boundary is reviewable:

• A concrete unified testing tool-surface inventory (test listing, page/app open, snapshot, cross-platform input, log and trace retrieval, case generation). references/project-harness/friendly-cli.md already owns the one-contract-many-surfaces rule and the MCP projection; duplicating a tool list would split that ownership. Only the verification-side consequence is adopted: cross-platform verdicts must normalize to one verdict domain and one result schema before the aggregate rules can run at all.
• A fixed five-file artifact set (step report, fix log, decision record, evidence directory, management one-pager). That is one methodology's output layout; prescribing filenames for other projects exceeds what a reference should own. Two transferable properties are adopted instead: evidence is addressed per run and append-only within a run, and probe output is layered for its three readers.
• The traditional-QA pain list and the three-way Web tool comparison. Selection background; the part that matters for a harness is already covered by the diagnostic-artifact filter.

Acceptance Scenarios

• VIR-AC-1 (intent as a discovery source): The Discover plane records intent sources — specification, acceptance criteria, prototype or design contract, interface contract — alongside the three existing machine sources, and states that the first three read the system as built while intent states what was promised.
• VIR-AC-2 (promised-but-absent behavior is a case): The reference states that the difference between promised behavior and implemented contract is itself coverage: such a case is admitted and judged, and the absence of a surface to observe makes it fail or blocked, never a reason to omit the case. It also states that verification coverage is bounded by the recorded intent, so unwritten behavior is unverified behavior.
• VIR-AC-3 (post-failure sequence): The reference defines the sequence that follows a non-passing verdict — reproduce on the same correlation handle, localize with the cheapest sufficient diagnostics, attribute to the smallest correct owner, repair, rerun the same case identity, record the round — and requires the rerun to use the same case id, assertions, and judging mode.
• VIR-AC-4 (round record and convergence bound): The reference states what each repair round records (round number, root-cause hypothesis, change scope, rerun verdict), requires each round to advance a different hypothesis, and requires a pre-agreed round bound after which the case becomes blocked for a human rather than retried further.
• VIR-AC-5 (same-cause sweep): The reference requires a repaired root cause to be swept across other cases and surfaces that share the same pattern, with matches entering the skeleton through the existing human gate.
• VIR-AC-6 (flaky registration): The reference requires a case whose verdict oscillates on the same revision to be registered as flaky, states that a flaky case contributes as unobserved to the aggregate verdict until the nondeterminism source is pinned, and adds flaky to the regression-skeleton status domain.
• VIR-AC-7 (layered probe output and append-only evidence): The Judge-plane design constraints require probe output layered for its three readers (machine verdict, diagnostic detail, human-readable summary) and require collected evidence to be addressed per run and append-only within a run.
• VIR-AC-8 (cross-platform verdict normalization): The reference states that cases driven by different per-platform tools must normalize to the same four-valued verdict domain and result schema before the aggregate rules apply, and routes wrapper and surface design to friendly-cli.md.
• VIR-AC-9 (diagnostic artifact as a second filter): ui-and-system-drivers.md records that drivers within one plane differ by the failure evidence they emit for an agent, classifies those artifacts, states what the agent can do after a fail with each, and states that the agent should drive a wrapped task-oriented command rather than a raw driver API.
• VIR-AC-10 (unreachable plane degradation): ui-and-system-drivers.md defines what to do when a tool boundary makes the chosen plane unreachable: degrade by changing evidence type rather than by lowering the drive plane, record the forfeited evidence class, the substitute evidence, and the owner and cadence of any human sampling, and record the decision. It states that such a case is not wholesale unobserved — the substitute evidence carries its own verdict and the forfeited part is a named residual gap in constraints.
• VIR-AC-11 (degraded runtime capability): verification-environment.md records a claim row for core-path completion under degraded target-runtime capability, with its minimum credible environment and what it does not prove, stating that the judgment is functional completion rather than pixel equality.
• VIR-AC-12 (anti-patterns): New anti-patterns are recorded for system-only coverage, repairing until green by loosening the judge, unbounded repair rounds, treating an oscillating verdict as a pass, and forcing an unstable driver at a known tool boundary instead of degrading and recording.
• VIR-AC-13 (link integrity): All relative Markdown links resolve and docs/better-harness-doc-links.mmd is regenerated so the routing graph is not stale.

Non-goals

• Add a new reference file. All three gaps land in existing owners; the domain README Owns list is unchanged.
• Prescribe artifact filenames, directory layouts, or a report schema for other projects.
• Define a testing tool-surface inventory or MCP tool list; friendly-cli.md owns surface design.
• Renumber or extend the four planes, the D0..D4 drive planes, the Fidelity Ladder rungs, or the four-valued verdict domain. flaky is added to the skeleton status field, which is an open enumeration, not to the verdict domain.
• Add rows to the Scenario Families table, or endorse any named tool or vendor.
• Change scoring models, report schemas, skills, scripts, templates, hooks, or host adapters. models/agent-work-loop.md keeps ownership of the Change Validation checks; this spec adds the reference-side execution sequence only.

Plan and Tasks

1. references/project-harness/agent-verify-loop.md Discover plane: add intent sources as a fourth source, restate how the four sources patch each other, and state the promised-versus-implemented difference as coverage (VIR-AC-1, VIR-AC-2).
2. Same file, Judge plane: extend the output design constraint to three reader layers and add the per-run append-only evidence rule (VIR-AC-7).
3. Same file: add a section after Plane 4 defining the post-failure sequence, the round record, the convergence bound, the same-cause sweep, and flaky registration; adjust the existing Rollout failure paragraph to point at it instead of restating it (VIR-AC-3..VIR-AC-6).
4. Same file: add flaky to the regression-skeleton status domain (VIR-AC-6), add the cross-platform verdict-normalization statement with the route to friendly-cli.md (VIR-AC-8), and add the new anti-patterns (VIR-AC-12).
5. references/project-harness/ui-and-system-drivers.md: add the diagnostic-artifact second filter after the technology-class table (VIR-AC-9) and a section for an unreachable plane (VIR-AC-10), plus the forced-plane anti-pattern (VIR-AC-12).
6. references/project-harness/verification-environment.md: add the degraded-capability runtime row to the claim table (VIR-AC-11).
7. Verify relative-link resolution and routing-graph freshness, then run the focused model test and the full suite (VIR-AC-13). All three edited files sit outside the skills/better-harness seed chain, so docs/better-harness-doc-links.mmd is expected to stay unchanged; confirm that rather than assuming it.
8. Run a Review Readiness Check over the local diff before review.

Test and Review Evidence

Implemented on 2026-08-04:

• node --test test/doc-link-graph.test.mjs — the repo-wide relative-link resolution check passes, covering the new routes to friendly-cli.md from both agent-verify-loop.md and ui-and-system-drivers.md (VIR-AC-13).
• docs/better-harness-doc-links.mmd needs no update for this change: the routing graph is seeded from skills/better-harness, and the three edited files are outside that seed chain (friendly-cli.md appears zero times in the committed graph). No node or edge in the graph belongs to this change.
• node --test test/maturity-models.test.mjs — 3/3 pass; consumer integrity for the edited references.
• npm test — 1148/1149 pass. The single failure, Better Harness skill's English-first Markdown chain stays Han-script-free, is not attributable to this change: it reports pre-existing Han text in docs/specs/2026-07-30-supported-host-entrypoints.md and docs/i18n/zh-Hans/**, pulled into the SKILL.md-reachable chain by a concurrent unrelated change in the same working tree that added a support-bootstrap.md -> references/agent-customize/routing.md edge. All three files edited here are Han-script-free and none is reachable from SKILL.md. Verified by stashing: the same test passes on HEAD alone.
• git diff --check — clean.
• VIR-AC-1..VIR-AC-12: manual contract review against the new headings, list items, tables, and anti-patterns in agent-verify-loop.md (After a Non-Passing Verdict, Discover source 1, Judge output constraints, Scenario Families normalization paragraph, skeleton status domain, four new anti-patterns), ui-and-system-drivers.md (Diagnostic Artifacts Are a Second Filter, When the Chosen Plane Cannot Be Reached, one new anti-pattern), and verification-environment.md (degraded-capability claim row plus the capability-degradation paragraph).