Harness Execution Closure: Revision, Materialization, Session, Evidence

Traceability

• Spec ID: SPEC-2026-08-15-harness-execution-closure
• Story: none (review-driven blocker fix on feat/harness-as-code-package)
• Status: Implemented

Intent

A source-level review of feat/harness-as-code-package accepted the v0.2 resource model but blocked the merge on four contracts that were not closed between Authored Harness → Locked Revision → Runtime Materialization → Execution Evidence.

This spec covers only those four blockers. The reviewer's larger restructuring asks — a separate HarnessLock artifact, splitting @qoder-ai/harness into core/adapters/devtools packages, moving binding strength out of the core DSL, and bridging HarnessRevision to HarnessComponentSnapshotV1 — are explicit non-goals here.

Non-goals

• Introducing a standalone HarnessLock artifact between IR and Revision.
• Splitting the package boundary along runtime dependencies.
• Removing binding strength / runtime execution claims from the core DSL grammar.
• Making Qoder or Pi materialize MCP connections, multi-session agents, or programmatic workflows.
• Unifying Studio/Compare evidence with the existing Inspector normalized trace.
• Hardening the harness-ui bind address (the review's separate High item on non-loopback serving requiring explicit auth or an unsafe flag).

Blocker 1 — HarnessRevision must be an execution closure

Problem: execute(revision, bundle, task) re-reads workflows and capabilities from whatever bundle the caller passes, without checking them against the hashes recorded in the revision. A revision resolved from bundle A could drive execution content from bundle B while still reporting A's revisionId. The revision was also a plain mutable object, assertRevisionHost ignored target.adapter, and capability source paths were never content-locked.

Acceptance

• AC1.1 validateRevisionAgainstBundle(revision, bundle) recomputes the harness, workflow, and every resolved capability content hash from the bundle and throws HarnessRevisionBundleMismatchError on any drift, naming the drifted entities.
• AC1.2 assertRevisionIntegrity(revision) recomputes revisionId from the revision body and throws HarnessRevisionTamperedError when a field was mutated after resolution.
• AC1.3 resolveHarness returns a deep-frozen revision; a mutation attempt does not change the observed value (and throws in strict mode).
• AC1.4 assertRevisionAdapter(revision, adapterId) throws HarnessAdapterMismatchError when revision.target.adapter names a different adapter package.
• AC1.5 doStart on both shipped adapters runs host, adapter, integrity, and bundle validation before the host SDK is loaded.
• AC1.6 Source locking: lockCapabilitySources(bundle, \{ root \}) digests each declared skill.source file or directory tree; a source-backed revision is verified at doStart and fails closed when the on-disk content drifted.
• AC1.7 A revision cannot resolve while a resolved source-backed capability lacks exactly one matching source lock. Locks digest raw bytes and are verified against an explicit source root, independent from the task working directory.
• AC1.8 Every public materializer runs revision integrity, adapter, bundle, and source-lock preflight before writing output; a Pi package cannot carry bundle B content under bundle A's revisionId.
• AC1.9 The revision locks the adapter contract version, implementation version, and descriptor hash; execution fails before SDK load when any of those facts drift.

Blocker 2 — Declared run semantics must not exceed adapter materialization

Problem: materializeV02() mapped every non-unsupported binding to advisory / prompt-preamble, so require tool, connect mcp, and programmatic workflows all resolved successfully while the executor only emitted a prompt line — and Qoder/Pi passed no tools at all.

Acceptance

• AC2.1 Materialization is capability-kind aware and adapter-owned. An AdapterRealizationDescriptor declares skill delivery, the DSL-tool → host-tool exposure map, MCP support, drivable workflow modes, agent isolation, and consumed setting keys.
• AC2.2 A tool requirement is never satisfied by prompt guidance. Without descriptor-declared exposure it realizes unsupported and resolution fails; with exposure it realizes wired through mechanism host-tool:<name>.
• AC2.3 connect mcp fails closed: no shipped adapter opens MCP connections in v0.2.
• AC2.4 A programmatic workflow fails resolution against a descriptor that can only drive declarative workflows, instead of resolving into a silent no-op.
• AC2.5 The Qoder adapter really exposes the host tools its revision requires (visible tools plus the run-time receipt), so an exposed tool requirement has a runtime effect.
• AC2.6 A multi-agent harness on a single-session adapter is recorded as an explicit workflow degradation with a run warning, not as satisfied orchestration.
• AC2.7 revision.permissions is renamed revision.requestedPermissions; a HarnessMaterializationReceipt records, per capability, materialized | degraded | unsupported plus mechanism, and separates requested from enforced permissions and consumed from ignored settings. Executors attach the receipt to HarnessRunResult.materialization.
• AC2.8 Public validators resolve against the selected built-in adapter descriptor. Published examples target only adapters that can realize their required capabilities.

Blocker 3 — Qoder adapter must be a real multi-turn session

Problem: every doPromptTurn called sdk.query() again with a fresh string prompt and no resume/session_id, so persistSession: true only suppressed the preamble; there was no proof turn 2 saw turn 1. There was also no in-flight turn mutex, doStop/doDestroy never closed or interrupted the query, and runOnce skipped doDestroy when doStop failed with a non-capability error.

Acceptance

• AC3.1 doStart opens exactly one query(\{ prompt: AsyncIterable<SDKUserMessage> \}); each turn pushes a user message onto that queue and drains the shared stream to its result message.
• AC3.2 A fake SDK that models the official Query lifecycle proves turn 2 can answer with information supplied only in turn 1, with a single query() call for the whole session.
• AC3.3 Concurrent doPromptTurn calls are rejected with HarnessConcurrentTurnError; the contract's "turns are sequential" claim is enforced, not documented.
• AC3.4 doStop ends the queue and awaits query.close(); doDestroy interrupts best-effort and then closes. A per-turn abortSignal calls query.interrupt().
• AC3.5 The session exposes the host sessionId; a turn that arrives after the host terminated the query reopens it with resume: sessionId and is delivered once, instead of being reported as a turn nobody ran or replaying a fresh context.
• AC3.6 runOnce destroys the session best-effort when doStop throws any error.
• AC3.7 The Qoder session does not impose a one-turn default on the whole query. Tests assert that an omitted maxTurns stays unbounded while explicit limits remain visible in the runtime receipt.

Blocker 4 — Compare must not accept on insufficient evidence

Problem: decideVerdict() accepted on any pass-rate or 5-point score improvement with no minimum sample size, compared two independent aggregates instead of paired trials, excluded infrastructure errors from the denominator (so one lucky completed trial could beat 20 real ones), compared total cost across unequal completed counts, allowed a manifest to move harness and runtime profile at once, and re-validated persisted verdicts without recomputing aggregates from trial rows.

Acceptance

• AC4.1 CompareStatus gains insufficient_evidence.
• AC4.2 A manifest must move exactly one treatment axis (harness or runtime profile); moving both is rejected at load. The verdict records the axis.
• AC4.3 Verdicts are decided on matched pairs (same trial index, both variants completed). Fewer than the required pairs → insufficient_evidence. A single matched pair can never accept, whatever the configured policy.
• AC4.4 An infrastructure error ratio above the policy threshold → infrastructure_error.
• AC4.5 Aggregates report costPerAttemptedTrialUsd, costPerCompletedTrialUsd, and costPerPassedTrialUsd; the cost guardrail compares cost per completed trial.
• AC4.6 parseHarnessCompareVerdict() recomputes completedTrials, passedTrials, infrastructureErrors, passRate, meanScore, totalCostUsd, totalCredits, the per-trial cost fields, and the matched-pair counts from the trial rows, and rejects any hand-edited aggregate that disagrees.
• AC4.7 Persisted status and reason are also recomputed from the validated policy and trial rows; a two-pair result cannot be relabelled accept under a five-pair policy.

Plan

1. src/ir/revision.ts — integrity, bundle validation, deep freeze, adapter assertion.
2. src/resolver/source-lock.ts — mandatory capability source digests and explicit-root verification for source-backed revisions.
3. src/resolver/adapter-descriptor.ts — adapter realization facts and the shipped descriptors.
4. src/resolver/resolve.ts — kind-aware materialization, requestedPermissions, receipt input. ResolveOptions.adapter also accepts a lookup keyed by the selected runtime, for callers that let the bundle choose its target; src/exec/built-in.ts exposes describeBuiltInAdapter.
5. src/exec/* — descriptor-driven preflight, materialization receipt, Qoder Query session, turn mutex, runOnce teardown.
6. src/compare/* — treatment axis, decision policy, paired verdict, recomputed parse.
7. packages/harness-ui resolves against the descriptor of the adapter that will run the revision; packages/harness-studio renders the treatment axis, matched-pair evidence with its threshold, and cost per completed trial, and styles insufficient_evidence.

Test evidence

The four tests the review asked for first, each failing before the change:

1. refuses execution against a bundle the revision was not resolved from (packages/harness/test/revision.test.ts).
2. detects a mutated revision before the host SDK loads (packages/harness/test/revision.test.ts).
3. answers the second turn from first-turn context inside one query (packages/harness/test/adapter.test.ts).
4. reports insufficient_evidence when a single matched pair is all the evidence there is (packages/harness/test/compare.test.ts).

Follow-ups also covered: a programmatic workflow fails resolution on a declarative-only adapter and a tool requirement cannot be satisfied by prompt-only realization (packages/harness/test/resolve.test.ts); missing/incomplete locks, raw-byte drift, bundle swaps at the public Pi materializer, adapter contract drift, and locked-source drift all fail closed before SDK load or package output (packages/harness/test/revision.test.ts, packages/harness/test/resolve.test.ts); the Qoder adapter refuses a tool exposure it cannot reproduce (packages/harness/test/exec.test.ts); concurrent turns, stop/destroy close and interrupt, resume after a terminated query, an unbounded omitted maxTurns, and a non-capability stop failure still tearing the session down (packages/harness/test/adapter.test.ts); and a persisted verdict whose aggregate, status, reason, or policy the trial rows do not support (packages/harness/test/compare.test.ts, packages/harness-studio/test/compare-model.test.ts).

Verified on 2026-08-15:

• Harness focused tests: 101 passed; full Harness tests: 120 passed; Harness build and typecheck passed.
• Harness UI: 20 passed; Harness Studio: 29 passed; repository suite: 94 files / 1,324 tests passed; doc-link graph: 6 passed and regenerated without stale output.
• Root preview returned ok from /health and served a 100,449-byte /canvas-module.js.
• Harness Studio production build and Playwright browser test passed. The Compare fixture was also opened at 1440x1000: the derived verdict, two matched pairs, cost guardrail, and trial rows rendered with zero console or page errors.
• A live Qoder SDK smoke kept the nonce cobalt-7319 across two doPromptTurn() calls in one adapter session and exposed one host session id.

Risk

• Resolution now fails closed for require tool on adapters without exposure and for connect mcp. Harnesses that previously resolved into a prompt-only no-op now need an adapter that really exposes the capability. This is the intended semantic correction.
• revision.permissions → revision.requestedPermissions is a breaking IR field rename inside the unreleased v0.2 surface.