Harness as Code Architecture Hardening: Sandbox, Grammar, Boundaries, Evidence

Traceability

• Spec ID: SPEC-2026-08-15-harness-as-code-architecture-hardening
• Story: none (review-driven follow-up on feat/harness-as-code-package)
• Status: Implemented

Intent

The source-level review of feat/harness-as-code-package raised four High findings beyond the four merge blockers. The blockers are closed by Harness Execution Closure, which explicitly deferred these four items as non-goals:

1. Execution boundary — the compare pipeline claims "network deny", but only the Quick Start example and export probe run under node --permission with a scrubbed environment; the npm test validation step inherits the full host process.env with no permission flags.
2. Adapter facts in the core DSL — binding strength (other than unsupported) and runtime execution claims are author declarations about adapter implementation facts; today they are silently ignored or trusted.
3. Package boundary — the main @qoder-ai/harness entry re-exports exec/compare/highlight, so a consumer that only parses .harness must install the Qoder SDK and Shiki; harness-ui imports concrete executors.
4. Second evidence platform — HarnessRevision / receipts / run traces / compare verdicts have no bridge to the existing HarnessComponentSnapshotV1 / NormalizedToolActivityV1 / Inspector chain.

The outcome of this spec: safety claims become recorded evidence rather than prose, the DSL cannot state facts it does not own, the dependency direction core ← adapters ← devtools is enforced by tests, and harness runs land in the same evidence chain the Inspector already reads.

Non-goals

• Publishing new npm packages (@qoder-ai/harness-core, adapter-qoder, …). This spec fixes the dependency direction inside the existing package; physical package splitting is a later, mechanical step.
• A container/VM sandbox implementation. This spec defines the TrialSandbox seam and ships the trusted-fixture policy; an isolated-container policy is a separate deliverable behind the same interface.
• Changing verdict statistics (matched pairs, insufficient_evidence, cost-per-trial) — already implemented by the execution-closure spec.
• Making Qoder/Pi adapters materialize MCP, multi-session agents, or programmatic workflows.
• Migrating existing Inspector reports or session-analysis storage formats.
• A HarnessLock artifact between IR and Revision (still deferred).

---

High 1 — Execution boundary: TrialSandbox and SandboxReceipt

Problem

packages/harness/src/compare/grader.ts hardens the Quick Start example and the export probe (node --permission --allow-fs-read=<trialRoot> plus safeExampleEnvironment(), a 7-key env allowlist), but the package-tests check calls runCommand(npmTest.command, …) with no env option, and runCommand in packages/harness/src/compare/process.ts defaults to env: process.env. Package scripts run with full host credentials, network, and filesystem. The agent-facing allowlist in compare/permissions.ts bounds the tool surface, not the process capability surface. The honest current claim is "bounded tool surface over a trusted fixture", not "network denied".

Design

Promote the execution boundary to a first-class seam owned by the compare runner, not scattered options inside the grader:

Grader / Runner
      │ never spawns directly
      ▼
TrialSandbox (interface)
  └── run(command, args, { cwd, timeoutMs }): Promise<CommandResult>
  └── describe(): SandboxReceipt
Policies:
  trusted-fixture  → env allowlist + node --permission where applicable
  (future) isolated → disposable container / clean fs + network namespace

SandboxReceipt is persisted into the trial evidence and surfaced in the verdict:

interface SandboxReceipt {
  policy: "trusted-fixture" | "isolated";
  envPolicy: "allowlist" | "inherited";
  envKeys: string[];                       // exact keys passed through
  networkPolicy: "denied" | "unverified";  // never claim more than enforced
  fsScope: "trial-root" | "host";
  permissionFlags: string[];               // e.g. ["--permission", "--allow-fs-read=…"]
}

Rules:

• The trusted-fixture policy runs every subprocess (including npm test) with the env allowlist. npm/node operational keys (PATH, TEMP/TMP/ TMPDIR, SystemRoot, ComSpec, PATHEXT, HOME for npm cache resolution, and explicit npm_config_* needed for offline install) are the whitelist; everything else — tokens, cloud credentials, proxy settings — is dropped.
• node --permission is applied where the command is a direct node invocation; npm test cannot carry it (npm spawns arbitrary scripts), so its receipt records networkPolicy: "unverified" — which is the honest fact.
• The verdict gains a sandbox field carrying the receipt. Report rendering must label any networkPolicy: "unverified" run as "trusted-fixture only — network not denied". README.md and package README wording is corrected to match.
• runCommand loses its process.env default: env becomes a required option, so a future call site cannot silently reintroduce inheritance.

Acceptance

• AC1.1 Every subprocess spawned during a compare trial goes through a TrialSandbox; a direct runCommand call from grader/runner without an explicit env fails to compile.
• AC1.2 Under the trusted-fixture policy, the env passed to npm test contains only allowlisted keys; a canary variable (e.g. AWS_SECRET_ACCESS_KEY, GITHUB_TOKEN) set in the parent env is absent from the child env.
• AC1.3 Each trial directory contains sandbox-receipt.json; parsing a verdict whose trials lack a sandbox receipt fails schema validation.
• AC1.4 The verdict HTML/CLI rendering shows the sandbox policy and labels networkPolicy: "unverified" runs as trusted-fixture only; no rendered output uses the phrase "network denied" for such runs.
• AC1.5 The Quick Start example and export probe keep their existing node --permission hardening and now also produce receipt entries with permissionFlags recorded.

---

High 2 — Remove adapter facts from the core DSL

Problem

The grammar (packages/harness/src/language/harness.langium) lets authors declare binding … \{ mechanism <QualifiedName> strength <Strength> \} and runtime … \{ execution programmatic.<lang> \}. Since the execution-closure work, materializeAgainstAdapter (resolver/resolve.ts) only honors strength unsupported as an author veto; any other declared strength is ignored and the adapter descriptor decides. That makes strength enforced a silent no-op — the worst kind of grammar surface. Similarly, whether a runtime can execute programmatic.deno is an adapter observation, not an authoring decision, yet the resolver trusts the DSL's execution claim (resolve.ts workflow checks).

Ownership model (from the review, already partially realized):

Harness DSL        → declares what a run needs (requirements, veto)
Adapter descriptor → declares what a runtime can provide (facts)
Receipt            → records what this run actually got (observations)

Design

1. Binding becomes veto-only. Grammar keeps binding <capability> for <runtimes> \{ unsupported? notes? \}. strength and mechanism are removed from the binding rule. During a deprecation window the validator reports a compile error (not a silent drop) for strength/mechanism with a fix-it message pointing at the adapter descriptor; the tokens are removed from the grammar in the following minor version.
2. execution moves to the descriptor. AdapterRealizationDescriptor (resolver/adapter-descriptor.ts) gains programmaticLanguages: readonly string[] alongside workflowModes. The resolver's programmatic-workflow gate reads the descriptor, not runtime.execution. The DSL keeps only the requirement side: a workflow may declare program deno "./loop.ts", and resolution fails closed when no descriptor supports that language (behavior already tested for unsupported adapters; the source of truth moves).
3. Descriptor registry. A pure-data registry (resolver/adapter-registry.ts) maps adapter package id → descriptor, loadable without importing any host SDK. QoderSdkAdapter.describe() / PiSdkAdapter.describe() return the same frozen descriptor objects the registry ships, and doStart asserts registry/instance descriptor equality so the two cannot drift.
4. RuntimeDeclaration.adapter stays: choosing which adapter package to bind is a deployment decision the author legitimately owns.

Affected v0.2 example files and skills/ DSL snippets are migrated in the same change.

Acceptance

• AC2.1 Compiling a .harness source containing strength wired or strength enforced in a binding produces a compile error whose message names the adapter descriptor as the owner of realization strength.
• AC2.2 binding <cap> for <rt> \{ unsupported \} still vetoes the capability on that runtime (existing behavior preserved; regression-tested).
• AC2.3 A programmatic workflow resolves only when the registry descriptor for the target adapter lists the language in programmaticLanguages; a DSL runtime … execution claim alone can no longer satisfy the gate. During the deprecation window, execution in a runtime block compiles with a deprecation diagnostic and has no resolution effect.
• AC2.4 resolveHarness can produce a revision for the built-in Qoder and Pi targets in a process that has never loaded @qoder-ai/qoder-agent-sdk or @earendil-works/pi-coding-agent (registry is pure data).
• AC2.5 For each shipped adapter, the registry descriptor deep-equals the live describe() output (drift test), and doStart fails closed on mismatch.

---

High 3 — Enforce the core ← adapters ← devtools dependency direction

Problem

compiler/, resolver/, ir/ already avoid host SDK imports, and package.json already exposes ./ir, ./exec, ./compare, ./compare/verdict, ./highlight subpaths. Four leaks defeat the layering:

1. src/index.ts re-exports ./exec/index.js, ./compare/index.js, and ./highlight/shiki.js, so the main entry's module graph pulls in the Qoder SDK, node:child_process, and Shiki. @qoder-ai/qoder-agent-sdk and shiki are hard dependencies.
2. packages/harness-ui/src/run.ts imports QoderSdkExecutor / PiSdkExecutor as values from the main entry, hard-binding the AG-UI layer to both hosts.
3. Adapter descriptors live in the same modules as adapter implementations (resolved by High 2's registry).
4. resolver/source-lock.ts imports node:fs/promises, so the core entry is not browser-safe.

Design

Fix the direction inside the existing package; do not split npm packages yet.

"."            core only: grammar, compile, IR, revision, canonical hash,
               resolver, adapter-descriptor types + registry   (browser-safe)
"./lock"       source locking (node:fs)                        [moved out of core]
"./exec"       adapter contract + Qoder/Pi adapters (host SDKs via dynamic import)
"./compare"    runner, grader, sandbox, verdict (node:child_process)
"./compare/verdict"  verdict parsing only (pure, browser-safe — Studio reads it)
"./highlight"  shiki

• src/index.ts drops the exec/compare/highlight re-exports and the source-lock value exports; consumers use subpaths. ./lock is added as a new subpath export for lockCapabilitySources / verifyRevisionSourceLocks.
• @qoder-ai/qoder-agent-sdk moves from dependencies to an optional peerDependency (mirroring the existing Pi arrangement); exec/qoder-sdk.ts already loads it by module id at runtime. shiki stays a dependency but is reachable only via ./highlight.
• harness-ui becomes injection-only: it keeps the existing HarnessUiExecutorFactory seam and stops importing concrete executors; the built-in Qoder/Pi factory moves to harness-studio's server (which already owns host wiring) and to the harness-ui CLI entry, both of which import @qoder-ai/harness/exec explicitly.
• An import-graph invariant test (see Test Evidence) makes the direction survivable. Per the repository test convention it must assert on the resolved module graph of the built entry points, not grep source text.

Acceptance

• AC3.1 Resolving the module graph of the built . entry (dist/index.js) reaches no module from dist/exec/, dist/compare/ (except nothing), dist/highlight/, shiki, @qoder-ai/qoder-agent-sdk, @earendil-works/pi-coding-agent, node:child_process, or node:fs.
• AC3.2 The graph of ./compare/verdict is equally pure (Studio can bundle it for the browser).
• AC3.3 npm install of @qoder-ai/harness without the Qoder SDK peer succeeds, and compileHarness + resolveHarness work; loading ./exec and constructing a Qoder adapter without the peer fails with the existing actionable module-load error.
• AC3.4 The module graph of @qoder-ai/harness-ui's main entry contains no concrete executor module; handleAguiRun requires an injected HarnessUiExecutorFactory and its tests pass with a fake factory only.
• AC3.5 harness-studio continues to run end-to-end (its server provides the factory), verified by the existing studio Playwright suite.

---

High 4 — One evidence chain: bridge runs into the Inspector platform

Problem

Two disjoint evidence platforms exist. Existing chain (canonical owners per ADR-0002 federation): HarnessComponentSnapshotV1 (scripts/harness-component-snapshot/), NormalizedToolActivityV1 + report model (scripts/harness-inspector/), session platform adapters (scripts/session-analysis/platforms/\{claude,codex,cursor,qoder\}.mjs), commit-session-link. New chain: HarnessRevision, materialization receipt, HarnessRunEvent, per-trial trace.jsonl / runtime-receipt.json, compare verdict.json, Studio. No bridge code exists in either direction, and a compare evidence directory is not self-contained (trials record revisionId but the directory holds no revision or resolution report).

Design

Converge at the persistence layer; do not merge schemas.

HarnessRevision + MaterializationReceipt   (config provenance; owner: packages/harness)
        ↓ run
HarnessRunEvent                            (live streaming only; owner: packages/harness)
        ↓ persisted as
trace.jsonl  ──ingested by──▶  NormalizedToolActivityV1   (owner: session-analysis/inspector)
        ↓
Inspector / Controlled Eval reports

1. Self-contained evidence directory. The compare runner writes revision.json (the full deep-frozen revision), resolution-report.json, and materialization-receipt.json at the variant level; each trial keeps trace.jsonl, runtime-receipt.json, sandbox-receipt.json (High 1), and permission-decisions.json. parseHarnessCompareVerdict gains a directory-level validation mode that fails when the revision file is absent or its revisionId disagrees with the trials.
2. New platform adapter. scripts/session-analysis/platforms/harness-run.mjs ingests a harness evidence directory the same way claude.mjs ingests Claude session files: it maps trace.jsonl events to NormalizedToolActivityV1 (tool calls, families, segments, timeline) and exposes the run as one more provider. The Inspector then reports harness runs with zero new report surface.
3. Cross-reference, not schema merge. HarnessRevision gains an optional componentSnapshotRef (snapshot id + digest) and HarnessComponentSnapshotV1 consumers may record revisionId in their population references. The snapshot answers "what exists in the project"; the revision answers "what this run was configured with".
4. Studio positioning. Studio remains the authoring/run workspace; live views consume HarnessRunEvent over SSE, historical views read the persisted evidence directory. Studio grows no second persistence format.
5. A short ADR under docs/adrs/ records the canonical owners of the two contracts and the bridge direction, extending the ADR-0002 federation, so a third platform does not appear later.

Acceptance

• AC4.1 After a compare run, the evidence directory contains revision.json, resolution-report.json, and materialization-receipt.json per variant; directory-level verdict validation fails on a missing or mismatched revision.json.
• AC4.2 The harness-run platform adapter converts a fixture trace.jsonl into a NormalizedToolActivityV1 whose call counts, tool families, and timeline match the fixture events (asserted on the parsed structure, not on rendered text).
• AC4.3 An Inspector report generated over a workspace containing a harness evidence directory lists the run as a session from provider harness-run, linked to the revisionId.
• AC4.4 A revision carrying componentSnapshotRef round-trips through canonical hashing and revision-integrity checks (the ref is part of the hashed body).
• AC4.5 The ADR exists, names both contracts, their owners, and the bridge direction, and is reachable from docs/adrs/README.md.

---

Plan and Tasks

Order chosen so grammar breaks land early and evidence contracts freeze last:

1. High 3 first (mechanical, unblocks High 2):
   • move source-lock exports to ./lock; trim src/index.ts; add subpath; demote Qoder SDK to optional peer; adjust harness-ui to injection-only; move built-in factory to studio server + harness-ui CLI.
   • add the import-graph invariant test.
   • files: packages/harness/src/index.ts, packages/harness/package.json, packages/harness-ui/src/run.ts, packages/harness-ui/src/cli.ts, packages/harness-studio/src/server/server.ts.
2. High 2 (breaking grammar change, cheapest now):
   • grammar: veto-only binding, deprecation diagnostics for strength/mechanism/execution; regenerate Langium artifacts (npm run langium:generate, check:generated stays green).
   • programmaticLanguages on the descriptor; registry module; drift test; migrate examples and skills/ snippets.
   • files: packages/harness/src/language/harness.langium, language/harness-validator.ts, resolver/adapter-descriptor.ts, resolver/adapter-registry.ts (new), resolver/resolve.ts, exec/qoder-sdk.ts, exec/pi-sdk.ts, examples/, skills/.
3. High 1 (verdict schema addition — before evidence freeze):
   • TrialSandbox interface + trusted-fixture policy; make env required on runCommand; thread sandbox through grader/runner; sandbox-receipt.json; verdict schema + rendering labels; README wording fix.
   • files: packages/harness/src/compare/sandbox.ts (new), process.ts, grader.ts, runner.ts, verdict.ts, aggregate.ts, report templates, packages/harness/README.md, root README.md (compare claims paragraph).
4. High 4 last (consumes contracts from 1–3):
   • self-contained evidence directory + directory validation; harness-run platform adapter; componentSnapshotRef; Studio historical view reads the directory; ADR.
   • files: packages/harness/src/compare/runner.ts, verdict.ts, packages/harness/src/ir/index.ts + ir/revision.ts, scripts/session-analysis/platforms/harness-run.mjs (new), scripts/harness-inspector/ (provider registration), docs/adrs/ (new ADR), docs/adrs/README.md.

Each step remains independently reviewable so the cross-platform Node matrix, package tests, and the Studio browser suite can serve as merge evidence.

Decision rationale:

• Dependency direction before grammar work: the registry (High 2) must live in a core that provably does not load host SDKs, so the graph invariant comes first.
• Deprecation-diagnostic window instead of instant token removal: existing .harness sources in examples, skills, and early adopters get one actionable compile error before the tokens disappear; silent acceptance is the failure mode being removed, so the diagnostic is an error, not a warning.
• Bridge at NormalizedToolActivityV1 rather than a new shared schema: the Inspector's platform-adapter seam already absorbs four providers; adding a fifth is the lowest-cost, lowest-risk convergence and keeps ADR-0002 federation intact.

Test and Review Evidence

Repository conventions apply: assert on behavior and parsed structures, never on source text patterns; no repo-wide literal scans.

AC	Evidence
AC1.1	TypeScript compile: env required on runCommand; grader/runner call-site test with a fake sandbox recording every spawn
AC1.2	Trial test sets GITHUB_TOKEN in parent env, asserts child env (captured by fake sandbox / echo probe) lacks it and matches the allowlist exactly
AC1.3	Verdict schema test: verdict without sandbox receipts fails parseHarnessCompareVerdict
AC1.4	Render test parses the generated report model (not HTML text) and asserts the policy label field for an unverified receipt
AC2.1–AC2.3	Compiler tests: fixture sources with strength wired, veto binding, programmatic workflow with/without registry support; assert diagnostics and resolution results
AC2.4	Test spawns a child Node process that imports only the core entry, resolves the Qoder target, and asserts require.cache/loaded-modules contains no host SDK
AC2.5	Deep-equal test between registry descriptor and describe() per adapter
AC3.1–AC3.2	Import-graph test: resolve the built entry's ESM graph (e.g. node --experimental-import-meta-resolve walker or es-module-lexer walk over dist/) and assert the forbidden-module set is unreachable
AC3.3	CI job (or test with temporarily hidden peer) exercising compile+resolve without the Qoder SDK installed
AC3.4	harness-ui tests run with fake factory only; module-graph assertion on its entry
AC3.5	Existing packages/harness-studio Playwright suite
AC4.1	Runner integration test over a fake executor; assert directory contents and directory-level validation failure on deleted revision.json
AC4.2	Fixture trace.jsonl → adapter → assert NormalizedToolActivityV1 fields
AC4.3	Inspector report-model test over a workspace fixture containing a harness evidence directory
AC4.4	Revision canonical-hash round-trip test including componentSnapshotRef
AC4.5	Doc-link test (npx vitest run test/skills-docs/doc-link-graph.test.mjs) after adding the ADR

Risk notes:

• R1 (High 1): over-tight env allowlist can break npm test on Windows (SystemRoot, ComSpec, PATHEXT) or on npm cache resolution (HOME, LOCALAPPDATA). Mitigation: allowlist is platform-aware and CI runs the compare suite on the existing four-platform matrix.
• R2 (High 2): grammar break invalidates existing .harness files. Mitigation: deprecation diagnostic with fix-it text; examples and skills migrate in the same commit; check:generated guards Langium artifacts.
• R3 (High 3): demoting the Qoder SDK to an optional peer changes install behavior for current consumers. Mitigation: ./exec keeps the existing actionable module-load error; README installation snippet updated in the same PR.
• R4 (High 4): trace.jsonl event vocabulary may not cover everything NormalizedToolActivityV1 expects. Mitigation: the adapter maps only what the trace proves and leaves absent fields empty rather than inventing them; gaps are recorded in the adapter's fixture tests.
• R5 (cross-cutting, deferred policy decision): whether npm test validation must eventually require the isolated-container policy before a compare verdict may render accept. This implementation preserves the specified trusted-fixture behavior and labels network denial as unverified.

Implementation Evidence

• npm test: 95 files and 1325 tests passed, including the new harness-run provider and Inspector integration coverage.
• Harness packages: build plus 127 harness, 21 harness-ui, and 29 harness-studio tests passed.
• npm run harness-studio:test:browser: the Studio Playwright interaction test passed in Chromium.
• npm run pack:verify: npm and runtime package contents passed verification.
• Cold install: the packed @qoder-ai/harness installed with --omit=peer; core compile and resolve succeeded without the Qoder SDK.
• npm run preview: /health and /canvas-module.js both returned successfully from http://127.0.0.1:58575.
• npx vitest run test/skills-docs/doc-link-graph.test.mjs, generated Langium checks, module-graph invariants, and git diff --check passed.