Compare live ACP agent runs

Traceability

• Spec ID: studio-realtime-acp-compare
• Status: Implemented

Intent

Allow Harness Studio Compare to execute its fresh Baseline and Candidate runs through one explicitly configured local ACP Agent, from the same locked checkpoint and in separate isolated worktrees. The workbench must stream canonical tool activity and bounded ACP protocol facts while retaining the raw, redacted protocol evidence on disk for later review.

Acceptance Scenarios

• AC-1: harness-experiment.v1 accepts either the existing qoder host or an acp host. Qoder manifests retain their existing profiles and tool policy; ACP manifests use the single honest acp-v1-stdio profile, declare no Harness-standard tools, and may compare harness or model while keeping the host fixed across every lane.
• AC-2: Resolving an ACP experiment uses the prompt-session @harness/adapter-acp descriptor. A manifest cannot claim that Agent-owned ACP Tool Calls satisfy Harness require tool contracts.
• AC-3: When Studio has both an ACP experiment and a server-configured ACP Agent, every fresh trial starts its own Agent process in its own detached worktree, creates a new ACP v1 session rooted there, applies the lane's model through session/set_config_option only when that Agent supports lane-owned model selection, and streams its neutral Harness events into the existing lane-scoped experiment SSE endpoint.
• AC-4: Compare projects ACP protocol frames into bounded browser-safe facts, including method, direction, session id, and permission requests. The workbench shows per-lane ACP frame/session evidence and lets the user resolve only choices offered by that exact pending request through the existing same-origin permission route.
• AC-5: Cancelling the comparison, closing its SSE client, or closing Studio aborts every active lane, sends session/cancel, cancels pending permissions, and reaps Agent processes after the bounded executor grace period.
• AC-6: Each trial persists its redacted neutral trajectory, including raw ACP protocol events, plus runtime, materialization, permission, patch, Git, grade, and result receipts. The runtime receipt identifies the requested and observed ACP session configuration without claiming standard Harness tools.
• AC-7: Studio fails before materializing lanes when an ACP experiment has no configured ACP Agent. Browser input cannot provide or change the executable, argv, environment, workspace root, or protocol version.
• AC-8: Focused contract, executor, runner, server, model, and browser tests cover Qoder compatibility, ACP resolution/execution, model configuration, live protocol projection, permission selection, cancellation/disconnect, evidence retention, responsive layout, keyboard focus, and console/page errors. A real local codex-acp smoke proves two fresh lanes start from the same checkpoint and finish with distinct ACP session ids.

Non-goals

• Comparing ACP against Qoder or another host in one experiment. Host remains a shared confounder boundary and is not introduced as a treatment axis.
• Discovering or accepting arbitrary Agent commands from the browser or from an experiment manifest.
• Treating observed ACP Tool Calls as a verified standard-tool capability.
• ACP v2, remote HTTP/WebSocket Agents, session resume, multi-turn chat, or restoration of external environment and network state.
• Changing the frozen harness-compare.v1 CLI/results path.

Plan and Tasks

1. Generalize the checkpoint experiment runtime contract to the bounded qoder | acp host set, with host-specific profile and visible-tool validation while preserving the existing schema version and Qoder inputs.
2. Resolve lane revisions through the descriptor for the declared host and select the matching executor factory. Keep ACP command ownership in Studio; the core runner accepts only an injected executor.
3. Extend AcpSdkExecutor with bounded session configuration applied after session/new, and include the applied configuration in the runtime receipt.
4. Bind Studio experiment lanes to its existing ACP run controls so permission and cancellation routes stay keyed by the exact lane runId.
5. Extend the canonical experiment stream with ACP protocol and permission facts, fold them into lane state, and render compact docked evidence and permission controls using existing Studio tokens.
6. Abort a run when its SSE client disconnects, retain per-trial protocol evidence, and add focused automated and real-runtime validation.

Test and Review Evidence

• AC-1/AC-2: manifest, resolver, axis, and experiment-runner tests for both hosts and for ACP tool-claim rejection.
• AC-3/AC-6: ACP executor fixture tests for acknowledged session/set_config_option, Agent-default model policy, unique sessions, worktree cwd, runtime receipt, and persisted protocol trajectory.
• AC-4/AC-5/AC-7: Studio server/model tests for protocol projection, permission choice, missing Agent failure, explicit cancel, and transport disconnect cleanup.
• AC-8: focused package typechecks/tests, existing Qoder experiment regression, built Studio Playwright at wide/compact/narrow widths with screenshots and no console/page errors, then one two-lane local codex-acp smoke in read-only or isolated-worktree mode.
• Risk: an ACP Agent remains a real local process. Only the server owns its executable and argv; every fresh lane receives a detached worktree and the browser can only choose an Agent-offered permission option.
• Risk: model selection can be falsely reported if an Agent ignores session configuration. Treat session/set_config_option success plus the returned current option as the observed receipt; fail the lane when the requested model is not acknowledged.
• Risk: protocol frames may contain credentials or large payloads. Preserve the existing redaction and size/count bounds, stream only the minimal canonical projection, and keep full retained evidence server-side.

Implemented evidence

• AC-1/AC-2/AC-3/AC-6/AC-7: Harness contract, runner, and ACP executor tests pass as part of 172 Harness tests. The real Studio run applied and observed gpt-5.4 and gpt-5.5 through session/set_config_option, produced distinct session ids, retained 163 and 157 ACP frames, and passed both isolated trials.
• AC-4/AC-5: Harness UI and Studio suites pass 31 and 275 tests. A live fixture run exposed two simultaneous lane-scoped permission requests, accepted each exact offered option, completed both lanes, and displayed 11 ACP frames per lane without browser console warnings or errors.
• AC-8: 486 focused/package/doc-link tests pass in total. Harness, Harness UI, and Studio production builds pass under Node 24, and generated Langium files are current. Browser checks at 1440x900 and 390x844 reported no horizontal overflow; screenshots cover idle, pending-permission, and narrow layouts.
• Environment limitation: the repository Canvas preview command cannot start on this host because no Canvas SDK runtime path is configured. This does not affect the built Studio surface or the live ACP Compare route.