CodeBuddy Code (cbc) harness

The configuration in this directory controls how cbc behaves inside each trial container. The agent implementation lives in src/workbuddy_bench/agents/cbc_agent.py (CbcAgent).

codebuddy-code/
  _defaults.yaml              shared identity (name/import_path/protocol) + params + models_file pointer + mount skeleton
  versions/<version>.yaml     per version: CBC_VERSION + settings_file + env (loader derives the mount tag from these)
  presets/models.json         static models.json fields (shared across all versions)
  presets/settings/<group>.json  settings presets grouped by version (deny lists vary by version)
  docker/Dockerfile           split-mount image build template

Scoped per version, never unioned: deny lists and env shift from one cbc version to the next, so each version maps explicitly to its own preset group. That mapping lives right in the settings_file / env fields of each versions/<v>.yaml — there's no separate index table. _defaults.yaml offers no fallback for settings_file, so a missing mapping fails fast.

Image delivery (split-mount)

cbc is not baked into the task image. Each version has a corresponding read-only image, mounted at /opt/codebuddy-code at runtime, and cbc_agent.install() symlinks cbc / codebuddy onto PATH. The loader derives the tag from versions/<version>.yaml as workbuddy-bench/harness/codebuddy-code:<version>.

scripts/harness/build-harness-mounts.sh --harness codebuddy-code/2.109.3            # build
scripts/harness/build-harness-mounts.sh --harness codebuddy-code/2.109.3 --dry-run  # just print the docker build

By default, scripts/run.sh just checks that the image exists and errors out if it's missing. You can let a run build it on the fly instead:

AUTO_BUILD_HARNESS_MOUNT=1 uv run ./scripts/run.sh --job <slug>

The two config files: settings.json vs models.json

File	What it controls	What the preset holds (static)	What cbc_agent fills in at runtime (dynamic)
settings.json	behavior, no endpoints	official documented evaluation-related fields	model's thinking_enabled → alwaysThinkingEnabled: true
models.json	model / endpoint	vendor / supportsToolCall / supportsImages / supportsReasoning / (optional) maxOutputTokens	id·name (addressing), url, apiKey, maxInputTokens (path A only, see below)

• id / name: for local_proxy, the route slug; for direct, the real backend id.
• url: the proxy address for local_proxy, or CBC_BASE_URL for direct (must end in /v1/chat/completions).
• apiKey: the dummy key for local_proxy, or CBC_API_KEY for direct.

The endpoint url + key are never written into the preset; they are assembled from the connection at runtime, and secrets never enter the manifest.

Context window + compaction (branches by version)

cbc's compaction mechanism changes across versions, so the window translation branches by version (verified against the dist source). The core logic:

resolveCompactTriggerAt(maxInputTokens, pct):
    return maxInputTokens ? maxInputTokens * pct     # path A
                          : getAutoCompactWindow()   # path B = clamp(AUTO_COMPACT_WINDOW ?? 200k, 100k, 1M)

The two paths are mutually exclusive: once maxInputTokens is set, AUTO_COMPACT_WINDOW is never read.

	path A (cbc < 2.103.4, e.g. 2.93.5)	path B (cbc ≥ 2.103.4, current default)
window →	models.json.maxInputTokens	CODEBUDDY_AUTO_COMPACT_WINDOW env
pct →	CODEBUDDY_AUTOCOMPACT_PCT_OVERRIDE env	inactive (no pct multiplication, env not passed)
maxInputTokens	written	not written (writing it disables the window env)
trigger point	window × pct (can express < 100k)	absolute window, clamped to [100k, 1M]
when maxInputTokens is missing	→ NaN → never compacts	—

The branch boundary is cbc_agent.py: _AUTOCOMPACT_WINDOW_ENV_MIN_VERSION = (2,103,4); versions that can't be parsed fall back to path B. Choose B for an absolute token window; use A's pct for a trigger point below 100k.

Input sources (priority job > bench; the model cap is validated)

Source	Field	Role
configs/bench/	context_window / context_compact_pct	global defaults
configs/jobs/<slug>.yaml	top-level fields of the same name	overrides bench
configs/models/<m>.yaml	model.context_window (optional)	physical cap; a resolved value exceeding it fails fast

After resolution, resolve_manifest._resolve_context_window stores the result in the manifest's context_window: {window, compact_pct, model_physical_max}, which cbc_agent then translates per the table above.

Version → deny list

Differences in permissions.deny across settings/<group>.json (tool existence verified against each version's dist/codebuddy.js):

Tool	legacy (2.93.5)	v2_103 (2.103.4)	v2_109 (2.109.x)	Introduced in
WebSearch / AskUserQuestion	deny	deny	deny	all versions
ComputerUse	absent	deny	deny	2.94
WaitForMcpServers	absent	deny	deny	~2.103
Workflow	absent	absent	deny	2.105
EnterPlanMode / ExitPlanMode	absent	absent	deny	2.109

• Denying a full tool name means cbc removes the tool from the tool table (log: filtered by permissions.deny), not just prompting for approval.
• Convention: only deny tools that exist in that version (denying a nonexistent tool is a no-op but misleads the reader).
• In a headless benchmark, PlanMode can enter an interactive approval flow, causing the rollout to write a plan and then stop producing the target artifact; hence 2.109+ explicitly denies EnterPlanMode / ExitPlanMode.
• 2.103.4+ additionally disables the ComputerUse / WaitForMcpServers gates in harness.env and skips title generation (CODEBUDDY_CODE_DISABLE_SESSION_TITLE_REFRESH=1).
• Fail-fast: cbc_agent.__init__ rejects an empty / missing settings_preset — if a version yaml omits settings_file, it errors out immediately, preventing WebSearch and the like from being silently allowed by an "empty deny list."

Auditing

The manifest's harness_runtime_config block records the final configuration that enters the container (settings_json / models_json / translated_env / max_turns / disallowed_tools / model_params), with sensitive values redacted (url=<proxy_url>, key=<dummy:proxy>) — so the run can be reproduced without reading the agent code.