Make the test suite prove behavior instead of implementation text

Traceability

• Spec ID: test-quality-cleanup
• Status: Implemented

Intent

Reduce the maintenance cost and false confidence created by tests that read source, templates, or prose and then lock individual words, private function names, CSS literals, or JSX spelling with regular expressions. Keep tests that prove observable behavior, versioned data contracts, security boundaries, package contents, documentation links, and intentionally stable public output.

The 2026-08-13 baseline has 95 *.test.mjs files, about 52,000 lines, and 2,020 assert.match / assert.doesNotMatch calls. Twelve hotspot files contain 1,089 of those text assertions. This cleanup is therefore organized by evidence quality rather than by mechanically banning regular expressions.

Acceptance Scenarios

• AC-1: The shipped Canvas template is accepted by the production artifact validator using representative findings and Canvas data without a test enumerating private component names, exact JSX layout literals, copy text, or helper-function spelling.
• AC-2: Harness Inspector tests assert the versioned report projection, self-contained document boundary, embedded data, privacy/redaction, and executable client script without enumerating private browser function names, CSS classes, or incidental visible copy in one omnibus assertion block.
• AC-3: Skill, model, template, and documentation checks retain structural invariants such as valid front matter, prompt budgets, heading/routing shape, resolvable links, locale coverage, and unsafe-content exclusions, but do not pin ordinary explanatory sentences word by word.
• AC-4: Tests that inspect module architecture use public exports, behavior, or a dedicated dependency/structure check. They do not infer ownership from the presence or absence of private identifier text when an executable seam is available.
• AC-5: Security, canonical-path, symlink, schema, packaging, privacy, parser, and failure-boundary regressions remain covered. A lower text-assertion count is not accepted if it removes these independent oracles.
• AC-6: Focused tests for every changed area, the documentation link graph, the full suite, package verification, and git diff --check pass from a cleanly identified workspace state. Visual behavior removed from source-text tests is either exercised in a browser or explicitly recorded as an unautomated verification gap.

Non-goals

• Rewriting all 95 test files in one change.
• Banning regular expressions in parser, validation, diagnostic, CLI-output, security, or intentionally stable serialization tests.
• Adding coverage percentage gates before a trustworthy behavioral baseline exists.
• Changing production behavior, report schemas, user-visible copy, package contents, or supported host declarations as part of the cleanup.
• Treating a green full-suite run as proof that the resulting tests are useful.

Plan and Tasks

1. Remove the largest implementation-text locks first: the shipped Canvas template test and the Harness Inspector omnibus HTML test. Preserve their executable validator, report-model, serialization, privacy, and script syntax checks.
2. Reduce the Better Harness Skill, model, style-template, and documentation suites to structural contracts and high-risk exclusions. Let the existing documentation graph and package verification own link and artifact reachability.
3. Replace source-token architecture checks with runtime exports, behavior, or one parser-backed dependency rule where the architecture itself is public.
4. Keep the suite organized around unit contracts, filesystem/CLI integration, artifact validation, documentation/package integrity, and browser/evaluation evidence. Introduce separate commands only when the file ownership boundary is clear enough that the command cannot silently omit tests.
5. After each slice, compare test count, assertion mix, changed behavior seams, and known verification gaps. Do not use assertion deletion alone as the success metric.

Test and Review Evidence

• AC-1: node --test test/harness-canvas-validation.test.mjs
• AC-2: node --test test/harness-inspector.test.mjs
• AC-3/AC-4: focused tests for every edited Skill, docs, template, model, or architecture file plus node --test test/doc-link-graph.test.mjs
• AC-5: focused security/schema/package tests remain unchanged unless replaced by an equal or stronger oracle.
• AC-6: npm test, npm run pack:verify, and git diff --check after focused tests. Browser-visible changes require preview health/module smoke checks, console inspection, and a saved screenshot under the repository's existing visual verification workflow.
• Review risk: removing phrase locks can reveal that some prompt or visual behavior has no executable oracle. Record those gaps in review evidence and add a scenario evaluation or browser check rather than recreating the same source-text assertion under another helper name.

Implementation Evidence

• Static workspace census changed from 52,298 to 50,913 test lines and from 2,020 to 1,066 assert.match / assert.doesNotMatch calls. This count includes a concurrent 13-line cleanup in test/read-only-command.test.mjs outside this implementation slice. Test files remain 95; security, schema, parser, filesystem, CLI, packaging, and privacy suites were not removed.
• Focused changed-area run: 62 passed, 0 failed. The docs runtime test was then rerun after warning isolation: 4 passed, 0 failed.
• node --test test/doc-link-graph.test.mjs: passed as part of the focused run; node scripts/doc-link-graph/cli.mjs skills/better-harness regenerated the same 39-file, 56-link graph.
• npm test -- --test-reporter=dot: exit 0.
• npm run pack:verify: passed with 504 npm entries and 526 runtime ZIP entries.
• git diff --check and node --check for every changed test file: passed.
• Known gap: Canvas composition, Inspector interactions, and LLM prompt quality are no longer represented as proven behavior merely because source tokens are present. This change does not alter production visuals or prompts. A future change to those behaviors still requires the repository's browser evidence or a dedicated scenario evaluation.
• Follow-up hardcode cleanup removed 147 assert.match / assert.doesNotMatch calls across six suites. It deleted private TSX identifier and style checks, packaging-script source inspection, a duplicate template spelling test, a private-function source slice, catalog ownership inferred from source tokens, and an exact HTML-markup test. Focused verification passed 191/191; npm test, npm run pack:verify (504 npm entries and 526 runtime ZIP entries), and git diff --check passed. Public package metadata, Canvas validators, CLI integration, catalog data, privacy, schema, and artifact checks remain.
• Follow-up gap: future-version Canvas sidecar merge behavior no longer has a focused unit test because its only seam was a source slice bounded by private function names. Restore that coverage only after the merge owner is exposed as a directly importable production function.