7. Quality Measurement

Self-Evaluation Questions

Run these questions against your CLAUDE.md periodically (quarterly at minimum):

Relevance:

• Does this rule still apply to the current stack, libraries, and team practices?
• Was this rule written for a problem that no longer exists?
• Would a new team member understand why this rule exists?

Specificity:

• Is this rule specific enough for Claude to know when it applies?
• Does this rule have at least one concrete example or counter-example?
• Could two developers interpret this rule differently?

Conflicts:

• Does this rule contradict another rule in the same file?
• Does this rule contradict a rule in a path-scoped module?
• Does this rule contradict a global rule without explicitly overriding it?

Coverage:

• Is this rule a specific case of a more general rule that already exists?
• Is this rule already implied by the architecture decisions stated elsewhere?

A rule that fails more than one of these checks is a candidate for update or removal.

Canary Checks

Canary checks are simple test prompts that verify Claude follows key conventions. Run them before and after major changes to CLAUDE.md to catch regressions.

Structure: 3-5 prompts that are simple enough to answer quickly, but specific enough to reveal adherence failures.

Example canary set for a React/TypeScript project:

# scripts/run-canaries.sh

PASS=0
FAIL=0

check() {
  local name="$1"
  local prompt="$2"
  local expected_pattern="$3"

  result=$(claude "$prompt" --output-format text 2>/dev/null)

  if echo "$result" | grep -qE "$expected_pattern"; then
    echo "PASS: $name"
    PASS=$((PASS + 1))
  else
    echo "FAIL: $name"
    echo "  Expected pattern: $expected_pattern"
    echo "  Got: $(echo "$result" | head -5)"
    FAIL=$((FAIL + 1))
  fi
}

check "TypeScript interfaces" \
  "Generate a React component that accepts a name and age prop" \
  "interface.*Props"

check "Named exports" \
  "Create a utility function that formats a date" \
  "^export (function|const)"

check "No any type" \
  "Write a function that processes user data" \
  "^((?!: any).)*$"

check "Error result type" \
  "Write a service function that fetches user data from an API" \
  "Result<"

echo ""
echo "Canaries: $PASS passed, $FAIL failed"
[[ $FAIL -eq 0 ]]

When to run canaries:

• Before merging changes to CLAUDE.md
• After adding a new path-scoped module
• When a team member reports unexpected Claude behavior
• As part of the CI drift detection job

Adherence Tracking

Informal but effective: for each key rule in CLAUDE.md, track how often Claude violates it across 10 consecutive interactions where the rule should apply.

Rule	Violations / 10	Status
TypeScript interfaces for props	1/10	Healthy
Result type for service functions	0/10	Healthy
No raw database IDs in API responses	3/10	Review rule
Structured logging with request context	5/10	Rule too vague
OWASP Top 10 check before shipping	8/10	Not actionable as stated

Rules with >20% violation rate are broken in one of three ways:

1. Too vague to apply consistently
2. Conflicting with another rule
3. Placed too late in the file to receive enough attention

Fix for "too vague": Add a concrete example of compliance and a counter-example of violation.

Fix for "conflicting": Find the conflict, decide which rule should win, update or remove the losing rule, and add an explicit note.

Fix for "placed too late": Move the rule to the top third of the file, or to a more prominent position in its section.

Context Debt Score

A single metric for the health of your context engineering system:

Context Debt Score = (total_rules / 150) × (conflicts_found / total_rules) × 100

Where:

• total_rules = count of distinct rules across all loaded config files
• 150 = the approximate attention ceiling
• conflicts_found = rules that contradict another rule

Score Range	Status	Action
< 30	Healthy	Standard quarterly audit
30 – 60	Degraded	Prune and deduplicate; fix conflicts
60 – 80	Poor	Major restructure needed
> 80	Critical	Start from scratch with top 30 rules

Running the score calculation:

# Count rules (approximate: lines starting with -)
TOTAL_RULES=$(grep -c "^- " CLAUDE.md 2>/dev/null || echo 0)

# Count conflicts requires manual review or an LLM audit pass
# Use: claude "Scan CLAUDE.md and count rules that contradict each other. Return the count."

echo "Total rules: $TOTAL_RULES"
echo "Run conflict audit manually or with Claude"

Context Drift Detection

The existing adherence metrics (canary checks, violation rates) require human interpretation: you know a rule is being violated when you notice it. Systematic drift detection is a complementary layer that detects behavioral shifts automatically, before they surface as bad outputs.

These methods come from ML observability. They are more relevant for teams running Claude in automated pipelines than for interactive use, but the concepts apply in both contexts.

Cosine distance method

The simplest production-ready approach. Embed model outputs (responses to fixed probe prompts) and measure cosine distance from a known-good baseline embedding.

1. Define 5-10 fixed probe prompts that test key conventions (equivalent to canary prompts).
2. At a stable point ("golden baseline"), capture outputs and compute their embeddings.
3. On each subsequent run, compute outputs for the same prompts and measure cosine distance from baseline.
4. Alert when average distance exceeds a threshold (typically 0.15-0.20 for sentence-level embeddings).

What this catches: gradual style drift, convention erosion, changes in output structure, all before violation rates increase.

Share of drifted features

More granular than cosine distance. Instead of a single distance metric, track which specific embedding dimensions have shifted beyond a threshold. This tells you which aspects of the output have changed (length, formality, code style) rather than just that something changed.

Practical implementation requires an embedding model and a monitoring store. Start with cosine distance; add feature-level tracking only if you need to diagnose what is drifting.

Maximum Mean Discrepancy (MMD)

A kernel-based method for comparing two distributions of outputs. MMD answers: "Are the outputs from this period statistically different from the baseline period?" It handles high-dimensional embeddings robustly and does not require specifying which features to track.

MMD is more setup cost than cosine distance but produces fewer false positives when output variance is naturally high. Relevant for teams with significant output volume (hundreds of Claude runs per day).

Statistical distance thresholds

Regardless of method, thresholds matter:

Distance metric	Alert threshold	Note
Cosine distance	> 0.15	Works for most sentence embeddings
Euclidean distance	Varies by dimensionality	Normalize embeddings first
Manhattan distance	Varies by dimensionality	More robust to outliers than Euclidean

These are starting points. Calibrate against your baseline variance: if your outputs naturally vary widely (creative tasks), use a looser threshold.

When to use drift detection

• Automated pipelines where human review is not per-output
• After CLAUDE.md changes, to verify behavior stayed stable
• When upgrading Claude model versions (behavior shifts between versions)
• Regression detection after any context configuration change

For interactive development with regular human review, canary checks and violation rate tracking (already above) are sufficient.

Useful Metrics to Track Over Time

Metric	How to Measure	Target
Always-on context size	wc -w CLAUDE.md ~/.claude/CLAUDE.md	< 5,000 words
Rule count	grep -c "^- " CLAUDE.md	< 150
File age	`git log --follow -p CLAUDE.md	head -20`
Violation rate per key rule	Manual spot checks	< 20% violation
Canary pass rate	./scripts/run-canaries.sh	100% (all pass)