Evaluation: Caliber (rely-ai-org/caliber)
Date: 2026-03-23 Evaluator: Claude (Sonnet 4.6) Status: ✅ Integrated — Configuration Quality section
📄 Résumé du contenu
- Caliber is a CLI tool that scores, generates, and continuously syncs AI agent configs (CLAUDE.md, MCP servers) with a codebase. Zero-install:
npx @rely-ai/caliber score - Scoring is 100% local and deterministic: 6 categories (Existence 25 + Quality 25 + Grounding 20 + Accuracy 15 + Freshness 10 + Bonus 7), 61 checks, no LLM calls, no network requests
- Drift detection is git-based: SHA256(file tree + dirty files) + git HEAD pointer cache, supports
caliber refreshfor automated doc updates after code changes - Learner module captures tool events via hooks → JSONL → incremental LLM analysis →
CALIBER_LEARNINGS.mdwith 70% similarity deduplication - GitHub: 65 stars (now 1,223 as of 2026-07-28), 12 forks, created 2026-03-10 (13 days old), MIT, TypeScript/Node.js ≥20
🎯 Score de pertinence (1-5)
| Score | Signification |
|---|---|
| 5 | Essentiel - Gap majeur dans le guide |
| 4 | Très pertinent - Amélioration significative |
| 3 | Pertinent - Complément utile |
| 2 | Marginal - Info secondaire |
| 1 | Hors scope - Non pertinent |
Score: 3/5
Justification: The guide already covers three config tools (claude-code-config, AIBlueprint, Packmind), but none solve the same problem as Caliber. The existing tools create or distribute config; Caliber scores existing config quality and detects when it drifts from the actual codebase. The drift detection feature (git-based, automatic) is genuinely absent from the guide's current ecosystem coverage. Score stays at 3 rather than 4 because the tool is 13 days old at time of evaluation — not enough community validation to claim "high value."
⚖️ Comparatif
| Aspect | Caliber | Guide actuel |
|---|---|---|
| Config quality scoring (0-100) | ✅ 61 checks, local, deterministic | ❌ Not covered |
| Config drift detection (git-based) | ✅ SHA256 tree + git HEAD cache | ❌ Not covered |
| CLAUDE.md generation from codebase | ✅ Codebase fingerprinting + LLM | ✅ AIBlueprint (scaffold only, no drift) |
| MCP server auto-suggestion | ✅ Based on detected deps | ❌ Not covered |
Continuous sync (caliber watch) | ✅ Loop: score → propose → review | ❌ Not covered |
| Org-scale distribution | ❌ Single-project only | ✅ Packmind |
| Machine-to-machine config sync | ❌ Does not solve this | ❌ Still a guide gap |
| GitHub Action / CI integration | ✅ PR scoring with delta + fail-below | ❌ Not covered |
| Backup + undo | ✅ .caliber/backups/ + manifest | ❌ Not covered |
| Multi-tool (Cursor, Codex, Copilot) | ✅ 4 tools in one | ✅ Mentioned separately |
🔬 Analyse technique (post-audit complet du code source)
Full source code audit conducted on a clone of Caliber. Key findings:
Scoring system
6 categories, 61 checks, ~102 points normalized to 100:
| Category | Max | Method |
|---|---|---|
| Existence | 25 | existsSync() checks on CLAUDE.md, skills, MCP, rules |
| Quality | 25 | Content parsing: code blocks, token budget, concreteness ratio, duplicate detection |
| Grounding | 20 | % of project dirs/files mentioned in config (2-level scan) |
| Accuracy | 15 | Path validation + git commits since last config update |
| Freshness | 10 | Git-based staleness + secret detection (13 regex patterns) |
| Bonus | 7 | Hooks, AGENTS.md, learned content, external sources |
Scoring is hybrid: file presence (40%), content analysis (50%), git heuristics (10%). All logic in src/scoring/checks/*.ts. Extractable as a standalone library.
Drift detection
Three independent signals:
- Tree signature:
SHA256(sorted file tree + git dirty files)— any add/remove/rename triggers invalidation - Git HEAD pointer: cache valid only if on same commit
- Git diff:
git diff <lastSha>..HEAD(code only, excluding CLAUDE.md/skills from drift consideration)
Cache stored at .caliber/cache/fingerprint.json. Stale if version mismatch, HEAD change, or signature mismatch.
Learner module
Events captured via hooks into .caliber/learning/session.jsonl → incremental LLM analysis every 50 events → CALIBER_LEARNINGS.md (bullet format: **[pattern|gotcha|fix|correction]** text). Deduplication at 70% normalized text similarity. ROI tracking compares failure rates with vs without learnings across sessions.
GitHub Action
github-action/action.yml + index.js (266 lines total). Inputs: agent, fail-below, comment, auto-refresh. Posts PR comment with score, grade, delta vs base branch. Fails CI if score < fail-below. Deterministic — no LLM calls in the Action itself.
Security note
Caliber has write access to CLAUDE.md in continuous sync mode (caliber watch, caliber refresh). Same risk class as Packmind: a compromised config generation or a malicious source can propagate instructions to AI sessions. Treat caliber init output with the same review discipline as a Packmind playbook update.
🔥 Challenge (technical-writer)
Score 3/5 confirmed. Key points raised:
- Scoring rubric opacity: The 100-point scale is deterministic but undocumented from a user-facing standpoint (no SCORING.md). A score of 74 tells you less than it should without knowing the rubric.
- Multi-tool depth concern: Supporting 4 tools (Claude Code + Cursor + Codex + Copilot) risks producing generically adequate configs rather than deeply Claude Code-specific ones.
- Write access risk underplayed in README: The security surface of
caliber watch(continuous writes to CLAUDE.md) deserves explicit mention. - Placement in guide: Should NOT sit alongside AIBlueprint (scaffolding) — different operational category. Belongs in its own "Configuration Quality" subsection, bridging "one-project drift" (Caliber) and "org-scale distribution" (Packmind).
- Risk of not integrating: Low to moderate. The drift detection gap is real but not critical for most users starting from scratch.
✅ Fact-Check
| Affirmation | Vérifiée | Source |
|---|---|---|
| Scoring 100% local, no LLM calls | ✅ | src/scoring/ — no HTTP calls, pure filesystem + git |
| 61 checks across 6 categories | ✅ | src/scoring/checks/ — counted directly |
Zero-install npx @rely-ai/caliber score | ✅ | README + package.json bin entry |
| MIT license | ✅ | LICENSE file + gh repo view |
| 65 stars, 12 forks, created 2026-03-10 | ✅ | gh repo view rely-ai-org/caliber --json |
Backup to .caliber/backups/ | ✅ | src/writers/backup.ts |
| Before/After 35→94 score example | ⚠️ | Marketing example, not independently verifiable |
| Node.js ≥20 required | ✅ | package.json engines field |
No hallucinated stats. The 35→94 progression is a marketing illustration, not a measured average.
📍 Décision et intégration
Score final: 3/5Action: IntegratedConfiance: Haute sur les faits techniques, réservée sur la pérennité (outil de 13 jours)
Placement: New ### Caliber entry under ## Configuration Management in guide/ecosystem/third-party-tools.md, in its own "Configuration Quality" subsection above "Engineering Standards Distribution". Explicit early-stage language, security note, and cross-ref to AIBlueprint (scaffolding) and Packmind (org-scale).
Gaps table update: Do NOT update the "Cross-platform config sync" row — Caliber solves config-to-codebase drift, not machine-to-machine sync. These are distinct problems.
Re-evaluation trigger: 200 stars or v1.0 release — whichever comes first.