9. Benchmarks and Evaluation
Academic Evaluation Frameworks
| Benchmark | What it measures |
|---|---|
| LongMemEval | Retention over time, graceful degradation when capacity is constrained |
| LoCoMo | Coherence across very long conversation histories (thousands of messages) |
| BEAM | Efficiency: performance vs. computational overhead tradeoff |
| MemoryAgentBench | Multi-turn: retention, recall, contextual adaptation, reasoning trace fidelity |
Numbers reported by tool vendors against these benchmarks should be read skeptically. Evaluation methodology varies. Ask for the corpus, adapter code, and reproduction steps.
Supermemory illustrates this concretely: its own site publishes two different Recall@15 figures for the same benchmark (LongMemEval_s), 95% on its research page and 85.4% on a blog post comparing itself to Pinecone. Both are self-published, and neither has been reproduced by a third party.
Representative Results (May 2026)
| System | R@5 (LongMemEval-S) | Source | Reproducible? |
|---|---|---|---|
| agentmemory | 95.2% | Tool author, own harness | Yes (eval/README.md) |
| BM25-only baseline | 86.2% | agentmemory harness | Yes |
| Letta | 83.2% | agentmemory harness | Partial |
| mem0 | 68.5% | agentmemory harness | Partial |
| MemPalace | 96.6% | Vendor | [UNVERIFIED] |
| mem0 (vendor-claimed) | >94% | Vendor | [APPROXIMATE] |
SimpleMem (arXiv:2601.02553) reports 43.24% F1 on LoCoMo with 30x fewer tokens. EvolveMem (arXiv:2605.13941) reports 54.3% F1 on LoCoMo via an evaluate-diagnose-propose-guard loop. Both IDs correspond to 2026; treat as recent preprints.
ReasoningBank (Google, arXiv 2509.25140) is the academic anchor for the success-and-failure memory the file-based playbook track (§3.7) implements. It distills generalizable strategies from both successful and failed trajectories, then retrieves a few relevant items at test time via embedding search. The paper is the credible source ORF cites; the practical divergence (ORF and this guide's retex drop the embeddings for lexical or LLM-read retrieval) is a deployment tradeoff, not a disagreement with the paper.
Memory Decay Models
Three documented approaches with different theoretical grounding:
ICM's importance-weighted decay maps to Anderson's ACT-R base-level activation formula: B_i = ln(sum(t_j^\{-d\})) where t_j is time since each retrieval and d is a decay constant (typically 0.5). ICM's "frequently recalled memories decay more slowly" is this formula simplified.
Auto Memory's 200-line hard limit is a UX heuristic, not a memory model. Works for solo use; fails when a load-bearing memory was written three months ago.
Zep's temporal edge windows implement the bitemporal model (Snodgrass, 1999). Right answer when point-in-time queries are a genuine requirement. Overhead is real: every retrieval becomes a temporal range scan.
The production-correct model combines all three: importance score at write time, exponential recency decay at read time, reinforcement on retrieval, and consolidation triggered by token budget rather than a fixed entry count. The MemGPT/Letta paging model (hot memories in context, warm in working store, cold in archive) is the most engineering-useful framing.