StaleBench: Isolating Stale Retrieval in Agent Memory, and Pricing the Fix
Organizations: Causal Dynamics Lab
Abstract
An agent that stores facts over time eventually stores several versions of the same fact. Asked a value-agnostic question - "which team does X play for now?" - a similarity retriever cannot prefer the current version, because every version matches the entity the query names. StaleBench isolates this stale-retrieval failure on 892 real updating facts, with currency labels from the source (Wikidata validity years via TempLAMA), never from a retrieval score. Our central result is that retrieval metrics can rank memory policies backwards: a retention policy and its retain-everything control return the identical statement at rank 1 for every test query, yet differ by 29.5 and 65.6 points in reader accuracy on two readers when no date is shown. Contamination@5, the metric we add, ranks them correctly where current@1 and current@5 do not. Withdrawing the lexical anchor that statements carry costs deployed keying mechanisms tens of points of freshness; entity resolution on both the write path and the query recovers most of the loss. Three deployed systems (Mem0, Graphiti, LangMem) and a reimplemented ERASE each handle supersession differently, and no single scoring convention describes all four. We release the data, a frozen evaluator, and reference baselines.
Figures & tables
| term | meaning |
|---|---|
| stale vs. miss | top-1 names the right fact with a superseded value, vs. the wrong fact |
| contamination@ | the top- holds both a current and a stale version of the queried fact |
| hedged | a context or answer names both the current and a superseded value |
| anchor | the subject’s full name on a statement; withdrawing it ( ) leaves a short form |
| keyer | decides whether an incoming statement describes a fact already stored |
| mechanism | what happens then: overwrite the slot, or invalidate and keep it |
| policy | current@1 | current@5 | stale@1 | miss@1 | contam@5 | slots |
|---|---|---|---|---|---|---|
| BM25 (store-all) | [0.355,0.444] | |||||
| BM25 + time-rerank | [0.912,0.957] | |||||
| Cosine (store-all) | [0.419,0.506] | |||||
| Cosine + time-rerank | [0.900,0.947] | |||||
| Cosine + recency-weight | [0.746,0.821] | |||||
| Reconsolidation ( ) | [0.827,0.889] |
| policy | keying deficit at | |||||
|---|---|---|---|---|---|---|
| BM25 (store-all) | — | |||||
| BM25 + time-rerank | — | |||||
| Cosine (store-all) | — | |||||
| + time-rerank | — | |||||
| + recency-weight | — | |||||
| Overwrite: threshold |
| retrieval | reader accuracy (no dates) | dates | ||||
|---|---|---|---|---|---|---|
| policy | current@1 | cont.@5 | qwen2.5:14b | glm-4.6 | qwen2.5:14b | glm-4.6 |
| Cosine (store-all) | ||||||
| + time-rerank | ||||||
| Overwrite: threshold | ||||||
| Overwrite: gazetteer | ||||||
| Read-path key (no overwrite) | ||||||
| anchored | anchor-free | |||||||
| system | cur@1 | by text | cont.@5 | reader | cur@1 | by text | cont.@5 | reader |
| Mem0 2.0.11 | ||||||||
| Mem0 0.1.118 | ||||||||
| Graphiti, as returned | ||||||||
| Graphiti, valid edges first | ||||||||
| LangMem, qwen2.5:14b | ||||||||
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
| line of work | asks | does not model |
|---|---|---|
| TempLAMA; FreshQA; RealTime QA; StreamingQA; DyKnow; GrowOVER ( Dhingra et al., 2022 ; Vu et al., 2024 ; Kasai et al., 2023 ; Liška et al., 2022 ; Mousavi et al., 2024 ; Ko et al., 2024 ) | is the model/corpus current with the world? | an agent-owned store; the retention decision; budgets |
| knowledge editing ( Meng et al., 2022 ; Meng et al., 2023 ) | can we rewrite weights ? | the store entirely |
| knowledge conflict ( Chen et al., 2022 ; Jin et al., 2024 ; Wu et al., 2024 ; Xu et al., 2024 ) | how does a reader behave given conflicting context? | which version reaches the context, and at what memory cost |
| entity linking, coreference, record linkage ( Wu et al., 2020 ; Joshi et al., 2019 ; Zaporojets et al., 2022 ; Fellegi and Sunter, 1969 ; Li et al., 2020 ) | are these two mentions/records the same entity? | the price of getting it wrong in a downstream retention decision |
| LoCoMo; LongMemEval ( Maharana et al., 2024 ; Wu et al., 2025 ) | broad long-term memory ability | staleness in isolation (multi-session gold dilutes it) |
| memory systems ( Packer et al., 2023 ; Chhikara et al., 2025 ; Rasmussen et al., 2025 ; Yan et al., 2026 ; Li et al., 2025 ; Jiménez Gutiérrez et al., 2024 ) | ship freshness mechanisms | an isolating measurement; oracle-priced keying |
| policy ( ) | cur@1 | stale@1 | miss@1 | cur@5 | cont.@5 | slots |
|---|---|---|---|---|---|---|
| BM25 (store-all) | ||||||
| BM25 + time-rerank | ||||||
| Cosine (store-all) | ||||||
| + time-rerank | ||||||
| + recency-weight | ||||||
| Overwrite: threshold |
| policy | clean | clock permuted | arrival permuted | both |
|---|---|---|---|---|
| Cosine (store-all) | ||||
| + time-rerank | ||||
| + recency-wt | ||||
| Overwrite: threshold | ||||
| Overwrite: gazetteer | ||||
| Invalidate: gazetteer |
| policy | current@1 | cont.@5 | current@1 | cont.@5 | current@1 | cont.@5 |
|---|---|---|---|---|---|---|
| Cosine (store-all) | [0.402,0.491] | [0.419,0.506] | [0.423,0.513] | |||
| Cosine + time-rerank | [0.940,0.976] | [0.900,0.947] | [0.921,0.964] | |||
| Reconsolidation | [0.857,0.915] | [0.827,0.889] | [0.795,0.863] | |||
| Keep-latest oracle | [0.994,1.000] | [0.966,0.991] | [0.953,0.985] | |||
| anchored ( ) | anchor-free ( ) | |||
|---|---|---|---|---|
| keyer | merge err. | % split | merge err. | % split |
| Threshold | ||||
| Gazetteer (exact) | ||||
| Fuzzy linker (no alias table) | ||||
| Linker + alias table | ||||
| True-key oracle | ||||
| overwrite (destructive) | temporal KG (invalidate) | |||||
|---|---|---|---|---|---|---|
| keyer | current@1 | cont.@5 | slots | current@1 | cont.@5 | slots |
| anchored, threshold | ||||||
| anchored, entity linker | ||||||
| anchored, true-key oracle | ||||||
| anchor-free, threshold | ||||||
| anchor-free, entity linker | ||||||
| anchored | anchor-free | |||
|---|---|---|---|---|
| policy | open-world | closed-world | open-world | closed-world |
| Cosine (store-all) | ||||
| + time-rerank | ||||
| Overwrite: threshold | ||||
| Overwrite: gazetteer | ||||
| Overwrite: fuzzy linker | ||||
| mechanical | natural | natural+fallback | ||||
|---|---|---|---|---|---|---|
| policy | current@1 | deficit | current@1 | deficit | current@1 | deficit |
| Overwrite: threshold | ||||||
| Overwrite: gazetteer | ||||||
| Overwrite: fuzzy linker | ||||||
| Read-path key (retain all) | ||||||
| Symmetric key | ||||||
| current@1 | % facts split | |||
|---|---|---|---|---|
| keyer | anchored | anchor-free | anchored | anchor-free |
| threshold | ||||
| gazetteer | ||||
| linker + alias table | ||||
| true-key oracle | ||||
| LLM keyer (glm-4.6) | ||||
| policy / system | version | cur@1 (anch.) | cur@1 (anch.-free) | cur@5 | cont.@5 | slots | ADD / UPDATE |
|---|---|---|---|---|---|---|---|
| Cosine (store-all) | --- | — | |||||
| + time-rerank | --- | — | |||||
| Overwrite: threshold | --- | — | |||||
| Overwrite: gazetteer | --- | — | |||||
| Overwrite: fuzzy linker | --- | — | |||||
| Symmetric key (read+write) | --- | — |
| mem0 release | vector store | writes | retrieval | returned | usable |
|---|---|---|---|---|---|
| 0.1.118 | chroma | ✓ | ✓ | ✓ | |
| 0.1.118 | faiss | ✓ | |||
| 0.1.118 | qdrant | ✓ | |||
| 2.0.11 | chroma | ✓ | ✓ | ✓ | |
| 2.0.11 | faiss | ✓ | ✓ | ✓ | |
| 2.0.11 | qdrant | ✓ | ✓ | ✓ |
| context source | as retrieved | query resolved | merged, freshest | oldest | |||
|---|---|---|---|---|---|---|---|
| cur@1 | reader | cur@1 | reader | cur@1 | reader | reader | |
| cosine-rag | |||||||
| time-rerank | |||||||
| reconsolidation | |||||||
| lexical-key | |||||||
| fuzzy-key | |||||||
| anchored | anchor-free | |||||||
|---|---|---|---|---|---|---|---|---|
| policy | cur@1 | by text | cont.@5 | reader | cur@1 | by text | cont.@5 | reader |
| Graphiti, as returned | ||||||||
| Graphiti, valid edges first | ||||||||
| Graphiti, valid edges only | — | — | ||||||
| Cosine (store-all) | ||||||||
| + time-rerank | ||||||||
| anchored | anchor-free | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| system, model, prompt | cur@1 | text | cont.@5 | slots | ops | cur@1 | text | cont.@5 | slots | ops |
| Mem0, qwen, default | 149+31 | 114+21 | ||||||||
| Mem0, glm, default | 2+1 | 7+1 | ||||||||
| Mem0, qwen, third-person | 155+23 | 110+12 | ||||||||
| Mem0, glm, third-person | 142+19 | 113+23 | ||||||||
| Graphiti, qwen, as returned | 157 | 137 | ||||||||
| anchored | anchor-free | |||||||
|---|---|---|---|---|---|---|---|---|
| policy | cur@1 | by text | cont.@5 (text) | reader | cur@1 | by text | cont.@5 (text) | reader |
| LangMem, qwen2.5:14b | ||||||||
| LangMem, glm-4.6 | ||||||||
| Cosine (store-all) | ||||||||
| + time-rerank | ||||||||
| Symmetric key | ||||||||
| anchored | anchor-free | |||||||
|---|---|---|---|---|---|---|---|---|
| policy | cur@1 | by text | cont.@5 (text) | reader | cur@1 | by text | cont.@5 (text) | reader |
| ERASE, qwen2.5:14b | ||||||||
| ERASE, glm-4.6 | ||||||||
| Cosine (store-all) | ||||||||
| + time-rerank | ||||||||
| Symmetric key | ||||||||
| no dates | dates | ||||
|---|---|---|---|---|---|
| constructed context | qwen2.5:14b | glm-4.6 | qwen2.5:14b | glm-4.6 | |
| no context (closed book) | |||||
| current only | |||||
| +1 stale, current first | |||||
| +1 stale, current last | |||||
| +2 stale, current middle | |||||
| anchored | anchor-free | |||||
|---|---|---|---|---|---|---|
| policy | current@1 | cont.@5 | set rec.@5 | current@1 | cont.@5 | set rec.@5 |
| Cosine (store-all) | ||||||
| + time-rerank | ||||||
| + recency-wt | ||||||
| Overwrite: threshold | ||||||
| Overwrite: gazetteer | ||||||
| value_gap | key_jitter | ||||||
|---|---|---|---|---|---|---|---|
| policy | |||||||
| Cosine (store-all) | 0.05 | 0.06 | 0.05 | 0.06 | 0.06 | 0.06 | 0.05 |
| + time-rerank | 0.03 | 0.04 | 0.04 | 0.03 | 0.04 | 0.06 | 0.07 |
| + recency-weight | 0.01 | 0.02 | 0.02 | 0.01 | 0.04 | 0.05 | 0.06 |
| Reconsolidation | 0.00 | 0.00 | 0.00 | 0.00 | 0.05 | 0.05 | 0.06 |
| Oracle | 0.00 | 0.00 | 0.00 | 0.00 | 0.01 | 0.05 | 0.05 |
| anchored ( ) | anchor-free ( ) | |||||
|---|---|---|---|---|---|---|
| policy | ||||||
| Cosine (store-all) | ||||||
| + time-rerank | ||||||
| Overwrite: threshold | ||||||
| Overwrite: gazetteer | ||||||
| Overwrite: fuzzy linker | ||||||
Agent Memory
Agent memory lets AI systems retain, update, and use facts, experiences, and skills across sessions and long-horizon tasks.
See all 208 papers