Memory-augmented LLM agents store and retrieve information from prior interactions, yet the relative importance of how memories are written versus how they are retrieved remains unclear. We introduce a diagnostic framework that analyzes how performance differences manifest across write strategies, retrieval methods, and memory utilization behavior, and apply it to a 3x3 study crossing three write strategies (raw chunks, Mem0-style fact extraction, MemGPT-style summarization) with three retrieval methods (cosine, BM25, hybrid reranking). On LoCoMo, retrieval method is the dominant factor: average accuracy spans 20 points across retrieval methods (57.1% to 77.2%) but only 3-8 points across write strategies. Raw chunked storage, which requires zero LLM calls, matches or outperforms expensive lossy alternatives, suggesting that current memory pipelines may discard useful context that downstream retrieval mechanisms fail to compensate for. Failure analysis shows that performance breakdowns most often manifest at the retrieval stage rather than at utilization. We argue that, under current retrieval practices, improving retrieval quality yields larger gains than increasing write-time sophistication. Code is publicly available at https://github.com/boqiny/memory-probe.
Figures & tables
Figure 1
Token F1
Accuracy (%)
Cos
BM25
Hyb
Cos
BM25
Hyb
Basic RAG
0.232
0.184
0.240
77.9
59.2
81.1
Extracted Facts
0.211
0.146
0.220
72.2
49.4
77.3
Summ. Episodes
0.197
0.173
0.202
70.1
62.7
73.3
Avg
0.213
0.168
0.221
73.4
57.1
77.2
Table 1: Token F1 and LLM-as-Judge accuracy across all nine configurations ( k=5 , 1,540 non-adversarial questions). Accuracy spread across retrieval methods (14–23 pts) consistently exceeds spread across write strategies (3–8 pts). Bold = best write strategy per retrieval method.
Figure 3
Probe 1
Probe 2: Utilization
Probe 3: Failure Mode
Precision
Beneficial
Ignored
Retrieval
Utilization
Halluc.
Basic RAG
Cosine
25.5
75.7
9.2
15.8
5.4
1.0
BM25
14.8
56.5
24.8
35.3
5.1
0.4
Hybrid
29.4
79.0
6.6
11.4
6.2
1.2
Ext. Facts
Cosine
21.2
70.4
9.7
21.2
5.9
0.6
BM25
10.6
46.2
29.2
46.3
3.9
0.5
Table 2: Diagnostic probes across all nine configurations (%). Probe 1: retrieval precision. Probe 2: memory utilization. Probe 3: failure modes as percentage of all questions (remainder = correct). See Section 3.3.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Component
Mimics / Mechanism
LLM Calls
Write
Basic RAG
Store raw 3-turn chunks
None
Extracted Facts
Mem0-style fact extraction + conflict resolution
1+/session
Summ. Episodes
MemGPT-style session summarization
1/session
Retrieval
Cosine Similarity
Embedding similarity
None
BM25
Keyword matching
None
Hybrid+Rerank
Cosine ∪ BM25, LLM rerank to top- k
1/query
Appendix
Table 3: Experimental design. We cross three write strategies (what is stored) with three retrieval methods (how it is retrieved), yielding a 3×3 evaluation grid.
Human: Correct
Human: Incorrect
Total
LLM: Correct
129
9
138
LLM: Incorrect
7
55
62
Total
136
64
200
Appendix
Table 4: Confusion matrix: LLM judge vs. human annotation for answer correctness (200 sampled questions, stratified by LLM judgment). Cohen’s κ=0.82 indicates substantial agreement.
Human \ LLM
Retrieval
Utilization
Hallucination
Total
Retrieval Failure
147
7
0
154
Utilization Failure
16
23
0
39
Hallucination
0
1
6
7
Total
163
31
6
200
Appendix
Table 5: Confusion matrix: LLM judge vs. human annotation for failure mode classification
Current LLM agents lack principled mechanisms for managing persistent memory across long interaction horizons. We present a biologically-grounded memory architecture comprising six cognitive mechanisms: (1) sleep-phase consolidation, (2) interference-based forgetting, (3) engram maturation, (4) reconsolidation upon retrieval, (5) entity knowledge graphs, and (6) hybrid multi-cue retrieval. Each mechanism addresses a specific failure mode of naive memory accumulation. We introduce a synthetic calibration methodology that derives all pipeline thresholds without benchmark data exposure, eliminating a common source of evaluation leakage. We evaluate on two benchmarks. First, a VSCode issue-tracking dataset (13K issues, 120K events) where deduplication-based consolidation achieves 97.2% retention precision with 58% store reduction (+21.8 pp over baseline). Second, the LongMemEval personal-chat benchmark where we conduct the first streaming M-tier evaluation (475 sessions, ~540K unique turns). At a 200K-token context budget, our pipeline matches raw retrieval accuracy (70.1% vs. 71.2%, overlapping 95% CI) while exposing a tunable accuracy/store-size operating curve. At S-tier scale (50 sessions), dedup-based consolidation yields a +13.3 pp improvement in preference recall.
Doga Kerestecioglu, Alexei Robsky, Clemens Vasters +2
Long-term memory enables LLM agents to reuse past experiences, yet retrieving relevant memories does not necessarily provide sufficient evidence for answering a query. Essential information may remain fragmented across memory records, obscured by abstraction, or disconnected from the relations required for reasoning. We identify this relevance-sufficiency gap and propose QEMem, a framework that reformulates memory utilization as the construction of query-specific evidence sets, shifting the focus from individual retrieval relevance to collective evidential support. QEMem integrates three complementary mechanisms: query-guided discovery retrieves evidence across multiple memory granularities; source-grounded reification organizes it into structured ViewCards with explicit evidential roles and source references; and sufficiency-guided refinement selectively recovers missing information. Operating on existing memory stores without modifying memory formation, QEMem achieves 91.43% LLM-judge accuracy on LoCoMo with Claude Opus 4.5, outperforming the strongest evaluated baseline, MemR3, by 7.6 percentage points and NEMORI by 10.7 points under a shared memory store. Further analyses demonstrate improved evidence coverage and strong performance on questions requiring cross-session evidence integration.
Shixuan Li, Wei Yang, Peiyu Zhang +7
University of Southern California · Georgia Institute of Technology
Long-term memory enables LLM agents to leverage past interactions, but dialogue histories quickly exceed the context window, forcing agents to retrieve relevant subsets at query time. Because useful evidence is sparse and scattered across verbose conversations, retrieval faces a fundamental tension: broadening recall improves coverage but floods downstream reasoning with noise, while compressing memories at write time eases retrieval but irreversibly discards details that future queries may need. We introduce LazyMem, which resolves this tension by deferring all memory construction to query time. Given a retrieved candidate pool, a lightweight model processes it in overlapping parallel windows, selectively retaining and compressing only query-relevant content. The model is trained with supervised fine-tuning followed by reinforcement learning, using a reward that jointly encourages the identification of relevant messages and the generation of compressions that are faithful to the source and useful for answering the query. On LongMemEval, LazyMem-4B achieves an LLM-judge accuracy of 0.85, outperforming the strongest non-oracle baseline while using only 213 answer-context memory tokens, 21.0 times fewer than the baseline. It further generalizes to LoCoMo without target-domain training and reduces mean latency relative to the prior query-time baseline. Code is available at https://github.com/allacnobug/LazyMem.
Jing Yu, Yibo Zhao, Jiaming Zhang +1
School of Data Science and Engineering, East China Normal University