Memory-augmented LLM agents store and retrieve information from prior interactions, yet the relative importance of how memories are written versus how they are retrieved remains unclear. We introduce a diagnostic framework that analyzes how performance differences manifest across write strategies, retrieval methods, and memory utilization behavior, and apply it to a 3x3 study crossing three write strategies (raw chunks, Mem0-style fact extraction, MemGPT-style summarization) with three retrieval methods (cosine, BM25, hybrid reranking). On LoCoMo, retrieval method is the dominant factor: average accuracy spans 20 points across retrieval methods (57.1% to 77.2%) but only 3-8 points across write strategies. Raw chunked storage, which requires zero LLM calls, matches or outperforms expensive lossy alternatives, suggesting that current memory pipelines may discard useful context that downstream retrieval mechanisms fail to compensate for. Failure analysis shows that performance breakdowns most often manifest at the retrieval stage rather than at utilization. We argue that, under current retrieval practices, improving retrieval quality yields larger gains than increasing write-time sophistication. Code is publicly available at https://github.com/boqiny/memory-probe.
Figures & tables
Figure 1
Token F1
Accuracy (%)
Cos
BM25
Hyb
Cos
BM25
Hyb
Basic RAG
0.232
0.184
0.240
77.9
59.2
81.1
Extracted Facts
0.211
0.146
0.220
72.2
49.4
77.3
Summ. Episodes
0.197
0.173
0.202
70.1
62.7
73.3
Avg
0.213
0.168
0.221
73.4
57.1
77.2
Table 1: Token F1 and LLM-as-Judge accuracy across all nine configurations ( k=5 , 1,540 non-adversarial questions). Accuracy spread across retrieval methods (14–23 pts) consistently exceeds spread across write strategies (3–8 pts). Bold = best write strategy per retrieval method.
Figure 3
Probe 1
Probe 2: Utilization
Probe 3: Failure Mode
Precision
Beneficial
Ignored
Retrieval
Utilization
Halluc.
Basic RAG
Cosine
25.5
75.7
9.2
15.8
5.4
1.0
BM25
14.8
56.5
24.8
35.3
5.1
0.4
Hybrid
29.4
79.0
6.6
11.4
6.2
1.2
Ext. Facts
Cosine
21.2
70.4
9.7
21.2
5.9
0.6
BM25
10.6
46.2
29.2
46.3
3.9
0.5
Table 2: Diagnostic probes across all nine configurations (%). Probe 1: retrieval precision. Probe 2: memory utilization. Probe 3: failure modes as percentage of all questions (remainder = correct). See Section 3.3.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Component
Mimics / Mechanism
LLM Calls
Write
Basic RAG
Store raw 3-turn chunks
None
Extracted Facts
Mem0-style fact extraction + conflict resolution
1+/session
Summ. Episodes
MemGPT-style session summarization
1/session
Retrieval
Cosine Similarity
Embedding similarity
None
BM25
Keyword matching
None
Hybrid+Rerank
Cosine ∪ BM25, LLM rerank to top- k
1/query
Appendix
Table 3: Experimental design. We cross three write strategies (what is stored) with three retrieval methods (how it is retrieved), yielding a 3×3 evaluation grid.
Human: Correct
Human: Incorrect
Total
LLM: Correct
129
9
138
LLM: Incorrect
7
55
62
Total
136
64
200
Appendix
Table 4: Confusion matrix: LLM judge vs. human annotation for answer correctness (200 sampled questions, stratified by LLM judgment). Cohen’s κ=0.82 indicates substantial agreement.
Human \ LLM
Retrieval
Utilization
Hallucination
Total
Retrieval Failure
147
7
0
154
Utilization Failure
16
23
0
39
Hallucination
0
1
6
7
Total
163
31
6
200
Appendix
Table 5: Confusion matrix: LLM judge vs. human annotation for failure mode classification