Long-term conversational memory poses a fundamental system challenge for LLM agents: reasoning over all past interactions improves coverage but incurs prohibitive token costs and noise, while retrieval-based methods are efficient yet often fail on multi-hop and temporal questions. Existing multi-step retrievers partially address this gap, but typically operate over an ever-growing textual history, causing context expansion and noise accumulation across iterations. We propose MemR3, a backend-agnostic closed-loop controller that reformulates conversational memory retrieval as explicit stateful decision-making. At each iteration, MemR3 maintains an evidence-gap state consisting of grounded information (evidence) and unresolved information requirements (gaps), and uses this state to route among three actions: retrieve, reflect, and answer. Newly retrieved snippets are incorporated into the state and then masked from subsequent prompts, so the model conditions on a compact summary of progress rather than the full retrieval history. This design turns multi-step memory retrieval into a bounded, inspectable control process whose query-time context scales with the state instead of raw accumulated text. Experiments on LongMemEvals show that MemR3 surpasses implicit multi-step retrieval baselines and often approaches or exceeds full-context prompting while retrieving only 1%-5% of the full conversation tokens on long conversations. On LoCoMo, MemR3 consistently improves over its underlying RAG- and Zep-based memory backbones. These results suggest that explicit evidence-gap tracking is an effective abstraction for token-efficient conversational memory.
Figures & tables
Figure 1: Illustration of three memory-usage paradigms. In this example, Full-Context exposes the LLM to many distractors, while Retrieve-then-Answer retrieves relevant snippets but still miscalculates. In contrast, MemR 3 iteratively retrieves and reflects using an explicit evidence-gap tracker (Acts 0-3). By absorbing information and discarding raw texts, it refines the query and produces the correct answer (3 months) without unbounded context expansion.
Figure 2: Pipeline of MemR 3 . MemR 3 transforms retrieval into a closed-loop process: a router dynamically switches between Retrieve, Reflect, and Answer nodes while a global evidence-gap tracker maintains what is known and what is still missing. This enables iterative query refinement, targeted retrieval, and early stopping, making MemR 3 an autonomous, backend-agnostic controller.
Figure 3: Results on LongMemEval s . (a) Overall accuracy across six LLM backends. (b) Retrieved-context tokens versus accuracy across six categories. (c) LLM-response latency. Repeated-run statistics and full query-time costs are reported in Appendices E.1 and E.4 .
Backend
RAG
Self-RAG
RCR
MemR 3
Full-Context
GPT-4o-mini
1.53
7.06
2.81
3.87
11.04
GPT-4.1-mini
1.72
10.71
3.10
5.59
7.55
GPT-5-mini
11.51
41.38
15.80
15.26
36.88
Table 2: Average end-to-end latency (seconds) on LongMemEval s .
Method
1. Multi-Hop
2. Temporal
3. Open-Domain
4. Single-Hop
Overall
4o
4.1
4o
4.1
4o
4.1
4o
4.1
4o
4.1
A-Mem ( Xu et al., 2025 )
61.70
71.99
64.49
74.77
40.62
58.33
76.63
79.88
69.06
76.00
LangMem ( LangChain Team, 2025 )
62.23
74.47
23.43
61.06
47.92
67.71
71.12
86.92
58.10
78.05
Mem0 ( Chhikara et al., 2025 )
67.13
62.41
55.51
57.32
51.15
44.79
72.93
66.47
66.88
62.47
Self-RAG ( Asai et al., 2024 )
69.15
75.89
64.80
75.08
34.38
54.17
88.31
90.12
76.46
82.08
RCR
71.28
80.85
71.03
81.62
42.71
62.50
86.99
90.12
77.96
84.89
Table 3: LLM-as-a-Judge Score (%) on LoCoMo. Categories are grouped, with GPT-4o-mini (4o) and GPT-4.1-mini (4.1) shown side-by-side. Δ denotes absolute improvements over backbones. MemR 3 rows are three-run means (Appendix E.1 ); baselines are single runs .
Figure 4: Granularity bias (failed case). The agent recovers the coarse fact “in 2022” and produces the correct answer, but the gap requires the exact date. Because of the granularity mismatch, the controller cannot close the loop and stops only at the iteration limit nmax .
Method
MH *
Temporal
OD *
SH *
Overall
RAG
68.79
65.11
58.33
83.86
75.54
MemR 3
71.39
76.22
61.11
89.44
81.55
w/o mask
62.41
68.54
55.21
72.17
68.54
w/o Δqk
66.67
75.08
60.42
83.37
77.11
w/o reflect
65.25
73.83
61.46
83.37
76.65
Table 4: Ablation studies on LoCoMo.
R
E
Interpretation
Questions
Human accuracy (%)
0
0
Retrieval failure
23
0.0
1
0
State/compression failure
24
4.2
1
1
Evidence preserved
105
90.5
Table 5: Human audit of retrieval and evidence preservation on 152 LoCoMo questions (GPT-4o-mini). R=1 : retrieved snippets contain answer-critical evidence; E=1 : the final state faithfully retains it.
Backend
Subset
n
Cumulative recall
J
Initial
Iter. 2
Iter. 3
Iter. 4
Iter. 5
GPT-4o-mini
R0=0
192
0.00
16.11
24.33
29.77
33.90
45.83
0<R0<1
206
48.00
52.49
56.92
58.66
59.99
70.87
R0=1
1,124
100.00
100.00
100.00
100.00
100.00
90.04
GPT-4.1-mini
R0=0
192
0.00
21.84
34.16
41.50
47.58
64.58
0<R0<1
206
48.00
54.72
59.54
63.32
65.23
78.64
Table 6: Cumulative evidence recall and final J score (%) on LoCoMo, grouped by initial retrieval recall .
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: LLM-as-a-Judge score (%) with different a) number of chunks per iteration, b) number of facts/entities, and c) max iterations.
RAG
Self-RAG
RCR
Full-Context
MemR 3
Mean ± SD
56.60±0.72
41.53±0.61
56.07±0.31
58.53±1.10
71.67±0.76
Run 1
57.40
42.20
56.00
59.80
72.00
Run 2
56.00
41.00
55.80
57.80
70.80
Run 3
56.40
41.40
56.40
58.00
72.20
Appendix
Table 7: Three-run J scores (%) on LongMemEval s with GPT-4o-mini.
Backend
Method
J (%)
Completion tokens
Individual runs (J)
GPT-4o-mini
FAIR-RAG
54.93±0.46
996±4
55.20 / 55.20 / 54.40
MemR 3
71.67±0.76
357±74
72.00 / 70.80 / 72.20
GPT-4.1-mini
FAIR-RAG
60.00±0.92
1,060±29
60.80 / 60.20 / 59.00
MemR 3
76.40±1.06
491±5
77.20 / 76.80 / 75.20
GPT-5-mini
FAIR-RAG
73.20±1.40
10,144±185
74.60 / 73.20 / 71.80
MemR 3
81.80±0.72
5,029±78
82.60 / 81.20 / 81.60
Appendix
Table 8: Three-run comparison with FAIR-RAG on LongMemEval s across six backends. The last column lists individual runs .
Method
GPT-4o-mini
GPT-4.1-mini
J (%)
Completion tokens
J (%)
Completion tokens
FAIR-RAG
68.96±0.23
891±5
78.46±0.20
786±2
MemR 3 (RAG)
81.55±0.34
300±6
86.75±0.46
431±1
Zep
74.69±0.24
45±2
78.94±0.15
64±2
MemR 3 (Zep)
76.26±0.34
390±1
80.88±0.46
496±4
Appendix
Table 9: Three-run results on LoCoMo. FAIR-RAG and MemR 3 (RAG) share the same RAG retriever; MemR 3 (Zep) is compared with its Zep backbone .
LLM
Retriever
Run
MH
T
OD
SH
Overall
GPT-4o-mini
Zep
1
68.09
73.52
68.75
80.72
76.13
2
69.86
72.59
67.71
80.36
76.00
3
70.21
75.39
64.58
80.72
76.65
Mean ± SD
69.39 ± 1.14
73.83 ± 1.43
67.01 ± 2.17
80.60 ± 0.21
76.26 ± 0.34
RAG
1
71.63
77.26
61.46
89.28
81.75
2
70.21
76.01
59.38
89.40
81.16
Appendix
Table 10: Individual MemR 3 runs on LoCoMo, with category means and sample standard deviations. MH = multi-hop; T = temporal; OD = open-domain; SH = single-hop.
Backend
Method
P
A
TR
MS
KU
U
Overall
GPT-4o-mini
RAG
56.67
94.64
42.11
33.83
70.51
87.14
57.40
Self-RAG
26.67
83.93
18.05
18.80
66.67
78.57
42.20
RCR
53.33
92.86
34.59
38.35
71.79
84.29
56.00
Full-Context
40.00
94.64
39.10
45.11
76.92
88.57
59.80
MemR 3
60.00
92.86
55.64
67.67
75.64
95.71
72.00
Improvement
20.00
-1.78
16.54
22.56
-1.28
7.14
12.20
Appendix
Table 11: J scores (%) on LongMemEval s . P/A/U = single-session preference/assistant/user; TR = temporal reasoning; MS = multi-session; KU = knowledge update. Improvements are absolute differences from Full-Context. Three-run results are in Tables 7 and 8 .
Backend
Decomp-RAG
Rewrite-RAG
MemR 3
GPT-4o-mini
62.00
63.40
72.00
GPT-4.1-mini
68.40
68.40
77.20
GPT-5-mini
79.00
76.40
82.60
Appendix
Table 12: J scores (%) for query decomposition and iterative rewriting on LongMemEval s .
Backend
Method
Input
Completion
Total
J (%)
GPT-4o-mini
RAG
1,403
8
1,411
57.40
Self-RAG
3,461
159
3,619
42.20
RCR
4,044
106
4,150
56.00
MemR 3
6,195
314
6,509
72.00
Full-Context
109,624
12
109,636
59.80
GPT-4.1-mini
MemR 3
6,596
486
7,082
77.20
Appendix
Table 13: Average query-time tokens per question on LongMemEval s , including all controller, state-generation, and answer-generation calls. GPT-5-mini completion counts include internal reasoning.
Figure 6: Average token usage of the retrieved context (y-axis) and LLM-as-a-Judge (J) Score (x-axis) of RAG, Self-RAG, RCR, MemR 3 , and Full-Context across four categories on LoCoMo.
Metric
RAG
Self-RAG
RCR
Full-Context
MemR 3 (default)
MemR 3 (concise)
EM
23.60
0.20
0.00
24.20
0.00
19.00
F1
43.21
14.28
13.90
46.48
17.72
43.49
J
57.40
42.20
56.00
59.80
72.00
68.40
Appendix
Table 14: Auxiliary metrics (%) on LongMemEval s with GPT-4o-mini.
Figure 7: Number of questions requiring different numbers of iterations before final answers, across four categories on LoCoMo (left) and six categories on LongMemEval s (right).
Method
RAG
Full-Context
MemR 3 (stall-2)
MemR 3 (stall-3)
MemR 3
J (%)
57.40
59.80
66.00
67.40
72.00
Appendix
Table 15: J scores (%) on LongMemEval s with GPT-4o-mini. Stall- m stops after m consecutive iterations without evidence-state updates.
Figure 8: A successful case executed by MemR 3 on LoCoMo.
Figure 9: A failed case executed by MemR 3 with coverage bias (false negatives). Although the retrieved snippets contain evidence supporting the requirement, the LLM-based tracker fails to extract and record it in Ek .
Figure 10: A failed case executed by MemR 3 with hallucination bias (false positives).
Figure 11: A case executed by MemR 3 with granularity bias. Although the final answer is correct, it cannot close the loop until the max iteration is reached.
Category
Category 1
Category 2
Category 3
Category 4
Category 5
Name
Multi-Hop
Temporal
Open-Domain
Single-Hop
Adversarial
# Questions
282
321
96
830
445
Appendix
Table 16: The alignment of the category orders and names in the LoCoMo dataset.
Efficient long-term conversational memory requires retrieving sufficient evidence without indiscriminately expanding the context presented to the language model. This is challenging because relevant evidence may be distributed across multiple sessions, while compression may discard details needed for answering. Different queries therefore require different forms of memory access. To capture these demands, we formulate memory access along two dimensions: discovery breadth, which controls how broadly evidence is searched, and reading fidelity, which controls whether evidence is read in compact form or recovered from the original conversation. Based on this formulation, we introduce JustMem, which stores conversation history as compact atomic memories and adapts memory access along these two dimensions to each query. Specifically, LOOKUP handles local evidence, COMPOSE broadens discovery for distributed evidence, and REPLAY increases reading fidelity for fidelity-sensitive evidence. On LoCoMo and LongMemEval-S, JustMem achieves the highest mean accuracy and retrieval recall among the compared memory systems while using substantially fewer generative-model tokens for memory construction and inference.
Long-term conversational memory requires retrieving evidence scattered across multiple sessions, yet single-pass retrieval fails on temporal and multi-hop questions. Existing iterative methods refine queries via generated content or document-level signals, but none explicitly diagnoses the evidence gap, namely what is missing from the accumulated retrieval set, leaving query refinement untargeted. We present EviMem, combining IRIS (Iterative Retrieval via Insufficiency Signals), a closed-loop framework that detects evidence gaps through sufficiency evaluation, diagnoses what is missing, and drives targeted query refinement, with LaceMem (Layered Architecture for Conversational Evidence Memory), a coarse-to-fine memory hierarchy supporting fine-grained gap diagnosis. On LoCoMo, EviMem improves Judge Accuracy over MIRIX on temporal (73.3% to 81.6%) and multi-hop (65.9% to 85.2%) questions at 4.5x lower latency. Code: https://github.com/AIGeeksGroup/EviMem.
Long-term conversational large language model (LLM) agents require memory systems that can recover relevant evidence from historical interactions without overwhelming the answer stage with irrelevant context. However, existing memory systems, including hierarchical ones, still often rely solely on vector similarity for retrieval. It tends to produce bloated evidence sets: adding many superficially similar dialogue turns yields little additional recall, but lowers retrieval precision, increases answer-stage context cost, and makes retrieved memories harder to inspect and manage. To address this, we propose HiGMem (Hierarchical and LLM-Guided Memory System), a two-level event-turn memory system that allows LLMs to use event summaries as semantic anchors to predict which related turns are worth reading. This allows the model to inspect high-level event summaries first and then focus on a smaller set of potentially useful turns, providing a concise and reliable evidence set through reasoning, while avoiding the retrieval overhead that would be excessively high compared to vector retrieval. On the LoCoMo10 benchmark, HiGMem achieves the best F1 on four of five question categories and improves adversarial F1 from 0.54 to 0.78 over A-Mem, while retrieving an order of magnitude fewer turns. Code is publicly available at https://github.com/ZeroLoss-Lab/HiGMem.
Shuqi Cao, Jingyi He, Fei Tan
East China Normal University, Shanghai, China · Shanghai Jiao Tong University, Shanghai, China