Under a limited context budget, a long-term agent must recall not merely memories that are individually relevant, but evidence that is collectively sufficient for the current query. Conventional memory access retrieves, scores, and truncates memories, often spending multiple context slots on redundant facts while leaving complementary evidence behind. We cast the read phase of long-term memory as query-conditioned evidence composition: each memory is valued by what it adds beyond the evidence already selected. We instantiate this operation with MemPursuit, a training-free mechanism that uses the residual of a sparse convex query reconstruction as a representation-space proxy for what the current memory set is missing. By reusing the query and candidate embeddings already available from retrieval, MemPursuit progressively composes nested evidence contexts without invoking a language model or cross-encoder to score each query--memory pair. The effectiveness of our method is verified on 3 popular benchmarks in comparison to strong baselines. In fixed-budget LoCoMo experiments, MemPursuit improves end-to-end accuracy by 7.2 percentage points at Top-10 and 5.5 percentage points at Top-20, with consistent gains across multiple memory frameworks, while achieving a 32.9× speedup in median reranking latency over fully batched BGE-v2-m3. Our implementation is available at https://github.com/FengQi-HITSZ/MemPursuit.
Figures & tables
Figure 1: A memory-reading example from LoCoMo. The query asks which of Deborah’s family and friends have passed away. The standard read returns repeated records about her father and a distractor about Anna’s mother, while omitting her friend Karlie. MemPursuit selects complementary records about her father, mother, and Karlie, enabling the complete answer shown. Dialogue and memory excerpts are shortened for presentation. Additional case studies appear in Appendix C.3 .
Figure 2: Overview of MemPursuit , illustrated with a query requiring two adoption dates. (1) Form the residual between the query embedding and the current memory reconstruction. (2) Select the memory with the highest residual alignment: the complementary record about Toby is preferred over another record about Buddy, despite its lower query similarity. (3) Jointly refit the selected memories’ nonnegative, sum-to-one weights and update the residual. Repeat until the memory budget is reached or the adaptive stopping condition is met. Scores and weights are illustrative.
MemPursuit
LLM scorer
Resource
Asymptotic cost
Instantiated scale
Asymptotic cost
Instantiated scale
Time
O(P2d+KP2+K4)
6.969×107
O(PL(nW2+n2W))
5.637×1012
Space
O(Pd+P2)
3.472×105
O(LW2+LnW+n2)
2.202×108
Table 1: Time and space complexity of final memory selection. Numerical scales use K=50 , P=200 , d=1536 , L=32 , W=2560 , and n=128 .
LoCoMo
LongMemEval-S
BEAM
Method
MH
Temp.
Open
SH
Avg.
KU
MS
SSA
SSP
SSU
TR
Avg.
Avg.
GPT-4o-mini
Full context
68.79
50.16
56.25
86.56
73.83
–
–
–
–
–
–
–
–
RAG
82.62
58.57
64.58
85.85
78.25
53.85
45.86
94.64
46.67
87.14
41.35
57.20
38.23
A-MEM
59.22
64.17
54.17
69.80
65.71
69.23
55.64
94.64
40.00
84.29
63.16
67.20
38.78
LightMem
79.08
73.83
62.50
86.56
81.04
80.77
72.93
12.50
46.67
88.57
77.44
69.20
41.42
Table 2: End-to-end results under two answerer backbones. LoCoMo and LongMemEval-S report binary-judge accuracy (%); BEAM reports its official mean score scaled to 0–100. Bold and underline denote the best and second-best result within each answerer panel. “–” denotes an unavailable run due to limited context length.
System
MemPursuit
Multi-hop
Temporal
Open-domain
Single-hop
Overall
LightMem
w/o
62.77
66.98
56.25
77.17
71.10
w/
69.15 ( +6.38 )
68.22 ( +1.25 )
57.29 ( +1.04 )
79.19 ( +2.02 )
73.70 ( +2.60 )
MemoryBank
w/o
53.55
43.30
51.04
65.87
57.99
w/
56.74 ( +3.19 )
39.25 ( −4.05 )
56.25 ( +5.21 )
70.27 ( +4.40 )
60.45 ( +2.47 )
SimpleMem
w/o
62.41
48.60
56.25
74.91
65.97
w/
70.21 ( +7.80 )
49.22 ( +0.62 )
60.42 ( +4.17 )
82.28 ( +7.37 )
71.82 ( +5.84 )
Table 3: Cross-framework comparisons on LoCoMo at K=10 . Each system is evaluated without (w/o) and with (w/) MemPursuit at the same selection budget. Entries report accuracy (%); parentheses in the w/ rows give the change in percentage points from the paired w/o result. For each system, the higher accuracy in each metric is bolded. Yellow highlights the w/ results. All configurations evaluate 1,540 questions using GPT-4o-mini as answerer and judge. Full results at K∈{10,20} appear in Table 4 (Appendix C.1 ).
Figure 3: Memory selection and evidence-session coverage on LoCoMo at Top-10 and Top-20. Panels (b) and (c) evaluate 1,536 gold-annotated questions. MemPursuit changes the selected context while increasing both average and complete evidence-session coverage.
Figure 4: LoCoMo performance with GPT-4o-mini as the answerer. The star in panel (b) marks adaptive context selection using the stopping rule in Section 3.3 with τ=10−5 , achieving 85.1% accuracy with an average of 15.65 selected memories. The 0.1% accuracy gap at Top-200 arises from differences in memory ordering.
Figure 5: LoCoMo accuracy versus median reranking latency at Top-20 with P=200 . Neural scorers use batch size 128. Latency includes final reranking only.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
K
System
MemPursuit
Multi-hop
Temporal
Open-domain
Single-hop
Overall
10
LightMem
w/o
62.77
66.98
56.25
77.17
71.10
w/
69.15 ( +6.38 )
68.22 ( +1.25 )
57.29 ( +1.04 )
79.19 ( +2.02 )
73.70 ( +2.60 )
MemoryBank
w/o
53.55
43.30
51.04
65.87
57.99
w/
56.74 ( +3.19 )
39.25 ( −4.05 )
56.25 ( +5.21 )
70.27 ( +4.40 )
60.45 ( +2.47 )
SimpleMem
w/o
62.41
48.60
56.25
74.91
65.97
w/
70.21 ( +7.80 )
49.22 ( +0.62 )
60.42 ( +4.17 )
82.28 ( +7.37 )
71.82 ( +5.84 )
Appendix
Table 4: Full cross-framework comparisons on LoCoMo at K∈{10,20} . Each system is evaluated without (w/o) and with (w/) MemPursuit at the same selection budget K (seed count for MemoryBank). Entries report accuracy (%); parentheses in the w/ rows give the change in percentage points from the paired w/o result. For each system and budget, the higher accuracy in each metric is bolded. Yellow highlights the w/ results. All configurations evaluate 1,540 questions using GPT-4o-mini as answerer and judge.
Figure 7: Reconstruction residual advantage of gold evidence over hybrid retrieval at matched set size. Positive Δq (green) indicates a lower residual norm for the gold evidence set; negative Δq (orange) indicates a lower residual for the baseline. The dashed line marks equal residuals. Both sets are evaluated with the same simplex refit.
Figure 8: Residual preference for missing evidence on LongMemEval with ∣S∣=1 . The histogram contains one mean utility difference per question (136 questions, 381 singleton comparisons). Positive values (green) favor remaining evidence over the semantic comparator, while negative values (orange) favor the comparator. The gray line marks zero.
Read policy
Multi-hop
Temporal
Open-domain
Single-hop
Overall
Original hybrid order
75.5
73.8
64.6
84.9
79.6
MemPursuit
80.9
77.9
63.5
90.7
85.1
Qwen3.5-0.8B
78.0
77.9
63.5
87.8
82.4
Qwen3.5-2B
81.6
79.4
66.7
91.3
85.5
Qwen3.5-4B
86.9
80.7
65.6
91.1
86.6
BGE-base
75.2
67.6
64.6
87.2
79.5
Appendix
Table 9: LoCoMo question-type accuracy (%) at Top-20 under the controlled LoCoMo evaluation setup, comparing hybrid ordering, MemPursuit , and pointwise neural rerankers.
Memory budget K
MemPursuit
MMR
10
81.04
77.08
20
85.06
83.31
50
87.08
86.10
Appendix
Table 10: LoCoMo accuracy (%) for MemPursuit and MMR with λMMR=0.5 , using the same P=200 candidate pool and downstream evaluation. Each row contains 1,540 questions; bold denotes the higher accuracy.
Embedding model
Top-10
Top-20
Top-50
text-embedding-3-small
81.04
85.06
87.08
E5-large-v2
80.32 ( −0.71 )
83.57 ( −1.49 )
86.49 ( −0.58 )
BGE-M3
80.00 ( −1.04 )
84.68 ( −0.39 )
86.10 ( −0.97 )
Qwen3-Embedding-0.6B
79.94 ( −1.10 )
82.40 ( −2.66 )
85.65 ( −1.43 )
Qwen3-Embedding-8B
83.18 ( +2.14 )
85.91 ( +0.84 )
86.49 ( −0.58 )
Appendix
Table 11: Robustness to embedding choice on LoCoMo with identical per-question candidate memories. Only the dense encoder used by MemPursuit changes. Entries report accuracy (%) over 1,540 questions; parentheses give changes in percentage points relative to text-embedding-3-small. Bold denotes the highest accuracy at each cutoff.
Latency (ms)
Speedup
Read policy
Batch
Mean
Median
P95
Mean latency
Median latency
MemPursuit
–
17.69
8.67
57.76
1.00 ×
1.00 ×
BGE-base
128
109.26
109.03
126.12
6.18 ×
12.58 ×
BGE-large
128
283.73
283.27
330.01
16.04 ×
32.67 ×
BGE-v2-m3
128
275.66
274.33
320.70
15.58 ×
31.64 ×
Qwen3.5-0.8B
128
1682.95
1675.98
1751.40
95.14 ×
193.31 ×
Appendix
Table 12: Final reranking latency at Top-20 with P=200 candidates. Neural scorers use batch size 128, a large candidate batch size supported by all evaluated BGE and Qwen models. Tail latency is P95 for every method. The two speedup columns divide each baseline’s mean or median latency by the corresponding MemPursuit value (17.69 or 8.67 ms).
Read policy
Mean ms
Median ms
P95 ms
Median speedup
MemPursuit
17.69
8.67
57.76
1.00 ×
BGE-base
111.34
110.48
129.88
12.74 ×
BGE-large
286.10
286.02
331.60
32.99 ×
BGE-v2-m3
286.45
285.61
332.61
32.94 ×
Qwen3.5-0.8B
1733.86
1724.74
1794.96
198.93 ×
Qwen3.5-2B
2129.20
2124.70
2222.55
245.06 ×
Appendix
Table 13: Full-batch reranking latency at Top-20 with P=200 candidates over 1,540 questions. Neural scorers use batch size 200. Latencies are in milliseconds. “Median speedup” divides each baseline median by the MemPursuit median of 8.67 ms.
Figure 9: Median reranking latency across batch sizes for BGE cross-encoders (blue) and Qwen3.5 LLM scorers (orange), with MemPursuit as a batch-independent reference at 8.67 ms (green dashed line). The horizontal axis uses a base-2 logarithmic scale and the vertical axis a base-10 logarithmic scale. BGE sweeps extend to batch size 200 and Qwen sweeps to 144.
Selected-memory budget K
Candidates P
10
20
50
100
100
4.54
6.30
11.79
22.75
200
7.27
8.40
13.62
23.99
300
9.73
12.16
17.11
26.49
500
18.04
22.47
30.67
38.58
Appendix
Table 14: Median MemPursuit reranking latency (ms per question) across candidate counts P and selected-memory budgets K . Each entry is measured in the P/K sweep.
Long-term agent memory requires recovering answer-relevant evidence dispersed across turns and sessions. Existing systems increasingly use structured memories and adaptive retrieval, yet often pass retrieved or expanded units directly to the reader, coupling memory addressing with evidence presentation and leaving the final context redundant or incomplete. To address this gap, we propose MemCompile, a query-conditioned evidence compilation framework for long-term conversational memory. Given a query, the Address Plane locates relevant support through compact source-linked descriptors and capacity-aware Cells, while a normalization boundary resolves heterogeneous retrieval hits to stable SourcePacket identities. Its Evidence Plane then allocates a bounded evidence budget across complementary packets, admits limited additional support, and renders the retained claims associated with each selected packet into an answer-ready context. MemCompile achieves 80.52% Judge Accuracy on LoCoMo and extends the empirical quality-cost Pareto frontier to a higher-quality regime. Its strong performance on LongMemEval-S further suggests that the benefits of query-conditioned compilation generalize across long-term conversational-memory benchmarks.
Shangze Li, Yanyu Yao, Derong Xu +2
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China, Hefei, China
Long-term memory enables language models to use past interactions in future conversations. However, evidence needed to answer a question may be scattered across distant turns, while the question itself omits clues needed to locate it. Retrieved facts can reveal these clues, motivating retrieval decisions conditioned on evidence already found. We introduce MERA (Missing-Evidence Retrieval Augmentation), which separates globally searchable memory from a question-specific evidence state. Verified evidence guides subsequent retrieval without restricting access to the global memory. We train a lightweight planner through reinforcement learning, rewarding queries that recover previously missing evidence. MERA achieves strong answer accuracy across Qwen3-30B and GPT-4o-mini backbones. With Qwen3-30B for evidence processing and answer generation, the trained 0.6B planner achieves 77.40% accuracy on LoCoMo and 71.29% on LongMemEval-S, exceeding a 30B planner without retrieval-grounded training by 4.10% and 3.96%, respectively. On LoCoMo, later retrieval rounds increase cumulative evidence recall from 55.5% to 80.5%.
Long-horizon agents can archive large histories, but future answers still incur retrieval, rereading, and context costs. When retained memory misses answer-relevant evidence, the system must return to larger portions of the raw history. We study budgeted evidence survival: before the query is known, which source evidence should be retained so that it remains recoverable and usable under a fixed retained source-evidence token budget? We instantiate this setting as Budgeted Pre-Query Retention, where memory is written during ingestion and later read without access to the full raw stream. We introduce EMBER, a learned retention policy that constructs a compact, source-backed evidence state. EMBER stores evidence capsules: verbatim source excerpts paired with retrieval keys and update metadata, preserving both grounding and read-time access. Post-query outcome feedback trains the writer to preserve evidence across the ingestion-retrieval-answer chain. On LongMemEval-RR, our LongMemEval-derived retained-evidence protocol, EMBER-14B reaches 0.3017 F1 at the 8192-token retained-evidence comparison point, compared with 0.1765 for the strongest non-EMBER budgeted baseline. Across retained source-evidence budgets, EMBER improves F1, Retain-Recall, and Read-Recall, indicating that long-horizon memory depends on retaining evidence within the budget rather than rereading larger histories.