Recall What Is Missing: Composing Evidence from Long-Term Agent Memory

Qi Feng; Jicong Fan

cs.AIOct 11, 2026

Recall What Is Missing: Composing Evidence from Long-Term Agent Memory

Authors: Qi Feng, Jicong Fan

Organizations: School of Data Science, The Chinese University of Hong Kong, Shenzhen

Abstract

Under a limited context budget, a long-term agent must recall not merely memories that are individually relevant, but evidence that is collectively sufficient for the current query. Conventional memory access retrieves, scores, and truncates memories, often spending multiple context slots on redundant facts while leaving complementary evidence behind. We cast the read phase of long-term memory as query-conditioned evidence composition: each memory is valued by what it adds beyond the evidence already selected. We instantiate this operation with MemPursuit, a training-free mechanism that uses the residual of a sparse convex query reconstruction as a representation-space proxy for what the current memory set is missing. By reusing the query and candidate embeddings already available from retrieval, MemPursuit progressively composes nested evidence contexts without invoking a language model or cross-encoder to score each query--memory pair. The effectiveness of our method is verified on 3 popular benchmarks in comparison to strong baselines. In fixed-budget LoCoMo experiments, MemPursuit improves end-to-end accuracy by 7.2 percentage points at Top-10 and 5.5 percentage points at Top-20, with consistent gains across multiple memory frameworks, while achieving a 32.9 $\times$ speedup in median reranking latency over fully batched BGE-v2-m3. Our implementation is available at https://github.com/FengQi-HITSZ/MemPursuit.

Figures & tables

Figure 1: A memory-reading example from LoCoMo. The query asks which of Deborah’s family and friends have passed away. The standard read returns repeated records about her father and a distractor about Anna’s mother, while omitting her friend Karlie. MemPursuit selects complementary records about her father, mother, and Karlie, enabling the complete answer shown. Dialogue and memory excerpts are shortened for presentation. Additional case studies appear in Appendix C.3 .

Figure 2: Overview of MemPursuit , illustrated with a query requiring two adoption dates. (1) Form the residual between the query embedding and the current memory reconstruction. (2) Select the memory with the highest residual alignment: the complementary record about Toby is preferred over another record about Buddy, despite its lower query similarity. (3) Jointly refit the selected memories’ nonnegative, sum-to-one weights and update the residual. Repeat until the memory budget is reached or the adaptive stopping condition is met. Scores and weights are illustrative.

	MemPursuit		LLM scorer
Resource	Asymptotic cost	Instantiated scale	Asymptotic cost	Instantiated scale
Time	$O(P^{2}d+KP^{2}+K^{4})$	$6.969\!\times\!10^{7}$	$O(PL(nW^{2}+n^{2}W))$	$5.637\!\times\!10^{12}$
Space	$O(Pd+P^{2})$	$3.472\!\times\!10^{5}$	$O(LW^{2}+LnW+n^{2})$	$2.202\!\times\!10^{8}$

Table 1: Time and space complexity of final memory selection. Numerical scales use

K=50

P=200

d=1536

L=32

W=2560

, and

n=128

GPT-4o-mini
	LoCoMo					LongMemEval-S							BEAM
Method	MH	Temp.	Open	SH	Avg.	KU	MS	SSA	SSP	SSU	TR	Avg.	Avg.
Full context	68.79	50.16	56.25	86.56	73.83	–	–	–	–	–	–	–	–
RAG	82.62	58.57	64.58	85.85	78.25	53.85	45.86	94.64	46.67	87.14	41.35	57.20	38.23
A-MEM	59.22	64.17	54.17	69.80	65.71	69.23	55.64	94.64	40.00	84.29	63.16	67.20	38.78
LightMem	79.08	73.83	62.50	86.56	81.04	80.77	72.93	12.50	46.67	88.57	77.44	69.20	41.42

Table 2: End-to-end results under two answerer backbones. LoCoMo and LongMemEval-S report binary-judge accuracy (%); BEAM reports its official mean score scaled to 0–100. Bold and underline denote the best and second-best result within each answerer panel. “–” denotes an unavailable run due to limited context length.

System	MemPursuit	Multi-hop	Temporal	Open-domain	Single-hop	Overall
LightMem	w/o	62.77	66.98	56.25	77.17	71.10
LightMem	w/	69.15 ( $+6.38$ )	68.22 ( $+1.25$ )	57.29 ( $+1.04$ )	79.19 ( $+2.02$ )	73.70 ( $+2.60$ )
MemoryBank	w/o	53.55	43.30	51.04	65.87	57.99
MemoryBank	w/	56.74 ( $+3.19$ )	39.25 ( $-4.05$ )	56.25 ( $+5.21$ )	70.27 ( $+4.40$ )	60.45 ( $+2.47$ )
SimpleMem	w/o	62.41	48.60	56.25	74.91	65.97
SimpleMem	w/	70.21 ( $+7.80$ )	49.22 ( $+0.62$ )	60.42 ( $+4.17$ )	82.28 ( $+7.37$ )	71.82 ( $+5.84$ )

Table 3: Cross-framework comparisons on LoCoMo at

K=10

. Each system is evaluated without (w/o) and with (w/) MemPursuit at the same selection budget. Entries report accuracy (%); parentheses in the w/ rows give the change in percentage points from the paired w/o result. For each system, the higher accuracy in each metric is bolded. Yellow highlights the w/ results. All configurations evaluate 1,540 questions using GPT-4o-mini as answerer and judge. Full results at

K\in\{10,20\}

appear in Table 4 (Appendix C.1 ).

Panel of Figure 3 — Figure 3: Memory selection and evidence-session coverage on LoCoMo at Top-10 and Top-20. Panels (b) and (c) evaluate 1,536 gold-annotated questions. MemPursuit changes the selected context while increasing both average and complete evidence-session coverage.

Panel of Figure 4 — Figure 4: LoCoMo performance with GPT-4o-mini as the answerer. The star in panel (b) marks adaptive context selection using the stopping rule in Section 3.3 with $\tau=10^{-5}$ , achieving 85.1% accuracy with an average of 15.65 selected memories. The 0.1% accuracy gap at Top-200 arises from differences in memory ordering.

Figure 5: LoCoMo accuracy versus median reranking latency at Top-20 with $P=200$ . Neural scorers use batch size 128. Latency includes final reranking only.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

$K$	System	MemPursuit	Multi-hop	Temporal	Open-domain	Single-hop	Overall
$10$	LightMem	w/o	62.77	66.98	56.25	77.17	71.10
	LightMem	w/	69.15 ( $+6.38$ )	68.22 ( $+1.25$ )	57.29 ( $+1.04$ )	79.19 ( $+2.02$ )	73.70 ( $+2.60$ )
	MemoryBank	w/o	53.55	43.30	51.04	65.87	57.99
	MemoryBank	w/	56.74 ( $+3.19$ )	39.25 ( $-4.05$ )	56.25 ( $+5.21$ )	70.27 ( $+4.40$ )	60.45 ( $+2.47$ )
	SimpleMem	w/o	62.41	48.60	56.25	74.91	65.97
	SimpleMem	w/	70.21 ( $+7.80$ )	49.22 ( $+0.62$ )	60.42 ( $+4.17$ )	82.28 ( $+7.37$ )	71.82 ( $+5.84$ )

Appendix

Table 4: Full cross-framework comparisons on LoCoMo at

K\in\{10,20\}

. Each system is evaluated without (w/o) and with (w/) MemPursuit at the same selection budget

K

(seed count for MemoryBank). Entries report accuracy (%); parentheses in the w/ rows give the change in percentage points from the paired w/o result. For each system and budget, the higher accuracy in each metric is bolded. Yellow highlights the w/ results. All configurations evaluate 1,540 questions using GPT-4o-mini as answerer and judge.

Figure 7: Reconstruction residual advantage of gold evidence over hybrid retrieval at matched set size. Positive $\Delta_{q}$ (green) indicates a lower residual norm for the gold evidence set; negative $\Delta_{q}$ (orange) indicates a lower residual for the baseline. The dashed line marks equal residuals. Both sets are evaluated with the same simplex refit.

Figure 8: Residual preference for missing evidence on LongMemEval with $|S|=1$ . The histogram contains one mean utility difference per question (136 questions, 381 singleton comparisons). Positive values (green) favor remaining evidence over the semantic comparator, while negative values (orange) favor the comparator. The gray line marks zero.

Read policy	Multi-hop	Temporal	Open-domain	Single-hop	Overall
Original hybrid order	75.5	73.8	64.6	84.9	79.6
MemPursuit	80.9	77.9	63.5	90.7	85.1
Qwen3.5-0.8B	78.0	77.9	63.5	87.8	82.4
Qwen3.5-2B	81.6	79.4	66.7	91.3	85.5
Qwen3.5-4B	86.9	80.7	65.6	91.1	86.6
BGE-base	75.2	67.6	64.6	87.2	79.5

Appendix

Table 9: LoCoMo question-type accuracy (%) at Top-20 under the controlled LoCoMo evaluation setup, comparing hybrid ordering, MemPursuit , and pointwise neural rerankers.

Memory budget $K$	MemPursuit	MMR
10	81.04	77.08
20	85.06	83.31
50	87.08	86.10

Appendix

Table 10: LoCoMo accuracy (%) for MemPursuit and MMR with

\lambda_{\mathrm{MMR}}=0.5

, using the same

P=200

candidate pool and downstream evaluation. Each row contains 1,540 questions; bold denotes the higher accuracy.

Embedding model	Top-10	Top-20	Top-50
text-embedding-3-small	81.04	85.06	87.08
E5-large-v2	80.32 ( $-0.71$ )	83.57 ( $-1.49$ )	86.49 ( $-0.58$ )
BGE-M3	80.00 ( $-1.04$ )	84.68 ( $-0.39$ )	86.10 ( $-0.97$ )
Qwen3-Embedding-0.6B	79.94 ( $-1.10$ )	82.40 ( $-2.66$ )	85.65 ( $-1.43$ )
Qwen3-Embedding-8B	83.18 ( $+2.14$ )	85.91 ( $+0.84$ )	86.49 ( $-0.58$ )

Appendix

Table 11: Robustness to embedding choice on LoCoMo with identical per-question candidate memories. Only the dense encoder used by MemPursuit changes. Entries report accuracy (%) over 1,540 questions; parentheses give changes in percentage points relative to text-embedding-3-small. Bold denotes the highest accuracy at each cutoff.

		Latency (ms)			Speedup
Read policy	Batch	Mean	Median	P95	Mean latency	Median latency
MemPursuit	–	17.69	8.67	57.76	1.00 $\times$	1.00 $\times$
BGE-base	128	109.26	109.03	126.12	6.18 $\times$	12.58 $\times$
BGE-large	128	283.73	283.27	330.01	16.04 $\times$	32.67 $\times$
BGE-v2-m3	128	275.66	274.33	320.70	15.58 $\times$	31.64 $\times$
Qwen3.5-0.8B	128	1682.95	1675.98	1751.40	95.14 $\times$	193.31 $\times$

Appendix

Table 12: Final reranking latency at Top-20 with

P=200

candidates. Neural scorers use batch size 128, a large candidate batch size supported by all evaluated BGE and Qwen models. Tail latency is P95 for every method. The two speedup columns divide each baseline’s mean or median latency by the corresponding MemPursuit value (17.69 or 8.67 ms).

Read policy	Mean ms	Median ms	P95 ms	Median speedup
MemPursuit	17.69	8.67	57.76	1.00 $\times$
BGE-base	111.34	110.48	129.88	12.74 $\times$
BGE-large	286.10	286.02	331.60	32.99 $\times$
BGE-v2-m3	286.45	285.61	332.61	32.94 $\times$
Qwen3.5-0.8B	1733.86	1724.74	1794.96	198.93 $\times$
Qwen3.5-2B	2129.20	2124.70	2222.55	245.06 $\times$

Appendix

Table 13: Full-batch reranking latency at Top-20 with

P=200

candidates over 1,540 questions. Neural scorers use batch size 200. Latencies are in milliseconds. “Median speedup” divides each baseline median by the MemPursuit median of 8.67 ms.

Figure 9: Median reranking latency across batch sizes for BGE cross-encoders (blue) and Qwen3.5 LLM scorers (orange), with MemPursuit as a batch-independent reference at 8.67 ms (green dashed line). The horizontal axis uses a base-2 logarithmic scale and the vertical axis a base-10 logarithmic scale. BGE sweeps extend to batch size 200 and Qwen sweeps to 144.

	Selected-memory budget $K$
Candidates $P$	10	20	50	100
100	4.54	6.30	11.79	22.75
200	7.27	8.40	13.62	23.99
300	9.73	12.16	17.11	26.49
500	18.04	22.47	30.67	38.58

Appendix

Table 14: Median MemPursuit reranking latency (ms per question) across candidate counts

P

and selected-memory budgets

K

. Each entry is measured in the

P/K

sweep.

Long-Term Conversational Memory

Long-term conversational memory helps AI assistants retain, update, and use information from past conversations across sessions.

See all 267 papers

From Memory Hits to Answerable Evidence: Query-Conditioned Memory Compilation for Long-Term Agent Memory
Oct 16, 2026Shangze Li, Yanyu Yao, Derong Xu +2LLM Agent Memory Memory Management for LLM Agents
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China, Hefei, China
Learning to Retrieve Missing Evidence for Long-Term Memory QA
Sep 29, 2026Yi-Xuan Deng, Yi Zhang, Wei Liu +2Long-Term Conversational Memory Evidence Retrieval
Tsinghua University
EMBER: Efficient Memory via Budgeted Evidence Retention for Long-Horizon Agents
Jun 4, 2026Yilong Li, Suman Banerjee, Tong ChePersistent Memory for Language Models Long-Term Conversational Memory
University of Wisconsin–Madison · NVIDIA Research

Oct 16, 2026J/K move · Enter open · S save

From Memory Hits to Answerable Evidence: Query-Conditioned Memory Compilation for Long-Term Agent Memory

Shangze Li, Yanyu Yao, Derong Xu, Zhi Zheng+1

State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China, Hefei, China

LLM Agent Memory Memory Management for LLM Agents Long-Term Conversational Memory

Open paper AskPDF arXiv

Long-term agent memory requires recovering answer-relevant evidence dispersed across turns and sessions. Existing systems increasingly use structured memories and adaptive retrieval, yet often pass retrieved or expanded units directly to the reader, coupling memory addressing with evidence presentation and leaving the final context redundant or incomplete. To address this gap, we propose MemCompile, a query-conditioned evidence compilation framework for long-term conversational memory. Given a query, the Address Plane locates relevant support through compact source-linked descriptors and capacity-aware Cells, while a normalization boundary resolves heterogeneous retrieval hits to stable SourcePacket identities. Its Evidence Plane then allocates a bounded evidence budget across complementary packets, admits limited additional support, and renders the retained claims associated with each selected packet into an answer-ready context. MemCompile achieves 80.52% Judge Accuracy on LoCoMo and extends the empirical quality-cost Pareto frontier to a higher-quality regime. Its strong performance on LongMemEval-S further suggests that the benefits of query-conditioned compilation generalize across long-term conversational-memory benchmarks.

Abstract

Long-Term Conversational Memory

Explore similar work

From Memory Hits to Answerable Evidence: Query-Conditioned Memory Compilation for Long-Term Agent Memory

Learning to Retrieve Missing Evidence for Long-Term Memory QA

EMBER: Efficient Memory via Budgeted Evidence Retention for Long-Horizon Agents