GraphMemix: Query-Aware Evidence Forests for Long-Term Multimodal Agent Memory

Geng Li; Yuhao Wang; Dong Li; Jianye Hao; Yuxin Peng

cs.AIAug 27, 2026

GraphMemix: Query-Aware Evidence Forests for Long-Term Multimodal Agent Memory

Authors: Geng Li, Yuhao Wang, Dong Li, Jianye Hao, Yuxin Peng

Organizations: Wangxuan Institute of Computer Technology, Peking University · MemoraX AI

Abstract

Organizing long-term memory for multimodal agents remains challenging because existing methods either suffer from expensive question-agnostic offline summaries or naive embedding similarity matching that introduces incomplete and redundant context. To address these issues, we propose GraphMemix, a combinatorial-optimization graph memory framework that models memory organization as query-aware evidence-forest construction. Specifically, our method consists of three key components:(1) candidate graph construction, which expands multi-view seed memories through schema and semantic relations to acquire query-aware original context; (2) evidence utility and activation costs, which decouples direct memory support from anchor-conditioned relation verification to suppress redundant or conflicting information; and (3) forest optimization, which jointly selects a forest-format memory context under a maximum evidence budget and its reliable relational structure. By organizing memory into a query-relevant subgraph, the method avoids substantial lifecycle cost and recovers low-similarity complementary evidence. Experimental results across four long-term multimodal memory benchmarks demonstrate significant improvements with different foundation models and establish a new Pareto frontier between accuracy and lifecycle cost.

Figures & tables

Figure 1: Overview of GraphMemix in three stages. (1) Candidate graph construction uses multi-view retrieval to identify seeds and bounded relation expansion to expose their local context. (2) Evidence utility and costs combines direct node support with ECV-validated, anchor-conditioned incremental relations. (3) Forest optimization jointly selects nodes and trusted edges under a maximum reader budget, then deterministically serializes the resulting evidence forest.

		ATM		Mem-Gallery		MemEye		H2HMem		Avg.
Method	Venue	J.Acc.	EM	J.Acc.	EM	J.Acc.	M.EM	J.Acc.	Recall	J.Acc.
A-MEM	NeurIPS’25	41.86	40.80	52.89	32.61	37.09	40.57	39.94	40.55	42.94
UniversalRAG	ACL’26	43.10	41.19	63.76	32.50	47.98	49.33	44.36	45.04	49.80
MemGuide	AAAI’26	48.47	47.80	48.10	32.73	41.40	44.61	39.94	35.81	44.48
LightMem	ICLR’26	22.70	24.52	38.28	23.20	36.23	39.76	31.04	27.82	32.06
VimRAG	ICML’26	33.81	32.76	48.33	16.13	28.25	28.23	17.89	17.01	32.07

Table 1: End-to-end results with Qwen3-VL-8B-Instruct, judged by GPT-5-mini. J.Acc. denotes Judge Accuracy (%). Each benchmark additionally reports its representative native metric. Avg. is the macro-average over the four Judge Accuracies.

		ATM		Mem-Gallery		MemEye		H2HMem		Avg.
Method	Venue	J.Acc.	EM	J.Acc.	EM	J.Acc.	M.EM	J.Acc.	Recall	J.Acc.
A-MEM	NeurIPS’25	42.43	41.95	50.03	28.40	24.42	30.93	44.20	39.97	40.27
UniversalRAG	ACL’26	49.04	48.18	64.82	32.14	58.17	38.07	48.34	41.31	55.09
MemGuide	AAAI’26	54.98	53.07	48.33	22.85	30.40	36.66	47.98	41.97	45.43
LightMem	ICLR’26	24.33	25.48	33.72	17.01	20.70	30.59	32.80	27.79	27.89
VimRAG	ICML’26	22.51	21.36	46.99	14.14	31.81	34.64	39.00	33.67	35.08

Table 2: End-to-end results with Gemma 4 12B Unified, judged by GPT-5-mini. J.Acc. denotes Judge Accuracy (%). Each benchmark additionally reports its representative native metric. Avg. is the macro-average over the four Judge Accuracies.

Figure 2: Optimization behavior over all 6,592 evaluation questions. (a) Mean improvement in the complete forest objective after accepted 1-swap updates. (b) Mean cumulative structural gain from accepted edges during the exact Kruskal update. Shaded regions are 95% confidence intervals.

Figure 3: Full-lifecycle cost and accuracy on ATM-Bench. (a) Estimated time to digest all 11,034 memories once and answer all 1,044 questions once. (b) Judge Accuracy versus the same lifecycle time. GraphMemix lies on the empirical Pareto frontier.

Configuration	MV	Node	ECV	Opt.	ATM	Gallery	MemEye	H2H	Avg.
(a) Embedding Top- $K$					42.80	62.50	46.70	44.80	49.20
(b) + Multi-view Retrieval	✓				47.40	67.90	49.10	50.00	53.60
(c) + Node Verifier	✓	✓			53.20	73.80	52.20	56.00	58.80
(d) + ECV Reranking	✓	✓	✓		54.00	74.60	52.80	57.40	59.70
(e) + Forest Optimization ( GraphMemix )	✓	✓	✓	✓	55.27	76.33	53.64	60.96	61.55

Table 3: Incremental ablation in Judge Accuracy (%). MV denotes multi-view retrieval and Opt. denotes set-level forest optimization. ECV Reranking uses the strongest verified relation as an independent candidate-level bonus, whereas the final row jointly optimizes the evidence set and its forest.

Dataset	Direct	Recoverable	No Access
ATM-Bench	68.50	15.14	16.35
Mem-Gallery	56.67	37.87	5.46
MemEye	42.08	39.70	18.21
H2HMem	15.61	33.93	50.46

Table 4: Accessibility of gold evidence from source top-10 and the actual bounded candidate graph (%).

Dataset	R.Q.	C.Q.		Share
Dataset	R.Q.	Count	Rate	Share
ATM-Bench	165	113	68.48%	12.01%
Mem-Gallery	1,142	629	55.08%	19.58%
MemEye	1,256	621	49.44%	19.17%
H2HMem	1,956	784	40.08%	19.01%

Table 5: Question-level recovery of gold evidence with Gemma 4. R.Q. denotes questions with recoverable gold in the candidate graph; C.Q. denotes questions gain recovered gold evidence by GraphMemix . Share is recovered evidence percentage among all retained gold pairs.

Dataset	Fixed Edges	ECV Edges	Fixed/ECV Components	Fixed Hit@10	ECV Hit@10	Gain
ATM-Bench	6.23	0.66	3.77 / 9.34	84.20	87.74	+3.54
Mem-Gallery	7.34	1.47	2.66 / 8.53	94.76	96.59	+1.83
MemEye	6.07	0.25	3.93 / 9.75	85.34	88.89	+3.56
H2HMem	6.75	1.00	3.25 / 9.00	89.46	95.06	+5.60

Table 6: Effect of ECV on coverage and forest sparsity. Edge and component counts are per question.

	Joint		Two-Call
Dataset	Acc.	R@10	Acc.	R@10
ATM-Bench	55	76.10	56	81.38
Mem-Gallery	66	64.10	71	71.68
MemEye	54	49.28	59	49.72
H2HMem	62	22.28	59	20.97
Macro Avg.	59.25	52.94	61.25	55.94

Table 7: Separating direct node utility from anchor-conditioned relation verification on random 100-question held-out subset of each benchmark. Accuracy uses the fixed GPT-5-mini judge protocol.

Figure 4: Qualitative evidence optimization on Mem-Gallery.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Contexts	Memories	Media	Experimental Q	Caption source
ATM-Bench	1	11,034	3,759 I + 533 V	1,044	Released derived fields
Mem-Gallery	20	7,944	1,490 I	1,711	Native captions
MemEye	16	3,392	442 I	1,855	Native memory captions
H2HMem	25	7,078	1,300 I	1,982	Qwen3-VL-8B generated

Appendix

Table 8: Dataset scale and actual experimental sample counts. I and V denote image and video assets. Media counts are unique assets in the unified snapshot.

Figure 5: Complete-lifecycle wall-clock estimates for the three benchmarks not shown in the main-paper ATM analysis. Left panels decompose one full memory digest plus one answer for every evaluation question; right panels pair the same estimated wall clock with Qwen3-VL Judge Accuracy. Dashed curves mark the empirical non-dominated envelope among the compared methods.

Selector	ATM-Bench	Mem-Gallery	MemEye	H2HMem	Avg.
Node Utility Top- $K$	82.27	68.99	58.19	18.04	56.87
MMR	82.27	69.39	57.94	17.76	56.84
Rel.–Red.	82.27	69.12	58.00	17.87	56.82
DPP	76.73	67.12	49.38	17.52	52.69
Facility Location	82.27	68.99	57.80	18.14	56.80
Anchor–Neighbor	82.27	70.43	58.19	18.45	57.33

Appendix

Table 9: Controlled fixed-

K

evidence selection on four held-out sets with frozen candidates and node utilities. Values are Recall@10 (%); Avg. is the dataset macro-average.

Selector	ATM-Bench	Mem-Gallery	MemEye	H2HMem	Total
Node Utility Top- $K$	0	0	0	0	0
MMR	0	+2	$-2$	$-6$	$-6$
Rel.–Red.	0	+1	$-1$	$-4$	$-4$
DPP	$-3$	$-4$	$-16$	$-13$	$-36$
Facility Location	0	0	$-3$	+1	$-2$
Anchor–Neighbor	0	+6	0	+8	+14

Appendix

Table 10: Net low-ranked gold evidence recovered over Node Utility Top-

K

on recoverable held-out questions. Positive values add gold evidence after accounting for any previously selected gold that is displaced.

Multimodal Memory

Multimodal memory lets AI systems retain, organize, and retrieve information across text, images, audio, and video for long-term reasoning and personalization.

See all 71 papers

MEMO: Multimodal Evidence Memory Organization for Long-Horizon LLM Agents
Sep 7, 2026Xian Gao, Jinpeng Wang, Jiacheng Ruan +3Multimodal Memory Long-Horizon LLM Agents
Shanghai Jiao Tong University · Tsinghua University · Institute of Automation of the Chinese Academy of Sciences
Temporal Evidence Forests for Long-Horizon Multi-Party Conversational Memory
Oct 9, 2026Junyou Guo, Chongyang Tao, Yan He +2LLM Agent Memory Conversational Memory
School of Computer Science and Engineering, Beihang University, Beijing, China · Alibaba Group
PMMC: Prospective Multimodal Memory Compilation for Long-Term LVLM Agents
Aug 2, 2026Jingyu Sun, Yan Lin, Yuyang Xue +10Memory-Augmented VLMs Multimodal Memory
The University of Manchester · The University of Newcastle · The University of Edinburgh +5

Sep 7, 2026J/K move · Enter open · S save

MEMO: Multimodal Evidence Memory Organization for Long-Horizon LLM Agents

Xian Gao, Jinpeng Wang, Jiacheng Ruan, Guangyu Cao+2

Shanghai Jiao Tong University · Tsinghua University · Institute of Automation of the Chinese Academy of Sciences

Multimodal Memory Long-Horizon LLM Agents Agent Memory Management Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

Long-running LLM agents rely on external memory to store and reuse information beyond a single context window, yet there is a fundamental tension between the continuous accumulation of interaction trajectories and the limited context capacity. The key challenge in agent memory is therefore not only to retrieve relevant records, but also to select necessary evidence under a given budget and organize it in an appropriate modality. Existing memory readout methods mainly use textual or visual forms. Text preserves high fidelity, but its linear token representation makes contents with different importance compete for the limited context at nearly uniform unit cost. Visual readout renders text into document-like images, which can use two-dimensional layouts to expose structure and emphasize key information, but it may lose fine-grained details during rendering and compression. To address this issue, we propose MEMO, a multimodal evidence memory organization method for LLM agents. MEMO first uses a trained evidence extractor to select relevant memory blocks and form evidence units with source information and presentation requirements. A trained query-conditioned memory manager assigns each unit to a textual, visual, or dual-channel carrier and selects a layout that matches the evidence structure. A deterministic memory construction module then generates the textual package and visual pages. The memory manager is trained with feedback from an offline reader that measures the utility of the guided memory plan, so that retention and presentation decisions align with downstream usage. We evaluate MEMO on four benchmarks, HotpotQA, 2WikiMultiHopQA, LoCoMo, and ALFWorld, with multiple reader backends. The results show that MEMO presents memory more efficiently with fewer memory tokens, improves downstream task performance, and builds more effective working memory under constrained budgets.

Abstract

Multimodal Memory

Explore similar work

MEMO: Multimodal Evidence Memory Organization for Long-Horizon LLM Agents

Temporal Evidence Forests for Long-Horizon Multi-Party Conversational Memory

PMMC: Prospective Multimodal Memory Compilation for Long-Term LVLM Agents