Remembering Through Change: Provenance-Preserving Hypergraph Memory for Multimodal Agents

Jielong Tang; Ling-An Zeng; Xujie Yuan; Xiao Dong; Zhiqian Xia; Qingfeng Li; Shixiong Kai; Kaichao Liang

cs.CLOct 21, 2026

Remembering Through Change: Provenance-Preserving Hypergraph Memory for Multimodal Agents

Authors: Jielong Tang, Ling-An Zeng, Xujie Yuan, Xiao Dong, Zhiqian Xia, Qingfeng Li, Shixiong Kai, Kaichao Liang,

Organizations: Sun Yat-sen University · Huawei Technologies, Noah Arks’ Lab · Shandong Normal University · Southeast University

Abstract

Long-term memory for multimodal agents remains challenging, as it must preserve fine-grained visual evidence while tracking entity evolution across sessions. Existing abstraction-based memories compress images into text, often discarding visually discriminative details, whereas pixel-preserving memories retain raw observations but struggle to reliably associate diverse observations with persistent entity identities and track their evolving states. To address these limitations, we propose MHG-Mem, a multimodal hypergraph memory that jointly organizes entity identity, state evolution, and provenance to visual evidence. Specifically, MHG-Mem represents distinct entity mentions and their observed facts as individual nodes, grouping identity-consistent mentions and facts through occurrence hyperedges. It then synthesizes aspect-specific evolution states from these grouped observations and links them to supporting entities, facts, and source evidence through evolutionary and evidential hyperedges. To navigate this structured memory, we introduce hypergraph-based agentic search, which adaptively selects an entry layer, traverses typed hyperedges across memory levels, and inspects source observations when fine-grained verification is required. These designs support both detailed visual verification and reasoning over cross-session state changes. Extensive experiments across three benchmarks demonstrate the effectiveness of MHG-Mem, improving performance by 8.20-9.98% over the strongest baseline.

Figures & tables

Figure 1: (a) Long-term agent–user multimodal interaction example. (b) Comparison of existing approaches: (i) Abstraction-based memory loses visual details (failing Q1, Q2); (ii) Pixel-preserving memory lacks persistent identity–evolution bindings (failing Q2); (iii) Our MHG-Mem organizes multimodal observations via typed hyperedges, preserving both fine-grained visual crops for inspection and traceable evolution across diverse entity mentions.

Figure 2: Overview of MHG-Mem . (a) Memory construction proceeds through three steps. (b) The three-layer MHG-Mem architecture and three types of hyperedges enable cross-layer traversal and provenance tracing. (c) Hypergraph-based agentic search employs four types of tools to adaptively construct query-aware evidence chains.

Type	Method	Venue	MemEye			Mem-Gallery			WorldMemArena-S			Avg. J.
Type	Method	Venue	EM	B-1	Judge	F1	B-1	Judge	F1	B-1	Judge	Avg. J.
Qwen3-VL-8B
Unimodal	A-Mem	NIPS’25	45.15	19.22	29.38	47.68	41.76	66.93	25.23	12.16	46.57	47.63
	MemoryOS	EMNLP’25	40.23	17.22	26.28	39.36	35.28	58.86	10.24	3.50	20.18	35.11
	SimpleMem	ICML’26	44.27	16.04	31.26	37.36	33.12	52.43	14.33	4.71	33.50	39.06
	HyperMem	ACL’26	42.31	16.84	28.42	41.05	32.80	62.56	20.06	12.33	40.45	43.81

Table 1: Comparison with state-of-the-art memory systems across different backbone models. We report Exact Match ( EM ), BLEU-1 ( B-1 ), and LLM-Judge ( Judge ) on MemEye , Mem-Gallery , and WorldMemArena-S . Avg. J. denotes the average Judge score across the three benchmarks. Bold and underlined results indicate the best and second-best performance, respectively.

Figure 3: (a) Memory quality and (b) fine-grained tasks performance on WorldMemArena-S (WMA-S) and Mem-Gallery with Qwen3.8-27B.

Figure 4: Case study in a home renovation scenario. MHG-Mem tracks wall paint color changes through evolution nodes and traces supporting facts to source images via evidential hyperedges, forming a verifiable evidence chain.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

$Symbol$	Meaning	$Symbol$	Meaning
$\mathcal{D}_{\leq T}$	Ordered multimodal history	$d_{t}$	Interaction round $t$
$x_{t}$	Dialogue text in round $t$	$\mathcal{I}_{t}$	Images in round $t$
$\delta_{t}$	Date of round $t$	$\mathcal{C}_{t}$	Recent within-session context
$q=(x_{q},\mathcal{I}_{q})$	Multimodal query	$\mathcal{W},\mathcal{S}$	Memory construction and retrieval operators
$E_{q},\hat{y}$	Retrieved evidence and generated answer	$\theta$	Fixed model parameters
$\mathcal{H}$	Attributed multimodal hypergraph	$\mathcal{M}_{T}$	Persistent memory at history length $T$

Appendix

Table 6

Type	Tool	Input	Returned information
Hypergraph	resolve_facts	Text query $w$	Ranked facts and source references.
	resolve_entity	Entity name or query image	Candidate referent groups and corresponding occurrence linking facts.
	resolve_evolution	Referent name, query image, or evolution_id	Evolutions and evidential linking facts.
Visual	search_image	Text query or query image	Matching images.
Inspection	get_raw_verbatim	Source-round identifier	Original dialogue and images.
Assembly	submit_evidence	Fact and image references; optional summary	Validation feedback; accepted evidence forms $E_{q}$ .

Appendix

Table 4: Four tool types for hypergraph-based agentic search. Query images, when supplied, are available to the visual lookups.

Capability	Mem-Gallery	WorldMemArena-S
Fact recall	FR (Factual Retrieval): retrieve an explicit fact from the conversation.	FR (Fact Recall): retrieve one concrete fact stated earlier in the trajectory.
Multi-entity reasoning	MR (Multi-entity Reasoning): combine retrieved information about multiple entities to infer an answer.	–
Temporal reasoning	TR (Temporal Reasoning): resolve a time-dependent question.	TR (Temporal Reasoning): reason about timing, ordering, or duration.
Visual or cross-modal reasoning	VR (Visual-centric Reasoning): use visual information in addition to the dialogue.	CMR (Cross-modal Reasoning): combine textual and visual memory.
Test-time learning	TTL (Test-time Learning): learn visual knowledge from historical images and apply it during QA.	TTL (Test-Time Learning): apply a rule or skill taught earlier in the trajectory.
Visual search	VS (Visual-centric Search): find the image(s) matching a query and return their image IDs.	VS (Visual Search): locate an object or attribute across visual memory.

Appendix

Table 5: Fine-grained task definitions used by Mem-Gallery and WorldMemArena-S. A dash indicates that the benchmark has no directly corresponding task label.

Figure 5: Average tool utilization on WorldMemArena-S with Qwen3.8-27B, reported overall and by 7 task types: OA (Overall Average), TTL (Test-time Learning), TR (Temporal Reasoning), CMR (Cross-modal Reasoning), KR (Knowledge Reasoning), DU (Dynamic Updating), VU (Visual Update), and MC (Memory Conflict).

Figure 6: Trade-off between token consumption and QA performance among different baselines. w/o A.S. represents replace MHG-Mem’s agentic search with top-k retrieval.

Figure 7: (a) Memory quality and (b) fine-grained tasks performance on WorldMemArena-S (WMA-S) and Mem-Gallery with Qwen3-VL-8B.

Figure 8: (a) Memory quality and (b) fine-grained tasks performance on WorldMemArena-S (WMA-S) and Mem-Gallery with GPT-4.1-mini.

Figure 9: Case study in an embodied agent scenario. MHG-Mem records the faucet’s state throughout the multimodal agent’s action history.

Figure 10: Case study in a GUI agent scenario. MHG-Mem records the count state of oak log in Minecraft throughout the multimodal agent’s action history.

Agent Memory

Agent memory lets AI systems retain, update, and use facts, experiences, and skills across sessions and long-horizon tasks.

See all 190 papers

ViSAGE: Constructing Self-Correcting Memories for Long-Form Video Understanding
Jul 29, 2026Xinkui Zhao, Enbo Chen, Yifan Zhang +4Agent Memory Multimodal Memory
Zhejiang University Hangzhou, China · Xidian University Xi’an, China
GraphMemix: Query-Aware Evidence Forests for Long-Term Multimodal Agent Memory
Aug 27, 2026Geng Li, Yuhao Wang, Dong Li +2Multimodal Memory Agent Memory Management
Wangxuan Institute of Computer Technology, Peking University · MemoraX AI
MemEye: A Visual-Centric Evaluation Framework for Multimodal Agent Memory
May 14, 2026Minghao Guo, Qingyue Jiao, Zeru Shi +14Multimodal Agent Evaluation Multimodal Memory
Rutgers · Notre Dame · Princeton +1

Jul 29, 2026J/K move · Enter open · S save

ViSAGE: Constructing Self-Correcting Memories for Long-Form Video Understanding

Xinkui Zhao, Enbo Chen, Yifan Zhang, Chang Liu+3

Zhejiang University Hangzhou, China · Xidian University Xi’an, China

Agent Memory Multimodal Memory Long-Video Understanding

Open paper AskPDF arXiv

Multimodal agents operating in long-horizon environments must build and continually update multimedia memories to support entity-consistent, temporally grounded reasoning. However, existing agentic memory approaches often discard fine-grained dentity cues under aggressive compression and segment-wise processing. They also rely heavily on vector similarity retrieval, which can surface semantically related yet identity-mismatched evidence, leading to entity confusion, error propagation, and hallucinated answers. We propose ViSAGE, a multimodal agentic memory framework that constructs self-correcting, entity-centric memories. Specifically, ViSAGE anchors entity identity via cross-modal binding over long temporal ranges. It then applies bidirectional memory refinement to propagate delayed identity evidence, retroactively unifying historical records and improving future reasoning. We also introduce multi-agent cross-verification to assess retrieved evidence under an identity-evidence alignment onstraint, enabling abstention instead of unsupported answers when evidence is missing. Extensive results demonstrate that ViSAGE consistently outperforms the strongest baseline, achieving 5.9% higher accuracy.

Abstract

Agent Memory

Explore similar work

ViSAGE: Constructing Self-Correcting Memories for Long-Form Video Understanding

GraphMemix: Query-Aware Evidence Forests for Long-Term Multimodal Agent Memory

MemEye: A Visual-Centric Evaluation Framework for Multimodal Agent Memory