Long-term memory for multimodal agents remains challenging, as it must preserve fine-grained visual evidence while tracking entity evolution across sessions. Existing abstraction-based memories compress images into text, often discarding visually discriminative details, whereas pixel-preserving memories retain raw observations but struggle to reliably associate diverse observations with persistent entity identities and track their evolving states. To address these limitations, we propose MHG-Mem, a multimodal hypergraph memory that jointly organizes entity identity, state evolution, and provenance to visual evidence. Specifically, MHG-Mem represents distinct entity mentions and their observed facts as individual nodes, grouping identity-consistent mentions and facts through occurrence hyperedges. It then synthesizes aspect-specific evolution states from these grouped observations and links them to supporting entities, facts, and source evidence through evolutionary and evidential hyperedges. To navigate this structured memory, we introduce hypergraph-based agentic search, which adaptively selects an entry layer, traverses typed hyperedges across memory levels, and inspects source observations when fine-grained verification is required. These designs support both detailed visual verification and reasoning over cross-session state changes. Extensive experiments across three benchmarks demonstrate the effectiveness of MHG-Mem, improving performance by 8.20-9.98% over the strongest baseline.
Figures & tables
Figure 1: (a) Long-term agent–user multimodal interaction example. (b) Comparison of existing approaches: (i) Abstraction-based memory loses visual details (failing Q1, Q2); (ii) Pixel-preserving memory lacks persistent identity–evolution bindings (failing Q2); (iii) Our MHG-Mem organizes multimodal observations via typed hyperedges, preserving both fine-grained visual crops for inspection and traceable evolution across diverse entity mentions.
Figure 2: Overview of MHG-Mem . (a) Memory construction proceeds through three steps. (b) The three-layer MHG-Mem architecture and three types of hyperedges enable cross-layer traversal and provenance tracing. (c) Hypergraph-based agentic search employs four types of tools to adaptively construct query-aware evidence chains.
Type
Method
Venue
MemEye
Mem-Gallery
WorldMemArena-S
Avg. J.
EM
B-1
Judge
F1
B-1
Judge
F1
B-1
Judge
Qwen3-VL-8B
Unimodal
A-Mem
NIPS’25
45.15
19.22
29.38
47.68
41.76
66.93
25.23
12.16
46.57
47.63
MemoryOS
EMNLP’25
40.23
17.22
26.28
39.36
35.28
58.86
10.24
3.50
20.18
35.11
SimpleMem
ICML’26
44.27
16.04
31.26
37.36
33.12
52.43
14.33
4.71
33.50
39.06
HyperMem
ACL’26
42.31
16.84
28.42
41.05
32.80
62.56
20.06
12.33
40.45
43.81
Table 1: Comparison with state-of-the-art memory systems across different backbone models. We report Exact Match ( EM ), BLEU-1 ( B-1 ), and LLM-Judge ( Judge ) on MemEye , Mem-Gallery , and WorldMemArena-S . Avg. J. denotes the average Judge score across the three benchmarks. Bold and underlined results indicate the best and second-best performance, respectively.
Figure 3: (a) Memory quality and (b) fine-grained tasks performance on WorldMemArena-S (WMA-S) and Mem-Gallery with Qwen3.8-27B.
Figure 4: Case study in a home renovation scenario. MHG-Mem tracks wall paint color changes through evolution nodes and traces supporting facts to source images via evidential hyperedges, forming a verifiable evidence chain.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Meaning
Symbol
Meaning
D≤T
Ordered multimodal history
dt
Interaction round t
xt
Dialogue text in round t
It
Images in round t
δt
Date of round t
Ct
Recent within-session context
q=(xq,Iq)
Multimodal query
W,S
Memory construction and retrieval operators
Eq,y^
Retrieved evidence and generated answer
θ
Fixed model parameters
H
Attributed multimodal hypergraph
MT
Persistent memory at history length T
Appendix
Table 6
Type
Tool
Input
Returned information
Hypergraph
resolve_facts
Text query w
Ranked facts and source references.
resolve_entity
Entity name or query image
Candidate referent groups and corresponding occurrence linking facts.
resolve_evolution
Referent name, query image, or evolution_id
Evolutions and evidential linking facts.
Visual
search_image
Text query or query image
Matching images.
Inspection
get_raw_verbatim
Source-round identifier
Original dialogue and images.
Assembly
submit_evidence
Fact and image references; optional summary
Validation feedback; accepted evidence forms Eq .
Appendix
Table 4: Four tool types for hypergraph-based agentic search. Query images, when supplied, are available to the visual lookups.
Capability
Mem-Gallery
WorldMemArena-S
Fact recall
FR (Factual Retrieval): retrieve an explicit fact from the conversation.
FR (Fact Recall): retrieve one concrete fact stated earlier in the trajectory.
Multi-entity reasoning
MR (Multi-entity Reasoning): combine retrieved information about multiple entities to infer an answer.
–
Temporal reasoning
TR (Temporal Reasoning): resolve a time-dependent question.
TR (Temporal Reasoning): reason about timing, ordering, or duration.
Visual or cross-modal reasoning
VR (Visual-centric Reasoning): use visual information in addition to the dialogue.
CMR (Cross-modal Reasoning): combine textual and visual memory.
Test-time learning
TTL (Test-time Learning): learn visual knowledge from historical images and apply it during QA.
TTL (Test-Time Learning): apply a rule or skill taught earlier in the trajectory.
Visual search
VS (Visual-centric Search): find the image(s) matching a query and return their image IDs.
VS (Visual Search): locate an object or attribute across visual memory.
Appendix
Table 5: Fine-grained task definitions used by Mem-Gallery and WorldMemArena-S. A dash indicates that the benchmark has no directly corresponding task label.
Figure 5: Average tool utilization on WorldMemArena-S with Qwen3.8-27B, reported overall and by 7 task types: OA (Overall Average), TTL (Test-time Learning), TR (Temporal Reasoning), CMR (Cross-modal Reasoning), KR (Knowledge Reasoning), DU (Dynamic Updating), VU (Visual Update), and MC (Memory Conflict).
Figure 6: Trade-off between token consumption and QA performance among different baselines. w/o A.S. represents replace MHG-Mem’s agentic search with top-k retrieval.
Figure 7: (a) Memory quality and (b) fine-grained tasks performance on WorldMemArena-S (WMA-S) and Mem-Gallery with Qwen3-VL-8B.
Figure 8: (a) Memory quality and (b) fine-grained tasks performance on WorldMemArena-S (WMA-S) and Mem-Gallery with GPT-4.1-mini.
Figure 9: Case study in an embodied agent scenario. MHG-Mem records the faucet’s state throughout the multimodal agent’s action history.
Figure 10: Case study in a GUI agent scenario. MHG-Mem records the count state of oak log in Minecraft throughout the multimodal agent’s action history.
Multimodal agents operating in long-horizon environments must build and continually update multimedia memories to support entity-consistent, temporally grounded reasoning. However, existing agentic memory approaches often discard fine-grained dentity cues under aggressive compression and segment-wise processing. They also rely heavily on vector similarity retrieval, which can surface semantically related yet identity-mismatched evidence, leading to entity confusion, error propagation, and hallucinated answers. We propose ViSAGE, a multimodal agentic memory framework that constructs self-correcting, entity-centric memories. Specifically, ViSAGE anchors entity identity via cross-modal binding over long temporal ranges. It then applies bidirectional memory refinement to propagate delayed identity evidence, retroactively unifying historical records and improving future reasoning. We also introduce multi-agent cross-verification to assess retrieved evidence under an identity-evidence alignment onstraint, enabling abstention instead of unsupported answers when evidence is missing. Extensive results demonstrate that ViSAGE consistently outperforms the strongest baseline, achieving 5.9% higher accuracy.
Xinkui Zhao, Enbo Chen, Yifan Zhang +4
Zhejiang University Hangzhou, China · Xidian University Xi’an, China
Organizing long-term memory for multimodal agents remains challenging because existing methods either suffer from expensive question-agnostic offline summaries or naive embedding similarity matching that introduces incomplete and redundant context. To address these issues, we propose GraphMemix, a combinatorial-optimization graph memory framework that models memory organization as query-aware evidence-forest construction. Specifically, our method consists of three key components:(1) candidate graph construction, which expands multi-view seed memories through schema and semantic relations to acquire query-aware original context; (2) evidence utility and activation costs, which decouples direct memory support from anchor-conditioned relation verification to suppress redundant or conflicting information; and (3) forest optimization, which jointly selects a forest-format memory context under a maximum evidence budget and its reliable relational structure. By organizing memory into a query-relevant subgraph, the method avoids substantial lifecycle cost and recovers low-similarity complementary evidence. Experimental results across four long-term multimodal memory benchmarks demonstrate significant improvements with different foundation models and establish a new Pareto frontier between accuracy and lifecycle cost.
Geng Li, Yuhao Wang, Dong Li +2
Wangxuan Institute of Computer Technology, Peking University · MemoraX AI
Long-term agent memory is increasingly multimodal, yet existing evaluations rarely test whether agents preserve the visual evidence needed for later reasoning. In prior work, many visually grounded questions can be answered using only captions or textual traces, allowing answers to be inferred without preserving the fine-grained visual evidence. Meanwhile, harder cases that require reasoning over changing visual states are largely absent. Therefore, we introduce MemEye, a framework that evaluates memory capabilities from two dimensions: one measures the granularity of decisive visual evidence (from scene-level to pixel-level evidence), and the other measures how retrieved evidence must be used (from single evidence to evolutionary synthesis). Under this framework, we construct a new benchmark across 8 life-scenario tasks, with ablation-driven validation gates for assessing answerability, shortcut resistance, visual necessity, and reasoning structure. By evaluating 13 memory methods across 4 VLM backbones, we show that current architectures still struggle to preserve fine-grained visual details and reason about state changes over time. Our findings show that long-term multimodal memory depends on evidence routing, temporal tracking, and detail extraction.