As multimodal agents accumulate increasingly long interaction histories, effectively retrieving and utilizing past information becomes critical for long-term interaction. Existing memory systems typically rely on semantic retrieval over compressed memories and expose predefined representations, which may miss relevant memories and overlook heterogeneous evidence needs across queries. We introduce \hivemem, a multimodal memory framework that interleaves Horizontal Memory Expansion with Vertical Evidence Composition. For horizontal memory expansion, \hivemem organizes memory episodes into a relational graph using textual and visual anchors, expanding semantically retrieved seeds with related neighbors to recover complementary information. For vertical evidence composition, each retrieved memory retains multiple evidence forms, including compact summaries, captions, grounded visual views, raw images, and interaction history. A lightweight trainable selector independently composes an evidence subset for each retrieved memory, enabling query-adaptive utilization of heterogeneous multimodal information. This two-dimensional design jointly broadens memory coverage and adapts the evidence contributed by individual memories. Experiments across three multimodal memory benchmarks show that \hivemem consistently improves task performance across base models while reducing token consumption by about 90% compared with the most efficient competing multimodal baselines.
Figures & tables
Figure 1: Overview of HiVe-Mem . (a) Horizontal Memory Expansion retrieves seed episodes through summary-based semantic matching and expands to graph neighbors ranked by the specificity and query relevance of their connecting anchors. (b) Vertical Evidence Composition uses an RL-trained selector to make binary inclusion decisions over compact summaries, image captions, grounded visual views, raw images, and raw interaction text, composing a query-dependent evidence subset for each retrieved episode to balance answer quality and token consumption.
Mem-Gallery
WorldMemArena
H2HMem
Avg.
Method
F1
LJ
Tok.
F1
LJ
Tok.
F1
LJ
Tok.
F1
LJ
Tok.
Qwen3-VL-4B-Instruct
Text-centric memory
LightMem
27.70
38.55
0.547
21.24
45.00
0.394
16.33
26.40
0.357
21.75
36.65
0.433
SimpleMem
31.76
36.27
0.462
16.31
38.50
0.598
18.30
24.11
0.433
22.12
32.96
0.498
BudgetMem
34.89
42.45
0.264
22.01
56.82
0.361
20.32
29.49
0.272
25.74
42.92
0.299
Table 1: Main results on multimodal memory benchmarks. We report F1 and LLM-Judge (LJ) for task performance, together with total token consumption (Tok.). Avg. denotes the average across three benchmarks. Best and second-best results among multimodal methods are shown in bold and underlined ; text-centric baselines are included for reference only, as they operate under a different modality setting without image processing during memory-bank construction.
Figure 2: Evidence composition under varying token-consumption preferences. Results are averaged across Mem-Gallery, WorldMemArena, and H2HMem using Qwen3-VL-4B-Instruct. (a) Token shares of selected evidence forms, normalized to 100%. (b) Selection rates of individual evidence forms. (c) LJ and token consumption per query under different λ values.
Horizontal Memory Expansion
Vertical Evidence Composition
Variant
LJ
Variant
LJ
Variant
LJ
HiVe-Mem (Full)
57.96
HiVe-Mem (Full)
57.96
HiVe-Mem (Full)
57.96
w/o Horizontal Expansion
56.51
w/o Vertical Composition
44.47
w/o Grounded Visual View
57.19
w/o Visual Anchors
57.30
w/o Compact Summary
56.86
w/o Raw Image
51.02
w/o IDF Weighting
56.55
w/o Image Caption
56.12
w/o Raw Text
49.68
Table 2: Ablation study of HiVe-Mem averaged equally across Mem-Gallery, WorldMemArena, and H2HMEM using Qwen3-VL-4B-Instruct. The best LJ score is in bold .
Figure 3: Latency and hyperparameter sensitivity analysis with Qwen3-VL-4B-Instruct on Mem-Gallery. (a) Time to first token (TTFT) across methods on a logarithmic scale. (b) Effect of graph degree Kedge on LJ with default Kret:Kexp=5:2 . (c) Effect of retrieval allocation on LJ with default Kret+Kexp=7 and Kedge=4 .
Figure 4: Evidence selection across query types. Using Qwen3-VL-4B-Instruct at λ=0 , we group the native query categories of each benchmark into (a) text-dominant, (b) visual-dominant, and (c) cross-modal queries, and report the selection rate of each evidence form. Rates are computed over all retrieved Top-7 memories and averaged equally across Mem-Gallery, WorldMemArena, and H2HMem. Error bars show ±1 population standard deviation across benchmarks.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Description
Embedding and Retrieval
Embedding model
Qwen3-VL-Embedding-2B
Multimodal encoder for queries and memory summaries.
Embedding dimension
2048
Dimension of embedding representations.
Kret
5
Number of memories obtained by semantic retrieval.
Kexp
2
Number of memories added by horizontal expansion.
Evidence Selector
Appendix
Table 3: Key hyperparameters used in our experiments.
Figure 5: Evidence composition under varying token-consumption preferences with GPT-5-mini. Results are averaged across Mem-Gallery, WorldMemArena, and H2HMem using the transferred evidence selector. (a) Token shares of selected evidence forms, normalized to 100%. (b) Selection rates of individual evidence forms. (c) LJ and token consumption per query under increasing λ . Displayed settings are equally spaced in increasing λ order.
Figure 6: Evidence selection across query types with GPT-5-mini. Native query categories are grouped into (a) text-dominant, (b) visual-dominant, and (c) cross-modal queries. We report evidence-selection rates using the transferred selector at λ=0 . Rates are averaged equally across Mem-Gallery, WorldMemArena, and H2HMem, with error bars showing ±1 population standard deviation across benchmarks.
Multimodal memory lets AI systems retain, organize, and retrieve information across text, images, audio, and video for long-term reasoning and personalization.
Long-term memory is essential for LVLM agents to maintain consistency and integrate information across extended multimodal interactions. Existing agent memory systems, however, often reduce visual experiences into textual summaries or rely on static retrieve-then-reason pipelines, which are inefficient at query time and brittle when questions require image-text binding, temporal updates, or visual details. We propose Prospective Multimodal Memory Compilation, a framework that shifts part of the memory reasoning process from query time to memory consolidation time. Given accumulated multimodal interactions, a Questioner predicts future question candidates, a Planner compiles question-conditioned multimodal memory programs, and a Doubter verifies whether the planned evidence path can support the predicted answer. The verified question-program pairs form a structured question bank for efficient query-time routing and evidence retrieval. Experiments on multimodal long-term memory benchmarks show that our method improves answer quality and visual evidence recall while reducing query-time token and latency costs. Extensive ablations analyze the effects of self-feedback, dynamic planning, raw-image access, and question bank coverage.
Jingyu Sun, Yan Lin, Yuyang Xue +10
The University of Manchester · The University of Newcastle · The University of Edinburgh +5
Organizing long-term memory for multimodal agents remains challenging because existing methods either suffer from expensive question-agnostic offline summaries or naive embedding similarity matching that introduces incomplete and redundant context. To address these issues, we propose GraphMemix, a combinatorial-optimization graph memory framework that models memory organization as query-aware evidence-forest construction. Specifically, our method consists of three key components:(1) candidate graph construction, which expands multi-view seed memories through schema and semantic relations to acquire query-aware original context; (2) evidence utility and activation costs, which decouples direct memory support from anchor-conditioned relation verification to suppress redundant or conflicting information; and (3) forest optimization, which jointly selects a forest-format memory context under a maximum evidence budget and its reliable relational structure. By organizing memory into a query-relevant subgraph, the method avoids substantial lifecycle cost and recovers low-similarity complementary evidence. Experimental results across four long-term multimodal memory benchmarks demonstrate significant improvements with different foundation models and establish a new Pareto frontier between accuracy and lifecycle cost.
Geng Li, Yuhao Wang, Dong Li +2
Wangxuan Institute of Computer Technology, Peking University · MemoraX AI
Long-running LLM agents rely on external memory to store and reuse information beyond a single context window, yet there is a fundamental tension between the continuous accumulation of interaction trajectories and the limited context capacity. The key challenge in agent memory is therefore not only to retrieve relevant records, but also to select necessary evidence under a given budget and organize it in an appropriate modality. Existing memory readout methods mainly use textual or visual forms. Text preserves high fidelity, but its linear token representation makes contents with different importance compete for the limited context at nearly uniform unit cost. Visual readout renders text into document-like images, which can use two-dimensional layouts to expose structure and emphasize key information, but it may lose fine-grained details during rendering and compression. To address this issue, we propose MEMO, a multimodal evidence memory organization method for LLM agents. MEMO first uses a trained evidence extractor to select relevant memory blocks and form evidence units with source information and presentation requirements. A trained query-conditioned memory manager assigns each unit to a textual, visual, or dual-channel carrier and selects a layout that matches the evidence structure. A deterministic memory construction module then generates the textual package and visual pages. The memory manager is trained with feedback from an offline reader that measures the utility of the guided memory plan, so that retention and presentation decisions align with downstream usage. We evaluate MEMO on four benchmarks, HotpotQA, 2WikiMultiHopQA, LoCoMo, and ALFWorld, with multiple reader backends. The results show that MEMO presents memory more efficiently with fewer memory tokens, improves downstream task performance, and builds more effective working memory under constrained budgets.
Xian Gao, Jinpeng Wang, Jiacheng Ruan +3
Shanghai Jiao Tong University · Tsinghua University · Institute of Automation of the Chinese Academy of Sciences