HiVe-Mem: Interleaving Horizontal Memory Expansion and Vertical Evidence Composition for Multimodal Agents
Organizations: Nanyang Technological University · Tsinghua University · Sun Yat-sen University · Aumovio Singapore Pte. Ltd.
Abstract
As multimodal agents accumulate increasingly long interaction histories, effectively retrieving and utilizing past information becomes critical for long-term interaction. Existing memory systems typically rely on semantic retrieval over compressed memories and expose predefined representations, which may miss relevant memories and overlook heterogeneous evidence needs across queries. We introduce \hivemem, a multimodal memory framework that interleaves Horizontal Memory Expansion with Vertical Evidence Composition. For horizontal memory expansion, \hivemem organizes memory episodes into a relational graph using textual and visual anchors, expanding semantically retrieved seeds with related neighbors to recover complementary information. For vertical evidence composition, each retrieved memory retains multiple evidence forms, including compact summaries, captions, grounded visual views, raw images, and interaction history. A lightweight trainable selector independently composes an evidence subset for each retrieved memory, enabling query-adaptive utilization of heterogeneous multimodal information. This two-dimensional design jointly broadens memory coverage and adapts the evidence contributed by individual memories. Experiments across three multimodal memory benchmarks show that \hivemem consistently improves task performance across base models while reducing token consumption by about 90% compared with the most efficient competing multimodal baselines.
Figures & tables
| Mem-Gallery | WorldMemArena | H2HMem | Avg. | |||||||||
| Method | F1 | LJ | Tok. | F1 | LJ | Tok. | F1 | LJ | Tok. | F1 | LJ | Tok. |
| Qwen3-VL-4B-Instruct | ||||||||||||
| Text-centric memory | ||||||||||||
| LightMem | 27.70 | 38.55 | 0.547 | 21.24 | 45.00 | 0.394 | 16.33 | 26.40 | 0.357 | 21.75 | 36.65 | 0.433 |
| SimpleMem | 31.76 | 36.27 | 0.462 | 16.31 | 38.50 | 0.598 | 18.30 | 24.11 | 0.433 | 22.12 | 32.96 | 0.498 |
| BudgetMem | 34.89 | 42.45 | 0.264 | 22.01 | 56.82 | 0.361 | 20.32 | 29.49 | 0.272 | 25.74 | 42.92 | 0.299 |
| Horizontal Memory Expansion | Vertical Evidence Composition | ||||
| Variant | LJ | Variant | LJ | Variant | LJ |
| HiVe-Mem (Full) | 57.96 | HiVe-Mem (Full) | 57.96 | HiVe-Mem (Full) | 57.96 |
| w/o Horizontal Expansion | 56.51 | w/o Vertical Composition | 44.47 | w/o Grounded Visual View | 57.19 |
| w/o Visual Anchors | 57.30 | w/o Compact Summary | 56.86 | w/o Raw Image | 51.02 |
| w/o IDF Weighting | 56.55 | w/o Image Caption | 56.12 | w/o Raw Text | 49.68 |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Value | Description |
| Embedding and Retrieval | ||
| Embedding model | Qwen3-VL-Embedding-2B | Multimodal encoder for queries and memory summaries. |
| Embedding dimension | 2048 | Dimension of embedding representations. |
| 5 | Number of memories obtained by semantic retrieval. | |
| 2 | Number of memories added by horizontal expansion. | |
| Evidence Selector | ||
Multimodal Memory
Multimodal memory lets AI systems retain, organize, and retrieve information across text, images, audio, and video for long-term reasoning and personalization.
See all 73 papers