Language models (LMs) and their extension, vision-language models (VLMs), have achieved remarkable performance across various tasks. However, they still struggle with complex reasoning tasks that require multimodal or multilingual real-world knowledge. To support such capabilities, an external memory system that can efficiently provide relevant multimodal information is essential. Existing approaches generally concatenate image and text tokens into a long sequence as memory, which, however, may drastically increase context length and even degrade performance. In contrast, we propose using continuous memory, a compact set of dense embeddings to more effectively and efficiently represent multimodal and multilingual knowledge. Our key insight is that a VLM can serve as its own continuous memory encoder. We empirically show that this design improves performance on complex multimodal reasoning tasks. Building on this, we introduce a data-efficient and parameter-efficient method to fine-tune the VLM into a memory encoder, requiring only 1.2% of the model's parameters and a small corpus of 15.6K self-synthesized samples. Our approach CoMEM utilizes VLM's original capabilities to encode arbitrary multimodal and multilingual knowledge into just 8 continuous embeddings. Since the inference-time VLM remains frozen, our memory module is plug-and-play and can be flexibly integrated as needed. Extensive experiments across eight multimodal reasoning benchmarks demonstrate the effectiveness of our approach.
Figures & tables
Figure 1: CoMEM architecture in comparison to the traditional RAG method.
Properties
Scenarios
Training Cost
Category
Method
Continuous
Pulg-and-Play
Multimodal
Multilingual
Data
Parameters
Multimodal-RAG
EchoSight Yan and Xie (2024)
✗
✓
✗Text
✗
900K
300M
ReflectiVA Cocchi et al. (2024)
✗
✗
✓Image+Text
✗
6.82M
8B
RoRA-VLM Qi et al. (2024)
✗
✗
✓Image+Text
✗
1M
7B
Context-Compression
xRAG Cheng et al. (2024)
✓
✓
✗Text
✗
3M
40M
KV-Distill Chari et al. (2025)
✗
✓
✗Text
✗
500K+
150M
Table 1: Comparison between our approach and other representative line of work.
InfoSeek
OKVQA
AOKVQA
Backbone Model
Method
Query
Entity
All
Qwen2.5-VL-Instruct
-
22.5
22.4
22.5
35.0
39.8
+RAG
17.7
18.8
18.2
31.3
34.9
+FastV
26.2
22.6
24.2
31.5
34.9
+VLM-as-Memory
29.3
28.0
28.6
37.3
44.4
+VLM-as-Memory+AS
30.0
25.3
27.5
37.9
41.8
Table 2: Comparison of training-free memory methods. Bold indicates the best performance. For VLM-as-Memory here, we use the cache KV from a VLM without fine-tuning, which differs from the main method described in Section 3 and is intended for preliminary exploration.
Figure 2: Qwen2.5 accuracy with varying attention-based compression rates.
Figure 3: Overview of the CoMEM architecture. Given a vision-language query, the system retrieves relevant multimodal knowledge via visual features. Retrieved image-text pairs are processed by a Memory Encoder—which consists of a VLM and Q-Former—to generate a dense continuous memory. This memory and the original query are fed into a frozen LM to produce accurate, grounded answers.
InfoSeek
OVEN
MRAG
OKVQA
AOKVQA
ViQuAE
Avg.
Model
Q
E
Q
E
LLaVA-v1.5
8.3
8.9
20.0
3.4
34.6
17.0
17.4
11.1
15.1
LLaVA-v1.6
10.3
9.1
17.9
1.8
33.4
31.4
31.7
18.7
19.3
LLaMA3
10.7
8.6
16.8
0.8
33.5
23.7
25.3
17.2
17.1
InternLM-2.5vl
13.4
10.8
14.5
3.3
34.8
29.1
32.8
29.7
19.5
mPLUG-Owl3
9.6
6.4
20.7
1.9
45.0
31.9
33.0
23.1
21.4
Table 3: Performance comparison with three types of baselines on knowledge-intensive VQA benchmarks. Bold indicates the best performance, and underscore denotes the second-best.
Language
Method
Qwen2.5-Instruct-VL
Qwen2-Instruct-VL
Multilingual InfoSeek
CVQA
Multilingual InfoSeek
CVQA
Unseen-Q
Unseen-E
All
Unseen-Q
Unseen-E
All
Chinese
-
17.4
13.8
15.4
82.32
15.1
10.9
12.6
74.60
+ RAG
14.8
9.8
11.8
74.60
11.5
8.9
10.1
72.35
+ CoMEM
22.5
21.5
22.0
78.46
23.1
19.5
21.1
73.31
Russian
-
15.6
14.7
15.1
66.50
13.0
13.8
13.4
71.00
Table 4: Performance comparison on Multilingual knowledge-intensive VQA benchmarks.
Figure 4: Comparison of Long Context Ability of RAG and Ours on Infoseek.
InfoSeek(%)
OVEN(%)
LLM
Unseen-Q
Unseen-E
All
Query
Entity
All
Avg.
Qwen2.5-Instruct
5.0
4.8
4.9
2.4
0.1
1.3
3.1
Qwen2.5-Instruct + RAG
13.4
10.3
11.9
1.8
2.7
2.2
7.0
Qwen2.5-Instruct + CoMEM (using VLM)
29.3
27.4
28.3
6.8
7.7
7.2
17.8
Table 5: Transferability Study of vision-language memory encoded by CoMEM on LLMs
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Metric
4-Emb
8-Emb
16-Emb
24-Emb
Unseen Question Score
29.32%
32.80%
31.95%
31.37%
Unseen Entity Score
29.64%
32.33%
30.03%
30.83%
Final Score
29.48%
32.74%
30.96%
31.10%
Appendix
Table 6: Performance on Infoseek of Different Embedding Sizes
Model
Method
BLEU-1
BLEU-4
METEOR
CIDEr
ROUGE-L
BERTScore-F
Trigram Diversity
qwen2.5-VL
Original
0.26
0.04
0.18
0.24
0.26
0.81
0.92
+RAG
0.30
0.05
0.20
0.45
0.27
0.81
0.94
+CoMEM
0.34
0.07
0.21
0.64
0.32
0.81
0.86
qwen2-VL
Original
0.34
0.08
0.21
0.79
0.34
0.81
0.84
+RAG
0.34
0.07
0.20
0.64
0.32
0.81
0.83
+CoMEM
0.36
0.08
0.23
0.76
0.32
0.82
0.88
Appendix
Table 7: Evaluation Metrics for Qwen2.5-VL and Qwen2-VL on COCO Captioning task
Metrics
Method
Accounting
Architecture
Clinical Med
Computer
Economics
Electronics
Management
Materials
Pharmacy
Accuracy
Qwen2.5-VL
26.7
23.3
43.3
26.7
46.7
23.3
40.0
30.0
30.0
+RAG
23.3
26.7
40.0
33.3
33.3
16.7
36.7
33.3
40.0
+CoMEM
40.0
26.7
43.3
40.0
46.7
30.0
36.7
33.3
40.0
Similarity
Qwen2.5-VL
26.7
26.7
43.3
26.7
46.7
26.7
40.0
30.0
30.0
+RAG
23.3
26.7
40.0
33.3
33.3
16.7
36.7
33.3
40.0
+CoMEM
40.0
33.3
43.3
40.0
50.0
30.0
43.3
33.3
43.3
Appendix
Table 8: Performance Comparison on Reasoning Task MMMU
Metrics
Method
Math
Textbook QA
Visual QA
Figure QA
Accuracy
Qwen2.5-VL
59
50
34
46
+RAG
63
53
37
54
+CoMEM
65
46
39
56
Similarity
Qwen2.5-VL
62
55
35
48
+RAG
63
57
37
56
+CoMEM
66
49
41
57
Appendix
Table 9: Performance Comparison on Reasoning Task MathVista
Token per Second
InfoSeek
OVEN
MRAG
OKVQA
AOKVQA
ViQuAE
Qwen2 models
Qwen2-VL
55.93
100.91
39.54
45.81
53.38
99.53
Qwen2-VL+RAG
40.71
49.27
8.38
34.66
23.60
37.50
Qwen2-VL+COMEM
48.93
53.29
18.61
40.42
51.60
41.50
Qwen2.5 models
Qwen2.5-VL
53.44
72.71
42.13
45.64
42.87
53.49
Appendix
Table 10: Inference Latency Evaluation accross different benchmarks
Token Number (Accuracy%)
Infoseek
Oven
MRAG
OKVQA
AOKVQA
ViQuAE
Qwen2-VL
357.6 (17.8)
408.6 (25.5)
84.5 (39.3)
404.8 (36.3)
418.4 (41.8)
424.7 (34.5)
Qwen2-VL+RAG
6163.8 (19.0)
6481.5 (24.7)
5882.4 (40.4)
6803.8 (41.9)
6715.4 (45.3)
6943.2 (33.6)
Qwen2-VL+COMEM
416.8 (33.1)
492.9 (30.5)
176.9 (35.1)
453.8 (57.7)
478.3 (60.6)
489.2 (36.3)
Appendix
Table 11: Token Number and Accuracy Across Datasets
Infoseek
Training Settings
Unseen-Q
Unseen-E
All
Original
32.8
28.5
30.7
Data
4x
34.8
28.4
31.3
2x
32.2
29.8
30.9
0.5x
26.5
24.4
25.4
0.25x
17.8
17.5
17.6
Appendix
Table 12: Performance of CoMEM on Qwen2.5-VL under different training data and parameter settings.
Figure 5: Case studies comparing CoMEM with baseline model and model with RAG.
Multimodal large language models (MLLMs) have recently made strong progress in vision-language reasoning, yet their performance often degrades as generations grow longer. A key factor is that they frequently lose track of earlier visual evidence and intermediate constraints under a monolithic growing context. Inspired by how humans separately recall what they see and what they infer when solving complex tasks, we propose DLMR, a parameter-efficient mechanism that equips MLLMs with Dual Latent Memories: a visual memory that compresses image evidence and a reasoning memory that tracks intermediate conclusions and constraints. A Router then dynamically decides which memory and how much to reuse during inference, preserving visual grounding while maintaining coherent long-horizon reasoning. DLMR is trained in three stages, from latent memory construction to selective router learning, while keeping the base MLLM frozen, yielding substantial gains on both general and reasoning benchmarks with only a small number of additional trainable parameters. Analyses further show interpretable, state-dependent routing with specialized memory roles and reduced decoding tokens over long generations. Code is available at https://github.com/Hunter-Wrynn/DLMR.
Hao-Xuan Ma, Jin-Fei Qi, Yicheng Xiao +1
School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China
Memory is essential for large vision-language models (LVLMs) to handle long, multimodal interactions, with two method directions providing this capability: long-context LVLMs and memory-augmented agents. However, no existing benchmark conducts a systematic comparison of the two on questions that genuinely require multimodal evidence. To close this gap, we introduce MEMLENS, a comprehensive benchmark for memory in multimodal multi-session conversations, comprising 789 questions across five memory abilities (information extraction, multi-session reasoning, temporal reasoning, knowledge update, and answer refusal) at four standard context lengths (32K-256K tokens) under a cross-modal token-counting scheme. An image-ablation study confirms that solving MEMLENS requires visual evidence: removing evidence images drops two frontier LVLMs below 2% accuracy on the 80.4% of questions whose evidence includes images. Evaluating 27 LVLMs and 7 memory-augmented agents, we find that long-context LVLMs achieve high short-context accuracy through direct visual grounding but degrade as conversations grow, whereas memory agents are length-stable but lose visual fidelity under storage-time compression. Multi-session reasoning caps most systems below 30%, and neither approach alone solves the task. These results motivate hybrid architectures that combine long-context attention with structured multimodal retrieval. Our code is available at https://github.com/xrenaf/MEMLENS.
Multimodal Large Language Models (MLLMs) have achieved remarkable performance by aligning pretrained visual representations with the linguistic knowledge embedded in Large Language Models (LLMs). However, existing approaches typically rely on final-layer visual features or learnable multi-layer fusion, which often fail to sufficiently exploit hierarchical visual cues without explicit cross-layer interaction design. In this work, we propose a Memory-Augmented Adapter (Mema) within the vision encoder. Specifically, Mema maintains a stateful memory that accumulates hierarchical visual representations across layers, with its evolution conditioned on both query embeddings and step-wise visual features. A portion of this memory is selectively injected into token representations via a feedback mechanism, thereby mitigating the attenuation of fine-grained visual cues from shallow layers. Designed as a lightweight and plug-and-play module, Mema integrates seamlessly into pretrained vision encoders without modifying the vanilla backbone architecture. Only a minimal set of additional parameters requires training, enabling adaptive visual feature refinement while reducing training overhead. Extensive experiments across multiple benchmarks demonstrate that Mema consistently improves performance, validating its effectiveness in complex multimodal reasoning tasks. The code have been released at https://github.com/Sisiliu312/Mema.
Ying Liu, Yudong Han, Kean Shi +1
Beijing Institute of Technology · Peking University · Yangtze Delta Region Academy of Beijing Institude of Technology, Jiaxing, China