Dual-Latent Memory Routing for Vision-Language Reasoning

Hao-Xuan Ma; Jin-Fei Qi; Yicheng Xiao; Han-Jia Ye

cs.CVSep 2, 2026

Dual-Latent Memory Routing for Vision-Language Reasoning

Authors: Hao-Xuan Ma, Jin-Fei Qi, Yicheng Xiao, Han-Jia Ye

Organizations: School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China

Abstract

Multimodal large language models (MLLMs) have recently made strong progress in vision-language reasoning, yet their performance often degrades as generations grow longer. A key factor is that they frequently lose track of earlier visual evidence and intermediate constraints under a monolithic growing context. Inspired by how humans separately recall what they see and what they infer when solving complex tasks, we propose DLMR, a parameter-efficient mechanism that equips MLLMs with Dual Latent Memories: a visual memory that compresses image evidence and a reasoning memory that tracks intermediate conclusions and constraints. A Router then dynamically decides which memory and how much to reuse during inference, preserving visual grounding while maintaining coherent long-horizon reasoning. DLMR is trained in three stages, from latent memory construction to selective router learning, while keeping the base MLLM frozen, yielding substantial gains on both general and reasoning benchmarks with only a small number of additional trainable parameters. Analyses further show interpretable, state-dependent routing with specialized memory roles and reduced decoding tokens over long generations. Code is available at https://github.com/Hunter-Wrynn/DLMR.

Figures & tables

Figure 1: Normalized attention to image tokens through generation step. Compared to the vanilla MLLM (Qwen2.5-vl-7B on MathVision Benchmark), DLMR preserves substantially more visual attention in later decoding, enabling more reliable revisiting of early visual evidence and often requiring fewer reasoning tokens to reach the final answer.

Figure 2: Overview of DLMR. Given an image $\mathbf{I}$ and text prompt $\mathbf{x}$ , a frozen MLLM backbone $\mathcal{M}_{\theta}$ performs autoregressive decoding. DLMR maintains two compact latent-space memories $\mathbf{Z}^{(v)}$ (visual evidence) and $\mathbf{Z}^{(r)}$ (reasoning constraints). At delimiter-eligible steps, a discrete router selects the memory type $s_{t}\in\{v,r\}$ and an injection budget $k_{t}\in\mathcal{K}^{+}$ (or a null action). A lightweight memory injector contextualizes the selected latents into $k_{t}$ memory tokens $\mathbf{M}_{t}$ , which are appended to the current context before predicting the next token. This enables state-dependent, budget-aware reuse without modifying the backbone.

Method	General				Reasoning
Method	MMVet	MMStar	RealWorldQA	Avg	MMMU	MathVerse	MathVision	MathVista	Avg
Qwen2.5-VL-7B
Vanilla	63.30	62.60	68.10	64.67	53.11	42.64	22.37	68.30	46.61
CoT	64.55	63.73	68.88	65.72	53.78	43.24	23.02	68.80	47.21
CCoT	64.67	64.11	69.12	65.97	53.33	43.31	24.67	67.90	47.30
SFT	64.61	62.98	69.27	65.62	54.16	43.38	23.68	68.90	47.53

Table 1: Comparison of DLMR with strong baselines on 3 general and 4 reasoning benchmarks using two MLLMs (Qwen2.5-VL-7B and InternVL-3-8B). DLMR (SFT) trains the injector with supervised learning, while DLMR (RL) uses GRPO-based training.

Figure 3: Disentanglement ablation on Qwen2.5-VL-7B evaluated on six benchmarks (two general: MMVet, MMStar; four reasoning: MMMU, MathVerse, MathVision, MathVista). We compare using the dual latent memory ( $\mathbf{Z}^{(v)},\mathbf{Z}^{(r)}$ ) and a single shared memory ( $\mathbf{Z}$ ).

Variant	MMMU	MVerse	MathV	MVista	Avg
Frozen injector	58.14	44.32	27.74	69.96	50.44
Trainable injector	61.00	46.95	35.32	72.10	53.84

Table 2: Injector ablation on Qwen2.5-VL-7B evaluated on four reasoning benchmarks. We compare the full model with a frozen injector variant (no injector training) while keeping the same dual memories and routing/injection interface.

Base Model	General				Reasoning
Base Model	MMVet	MMStar	RealWorldQA	Avg	MMMU	MathVerse	MathVision	MathVista	Avg
LLaVA-OV-1.5-4B	66.0	62.6	68.1	64.6	56.0	42.6	22.3	68.3	46.6
+ DLMR	75.1 $\uparrow$ 9.1	68.9 $\uparrow$ 6.3	71.3 $\uparrow$ 3.2	71.4 $\uparrow$ 6.8	63.9 $\uparrow$ 7.9	46.9 $\uparrow$ 4.3	35.3 $\uparrow$ 13.0	72.1 $\uparrow$ 3.8	53.8 $\uparrow$ 7.2
Qwen2.5-VL-3B	61.2	55.1	65.3	60.6	51.2	31.2	21.9	61.2	41.4
+ DLMR	69.8 $\uparrow$ 8.6	60.6 $\uparrow$ 5.5	70.1 $\uparrow$ 4.8	66.8 $\uparrow$ 6.2	58.9 $\uparrow$ 7.7	33.2 $\uparrow$ 2.0	36.5 $\uparrow$ 4.6	68.6 $\uparrow$ 7.4	49.3 $\uparrow$ 7.9
InternVL-3-2B	65.1	61.3	64.5	63.6	48.6	24.1	21.7	54.7	37.3

Table 3: Cross-backbone generalization of DLMR on three additional MLLMs (LLaVA-OV-1.5-4B, Qwen2.5-VL-3B, InternVL-3-2B), evaluated on General and Reasoning benchmarks.

\uparrow

indicates the performance improvement over the corresponding base model. DLMR yields consistent gains across all three backbones, indicating that the proposed memory augmentation is largely base-model agnostic.

Figure 4: Accuracy–length frontier under different router budget caps

k_{\max}

on Qwen2.5-VL-7B. General tasks peak at

k_{\max}=16

, while Reasoning tasks favor a larger headroom and peak at

k_{\max}=32

, indicating that adaptive routing uses high budgets only on hard states.

Variant	Avg Acc	Avg token length
No router, fixed $k{=}4$	51.55	664
No router, fixed $k{=}8$	52.71	732
No router, fixed $k{=}16$	52.04	765
Full DLMR (adaptive $k_{t}$ )	53.84	677

Table 4: Budget ablation ( Qwen2.5-VL-7B ). We compare fixed injection budgets

k

(no adaptive routing) with the full DLMR router. Avg Acc is averaged over four reasoning benchmarks; Avg token length is the average number of generated tokens per sample.

Figure 5: Cross-modality transfer of memory routing to text-only benchmarks (MMLU, GSM8K, ARC-Challenge) on Qwen2.5-VL-7B , compared against the vanilla backbone, GRPO, and Visual-RFT baselines.

Figure 6: Average generated tokens per sample on four reasoning-centric benchmarks using Qwen2.5-VL-7B . By selectively reusing latent memories, DLMR shortens generations compared to strong baselines.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A1 : Router frequency over generation position. We plot the routing ratio (injection frequency) of the visual memory and the reasoning memory as a function of relative decoding position, aggregated over reasoning benchmarks (top) and general benchmarks (bottom). The router invokes reasoning memory more often on reasoning tasks, while it prefers visual memory on general tasks, indicating task-adaptive and interpretable routing.

Model & Method	General		Reasoning
	Time (s)	Acc (%)	Time (s)	Acc (%)
Qwen2.5-VL-7B
Vanilla	10.1	64.7	28.5	46.6
SFT	5.6	65.6	14.0	47.5
DLMR	5.4	71.4	11.5	53.8
InternVL3-8B

Appendix

Table A1 : Inference time and accuracy on the General/Reasoning benchmark suites. Accuracy is averaged over the three General benchmarks (MMVet, MMStar, RealWorldQA) and the four Reasoning benchmarks (MMMU, MathVerse, MathVision, MathVista) from Tab. 1 ; time is the average end-to-end inference time per sample.

Method	LogicVista	OCR-Bench	MMT-Bench
Vanilla	43.1	86.1	51.5
CoT	47.5	89.1	55.0
SFT	49.0	90.1	56.5
GRPO	50.2	88.4	58.0
DLMR	55.0	92.0	62.5

Appendix

Table A2 : Transfer to unseen benchmarks. We evaluate models trained under the default setting on unseen datasets spanning reasoning ( LogicVista ) and general perception/OCR ( OCR-Bench , MMT-Bench ). All numbers are task scores in % (higher is better).

Configurations	Parameters	Qwen2.5-VL-7B	InternVL3-8B
Text backbone	hidden_size $d$	3584	3584
	num_layers $L$	28	28
	num_heads $H$	28	28
	num_kv_heads $H_{\mathrm{kv}}$	4	4
	mlp_intermediate $d_{\mathrm{ff}}$	18944	18944
	max_position_emb	128000	32768

Appendix

Table A3: Backbone model configurations. Key architectural hyperparameters of the two backbone MLLMs used in our experiments/analysis: Qwen2.5-VL-7B-Instruct and InternVL3-8B.

Benchmark	Domain	# Eval	Answer fmt.	Metric
General (vision–language)
MMVet ( Yu et al., 2023 )	Composite VL skills	218	Open-ended	LLM-judge acc.
MMStar ( Chen et al., 2024 )	Vision-indispensable VL	1,500	MCQ	Accuracy
RealWorldQA	Real-world VQA	765	MCQ / short ans.	Accuracy
Reasoning (vision–language)
MMMU ( Yue et al., 2024 )	College-level multidisciplinary	900 (val)	MCQ / open	Accuracy

Appendix

Table A4: Evaluation benchmarks. # Eval refers to the size of the split we evaluate on.

Configurations	Parameters	Stage 1	Stage 2	Stage 3
Core	backbone	Qwen2.5-VL-7B / InternVL3-8B	Qwen2.5-VL-7B / InternVL3-8B	Qwen2.5-VL-7B / InternVL3-8B
	visual_latents_len $M_{v}$	8	8	–
	reasoning_latents_len $M_{r}$	16	16	–
	max_visual_aug_num	1	1	1
	max_reasoning_aug_num $N_{\max}$	5	5	5
	random_reasoning_aug	True	–	–

Appendix

Table A5: Hyperparameters across three training stages. Stage 1: latent cold-start (optimize latent banks only). Stage 2: injector SFT (PEFT, backbone frozen). Stage 3: router RL (GRPO, backbone frozen). Values shared across both backbones (Qwen2.5-VL-7B and InternVL3-8B) unless stated otherwise. Batch sizes are per-device; gradient accumulation is used when needed.

Memory-Augmented VLMs

Memory-augmented vision-language models use persistent or retrieved context to improve reasoning across long inputs and interactions.

See all 151 papers

ReMAP: Restoring the Perceptual Cycle with Reasoning-Time Latent Visual Memory
Oct 4, 2026Hao Jiang, Zhanyu Guo, Chenwei Wu +11Memory-Augmented VLMs Efficient Multimodal Inference
University of Electronic Science and Technology of China · University of Michigan, Ann Arbor · Nanyang Technological University +5
DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
Jun 13, 2026David Huang, Lianlei ShanEfficient Multimodal Inference Multimodal Reasoning
University of Toronto · Tsinghua University
Retrieve, Integrate, and Synthesize: Spatial-Semantic Grounded Latent Visual Reasoning
May 8, 2026Jin Cui, Xinyue Long, Xunyong Zhang +5Visual Spatial Reasoning Visual Reasoning
1State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, and Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University · School of Software Engineering, State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University

Oct 4, 2026J/K move · Enter open · S save

ReMAP: Restoring the Perceptual Cycle with Reasoning-Time Latent Visual Memory

Hao Jiang, Zhanyu Guo, Chenwei Wu, Yichen Guo+10

University of Electronic Science and Technology of China · University of Michigan, Ann Arbor · Nanyang Technological University+5

Memory-Augmented VLMs Efficient Multimodal Inference Latent Visual Reasoning Visually Grounded Reasoning

Open paper AskPDF arXiv

As multimodal large language models (MLLMs) reason for longer, attention to the initial visual input diminishes, weakening visual grounding. Visual memory reintroduces visual evidence during reasoning. We conduct a controlled analysis of visual memory along three axes: curation, organization, and access. We find that local evidence benefits from global context, compact latent representations balance accuracy and visual-context cost, and the utility of memory access depends on the reasoning state. Guided by these findings, we propose ReMAP (Reasoning-Time Memory-Augmented Perception), which couples two complementary latent memories: a static, question-conditioned Global memory that preserves scene and cross-image context, and a dynamic Local memory that uses this context as an anchor while selecting and re-encoding region-level evidence according to the current reasoning state. Both memories return compact latent tokens inserted into the reasoning sequence, and a reinforcement-learning access policy trained with branched rollouts decides when to continue reasoning or invoke Global or Local memory. On ten benchmark families, ReMAP outperforms prior visual-memory methods on all four multi-image benchmarks, exceeding the strongest prior results on MuirBench and MIMIC by 8.38 and 14.84 percentage points. Across four backbone families, enabling memory access improves over the same trained model with memory disabled, and on shared V*Bench, CV-Bench-2D, and MuirBench questions ReMAP reduces the visual tokens entering the reasoning sequence by 51.0-76.8% relative to the native-resolution backbone. Further analyses show that Global and Local memory form distinct yet complementary latent representations. Together, these components restore the perceptual cycle by letting the reasoning state trigger targeted visual retrieval, with the retrieved evidence guiding subsequent reasoning.

Abstract

Memory-Augmented VLMs

Explore similar work

ReMAP: Restoring the Perceptual Cycle with Reasoning-Time Latent Visual Memory

DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning

Retrieve, Integrate, and Synthesize: Spatial-Semantic Grounded Latent Visual Reasoning