ReMAP: Restoring the Perceptual Cycle with Reasoning-Time Latent Visual Memory

Hao Jiang; Zhanyu Guo; Chenwei Wu; Yichen Guo; Qizhe Zhang; Junchi Yao; Jixian Wu; Jinhao You

cs.CVOct 4, 2026

ReMAP: Restoring the Perceptual Cycle with Reasoning-Time Latent Visual Memory

Authors: Hao Jiang, Zhanyu Guo, Chenwei Wu, Yichen Guo, Qizhe Zhang, Junchi Yao, Jixian Wu, Jinhao You, +6 more

Organizations: University of Electronic Science and Technology of China · University of Michigan, Ann Arbor · Nanyang Technological University · Peking University · Mohamed bin Zayed University of Artificial Intelligence · New York University · Harvard University · Massachusetts Institute of Technology

Abstract

As multimodal large language models (MLLMs) reason for longer, attention to the initial visual input diminishes, weakening visual grounding. Visual memory reintroduces visual evidence during reasoning. We conduct a controlled analysis of visual memory along three axes: curation, organization, and access. We find that local evidence benefits from global context, compact latent representations balance accuracy and visual-context cost, and the utility of memory access depends on the reasoning state. Guided by these findings, we propose ReMAP (Reasoning-Time Memory-Augmented Perception), which couples two complementary latent memories: a static, question-conditioned Global memory that preserves scene and cross-image context, and a dynamic Local memory that uses this context as an anchor while selecting and re-encoding region-level evidence according to the current reasoning state. Both memories return compact latent tokens inserted into the reasoning sequence, and a reinforcement-learning access policy trained with branched rollouts decides when to continue reasoning or invoke Global or Local memory. On ten benchmark families, ReMAP outperforms prior visual-memory methods on all four multi-image benchmarks, exceeding the strongest prior results on MuirBench and MIMIC by 8.38 and 14.84 percentage points. Across four backbone families, enabling memory access improves over the same trained model with memory disabled, and on shared V*Bench, CV-Bench-2D, and MuirBench questions ReMAP reduces the visual tokens entering the reasoning sequence by 51.0-76.8% relative to the native-resolution backbone. Further analyses show that Global and Local memory form distinct yet complementary latent representations. Together, these components restore the perceptual cycle by letting the reasoning state trigger targeted visual retrieval, with the retrieved evidence guiding subsequent reasoning.

Figures & tables

Figure 1: Investigating memory curation, organization, and access. (a) An interaction with Qwen2.5-VL-7B-Instruct on a MuirBench example illustrates how global context helps resolve ambiguous local evidence. (b) A comparison of frozen latents, crop pixels, and fresh latents on a random 25% subset of V*Bench examines the trade-off between accuracy and incremental GPU-memory overhead. (c) An easy recognition example and cases from V*Bench and MuirBench show how the need for memory and its invocation timing vary across queries and reasoning states. — Figure 1: Investigating memory curation, organization, and access. (a) An interaction with Qwen2.5-VL-7B-Instruct on a MuirBench example illustrates how global context helps resolve ambiguous local evidence. (b) A comparison of frozen latents, crop pixels, and fresh latents on a random 25% subset of VBench examines the trade-off between accuracy and incremental GPU-memory overhead. (c) An easy recognition example and cases from VBench and MuirBench show how the need for memory and its invocation timing vary across queries and reasoning states.

Figure 2: Overview of ReMAP. Global memory forms a question-conditioned summary of individual images and their relations before decoding. During reasoning, a Local request uses the current state to select sources and regions, re-encodes selected pixels, and compresses them with cached Global context. Global and Local returns are inserted as continuous latent blocks after their request markers, followed by end markers before autoregressive decoding resumes.

Figure 3: Local memory and adaptive invocation. (a) The request state $h_{t}^{12}$ selects source images $\mathcal{S}_{t}$ , Twig uses the prefix states $H_{<t}^{18}$ to localize a crop in each, and the Local Weaver compresses the crops with the Global anchor $Z_{G}$ into $K$ Local latents. (b) Stage I trains the Local Weaver with $\mathcal{L}_{\mathrm{suffix}}$ and Twig with $\mathcal{L}_{\mathrm{ROI}}$ , adapts both to teacher-reviewed student states, and warms up the memory tokens. (c) Each root branches where siblings are most likely to take a different route, and the invocation gain compares requesting memory with the default continuation $\mathrm{N}$ at the same state.

Pixel reinspection
	Single-image evidence acquisition					Multi-image relational reasoning				Held-out generalization
Method	V ⋆	CV-2D	CV-3D	HR-4K	HR-8K	BLINK	Muir	MMSI	MIMIC Avg.	RWQA	MathVista	MMVet
Native	76.44	74.41	76.33	72.88	67.75	56.62	57.85	28.10	45.17	67.97	59.90	46.06
Coarse	67.54	73.10	75.42	61.13	54.62	57.79	58.15	28.40	43.56	67.19	59.50	47.06
Q-Zoom	79.06	76.35	74.08	73.00	66.63	53.97	50.38	25.40	42.29	70.85	54.60	47.43
DeepEyes	84.29	78.61	79.42	74.63	68.75	49.24	48.50	24.10	39.27	69.67	62.00	51.01

Table 1: Comparison with existing visual memory methods across ten benchmark families. Results cover single-image perception, multi-image reasoning, and broader visual understanding. MIMIC Avg. is the mean across four subsets. The top-2 results in each column are highlighted as first and second , with the best results also shown in bold .

Panel of Figure 4 — Figure 4: Global and Local memory representations. (a) High-resolution and (b) multi-image benchmark groups. Joint t-SNE projections show mean-pooled Global and Local memory vectors; colors denote benchmark groups, circles and triangles denote Global and Local, respectively, and shaded contours show density.

Figure 5: Local refinement and Global–Local interaction. (a) HR-Bench-4K #0347: repeated Local calls narrow the inspection from the street scene to a speed-limit sign, yielding 30 km/h rather than 40 km/h without memory. (b) MMSI #0377: two Local calls followed by a Global call accompany the northeast answer, whereas the paired no-call trajectory answers due east.

Panel of Figure 6 — Figure 6: Memory invocation and visual-context cost. (a) Mean total visual tokens entering the sequence (bars) and performance (lines) on shared V*Bench, CV-Bench-2D, and MuirBench questions; the shaded column marks ReMAP. (b) Global (solid) and Local (dashed) invocation ratios over the normalized reasoning trajectory.

Memory curation
Method	V ⋆	CV-3D	Muir	BLINK	MM-Vet
Base	67.54	75.42	58.15	57.81	47.06
SFT	58.64	75.58	55.73	54.13	55.05
Stage I sft	76.44	83.33	57.81	58.92	48.35
w/o Memory	79.06	80.00	61.00	59.02	57.80
w/ Global memory	79.58	83.00	64.00	59.50	60.55

Table 2: Training, curation, and access ablations (%). Base is Coarse; BLINK reports micro accuracy and MM-Vet mean graded scores.

Figure 7: Latent-capacity sensitivity. Performance on 20% holdout splits from HR-Bench-4K, MuirBench, and MMSI-Bench. Curves show benchmark scores and their equal-weight macro average; shaded bands are item-level 95% intervals from one training seed. Dashed guides mark the selected

K=32

and

4N+4

settings.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Pixel / Region Reacquisition	Compact Visual Memory	State- Conditioned Access	Cross-Image Context	Global / Local Split	Invocation vs. Continuation
DeepEyes	✓	✗	–	–	✗	✗
Pixel Reasoner	✓	✗	–	–	✗	✗
VLM-R 3	✓	✗	–	–	✗	✗
CMMCoT	✓	✓	–	✓	✗	✗
Qwen Look Again	✗	✓	–	–	✗	✗
ICoT / MINT-CoT	✗	✓	–	–	✗	✗

Appendix

Table 3: Qualitative positioning of representative visual-memory methods. Green checks indicate explicit support, red crosses indicate non-core mechanisms, and gray dashes indicate dimensions not emphasized in the cited work.

Figure 8: Five-step training pipeline. Each row lists the inputs of one step, the updated modules (flame), the frozen components (snowflake), the objective, and the output. Steps 1–4 form Stage I, and step 5 is Stage II access learning, in which the decoder LoRA $\phi$ and the request interface $\psi$ receive gradients only from $\mathcal{L}_{\mathrm{dec}}$ and $\mathcal{L}_{\mathrm{access}}$ , respectively.

Stage	Parameter group or setting	Value
I: Global formation	Image and Relation Weavers	$10^{-6}$
	Output projection	$10^{-5}$
I: Local formation	Source Selector, Twig, and Local Weaver	$10^{-6}$
I: interface warm-up	Marker and request rows	$10^{-4}$
II: access learning	Decoder LoRA	$10^{-6}$
	Decoder LoRA rank $r$ / $\alpha$	$16$ / $32$

Appendix

Table 4: Optimization settings for memory formation and access learning.

Source	Content and use
Visual-CoT (GQA) ( Shao et al., 2024a )	Object, attribute, and spatial-relation questions with region annotations.
COCO train2017 ( Lin et al., 2014 )	Multi-image questions built from instance annotations.
Mantis-Instruct ( Jiang et al., 2024 ) : ChartQA, Spot-the-Diff, ImageCoDe, NLVR2, and Birds-to-Words	Chart reading and arithmetic, image comparison, description-to-image matching, and cross-image relational judgments.
Q-Zoom’s VStar-COCO data ( Shi et al., 2026 )	Spatial-relation questions on COCO train2017 images.
ScanQA via M4-Instruct ( Li et al., 2024 )	Multi-view scene question answering, restricted to examples from the original ScanQA training split.

Appendix

Table 5: Training data sources. Source datasets provide question–answer pairs, region annotations, or instance annotations from which additional questions are constructed.

Evaluation unit	Examples	Aggregated metric
Single-image perception
V*Bench, released test	191	Micro accuracy
CV-Bench, test-2D	1,438	Mean source accuracy
CV-Bench, test-3D	1,200	Micro accuracy
HR-Bench, 4K	800	Mean category accuracy
HR-Bench, 8K	800	Mean category accuracy

Appendix

Table 6: Evaluation units and aggregation rules. Scores are reported as percentages. The example count is the full evaluation denominator.

Method	Common	Counting	Odd-One-Out	Listing	Mean
Native	62.60	44.98	60.40	12.69	45.17
Coarse	61.10	38.42	54.70	20.02	43.56
Q-Zoom †	51.90	37.44	54.80	25.00	42.29
DeepEyes †	52.60	40.38	53.20	10.90	39.27
Pixel Reasoner	64.50	43.34	30.00	41.46	44.83
ReGround	53.60	36.76	13.00	8.66	28.01

Appendix

Table 7: MIMIC component scores. A dagger denotes an ordered contact-sheet adaptation. VisMem is the public-code reproduction used in the main table.

Figure 9: Visual influence decays along the reasoning trajectory. Rate at which the next-token top-1 decision changes when the input images are replaced, in eight bins of normalized reasoning position. (a) Patch-shuffled and blank images, pooled over V*Bench, BLINK, and MuirBench, with 95% bootstrap intervals. (b) Patch-shuffled images for each cohort.

Cohort	Eligible / selected	First $\to$ last bin	Paired change
V*Bench	$189/191$	$15.43\%\to 2.36\%$	$-8.79$ [ $-10.48,-7.00$ ] pp
BLINK	$254/256$	$13.27\%\to 2.45\%$	$-10.88$ [ $-12.15,-9.63$ ] pp
MuirBench	$254/256$	$7.43\%\to 1.96\%$	$-4.50$ [ $-5.48,-3.55$ ] pp
Pooled	$697/703$	$11.73\%\to 2.25\%$	$-7.98$ [ $-8.78,-7.21$ ] pp

Appendix

Table 8: Patch-shuffle visual influence across reasoning. Eligible/selected counts the trajectories analyzed out of the sampled examples. First

\to

last bin gives the flip rate in the first and eighth bins. Paired change is the per-example mean flip rate over the last quarter of the trajectory minus that over the first quarter, with a 95% bootstrap interval.

Cohort	Contrast	Estimate	95% CI
V*Bench ( $n=125$ )	late–early correct-content utility	$+0.1771$	$[+0.1327,+0.2227]$
V*Bench ( $n=125$ )	correct–random-wrong RoI	$-0.0112$	$[-0.0754,+0.0486]$ NS
CV-Bench-3D ( $n=152$ )	late–early current-scene utility	$+0.0726$	$[+0.0405,+0.1050]$
CV-Bench-3D ( $n=152$ )	current–matched-wrong scene	$+0.2265$	$[+0.1361,+0.3171]$
CV-Bench-3D ( $n=152$ )	current–shuffled scene	$+0.0264$	$[-0.0358,+0.0882]$ NS

Appendix

Table 9: State and content contrasts in gold-answer log probability. Late–early contrasts insert the same content at the later and the earlier selected state. Content contrasts compare the correct region or scene with a random wrong region, a matched wrong scene, or a shuffled scene. NS marks an interval that contains zero.

Figure 10: Pixel re-encoding versus feature pooling on V*Bench. Both arms return the gold RoI with

K

inserted tokens at the same insertion point on 191 examples. (a) Accuracy. (b) Pixel minus feature gold-answer log probability, with 95% CIs.

Contrast	Estimate	95% CI
Role change $-$ appearance control	$+0.0482$	$[+0.0254,+0.0725]$
Late $-$ early role sensitivity	$+0.0039$	$[-0.0056,+0.0138]$ NS
Late $-$ early fixation rate	$+0.0182$	$[-0.0026,+0.0391]$ NS
Oracle $-$ no context	$+2.5797$	$[+2.1222,+2.9881]$
Oracle $-$ blank block	$+2.4765$	$[+2.1251,+2.8061]$
Oracle $-$ zero block	$+2.8028$	$[+2.5116,+3.0891]$

Appendix

Table 10: Relational-role probe. The upper rows use total-variation sensitivity or fixation rate on 192 pairs. The lower rows compare the oracle relational context with each control in gold-answer log probability on the 71 selected late failure states. NS marks an interval that contains zero.

Figure 11: Relational-role probe. (a) Total-variation sensitivity to a role change and to an appearance change at four reasoning positions on 192 pairs, with 95% intervals. (b) Accuracy on the 71 selected late failure states with the oracle relational context and six controls; without context, every state is wrong by construction.

Figure 12: Branch-point selection with a fixed trajectory budget. (a) Routing Yield@H16, the fraction of branch families whose trajectories make different Global and Local requests within 16 positions after the branch. (b) Joint EAY, the reward variation that co-occurs with this routing diversity. Each point averages four root families of one prompt; diamonds and bars mark the mean and 95% bootstrap interval. All rules use the same roots, candidate states, and 12 trajectories per prompt on 192 prompts, with the policy frozen. ARPO-style $\Delta H$ is the fixed-budget adaptation of ARPO’s entropy-change signal.

Figure 13: Complete paired reasoning traces. Cases 1 and 2 expand panels (a) and (b) of Figure 5 with the full recorded reasoning of the paired no-call and memory-enabled trajectories. Each memory return is marked by its request and end tokens. Case 1 (HR-Bench-4K #0347): without memory, the model reads 40 km/h and answers B; after three Local calls, it reads 30 km/h and answers D. Case 2 (MMSI #0377): without memory, the model answers due east (C); after two Local calls and one Global call, it answers northeast (B).

Benchmark	Method	Initial	Added	Total
V*Bench ( $n=78$ )	Native	4,417.9	0.0	4,417.9
	Q-Zoom	1,008.0	331.8	1,339.8
	DeepEyes	4,417.9	35.9	4,453.8
	VisMem	4,417.9	0.0	4,417.9
	ReMAP	1,008.0	16.0	1,024.0
CV-Bench-2D ( $n=60$ )	Native	906.0	0.0	906.0

Appendix

Table 11: Visual tokens entering the decoder sequence per question. Means over the questions shared by all five methods, with the question count given for each benchmark. Added counts crop tokens and memory latents returned during reasoning. Totals are computed before rounding.

Explore similar work

May 8, 2026cs.CL

Retrieve, Integrate, and Synthesize: Spatial-Semantic Grounded Latent Visual Reasoning

Multimodal Large Language Models (MLLMs) have made remarkable progress on vision-language reasoning, yet most methods still compress visual evidence into discrete textual thoughts, creating an information bottleneck for fine-grained perception. Recent latent visual reasoning methods attempt to reason in continuous hidden states, but we find that they suffer from insufficient manifold compatibility: latent trajectories drift away from pretrained reasoning circuits, collapse into instance-agnostic patterns, and are often bypassed during answer generation. To address these issues, we propose RIS (Retrieve, Integrate, and Synthesize), a spatial-semantic grounded framework that develops latent reasoning as a compatible extension of pretrained MLLM computation. We first construct a step-wise grounded reasoning dataset with bounding boxes and region-specific semantic descriptions. Built on this supervision, RIS anchors latent tokens to both spatial and semantic evidence, enforces their causal role through a progressive attention bottleneck, and introduces short language transition tokens to bridge synthesized latent states back to vocabulary-aligned decoding. Experiments on V*, HRBench4K, HRBench8K, MMVP, and BLINK show consistent improvements over closed/open-source and latent reasoning baselines. Further analyses demonstrate that RIS learns diverse, interpretable, and progressively integrated latent trajectories, offering a practical path toward faithful internal visual reasoning in MLLMs.

Jin Cui, Xinyue Long, Xunyong Zhang +5

1State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, and Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University · School of Software Engineering, State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University

Latent Visual Reasoning Recent Vision-Language Models Multimodal Large Language Models

Ask PDF

May 31, 2026cs.CV

Beyond Visual Memory: Mechanistic Diagnostics of Latent Visual Reasoning

Recent latent visual reasoning methods achieve substantial gains by inserting continuous latent tokens into multimodal language models. These gains are commonly attributed to the tokens encoding visual evidence; recent analyses, however, reveal a paradox: the tokens are loosely tied to the image and contribute little to the answer. Critically, these analyses treat latent tokens as a single unit, obscuring the true source of the gains. We therefore decompose latent tokens into three testable components: latent slots, boundary markers, and format, and develop a state-of-the-art method as a probe under favorable conditions. Across six method-stage settings and four perception-heavy benchmarks, latent slots fail every prediction of the visual-memory account. Strikingly, retaining only the boundary markers preserves 78 to 100% of the gain in several settings, while the model attends to the image more narrowly at latent positions than at answer positions. The gain therefore comes from boundary markers, format, and this attention pattern, not from latent slots. How each method engages this mechanism depends on its training supervision: at matched accuracy, mechanisms can still differ markedly. Latent visual reasoning thus needs evaluation not only by accuracy but by what the model actually relies on.

Garvin Guo, Yu Chen, Xiang Wang +4

Amap, Alibaba Group · Shanghai Innovation Institute

Latent Visual Reasoning Compact Latent Sensor Tokens Visual Memory Multimodal Large Language Models

Ask PDF

Aug 2, 2026cs.CV

Remember-R1: Mitigating Long-Context Visual Forgetting through Reinforcement Learning

Multimodal large language models (MLLMs) increasingly rely on long chain-of-thought reasoning for complex tasks. However, as reasoning sequences lengthen, models may gradually rely less on visual evidence and more on accumulated textual context, leading to visual forgetting. Existing approaches do not directly constrain how visual evidence is used and maintained along the original reasoning trajectory, leaving long-context visual forgetting insufficiently addressed. To address this issue, we propose Remember-R1, a reinforcement learning framework that mitigates long-context visual forgetting by applying process-level supervision directly on the original reasoning trajectory. Specifically, Remember-R1 introduces rewards that encourage broader coverage of matched visual keywords, stronger persistence of visual dependence in later reasoning steps, and greater focus on question-relevant image regions. Experiments across multiple model scales and diverse multimodal benchmarks demonstrate that Remember-R1 consistently improves reasoning performance. Additional analyses further show that it slows the decline of visual attention during generation, supporting its effectiveness in mitigating long-context visual forgetting.

Jianmin Chen, Jiaqi Tang, Wei Wei +9

Hong Kong University of Science and Technology Hong Kong, China · Northwestern Polytechnical University Xi’an, China · Zhejiang University Hangzhou, China

Latent Visual Reasoning Visual Memory Recent Vision-Language Models Catastrophic Forgetting Forgetting Multimodal Benchmarks+1

Ask PDF