cs.CVOct 4, 2026
SaveReMAP: Restoring the Perceptual Cycle with Reasoning-Time Latent Visual Memory
Organizations: University of Electronic Science and Technology of China · University of Michigan, Ann Arbor · Nanyang Technological University · Peking University · Mohamed bin Zayed University of Artificial Intelligence · New York University · Harvard University · Massachusetts Institute of Technology
Abstract
As multimodal large language models (MLLMs) reason for longer, attention to the initial visual input diminishes, weakening visual grounding. Visual memory reintroduces visual evidence during reasoning. We conduct a controlled analysis of visual memory along three axes: curation, organization, and access. We find that local evidence benefits from global context, compact latent representations balance accuracy and visual-context cost, and the utility of memory access depends on the reasoning state. Guided by these findings, we propose ReMAP (Reasoning-Time Memory-Augmented Perception), which couples two complementary latent memories: a static, question-conditioned Global memory that preserves scene and cross-image context, and a dynamic Local memory that uses this context as an anchor while selecting and re-encoding region-level evidence according to the current reasoning state. Both memories return compact latent tokens inserted into the reasoning sequence, and a reinforcement-learning access policy trained with branched rollouts decides when to continue reasoning or invoke Global or Local memory. On ten benchmark families, ReMAP outperforms prior visual-memory methods on all four multi-image benchmarks, exceeding the strongest prior results on MuirBench and MIMIC by 8.38 and 14.84 percentage points. Across four backbone families, enabling memory access improves over the same trained model with memory disabled, and on shared V*Bench, CV-Bench-2D, and MuirBench questions ReMAP reduces the visual tokens entering the reasoning sequence by 51.0-76.8% relative to the native-resolution backbone. Further analyses show that Global and Local memory form distinct yet complementary latent representations. Together, these components restore the perceptual cycle by letting the reasoning state trigger targeted visual retrieval, with the retrieved evidence guiding subsequent reasoning.
Figures & tables
Figure 1: Investigating memory curation, organization, and access. (a) An interaction with Qwen2.5-VL-7B-Instruct on a MuirBench example illustrates how global context helps resolve ambiguous local evidence. (b) A comparison of frozen latents, crop pixels, and fresh latents on a random 25% subset of VBench examines the trade-off between accuracy and incremental GPU-memory overhead. (c) An easy recognition example and cases from VBench and MuirBench show how the need for memory and its invocation timing vary across queries and reasoning states.
Figure 2: Overview of ReMAP. Global memory forms a question-conditioned summary of individual images and their relations before decoding. During reasoning, a Local request uses the current state to select sources and regions, re-encodes selected pixels, and compresses them with cached Global context. Global and Local returns are inserted as continuous latent blocks after their request markers, followed by end markers before autoregressive decoding resumes.
Figure 3: Local memory and adaptive invocation. (a) The request state selects source images , Twig uses the prefix states to localize a crop in each, and the Local Weaver compresses the crops with the Global anchor into Local latents. (b) Stage I trains the Local Weaver with and Twig with , adapts both to teacher-reviewed student states, and warms up the memory tokens. (c) Each root branches where siblings are most likely to take a different route, and the invocation gain compares requesting memory with the default continuation at the same state.
| Single-image evidence acquisition | Multi-image relational reasoning | Held-out generalization | ||||||||||
| Method | V ⋆ | CV-2D | CV-3D | HR-4K | HR-8K | BLINK | Muir | MMSI | MIMIC Avg. | RWQA | MathVista | MMVet |
| Native | 76.44 | 74.41 | 76.33 | 72.88 | 67.75 | 56.62 | 57.85 | 28.10 | 45.17 | 67.97 | 59.90 | 46.06 |
| Coarse | 67.54 | 73.10 | 75.42 | 61.13 | 54.62 | 57.79 | 58.15 | 28.40 | 43.56 | 67.19 | 59.50 | 47.06 |
| Pixel reinspection | ||||||||||||
| Q-Zoom | 79.06 | 76.35 | 74.08 | 73.00 | 66.63 | 53.97 | 50.38 | 25.40 | 42.29 | 70.85 | 54.60 | 47.43 |
| DeepEyes | 84.29 | 78.61 | 79.42 | 74.63 | 68.75 | 49.24 | 48.50 | 24.10 | 39.27 | 69.67 | 62.00 | 51.01 |
Table 1: Comparison with existing visual memory methods across ten benchmark families. Results cover single-image perception, multi-image reasoning, and broader visual understanding. MIMIC Avg. is the mean across four subsets. The top-2 results in each column are highlighted as first and second , with the best results also shown in bold .
Figure 4: Global and Local memory representations. (a) High-resolution and (b) multi-image benchmark groups. Joint t-SNE projections show mean-pooled Global and Local memory vectors; colors denote benchmark groups, circles and triangles denote Global and Local, respectively, and shaded contours show density.
Figure 5: Local refinement and Global–Local interaction. (a) HR-Bench-4K #0347: repeated Local calls narrow the inspection from the street scene to a speed-limit sign, yielding 30 km/h rather than 40 km/h without memory. (b) MMSI #0377: two Local calls followed by a Global call accompany the northeast answer, whereas the paired no-call trajectory answers due east.
Figure 6: Memory invocation and visual-context cost. (a) Mean total visual tokens entering the sequence (bars) and performance (lines) on shared V*Bench, CV-Bench-2D, and MuirBench questions; the shaded column marks ReMAP. (b) Global (solid) and Local (dashed) invocation ratios over the normalized reasoning trajectory.
| Method | V ⋆ | CV-3D | Muir | BLINK | MM-Vet |
|---|---|---|---|---|---|
| Base | 67.54 | 75.42 | 58.15 | 57.81 | 47.06 |
| SFT | 58.64 | 75.58 | 55.73 | 54.13 | 55.05 |
| Stage I sft | 76.44 | 83.33 | 57.81 | 58.92 | 48.35 |
| Memory curation | |||||
| w/o Memory | 79.06 | 80.00 | 61.00 | 59.02 | 57.80 |
| w/ Global memory | 79.58 | 83.00 | 64.00 | 59.50 | 60.55 |
Table 2: Training, curation, and access ablations (%). Base is Coarse; BLINK reports micro accuracy and MM-Vet mean graded scores.
Figure 7: Latent-capacity sensitivity. Performance on 20% holdout splits from HR-Bench-4K, MuirBench, and MMSI-Bench. Curves show benchmark scores and their equal-weight macro average; shaded bands are item-level 95% intervals from one training seed. Dashed guides mark the selected and settings.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Pixel / Region Reacquisition | Compact Visual Memory | State- Conditioned Access | Cross-Image Context | Global / Local Split | Invocation vs. Continuation |
| DeepEyes | ✓ | ✗ | – | – | ✗ | ✗ |
| Pixel Reasoner | ✓ | ✗ | – | – | ✗ | ✗ |
| VLM-R 3 | ✓ | ✗ | – | – | ✗ | ✗ |
| CMMCoT | ✓ | ✓ | – | ✓ | ✗ | ✗ |
| Qwen Look Again | ✗ | ✓ | – | – | ✗ | ✗ |
| ICoT / MINT-CoT | ✗ | ✓ | – | – | ✗ | ✗ |
Table 3: Qualitative positioning of representative visual-memory methods. Green checks indicate explicit support, red crosses indicate non-core mechanisms, and gray dashes indicate dimensions not emphasized in the cited work.
Figure 8: Five-step training pipeline. Each row lists the inputs of one step, the updated modules (flame), the frozen components (snowflake), the objective, and the output. Steps 1–4 form Stage I, and step 5 is Stage II access learning, in which the decoder LoRA and the request interface receive gradients only from and , respectively.
| Stage | Parameter group or setting | Value |
|---|---|---|
| I: Global formation | Image and Relation Weavers | |
| Output projection | ||
| I: Local formation | Source Selector, Twig, and Local Weaver | |
| I: interface warm-up | Marker and request rows | |
| II: access learning | Decoder LoRA | |
| Decoder LoRA rank / | / |
Table 4: Optimization settings for memory formation and access learning.
| Source | Content and use |
|---|---|
| Visual-CoT (GQA) ( Shao et al., 2024a ) | Object, attribute, and spatial-relation questions with region annotations. |
| COCO train2017 ( Lin et al., 2014 ) | Multi-image questions built from instance annotations. |
| Mantis-Instruct ( Jiang et al., 2024 ) : ChartQA, Spot-the-Diff, ImageCoDe, NLVR2, and Birds-to-Words | Chart reading and arithmetic, image comparison, description-to-image matching, and cross-image relational judgments. |
| Q-Zoom’s VStar-COCO data ( Shi et al., 2026 ) | Spatial-relation questions on COCO train2017 images. |
| ScanQA via M4-Instruct ( Li et al., 2024 ) | Multi-view scene question answering, restricted to examples from the original ScanQA training split. |
Table 5: Training data sources. Source datasets provide question–answer pairs, region annotations, or instance annotations from which additional questions are constructed.
| Evaluation unit | Examples | Aggregated metric |
| Single-image perception | ||
| V*Bench, released test | 191 | Micro accuracy |
| CV-Bench, test-2D | 1,438 | Mean source accuracy |
| CV-Bench, test-3D | 1,200 | Micro accuracy |
| HR-Bench, 4K | 800 | Mean category accuracy |
| HR-Bench, 8K | 800 | Mean category accuracy |
Table 6: Evaluation units and aggregation rules. Scores are reported as percentages. The example count is the full evaluation denominator.
| Method | Common | Counting | Odd-One-Out | Listing | Mean |
|---|---|---|---|---|---|
| Native | 62.60 | 44.98 | 60.40 | 12.69 | 45.17 |
| Coarse | 61.10 | 38.42 | 54.70 | 20.02 | 43.56 |
| Q-Zoom † | 51.90 | 37.44 | 54.80 | 25.00 | 42.29 |
| DeepEyes † | 52.60 | 40.38 | 53.20 | 10.90 | 39.27 |
| Pixel Reasoner | 64.50 | 43.34 | 30.00 | 41.46 | 44.83 |
| ReGround | 53.60 | 36.76 | 13.00 | 8.66 | 28.01 |
Table 7: MIMIC component scores. A dagger denotes an ordered contact-sheet adaptation. VisMem is the public-code reproduction used in the main table.
Figure 9: Visual influence decays along the reasoning trajectory. Rate at which the next-token top-1 decision changes when the input images are replaced, in eight bins of normalized reasoning position. (a) Patch-shuffled and blank images, pooled over V*Bench, BLINK, and MuirBench, with 95% bootstrap intervals. (b) Patch-shuffled images for each cohort.
| Cohort | Eligible / selected | First last bin | Paired change |
|---|---|---|---|
| V*Bench | [ ] pp | ||
| BLINK | [ ] pp | ||
| MuirBench | [ ] pp | ||
| Pooled | [ ] pp |
Table 8: Patch-shuffle visual influence across reasoning. Eligible/selected counts the trajectories analyzed out of the sampled examples. First last bin gives the flip rate in the first and eighth bins. Paired change is the per-example mean flip rate over the last quarter of the trajectory minus that over the first quarter, with a 95% bootstrap interval.
| Cohort | Contrast | Estimate | 95% CI |
|---|---|---|---|
| V*Bench ( ) | late–early correct-content utility | ||
| V*Bench ( ) | correct–random-wrong RoI | NS | |
| CV-Bench-3D ( ) | late–early current-scene utility | ||
| CV-Bench-3D ( ) | current–matched-wrong scene | ||
| CV-Bench-3D ( ) | current–shuffled scene | NS |
Table 9: State and content contrasts in gold-answer log probability. Late–early contrasts insert the same content at the later and the earlier selected state. Content contrasts compare the correct region or scene with a random wrong region, a matched wrong scene, or a shuffled scene. NS marks an interval that contains zero.
Figure 10: Pixel re-encoding versus feature pooling on V*Bench. Both arms return the gold RoI with inserted tokens at the same insertion point on 191 examples. (a) Accuracy. (b) Pixel minus feature gold-answer log probability, with 95% CIs.
| Contrast | Estimate | 95% CI |
|---|---|---|
| Role change appearance control | ||
| Late early role sensitivity | NS | |
| Late early fixation rate | NS | |
| Oracle no context | ||
| Oracle blank block | ||
| Oracle zero block |
Table 10: Relational-role probe. The upper rows use total-variation sensitivity or fixation rate on 192 pairs. The lower rows compare the oracle relational context with each control in gold-answer log probability on the 71 selected late failure states. NS marks an interval that contains zero.
Figure 11: Relational-role probe. (a) Total-variation sensitivity to a role change and to an appearance change at four reasoning positions on 192 pairs, with 95% intervals. (b) Accuracy on the 71 selected late failure states with the oracle relational context and six controls; without context, every state is wrong by construction.
Figure 12: Branch-point selection with a fixed trajectory budget. (a) Routing Yield@H16, the fraction of branch families whose trajectories make different Global and Local requests within 16 positions after the branch. (b) Joint EAY, the reward variation that co-occurs with this routing diversity. Each point averages four root families of one prompt; diamonds and bars mark the mean and 95% bootstrap interval. All rules use the same roots, candidate states, and 12 trajectories per prompt on 192 prompts, with the policy frozen. ARPO-style is the fixed-budget adaptation of ARPO’s entropy-change signal.
Figure 13: Complete paired reasoning traces. Cases 1 and 2 expand panels (a) and (b) of Figure 5 with the full recorded reasoning of the paired no-call and memory-enabled trajectories. Each memory return is marked by its request and end tokens. Case 1 (HR-Bench-4K #0347): without memory, the model reads 40 km/h and answers B; after three Local calls, it reads 30 km/h and answers D. Case 2 (MMSI #0377): without memory, the model answers due east (C); after two Local calls and one Global call, it answers northeast (B).
| Benchmark | Method | Initial | Added | Total |
|---|---|---|---|---|
| V*Bench ( ) | Native | 4,417.9 | 0.0 | 4,417.9 |
| Q-Zoom | 1,008.0 | 331.8 | 1,339.8 | |
| DeepEyes | 4,417.9 | 35.9 | 4,453.8 | |
| VisMem | 4,417.9 | 0.0 | 4,417.9 | |
| ReMAP | 1,008.0 | 16.0 | 1,024.0 | |
| CV-Bench-2D ( ) | Native | 906.0 | 0.0 | 906.0 |
Table 11: Visual tokens entering the decoder sequence per question. Means over the questions shared by all five methods, with the question count given for each benchmark. Added counts crop tokens and memory latents returned during reasoning. Totals are computed before rounding.