ReAccess: Look Once, Stay Focused with Efficient Test Time Memory
Organizations: KAIST · Tianjin University · National University of Singapore
Abstract
Despite recent advances, vision-language models (VLMs) still struggle with fine-grained visual understanding, often failing to effectively utilize critical evidence already encoded in their visual representations. Existing approaches address this limitation through repeated visual processing or explicit reasoning, introducing substantial computational overhead. In this paper, we propose ReAccess, a lightweight framework that preserves and reuses question-relevant visual evidence through compact associative memory. Constructed during a single visual prefill, the memory supports progressive evidence retrieval that steers both attention queries and hidden representations, enabling effective visual grounding without repeated image encoding or explicit reasoning traces. Extensive experiments demonstrate consistent improvements in both general visual understanding and fine-grained perception, while achieving substantially faster inference than existing reasoning-intensive approaches, demonstrating a promising path to efficient multimodal reasoning.
Figures & tables
| Baselines | Matched Image | |||||
|---|---|---|---|---|---|---|
| Evidence Type | Question Only | Unrelated Image | Full State | PCA (16D) | PCA (64D) | Random (16D) |
| Visual Appearance | 23.5 | 21.3 | 64.5 | 64.3 | 65.4 | 44.4 |
| Attribute Grounding | 51.5 | 53.5 | 85.9 | 86.3 | 86.6 | 84.3 |
| Spatial Structure | 53.2 | 51.0 | 89.5 | 90.5 | 89.8 | 78.8 |
| Physical Properties | 41.6 | 40.1 | 75.2 | 73.7 | 70.1 | 45.3 |
| Backbone | GQA | MMBench | MMStar | TextVQA | DocVQA | InfoVQA | ChartQA | AI2D | OCRBench | MMMU | LogicVista | POPE |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen2.5-VL-7B [ 4 ] | 60.4 | 82.2 | 56.7 | 83.2 | 93.3 | 78.6 | 83.3 | 78.7 | 84.3 | 50.1 | 39.1 | 86.1 |
| +Ours | 62.1 1.7 | 83.7 1.5 | 61.6 4.9 | 85.0 1.8 | 95.2 1.9 | 80.5 1.9 | 87.6 4.3 | 82.6 3.9 | 86.7 2.4 | 51.0 0.9 | 42.4 3.3 | 87.9 1.8 |
| LLaVA-NeXT-7B [ 19 ] | 63.0 | 64.9 | 33.5 | 65.4 | 66.1 | 30.7 | 53.5 | 63.4 | 52.4 | 35.8 | 26.3 | 86.1 |
| +Ours | 64.6 1.6 | 67.4 2.5 | 36.5 3.0 | 67.8 2.4 | 70.2 4.1 | 35.8 5.1 | 55.7 2.2 | 65.2 1.8 | 53.7 1.3 | 37.4 1.6 | 29.2 2.9 | 88.2 2.1 |
| Qwen3-VL-8B [ 3 ] | 61.8 | 83.8 | 61.1 | 82.7 | 94.3 | 80.8 | 84.6 | 82.9 | 85.8 | 54.0 | 35.3 | 87.3 |
| +Ours | 63.0 1.2 | 86.4 2.6 | 65.5 4.4 | 84.6 1.9 | 95.8 1.5 | 83.6 2.8 | 89.9 5.3 | 84.3 1.4 | 88.6 2.8 | 55.7 1.7 | 39.5 4.2 | 88.5 1.2 |
| Method | HR-Bench 4K | HR-Bench 8K | V ∗ | CV-Bench | ||
|---|---|---|---|---|---|---|
| Attribute | Spatial | Overall | ||||
| Multi-turn | ||||||
| Thyme-RL [ 47 ] | 77.0 | 72.0 | 84.4 | 81.6 | 83.3 | 78.2 |
| DeepEyes [ 48 ] | 73.9 | 71.0 | 88.7 | 79.0 | 84.8 | 77.8 |
| Two-stage | ||||||
| API [ 45 ] | 71.5 | 68.5 | 80.9 | 75.0 | 78.5 | 78.1 |
| Training Objectives | Architecture | Evaluation Benchmarks | ||||||||||||
| Configuration | Q | O | Q-Cond. | Prog. | HR-4K | HR-8K | V ∗ -Attr. | V ∗ -Spat. | V ∗ -Overall | TextVQA | POPE | |||
| Full | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 73.3 | 71.8 | 84.4 | 82.9 | 83.8 | 85.0 | 87.9 |
| Training objective ablations | ||||||||||||||
| Evidence reconstruction | ✗ | ✓ | ✓ | 71.6 | 69.8 | 82.6 | 79.0 | 81.2 | 83.4 | 87.2 | ||||
| Grounding | ✓ | ✗ | ✓ | 72.0 | 67.9 | 80.9 | 77.6 | 79.6 | 84.2 | 86.7 | ||||
| Diversity | ✓ | ✓ | ✗ | 72.4 | 70.8 | 83.5 | 80.3 | 82.2 | 84.6 | 87.3 | ||||
| HR-Bench | V* | ||||
|---|---|---|---|---|---|
| Memory Setting | 4K | 8K | Attr. | Spatial | Overall |
| Original | 73.3 | 71.8 | 84.4 | 82.9 | 83.8 |
| Wrong-Image | 69.6 | 66.1 | 80.0 | 76.3 | 78.5 |
| Wrong-Question | 71.3 | 68.4 | 82.6 | 79.0 | 81.2 |
| Zero-Memory | 70.9 | 66.9 | 81.7 | 77.6 | 80.1 |
| Memory Size | HR-Bench | V* | TextVQA | |||
|---|---|---|---|---|---|---|
| 4K | 8K | Attr. | Spatial | Overall | ||
| 71.8 | 69.8 | 81.7 | 79.0 | 80.6 | 84.7 | |
| 73.3 | 71.8 | 84.4 | 82.9 | 83.8 | 85.0 | |
| 73.1 | 71.5 | 83.5 | 81.6 | 82.7 | 84.6 | |
| 72.8 | 71.0 | 82.6 | 81.6 | 82.2 | 84.1 | |
| Depth | HR-Bench | V* | TextVQA | |||
|---|---|---|---|---|---|---|
| 4K | 8K | Attr. | Spatial | Overall | ||
| Full | 73.3 | 71.8 | 84.4 | 82.9 | 83.8 | 85.0 |
| Shallow | 72.0 | 68.9 | 80.9 | 75.0 | 78.5 | 84.6 |
| Middle | 72.5 | 69.0 | 83.5 | 80.3 | 82.2 | 84.0 |
| Deep | 72.1 | 68.8 | 81.7 | 77.6 | 80.1 | 82.9 |
| Configuration | Parameter | Value |
| Core | Backbone | Qwen2.5-VL-7B |
| Memory dimension | 64 | |
| Memory layers | 28 | |
| Memory reads | 2 | |
| Spatial kernel | Depthwise | |
| Trainable parameters | 57.9M ( 0.7%) |
| HR-Bench | V* | MMMU | Anomalies | ||
| Method | 8K | 4K | Latency | Latency | /400 |
| ReAccess | 3.98 (0) | 3.34 (0) | 0.95 (0) | 0.31 (0) | 0 |
| Frozen | 3.79 (0) | 3.17 (0) | 0.89 (0) | 0.28 (0) | 0 |
| LVR | 4.26 (0) | 3.78 (0) | 1.38 (0) | 1.00 (0) | 0 |
| API | 8.26 (0) | 4.98 (0) | 2.14 (0) | 1.24 (0) | 0 |
| FGVP-Mask | 8.28 (0) | 5.93 (0) | 2.86 (0) | 1.70 (0) | 0 |
| Method | Anomalies | Failure Cause |
|---|---|---|
| Thyme-RL | 6 | Repetitive outputs reaching token limits and triggering repeated retries. |
| DeepEyes | 2 | Repeated unsuccessful action requests with excessive token generation. |
| Vision-R1 | 29 | Overlong, repetitive reasoning and retry loops during sampled decoding. |
| PixelReasoner | 1 | Repetitive generation exhausting the output token limit. |
| HyLaR | 10 | Erroneous generation continuing until the maximum token limit. |
| SkiLa | 2 | Erroneous generation continuing until the maximum token limit. |
Memory-Augmented VLMs
Memory-augmented vision-language models use persistent or retrieved context to improve reasoning across long inputs and interactions.
See all 175 papers