Keep Images in Sight: Visual Position Re-Anchoring for Long-Context MLLMs
Organizations: Tsinghua University
Abstract
Long-context multimodal intelligence requires revisiting earlier images as conversations and documents accumulate, yet retaining an image does not guarantee reliable access to its details. By varying context length, redistributing a fixed text passage around the image, and changing positions without adding text, we identify the positional relation between images and questions as a direct influence on visual access. To address this gap, we propose Visual Position Re-Anchoring (VPR), a training-free method that shifts cached visual-key rotary phases toward a short-context reference before question processing. It operates directly on the existing KV cache, preserving standard attention computation without extracting attention maps or adding model forward passes. We develop static and continuous compensation schedules, using the static schedule by default. Across ten checkpoints and ten benchmarks, static VPR improves 97 of 100 model--task scores. Further analysis shows greater attention to regions containing answer evidence across six checkpoints, with minimal runtime overhead under fixed rotary frequencies. These findings demonstrate an efficient way to sustain visual access through positional adjustment within the cache.
Figures & tables
| Task | Method | Qwen2.5-VL | Qwen3-VL | InternVL3 | LLaVA | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| MRoPE | MRoPE-I | 1D + NTK | 1D: L3.1 / Qwen2 | ||||||||
| 3B-I | 7B-I | 2B-I | 4B-I | 8B-I | 8B-T | 2B | 8B | L3.1-8B | OV-7B† | ||
| POPE | Vanilla | 64.6 | 51.6 | 75.4 | 85.0 | 77.8 | 82.0 | 50.0 | 79.0 | 58.2 | 64.4 |
| VPR | 82.2 (+17.6) | 63.4 (+11.8) | 84.0 (+8.6) | 86.0 (+1.0) | 87.0 (+9.2) | 83.0 (+1.0) | 63.0 (+13.0) | 84.0 (+5.0) | 84.6 (+26.4) | 83.8 (+19.4) | |
| CountBench | Vanilla | 43.6 | 44.2 | 64.0 | 64.0 | 59.0 | 79.0 | 5.0 | 57.0 | 36.6 | 62.0 |
| VPR | 72.0 (+28.4) | 71.2 (+27.0) | 68.0 (+4.0) | 69.0 (+5.0) | 67.0 (+8.0) | 86.0 (+7.0) | 32.0 (+27.0) | 58.0 (+1.0) | 54.4 (+17.8) | 76.8 (+14.8) | |
| Task | Model | Vanilla | Static | Cont. | Diff. |
|---|---|---|---|---|---|
| TextVQA | Qwen2.5-7B | 43.8 | 62.2 | 61.8 | -0.4 |
| Qwen3-8B-I | 70.8 | 74.8 | 75.0 | +0.2 | |
| InternVL3-2B | 37.2 | 53.2 | 54.3 | +1.1 | |
| LLaVA-L3.1-8B | 29.8 | 55.4 | 54.2 | -1.2 | |
| MathVista | Qwen2.5-7B | 38.4 | 47.4 | 49.4 | +2.0 |
| Qwen3-8B-I | 61.0 | 63.0 | 60.2 | -2.8 |
| Vanilla | Static VPR | Continuous VPR | ||||
|---|---|---|---|---|---|---|
| Output | Model | Time (s) | Time (s) | Extra (%) | Time (s) | Extra (%) |
| 128 | Qwen2.5-7B | 3.465 | 3.470 | +0.13 | 3.709 | +7.03 |
| Qwen3-8B-I | 4.738 | 4.741 | +0.06 | 5.033 | +6.23 | |
| InternVL3-2B | 2.283 | 2.934 | +28.51 | 2.941 | +28.82 | |
| LLaVA-L3.1-8B | 3.991 | 3.997 | +0.14 | 4.251 | +6.52 | |
| 512 | Qwen2.5-7B | 9.416 | 9.447 | +0.33 | 10.443 | +10.91 |
| POPE accuracy (%) | Cost | ||||||
|---|---|---|---|---|---|---|---|
| Method | 0K | 8K | 16K | 24K | 32K | Time (s) | Extra (%) |
| Vanilla | 85.8 | 72.8 | 63.0 | 56.4 | 51.6 | 3.465 | ref. |
| VPR | 85.8 | 77.8 | 74.6 | 68.4 | 63.4 | 3.470 | +0.13 |
| VAF | 86.0 | 76.6 | 73.6 | 67.2 | 62.0 | 3.631 | +4.8 |
| VGA | 85.8 | 72.8 | 63.0 | 56.4 | 52.8 | 4.081 | +17.8 |
| VAR | 85.4 | 73.4 | 63.4 | 56.8 | 54.2 | 19.408 | +460.1 |
| Joint lookup accuracy (%) | |||||
|---|---|---|---|---|---|
| Model | Method | 8K | 16K | 24K | 32K |
| Qwen2.5-7B | Vanilla | 18.75 | 14.25 | 11.75 | 11.00 |
| VPR | 23.25 | 21.25 | 18.00 | 17.00 | |
| +4.50 | +7.00 | +6.25 | +6.00 | ||
| Qwen3-8B-I | Vanilla | 72.25 | 64.25 | 60.50 | 65.75 |
| VPR | 77.50 | 70.25 | 64.50 | 69.25 | |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Checkpoint | RoPE | Training * | Config. | Eval. ceiling | Output |
|---|---|---|---|---|---|
| Qwen2.5-VL-3B-Instruct | MRoPE | 8K 32K a | 128,000 | 128,000 | 1,024 |
| Qwen2.5-VL-7B-Instruct | MRoPE | 8K 32K a | 128,000 | 128,000 | 1,024 |
| Qwen3-VL-2B-Instruct | MRoPE-I | 8K 256K b | 262,144 | 262,144 | 1,024 |
| Qwen3-VL-4B-Instruct | MRoPE-I | 8K 256K b | 262,144 | 262,144 | 1,024 |
| Qwen3-VL-8B-Instruct | MRoPE-I | 8K 256K b | 262,144 | 262,144 | 1,024 |
| Qwen3-VL-8B-Thinking | MRoPE-I | 8K 256K b | 262,144 | 262,144 | 4,096 |
| Study | Conditions and models | Questions per score |
|---|---|---|
| Main tasks | Ten checkpoints, ten tasks, 32K | 500 |
| POPE lengths | Six checkpoints, 0/1/2/4/8/16/32K | 500 |
| TextVQA lengths | Six checkpoints, seven lengths | 500 |
| Five layouts | Six checkpoints, 32K total text, 0/8/16/24/32K after image | 500 |
| Strength | Six checkpoints, 32K, to 1.5 | 500 |
| Update schedule | Qwen2.5-7B, Qwen3-8B-I, InternVL3-2B, LLaVA. TextVQA/MathVista, 32K | 500 |
| Model | Input K | Output | Method | Time | Overhead | 95% interval | TTFT | Rate | Peak |
|---|---|---|---|---|---|---|---|---|---|
| InternVL3-2B | 8 | 128 | Vanilla | 1.818 | +0.00 | [+0.00, +0.00] | 0.107 | 74.3 | 4.716 |
| InternVL3-2B | 8 | 128 | Continuous | 2.459 | +35.29 | [+34.82, +35.66] | 0.115 | 54.2 | 4.716 |
| InternVL3-2B | 8 | 128 | Static | 1.817 | -0.06 | [-0.38, +0.27] | 0.115 | 74.7 | 4.716 |
| InternVL3-2B | 8 | 512 | Vanilla | 7.001 | +0.00 | [+0.00, +0.00] | 0.107 | 74.2 | 4.716 |
| InternVL3-2B | 8 | 512 | Continuous | 9.571 | +36.71 | [+36.25, +37.15] | 0.115 | 54.1 | 4.716 |
| InternVL3-2B | 8 | 512 | Static | 7.004 | +0.04 | [-0.16, +0.22] | 0.115 | 74.2 | 4.716 |
| Model | Contrast | Method time | Native time | Overhead | 95% interval |
|---|---|---|---|---|---|
| Qwen2.5-7B | split_vs_native | 3.465 | 3.392 | +2.16 | [+2.07, +2.38] |
| Qwen2.5-7B | static_vs_native | 3.470 | 3.392 | +2.29 | [+2.18, +2.45] |
| Qwen2.5-7B | continuous_vs_native | 3.709 | 3.392 | +9.34 | [+9.27, +9.51] |
| Qwen3-8B-I | split_vs_native | 4.738 | 4.635 | +2.22 | [+2.15, +2.45] |
| Qwen3-8B-I | static_vs_native | 4.741 | 4.635 | +2.29 | [+2.18, +2.56] |
| Qwen3-8B-I | continuous_vs_native | 5.033 | 4.635 | +8.59 | [+8.44, +8.88] |
| Native frequency | Cache updates | ||||
|---|---|---|---|---|---|
| Models | Input K | Output | changes | Static | Continuous |
| Qwen2.5 / Qwen3 / LLaVA | 8 / 32 | 128 | 0 | 1 | 128 |
| 8 / 32 | 512 | 0 | 1 | 512 | |
| InternVL3-2B | 8 | 128 | 0 | 1 | 128 |
| 8 | 512 | 0 | 1 | 512 | |
| 32 | 128 | 129 | 129 | 129 | |
Long-Context VLMs
Long-context vision-language models (VLMs) process and reason over extended image, video, and document inputs.
See all 49 papers