Long-context multimodal intelligence requires revisiting earlier images as conversations and documents accumulate, yet retaining an image does not guarantee reliable access to its details. By varying context length, redistributing a fixed text passage around the image, and changing positions without adding text, we identify the positional relation between images and questions as a direct influence on visual access. To address this gap, we propose Visual Position Re-Anchoring (VPR), a training-free method that shifts cached visual-key rotary phases toward a short-context reference before question processing. It operates directly on the existing KV cache, preserving standard attention computation without extracting attention maps or adding model forward passes. We develop static and continuous compensation schedules, using the static schedule by default. Across ten checkpoints and ten benchmarks, static VPR improves 97 of 100 model--task scores. Further analysis shows greater attention to regions containing answer evidence across six checkpoints, with minimal runtime overhead under fixed rotary frequencies. These findings demonstrate an efficient way to sustain visual access through positional adjustment within the cache.
Figures & tables
Figure 1: Visual evidence drift in long contexts. The correct short-context answer, “Tongue in cheek,” becomes “Sign in protest” in a long context, confusing the subtitle with the headline.
Figure 2: Overview of VPR. (a) Left: image keys are rephased before question processing; values and cache slots are preserved. One rotary pair is shown schematically. (b) Right: ρ=1 restores the initial short context separation. The offset stays fixed as decoding queries advance.
Figure 3: Visual access across context lengths. Vanilla POPE accuracy and image-attention mass measure task performance and allocation, respectively; they use separate cohorts (Appendix C.4 ).
Figure 4: Image position at a fixed 32K text budget. (a) The same text is split before and after the image. (b) Dotted lines mark each model’s 0K POPE accuracy from Figure 3 . Middle-row panels use 70–90% accuracy; other panels use 45–95%.
Figure 5: Position shifts on identical short inputs. Bars show the score loss relative to each model’s unshifted score (0K) as the question moves later or the visual keys move earlier.
Task
Method
Qwen2.5-VL
Qwen3-VL
InternVL3
LLaVA
MRoPE
MRoPE-I
1D + NTK
1D: L3.1 / Qwen2
3B-I
7B-I
2B-I
4B-I
8B-I
8B-T
2B
8B
L3.1-8B
OV-7B†
POPE
Vanilla
64.6
51.6
75.4
85.0
77.8
82.0
50.0
79.0
58.2
64.4
VPR
82.2 (+17.6)
63.4 (+11.8)
84.0 (+8.6)
86.0 (+1.0)
87.0 (+9.2)
83.0 (+1.0)
63.0 (+13.0)
84.0 (+5.0)
84.6 (+26.4)
83.8 (+19.4)
CountBench
Vanilla
43.6
44.2
64.0
64.0
59.0
79.0
5.0
57.0
36.6
62.0
VPR
72.0 (+28.4)
71.2 (+27.0)
68.0 (+4.0)
69.0 (+5.0)
67.0 (+8.0)
86.0 (+7.0)
32.0 (+27.0)
58.0 (+1.0)
54.4 (+17.8)
76.8 (+14.8)
Table 1: Ten checkpoints across ten visual tasks at 32K. Parentheses show VPR gains: red for increases, green for decreases. DocVQA uses ANLS, TextVQA uses consensus scores, and other tasks use accuracy, all on a 0–100 scale. I/T: Instruct/Thinking; L3.1/OV: LLaVA-Llama3.1/OneVision. † Context extrapolation.
Figure 6: TextVQA scores as the passage between the image and question grows. Static VPR scores higher than Vanilla at every nonzero length shown. Each panel uses its own vertical scale.
Figure 7: VPR recovers the license-plate answer. Dashed boxes highlight the plate region; attention maps share a continuous scale. Red and blue in the difference map indicate increases and decreases, respectively.
Figure 8: Attention share on answer-bearing text within the image. Bars show means with 95% bootstrap intervals.
Table 2: Long-context POPE accuracy across compensation strengths. Shading marks the default ρ=1 . Bold marks row maxima. All six checkpoints contribute equally to the mean.
Task
Model
Vanilla
Static
Cont.
Diff.
TextVQA
Qwen2.5-7B
43.8
62.2
61.8
-0.4
Qwen3-8B-I
70.8
74.8
75.0
+0.2
InternVL3-2B
37.2
53.2
54.3
+1.1
LLaVA-L3.1-8B
29.8
55.4
54.2
-1.2
MathVista
Qwen2.5-7B
38.4
47.4
49.4
+2.0
Qwen3-8B-I
61.0
63.0
60.2
-2.8
Table 3: Static and continuous VPR. Diff. denotes Continuous minus Static.
Vanilla
Static VPR
Continuous VPR
Output
Model
Time (s)
Time (s)
Extra (%)
Time (s)
Extra (%)
128
Qwen2.5-7B
3.465
3.470
+0.13
3.709
+7.03
Qwen3-8B-I
4.738
4.741
+0.06
5.033
+6.23
InternVL3-2B
2.283
2.934
+28.51
2.941
+28.82
LLaVA-L3.1-8B
3.991
3.997
+0.14
4.251
+6.52
512
Qwen2.5-7B
9.416
9.447
+0.33
10.443
+10.91
Table 4: Inference time and overhead relative to Vanilla with matched split prefill.
POPE accuracy (%)
Cost
Method
0K
8K
16K
24K
32K
Time (s)
Extra (%)
Vanilla
85.8
72.8
63.0
56.4
51.6
3.465
ref.
VPR
85.8
77.8
74.6
68.4
63.4
3.470
+0.13
VAF
86.0
76.6
73.6
67.2
62.0
3.631
+4.8
VGA
85.8
72.8
63.0
56.4
52.8
4.081
+17.8
VAR
85.4
73.4
63.4
56.8
54.2
19.408
+460.1
Table 5: Training-free interventions on Qwen2.5-VL-7B-Instruct across POPE context lengths. Cost reports inference time and overhead relative to Vanilla.
Joint lookup accuracy (%)
Model
Method
8K
16K
24K
32K
Qwen2.5-7B
Vanilla
18.75
14.25
11.75
11.00
VPR
23.25
21.25
18.00
17.00
Δ
+4.50
+7.00
+6.25
+6.00
Qwen3-8B-I
Vanilla
72.25
64.25
60.50
65.75
VPR
77.50
70.25
64.50
69.25
Table 6: Joint image and text lookup across context lengths. Scores measure final answer code accuracy; Δ is the static VPR gain in percentage points.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Checkpoint
RoPE
Training *
Config.
Eval. ceiling
Output
Qwen2.5-VL-3B-Instruct
MRoPE
8K → 32K a
128,000
128,000
1,024
Qwen2.5-VL-7B-Instruct
MRoPE
8K → 32K a
128,000
128,000
1,024
Qwen3-VL-2B-Instruct
MRoPE-I
8K → 256K b
262,144
262,144
1,024
Qwen3-VL-4B-Instruct
MRoPE-I
8K → 256K b
262,144
262,144
1,024
Qwen3-VL-8B-Instruct
MRoPE-I
8K → 256K b
262,144
262,144
1,024
Qwen3-VL-8B-Thinking
MRoPE-I
8K → 256K b
262,144
262,144
4,096
Appendix
Table 7: Model releases, training context, and evaluation settings. Checkpoint names link to the evaluated releases. Training entries refer to the sourced recipes below; Config. is the archived maximum-position setting, and Eval. ceiling is the configured evaluation ceiling, not the tested input length. Output is the generation cap.
Study
Conditions and models
Questions per score
Main tasks
Ten checkpoints, ten tasks, 32K
500
POPE lengths
Six checkpoints, 0/1/2/4/8/16/32K
500
TextVQA lengths
Six checkpoints, seven lengths
500
Five layouts
Six checkpoints, 32K total text, 0/8/16/24/32K after image
Figure 9: POPE accuracy across seven context lengths. Solid lines show Vanilla; dashed lines show static VPR. Each panel uses its own vertical scale.
Figure 10: ChartQA accuracy across seven context lengths. Panels use model-specific vertical ranges; red labels show 32K gains in percentage points.
Figure 11: Layer-wise change in image-attention mass, in percentage points, using a shared color scale. Layer indices are model-local and decoder depths differ.
Figure 12: TextVQA image-attention mass across lengths (top) and layers at 32K (bottom). The same image questions are used throughout.
Model
Input K
Output
Method
Time
Overhead
95% interval
TTFT
Rate
Peak
InternVL3-2B
8
128
Vanilla
1.818
+0.00
[+0.00, +0.00]
0.107
74.3
4.716
InternVL3-2B
8
128
Continuous
2.459
+35.29
[+34.82, +35.66]
0.115
54.2
4.716
InternVL3-2B
8
128
Static
1.817
-0.06
[-0.38, +0.27]
0.115
74.7
4.716
InternVL3-2B
8
512
Vanilla
7.001
+0.00
[+0.00, +0.00]
0.107
74.2
4.716
InternVL3-2B
8
512
Continuous
9.571
+36.71
[+36.25, +37.15]
0.115
54.1
4.716
InternVL3-2B
8
512
Static
7.004
+0.04
[-0.16, +0.22]
0.115
74.2
4.716
Appendix
Table 9: Complete inference measurements: time and TTFT in seconds, overhead relative to split Vanilla in percent, decode rate in tokens/s, peak allocated memory in GiB.
Model
Contrast
Method time
Native time
Overhead
95% interval
Qwen2.5-7B
split_vs_native
3.465
3.392
+2.16
[+2.07, +2.38]
Qwen2.5-7B
static_vs_native
3.470
3.392
+2.29
[+2.18, +2.45]
Qwen2.5-7B
continuous_vs_native
3.709
3.392
+9.34
[+9.27, +9.51]
Qwen3-8B-I
split_vs_native
4.738
4.635
+2.22
[+2.15, +2.45]
Qwen3-8B-I
static_vs_native
4.741
4.635
+2.29
[+2.18, +2.56]
Qwen3-8B-I
continuous_vs_native
5.033
4.635
+8.59
[+8.44, +8.88]
Appendix
Table 10: Complete-prefill native-path control at 32K and 128 output tokens. Time in seconds; overhead and intervals in percent.
Native frequency
Cache updates
Models
Input K
Output
changes
Static
Continuous
Qwen2.5 / Qwen3 / LLaVA
8 / 32
128
0
1
128
8 / 32
512
0
1
512
InternVL3-2B
8
128
0
1
128
8
512
0
1
512
32
128
129
129
129
Appendix
Table 11: Native frequency changes and visual-cache updates per request. Counts are identical for Qwen2.5-7B, Qwen3-8B-Instruct and LLaVA-Llama3.1-8B. A logical cache update writes visual keys across all layers; Vanilla performs no VPR cache updates.
Shenzhen International Center for Industrial and Applied Mathematics · Shenzhen Research Institute of Big Data · The Chinese University of Hong Kong, Shenzhen +1