A vision-language model may take left-right direction from the orientation of image content within patches, its vision transformer's (ViT) position code, or token positions and order in its language model (LLM). Ablations show the ViT code is necessary; we test sufficiency by mirroring only a position code. In Qwen2.5-VL-7B, Qwen3-VL-4B and InternVL3-2B, mirroring ViT width positions reproduces the flipped-image answer on 84.3-93.2% of CV-Bench left-right questions (Qwen2.5-VL-3B: 87.5%, partly a left fallback). With two options, this also counts disruption; a half-grid shift flips most answers too. With the ViT code intact, LLM-side id and token-order mirrors change few left-right answers in all three series (a weak id test in Qwen2.5-VL). In Qwen2.5-VL-7B the effect sits in the global-attention ViT blocks, mostly the last. On What'sUp, flipping content and token order, not the position code, keeps more on/under than left-right answers at the flipped-image answer. Post hoc analyses qualify these results. On CV-Bench items with located objects, random permutations flip 18.9-31.0 points more answers when reversing the objects' order than when keeping it. On What'sUp, in Qwen2.5-VL and Qwen3-VL, the share of the left-right changes made by random permutations that land on the mirror relation is mostly below chance. Counted as returns to the original answer, the What'sUp contrast of support with left-right relations is clear only in InternVL3-2B. Most remaining errors give the same relation whichever object is asked about, suggesting that repair may target relation binding, object localization or use of the reference object.
Figures & tables
Work
Evidence
Intervention
Not addressed
[ 5 ]
necessity (ablation); accuracy per relation
mask / shuffle / constant per axis, ViT ( h , w ) and LLM ( t , x , y ), pixels fixed; What’sUp left / right / on / under
per-item comparison with a real flip; ids vs. token order; block localization; split by object order
[ 6 ]
geometry; correlation; method
X/Y decomposition of 2D-RoPE attention; embedding-level interventions; frequency scaling
direction-predicting edit of the position code; LLM-side ids
[ 27 ]
attention decomposition; ablation
position × token interaction in CLIP-style encoders (synthetic 1D scenes)
pretrained VLMs; LLM side
[ 7 ]
content patching; interchange (sufficiency-type)
visual-token content; left and right regions swapped with a counterfactual image
position code; LLM-side ids and order
[ 8 ]
activation patching (sufficiency-type); mean-difference spatial IDs and ID steering
activations from a mirrored-image run patched into the original run; spatial IDs
input-side position code (ViT or LLM ids)
[ 31 ]
state patching with horizontal flips (sufficiency-type)
image, query-text and final-token states between an image and its flip (Qwen3-VL-4B)
position code; heads and MLPs
Table 1: Closest prior work. The sufficiency-type interventions among the listed works patch activations or visual-token content from a counterfactual image; none of the listed works compares the answer after an edit of the position code alone, item by item, with the answer to a real image transform.
Figure 1: Position-only counterfactuals keep every pixel and mirror only a position code. Rows: real horizontal flip ( H-flip , reference); ViT-W : width index w→W−1−w of each patch in the ViT; LLM-W : LLM-side width ids of the visual tokens mirrored. Answers measured on this photograph; percentages: changed geometry-consistent left–right answers, four models ( Figure 2 a), not agreement with H-flip ; last-block note: Figure 3 .
Qwen2.5-VL
Qwen2.5-VL
InternVL3
Qwen3-VL
InternVL3
3B
7B
2B
4B
8B
ViT position code
2D-RoPE
2D-RoPE
learned abs.
abs. + 2D-RoPE
learned abs.
LLM position ids
(t,h,w)
(t,h,w)
1D
(t,h,w)
1D
Accuracy, left–right (original)
73.1%
90.1%
72.1%
93.2%
93.2%
Accuracy, up–down (original)
74.2%
89.5%
78.3%
–
89.5%
Agreement with the real horizontal flip (all 383 left–right items)
Table 2: Position-only counterfactuals on CV-Bench Relation (383 left–right, 267 up–down questions). Agreement: share of questions whose counterfactual answer equals the answer under the matching real transform. Flip rate: share of geometry-consistent items (correct on the original and on the real transform) whose answer changes. Brackets: 95% Wilson intervals (integer percent). In Qwen2.5-VL-3B the original answers already agree with the flipped-image answers on 46.0% of the left–right questions, mostly because both are left , which inflates its agreement. InternVL3-8B: scale check; equal values in the 7B and 8B columns are coincidences (their answers differ on 42 of 650 original and 74 ViT-W questions). † Swapped options: 58.9%; mean of both orders 52.2% ( C ). Qwen3-VL-4B: same bf16 scoring; –: the 180∘ conditions and the off-axis ViT-H control were not run, and up–down accuracy and the all-item off-axis rate were not computed; its up–down subsets are defined by the vertical flip; the tie-removed analyses ( C ) do not include it; bf16 ties on 14 of its 344 geometry-consistent left–right items. ‡ Original answers without the frame sentence that the 180∘ conditions add ( Section 3.4 ).
Figure 2: Mirroring the ViT width positions flips most left–right answers; mirroring the LLM-side positions changes almost none of them. Share of geometry-consistent items whose answer changes, with 95% Wilson intervals; color: condition, open: LLM-side mirror, shape: model. (a) CV-Bench Relation, Qwen2.5-VL-3B in the original option order. (b) What’sUp: change to the mirror relation (markers) and any change ( × ); Qwen3-VL-4B: left–right conditions only, and for its LLM-W only any change. Subset sizes in Tables 2 and 4 .
Qwen2.5-VL
Qwen2.5-VL
3B
7B
Left–right items (383), all items
order only, width (ids follow tokens)
2.3%
2.9%
order only, height (off-axis control)
3.7%
4.4%
id only, width (the LLM-W edit; all items)
0.8%
0.3%
order and id, width
2.3%
2.3%
Table 3: LLM-side token order in Qwen2.5-VL: fraction of answers that change when the visual tokens are reordered in the LLM input ( order only : each token keeps its own M-RoPE id; id only : tokens stay in place and their ids are mirrored). Pooling the three shuffle seeds overstates the precision of that control; in neither model does the mirrored row order change clearly more up–down answers than both non-mirror permutations, and in 3B only about half of its flips agree with the real vertical flip.
Figure 3: The left–right effect of the width mirror is concentrated in the global-attention ViT blocks, chiefly the last, whereas in the 3B model the up–down effect of the height mirror builds up across blocks (Qwen2.5-VL, all CV-Bench Relation items, 95% Wilson intervals). (a) ViT-W in single blocks or block sets (shape: model); ‘Global’: the four full-attention blocks 7, 15, 23 and 31. (b) Dose–response in the 3B model, mirroring the first or the last L blocks.
Qwen2.5-VL
Qwen2.5-VL
InternVL3
Qwen3-VL
3B
7B
2B
4B
Accuracy, left–right / up–down (original)
95.1% / 94.7%
85.0% / 98.1%
98.1% / 92.7%
99.0% / –
Left–right items, geometry-consistent subset
ViT-W : answer changes
100.0%
100.0%
99.0%
99.5%
ViT-W : changes to the mirror relation
82.3% [76, 87]
72.1% [65, 79]
94.9% [91, 97]
90.6% [86, 94]
LLM-W : answer changes
0.0%
0.0%
1.0%
1.5%
Table 4: What’sUp (Controlled Images A; 412 questions with four options: left , right , on , under ). The main metric is the fraction that changes to the mirror relation (left ↔ right or on ↔ under); a change to an off-axis option is not a directional effect. On up–down items, random row permutations also turn 86–100% (Qwen2.5-VL) and 70–86% (InternVL3-2B) of the answers they change into on ↔ under, so the up–down mirror-relation row does not show direction. Qwen3-VL-4B: left–right conditions only; –: not run.
Figure 4: Up–down answers follow the position code less than left–right answers, and depth answers that follow the height cue depend on the ViT height positions. (a) Blank-image decomposition of all CV-Bench Relation items into answers that follow position (flip under the matching ViT mirror, disruption included), follow content, or neither (which includes image-dependent answers). (b) CV-Bench 3D depth, Qwen2.5-VL, validation and test halves (post hoc): answers changed by ViT-H and ViT-W when the original answer follows the ‘higher means farther’ heuristic (left of each pair) or not (right); 95% Wilson intervals.
Qwen2.5-VL
Qwen2.5-VL
3B
7B
Test half
ViT-H
37.3% [32.1, 42.7]
43.9% [38.6, 49.5]
ViT-W
27.1% [22.5, 32.2]
19.7% [15.7, 24.5]
LLM-H
5.7%
6.1%
n
314
314
Table 5: Depth questions of CV-Bench 3D. Top: fraction of answers that change under each counterfactual on the held-out test half (95% Wilson intervals). Bottom (post hoc): change rates split by whether the original answer follows the ‘higher in the image means farther’ heuristic (following) or not (against); group sizes in parentheses (they differ slightly between conditions because tied items are excluded per condition). The remaining test-half items are ambiguous for the heuristic.
Figure 5: In four public fine-tuned Qwen2.5-VL-3B checkpoints the position-code effect is preserved, and the gain of SpatialLadder follows its LLM. (a) Agreement of ViT-W with H-flip on left–right items that the real flip changes, with 95% Wilson intervals; dashed line: base model. (b) CV-Bench Relation accuracy change over the base model for the public checkpoints and for hybrids that take only the named components from SpatialLadder; 95% paired-bootstrap intervals.
Figure 6: The ViT mirrors impair counting, while swapping the option order moves the left–right results by less than 5 points (Qwen2.5-VL, re-run; 95% paired-bootstrap intervals). (a) Accuracy drop on CV-Bench Count. (b) Swapped minus original option order for six main CV-Bench Relation quantities; gray band: ± 5 points. Only the point estimate of the 3B ViT-H up–down flip rate lies outside the band; its mean over both orders is reported.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Qwen2.5-VL
Qwen2.5-VL
InternVL3
3B
7B
2B
ViT-W vs. H-flip agreement (left–right)
87.5% → 89.9%
93.2% → 94.4%
84.3% → 85.9%
ViT-W flip rate (geometry-consistent)
83.3% → 88.1%
95.4% → 95.7%
88.5% → 89.7%
LLM-W flip rate (geometry-consistent)
0.5% → 0.0%
0.3% → 0.0%
2.5% → 1.7%
Appendix
Table C.1: Main CV-Bench metrics before and after removing every item that involves a bf16 tie between the two option logits.
Qwen2.5-VL
Qwen2.5-VL
InternVL3
3B
7B
2B
Flip rate, original answer left
62.2%
92.8%
85.5%
Flip rate, original answer right
100.0%
97.7%
91.0%
left answers under ViT-W
82.2%
55.6%
54.6%
left answers, original
71.0%
50.4%
50.7%
Appendix
Table C.2: Post hoc: CV-Bench left–right flip rate under ViT-W on the geometry-consistent subset, split by the original answer, and the share of left answers under ViT-W (all left–right items). A model that fell back to a left prior would flip original right answers but not original left answers.
Qwen2.5-VL-3B
Qwen2.5-VL-7B
InternVL3-2B
orig.
swapped
difference
orig.
swapped
difference
orig.
swapped
diff.
ViT-W vs. H-flip , agreement, left–right
87.5%
86.4%
− 1.0 [ − 4.4, +2.2]
93.2%
89.3%
− 3.9 [ − 6.6, − 1.3]
84.3%
88.5%
+4.2
ViT-W flip, left–right
83.3%
85.6%
+2.3 [ − 2.5, +7.1]
95.4%
91.7%
− 3.7 [ − 6.3, − 1.2]
88.5%
92.8%
+4.2
LLM-W flip, left–right
0.5%
1.4%
+0.9 [ − 1.0, +2.8]
0.3%
0.6%
+0.3 [ − 0.6, +1.5]
2.5%
2.6%
+0.1
ViT-H flip, up–down
45.5%
58.9%
+13.4 [+1.2, +25.8]
51.4%
54.5%
+3.0 [ − 4.4, +10.4]
69.8%
66.4%
− 3.4
LLM-H flip, up–down
9.1%
7.8%
− 1.3 [ − 8.6, +6.1]
12.1%
10.7%
− 1.5 [ − 6.5, +3.5]
4.2%
12.6%
+8.4
Appendix
Table C.3: Main CV-Bench metrics with the original and the swapped option order (re-runs; flip rates on the geometry-consistent subset of each order). Difference: swapped minus original; for Qwen2.5-VL with a joint bootstrap interval over the 650 items (10,000 resamples, seed 0), for InternVL3-2B point estimates only. bf16 ties: 754 / 721 (3B) and 239 / 258 (7B) of 12,350 rows per order. For InternVL3-2B, the off-axis up–down row uses the up–down items answered correctly on the original and the vertically flipped image, so it is not comparable with Table 2 , whose subset is defined by the horizontal flip.
Model
Agreement
LLM-W change
Gain (points)
3D conflict
Base (Qwen2.5-VL-3B)
79.6% (225)
0.0%
–
54.7%
SpatialLadder-3B [ 56 ]
87.5% (279)
0.0%
+8.0 [+5.9, +10.0]
45.3%
SpaceQwen2.5-VL-3B [ 57 ]
81.6% (136)
0.8%
− 3.2 [ − 4.8, − 1.5]
50.0%
SpaceR-3B [ 58 ]
77.3% (203)
0.3%
− 0.8 [ − 1.9, +0.2]
50.0%
VLM-R1-REC-500steps [ 59 ]
83.9% (254)
1.6%
+1.8 [+0.8, +2.8]
56.2%
Appendix
Table E.1: Public spatially fine-tuned versions of Qwen2.5-VL-3B. Agreement: share of left–right items, among those whose answer changes under H-flip , on which ViT-W gives the H-flip answer ( n in parentheses). Gain: CV-Bench accuracy relative to the base model (paired bootstrap, 2,000 resamples, both option orders). 3D conflict: accuracy on the 64 CV-Bench 3D depth items where the heuristic is wrong.
Vision side / LLM
Accuracy
− base (points)
Share of gain
Agreement
B / B (base)
76.1%
–
–
79.6%
S / S (SpatialLadder)
84.1%
+8.0 [+5.9, +10.0]
1
87.5%
B / S
85.3%
+9.2 [+7.2, +11.2]
1.15 [0.98, 1.40]
88.4%
S / B
72.0%
− 4.1 [ − 5.7, − 2.4]
− 0.51 [ − 0.86, − 0.26]
75.4%
ViT blocks 28–31 from S, rest B
75.9%
− 0.2 [ − 0.9, +0.6]
− 0.02 [ − 0.13, 0.08]
80.0%
Appendix
Table E.2: Component swap between SpatialLadder-3B (S) and its base model Qwen2.5-VL-3B (B) on CV-Bench 2D (650 questions, both option orders, n=1300 ; paired bootstrap 95% intervals, 2,000 resamples). Vision side: vision encoder and merger. Share: difference to the base model divided by the gain of SpatialLadder. Agreement: left–right agreement of ViT-W with H-flip (original option order only, as is the LLM-W change). Accuracy on the 64 CV-Bench 3D items where the ‘higher means farther’ heuristic is wrong: 54.7% (B/B), 45.3% (S/S), 39.1% (B/S), 56.2% (S/B).