A vision-language model may take left-right direction from the orientation of image content within patches, its vision transformer's (ViT) position code, or token positions and order in its language model (LLM). Ablations show the ViT code is necessary; we test sufficiency by mirroring only a position code. In Qwen2.5-VL-7B, Qwen3-VL-4B and InternVL3-2B, mirroring ViT width positions reproduces the flipped-image answer on 84.3-93.2% of CV-Bench left-right questions (Qwen2.5-VL-3B: 87.5%, partly a left fallback). With two options, this also counts disruption; a half-grid shift flips most answers too. With the ViT code intact, LLM-side id and token-order mirrors change few left-right answers in all three series (a weak id test in Qwen2.5-VL). In Qwen2.5-VL-7B the effect sits in the global-attention ViT blocks, mostly the last. On What'sUp, flipping content and token order, not the position code, keeps more on/under than left-right answers at the flipped-image answer. Post hoc analyses qualify these results. On CV-Bench items with located objects, random permutations flip 18.9-31.0 points more answers when reversing the objects' order than when keeping it. On What'sUp, in Qwen2.5-VL and Qwen3-VL, the share of the left-right changes made by random permutations that land on the mirror relation is mostly below chance. Counted as returns to the original answer, the What'sUp contrast of support with left-right relations is clear only in InternVL3-2B. Most remaining errors give the same relation whichever object is asked about, suggesting that repair may target relation binding, object localization or use of the reference object.
Figures & tables
Work
Evidence
Intervention
Not addressed
[ 5 ]
necessity (ablation); accuracy per relation
mask / shuffle / constant per axis, ViT ( h , w ) and LLM ( t , x , y ), pixels fixed; What’sUp left / right / on / under
per-item comparison with a real flip; ids vs. token order; block localization; split by object order
[ 6 ]
geometry; correlation; method
X/Y decomposition of 2D-RoPE attention; embedding-level interventions; frequency scaling
direction-predicting edit of the position code; LLM-side ids
[ 27 ]
attention decomposition; ablation
position × token interaction in CLIP-style encoders (synthetic 1D scenes)
pretrained VLMs; LLM side
[ 7 ]
content patching; interchange (sufficiency-type)
visual-token content; left and right regions swapped with a counterfactual image
position code; LLM-side ids and order
[ 8 ]
activation patching (sufficiency-type); mean-difference spatial IDs and ID steering
activations from a mirrored-image run patched into the original run; spatial IDs
input-side position code (ViT or LLM ids)
[ 31 ]
state patching with horizontal flips (sufficiency-type)
image, query-text and final-token states between an image and its flip (Qwen3-VL-4B)
position code; heads and MLPs
Table 1: Closest prior work. The sufficiency-type interventions among the listed works patch activations or visual-token content from a counterfactual image; none of the listed works compares the answer after an edit of the position code alone, item by item, with the answer to a real image transform.
Figure 1: Position-only counterfactuals keep every pixel and mirror only a position code. Rows: real horizontal flip ( H-flip , reference); ViT-W : width index w→W−1−w of each patch in the ViT; LLM-W : LLM-side width ids of the visual tokens mirrored. Answers measured on this photograph; percentages: changed geometry-consistent left–right answers, four models ( Figure 2 a), not agreement with H-flip ; last-block note: Figure 3 .
Qwen2.5-VL
Qwen2.5-VL
InternVL3
Qwen3-VL
InternVL3
3B
7B
2B
4B
8B
ViT position code
2D-RoPE
2D-RoPE
learned abs.
abs. + 2D-RoPE
learned abs.
LLM position ids
(t,h,w)
(t,h,w)
1D
(t,h,w)
1D
Accuracy, left–right (original)
73.1%
90.1%
72.1%
93.2%
93.2%
Accuracy, up–down (original)
74.2%
89.5%
78.3%
–
89.5%
Agreement with the real horizontal flip (all 383 left–right items)
Table 2: Position-only counterfactuals on CV-Bench Relation (383 left–right, 267 up–down questions). Agreement: share of questions whose counterfactual answer equals the answer under the matching real transform. Flip rate: share of geometry-consistent items (correct on the original and on the real transform) whose answer changes. Brackets: 95% Wilson intervals (integer percent). In Qwen2.5-VL-3B the original answers already agree with the flipped-image answers on 46.0% of the left–right questions, mostly because both are left , which inflates its agreement. InternVL3-8B: scale check; equal values in the 7B and 8B columns are coincidences (their answers differ on 42 of 650 original and 74 ViT-W questions). † Swapped options: 58.9%; mean of both orders 52.2% ( C ). Qwen3-VL-4B: same bf16 scoring; –: the 180∘ conditions and the off-axis ViT-H control were not run, and up–down accuracy and the all-item off-axis rate were not computed; its up–down subsets are defined by the vertical flip; the tie-removed analyses ( C ) do not include it; bf16 ties on 14 of its 344 geometry-consistent left–right items. ‡ Original answers without the frame sentence that the 180∘ conditions add ( Section 3.4 ).
Figure 2: Mirroring the ViT width positions flips most left–right answers; mirroring the LLM-side positions changes almost none of them. Share of geometry-consistent items whose answer changes, with 95% Wilson intervals; color: condition, open: LLM-side mirror, shape: model. (a) CV-Bench Relation, Qwen2.5-VL-3B in the original option order. (b) What’sUp: change to the mirror relation (markers) and any change ( × ); Qwen3-VL-4B: left–right conditions only, and for its LLM-W only any change. Subset sizes in Tables 2 and 4 .
Qwen2.5-VL
Qwen2.5-VL
3B
7B
Left–right items (383), all items
order only, width (ids follow tokens)
2.3%
2.9%
order only, height (off-axis control)
3.7%
4.4%
id only, width (the LLM-W edit; all items)
0.8%
0.3%
order and id, width
2.3%
2.3%
Table 3: LLM-side token order in Qwen2.5-VL: fraction of answers that change when the visual tokens are reordered in the LLM input ( order only : each token keeps its own M-RoPE id; id only : tokens stay in place and their ids are mirrored). Pooling the three shuffle seeds overstates the precision of that control; in neither model does the mirrored row order change clearly more up–down answers than both non-mirror permutations, and in 3B only about half of its flips agree with the real vertical flip.
Figure 3: The left–right effect of the width mirror is concentrated in the global-attention ViT blocks, chiefly the last, whereas in the 3B model the up–down effect of the height mirror builds up across blocks (Qwen2.5-VL, all CV-Bench Relation items, 95% Wilson intervals). (a) ViT-W in single blocks or block sets (shape: model); ‘Global’: the four full-attention blocks 7, 15, 23 and 31. (b) Dose–response in the 3B model, mirroring the first or the last L blocks.
Qwen2.5-VL
Qwen2.5-VL
InternVL3
Qwen3-VL
3B
7B
2B
4B
Accuracy, left–right / up–down (original)
95.1% / 94.7%
85.0% / 98.1%
98.1% / 92.7%
99.0% / –
Left–right items, geometry-consistent subset
ViT-W : answer changes
100.0%
100.0%
99.0%
99.5%
ViT-W : changes to the mirror relation
82.3% [76, 87]
72.1% [65, 79]
94.9% [91, 97]
90.6% [86, 94]
LLM-W : answer changes
0.0%
0.0%
1.0%
1.5%
Table 4: What’sUp (Controlled Images A; 412 questions with four options: left , right , on , under ). The main metric is the fraction that changes to the mirror relation (left ↔ right or on ↔ under); a change to an off-axis option is not a directional effect. On up–down items, random row permutations also turn 86–100% (Qwen2.5-VL) and 70–86% (InternVL3-2B) of the answers they change into on ↔ under, so the up–down mirror-relation row does not show direction. Qwen3-VL-4B: left–right conditions only; –: not run.
Figure 4: Up–down answers follow the position code less than left–right answers, and depth answers that follow the height cue depend on the ViT height positions. (a) Blank-image decomposition of all CV-Bench Relation items into answers that follow position (flip under the matching ViT mirror, disruption included), follow content, or neither (which includes image-dependent answers). (b) CV-Bench 3D depth, Qwen2.5-VL, validation and test halves (post hoc): answers changed by ViT-H and ViT-W when the original answer follows the ‘higher means farther’ heuristic (left of each pair) or not (right); 95% Wilson intervals.
Qwen2.5-VL
Qwen2.5-VL
3B
7B
Test half
ViT-H
37.3% [32.1, 42.7]
43.9% [38.6, 49.5]
ViT-W
27.1% [22.5, 32.2]
19.7% [15.7, 24.5]
LLM-H
5.7%
6.1%
n
314
314
Table 5: Depth questions of CV-Bench 3D. Top: fraction of answers that change under each counterfactual on the held-out test half (95% Wilson intervals). Bottom (post hoc): change rates split by whether the original answer follows the ‘higher in the image means farther’ heuristic (following) or not (against); group sizes in parentheses (they differ slightly between conditions because tied items are excluded per condition). The remaining test-half items are ambiguous for the heuristic.
Figure 5: In four public fine-tuned Qwen2.5-VL-3B checkpoints the position-code effect is preserved, and the gain of SpatialLadder follows its LLM. (a) Agreement of ViT-W with H-flip on left–right items that the real flip changes, with 95% Wilson intervals; dashed line: base model. (b) CV-Bench Relation accuracy change over the base model for the public checkpoints and for hybrids that take only the named components from SpatialLadder; 95% paired-bootstrap intervals.
Figure 6: The ViT mirrors impair counting, while swapping the option order moves the left–right results by less than 5 points (Qwen2.5-VL, re-run; 95% paired-bootstrap intervals). (a) Accuracy drop on CV-Bench Count. (b) Swapped minus original option order for six main CV-Bench Relation quantities; gray band: ± 5 points. Only the point estimate of the 3B ViT-H up–down flip rate lies outside the band; its mean over both orders is reported.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Qwen2.5-VL
Qwen2.5-VL
InternVL3
3B
7B
2B
ViT-W vs. H-flip agreement (left–right)
87.5% → 89.9%
93.2% → 94.4%
84.3% → 85.9%
ViT-W flip rate (geometry-consistent)
83.3% → 88.1%
95.4% → 95.7%
88.5% → 89.7%
LLM-W flip rate (geometry-consistent)
0.5% → 0.0%
0.3% → 0.0%
2.5% → 1.7%
Appendix
Table C.1: Main CV-Bench metrics before and after removing every item that involves a bf16 tie between the two option logits.
Qwen2.5-VL
Qwen2.5-VL
InternVL3
3B
7B
2B
Flip rate, original answer left
62.2%
92.8%
85.5%
Flip rate, original answer right
100.0%
97.7%
91.0%
left answers under ViT-W
82.2%
55.6%
54.6%
left answers, original
71.0%
50.4%
50.7%
Appendix
Table C.2: Post hoc: CV-Bench left–right flip rate under ViT-W on the geometry-consistent subset, split by the original answer, and the share of left answers under ViT-W (all left–right items). A model that fell back to a left prior would flip original right answers but not original left answers.
Qwen2.5-VL-3B
Qwen2.5-VL-7B
InternVL3-2B
orig.
swapped
difference
orig.
swapped
difference
orig.
swapped
diff.
ViT-W vs. H-flip , agreement, left–right
87.5%
86.4%
− 1.0 [ − 4.4, +2.2]
93.2%
89.3%
− 3.9 [ − 6.6, − 1.3]
84.3%
88.5%
+4.2
ViT-W flip, left–right
83.3%
85.6%
+2.3 [ − 2.5, +7.1]
95.4%
91.7%
− 3.7 [ − 6.3, − 1.2]
88.5%
92.8%
+4.2
LLM-W flip, left–right
0.5%
1.4%
+0.9 [ − 1.0, +2.8]
0.3%
0.6%
+0.3 [ − 0.6, +1.5]
2.5%
2.6%
+0.1
ViT-H flip, up–down
45.5%
58.9%
+13.4 [+1.2, +25.8]
51.4%
54.5%
+3.0 [ − 4.4, +10.4]
69.8%
66.4%
− 3.4
LLM-H flip, up–down
9.1%
7.8%
− 1.3 [ − 8.6, +6.1]
12.1%
10.7%
− 1.5 [ − 6.5, +3.5]
4.2%
12.6%
+8.4
Appendix
Table C.3: Main CV-Bench metrics with the original and the swapped option order (re-runs; flip rates on the geometry-consistent subset of each order). Difference: swapped minus original; for Qwen2.5-VL with a joint bootstrap interval over the 650 items (10,000 resamples, seed 0), for InternVL3-2B point estimates only. bf16 ties: 754 / 721 (3B) and 239 / 258 (7B) of 12,350 rows per order. For InternVL3-2B, the off-axis up–down row uses the up–down items answered correctly on the original and the vertically flipped image, so it is not comparable with Table 2 , whose subset is defined by the horizontal flip.
Model
Agreement
LLM-W change
Gain (points)
3D conflict
Base (Qwen2.5-VL-3B)
79.6% (225)
0.0%
–
54.7%
SpatialLadder-3B [ 56 ]
87.5% (279)
0.0%
+8.0 [+5.9, +10.0]
45.3%
SpaceQwen2.5-VL-3B [ 57 ]
81.6% (136)
0.8%
− 3.2 [ − 4.8, − 1.5]
50.0%
SpaceR-3B [ 58 ]
77.3% (203)
0.3%
− 0.8 [ − 1.9, +0.2]
50.0%
VLM-R1-REC-500steps [ 59 ]
83.9% (254)
1.6%
+1.8 [+0.8, +2.8]
56.2%
Appendix
Table E.1: Public spatially fine-tuned versions of Qwen2.5-VL-3B. Agreement: share of left–right items, among those whose answer changes under H-flip , on which ViT-W gives the H-flip answer ( n in parentheses). Gain: CV-Bench accuracy relative to the base model (paired bootstrap, 2,000 resamples, both option orders). 3D conflict: accuracy on the 64 CV-Bench 3D depth items where the heuristic is wrong.
Vision side / LLM
Accuracy
− base (points)
Share of gain
Agreement
B / B (base)
76.1%
–
–
79.6%
S / S (SpatialLadder)
84.1%
+8.0 [+5.9, +10.0]
1
87.5%
B / S
85.3%
+9.2 [+7.2, +11.2]
1.15 [0.98, 1.40]
88.4%
S / B
72.0%
− 4.1 [ − 5.7, − 2.4]
− 0.51 [ − 0.86, − 0.26]
75.4%
ViT blocks 28–31 from S, rest B
75.9%
− 0.2 [ − 0.9, +0.6]
− 0.02 [ − 0.13, 0.08]
80.0%
Appendix
Table E.2: Component swap between SpatialLadder-3B (S) and its base model Qwen2.5-VL-3B (B) on CV-Bench 2D (650 questions, both option orders, n=1300 ; paired bootstrap 95% intervals, 2,000 resamples). Vision side: vision encoder and merger. Share: difference to the base model divided by the gain of SpatialLadder. Agreement: left–right agreement of ViT-W with H-flip (original option order only, as is the LLM-W change). Accuracy on the 64 CV-Bench 3D items where the ‘higher means farther’ heuristic is wrong: 54.7% (B/B), 45.3% (S/S), 39.1% (B/S), 56.2% (S/B).
Where should the question go in a vision-language model (VLM) prompt: before the image or after it? Intuition says before: knowing what is asked should tell the model where to look. Yet across visual question answering benchmarks, question-first prompting consistently underperforms the image-first ordering recommended for frontier VLMs, a phenomenon we term the question-first paradox. We trace this paradox to a conflict between two stages of VLM computation. Logit-lens and attention probes show that question-first prompting steers perception, shifting image patch representations toward question-relevant concepts. But downstream, stranded behind hundreds of image tokens, the question is barely attended by the answer token, which instead commits to image-driven, often wrong answers. Causal attention knockout confirms that the answer reads the question only when it follows the image. This diagnosis yields a training-free fix: question echoing, restating the question on both sides of the image so one copy steers perception while the other is available at answer time. A similar division of labor appears in a fifty-year-old finding on human 'adjunct questions', where repeating a question before and after a passage improves comprehension. Echoing the image as well brings further gains by restoring the whole-image view otherwise lost by a causal decoder. The paradox holds across five open VLMs, costing up to 17.5 group-accuracy points. Echoed prompts recover most of the gap and, on NaturalBench and Winoground, surpass the best single-pass ordering by up to 19 group-accuracy points on Winoground, with no training, fine-tuning, or architecture change. The paradox reveals a tension between steering what a model sees and preserving access to what it was asked; echoing resolves this through prompt design. Project Page: https://rakshanda-cmu.github.io/ask-twice-look-twice/
Rakshanda Hassan Abhinandan, John Galeotti, Deva Ramanan +1
Vision-language models (VLMs) achieve strong visual reasoning performance, yet subtle changes from routine image capture and processing can alter their reasoning trajectories even when images appear nearly identical. In long-horizon generation, the resulting activation shifts may accumulate across decoding steps, progressively altering reasoning tokens and ultimately changing the final answer, a phenomenon referred to as answer flips. To address this instability, we propose FlipDir (Flip-Direction Steering), a training-free inference-time method that estimates a low-rank flip-inducing activation subspace from contrastive pairs of original and answer-flipping inputs and selectively steers hidden states during decoding. A margin-based gate limits subspace attenuation to uncertain decoding steps, recovering original predictions while preserving stable ones. To evaluate robustness beyond accuracy or consistency on fixed test sets, we introduce VisFlip, a benchmark framework that constructs evaluation groups for a target model and visual variation setting to separately assess recovery of original predictions and preservation of stable ones. VisFlip spans nine dataset-variation combinations across scientific reasoning, robot-scene understanding, and medical VQA, covering subtle visual variations common in each domain. Experiments across 18 settings demonstrate that FlipDir consistently outperforms existing methods on the combined recovery and preservation metric. We will make our code publicly available.
Yeonsung Jung, Joonhyun Jeong, Hoang Pham +4
Graduate School of AI, KAIST · NAVER Cloud · Department of Computer Science and Engineering, The Ohio State University +3
We find that vision-language models are sensitive to a specific semantically irrelevant change: the order in which the image and question are presented. Across three models and three benchmarks, image first prompting consistently outperforms question-first prompting, revealing a repeatable modality order failure. We use this gap to design an order-consistent test-time training method. Our method substantially closes the modality-order gap across all evaluated settings. Surprisingly, it also yields consistent improvements in the stronger image-first branch over the baseline, hence bootstrapping both orderings toward mutual consistency. Activation patching localizes the ordering failure to a narrow mid-network region where representations diverge sharply between prompt orders. We find that the test-time training method repairs this misalignment across layers. Together, our results identify modality-order sensitivity as a circuit-level failure in VLMs and demonstrate that simple, asymmetric test-time adaptation can effectively mitigate it and even improve performance over the baseline.