Vision-language models (VLMs) are often asked several questions about the same image in a single prompt (co-asking), yet existing evaluations report conflicting accuracy gains and hallucinations at the prompt level without tracking how individual answers change. By pairing each question's single-question and co-asked answers across 22 open-weight VLMs (1B-35B), we find that erratic net shifts reflect two opposing effects: on Densely Captioned Images (DCI), every model repairs 15.4%-29.8% of its single-question errors under co-asking, while the share of correct answers turned wrong falls from 30.8% on weaker models to 4.7% on stronger ones. Across all 14 models supporting custom attention masks, we show through attention readouts and layer-window interventions that earlier same-image questions read visual information into their question tokens in the early-to-middle layers during prefill, helping later questions concentrate image attention onto their queried targets. This visual help accumulates across the scene even across non-overlapping regions, whereas blocking earlier questions from attending to the image lowers both target localization and accuracy in all 14 models. While cross-question help is established during shared question reading, part of the co-asking damage enters later during answer decoding when the decoder attends to other question stems. We therefore introduce Read Together, Answer Apart, a training-free decoding mask that keeps shared prefill intact and isolates each answer from other stems, improving co-asking accuracy on 12 of 14 models on DCI (up to +4.71 percentage points, closing 70% of the pooled gap to asking alone) and on 10, 8, and 5 of the 14 models on WebUI, FinTabNet, and GQA (up to +11.4, +5.3, and +1.1 points).
Figures & tables
Figure 1: Earlier questions help a later question find its target. Two repaired DCI examples on Gemma-4-12B (layers 17,23,29 of 48 at the final question token; answers abbreviated). Each pair shares a display scale; white boxes mark targets and percentages report unsmoothed in-box shares of image attention (cropped for display; full frames in Supp. Figs. S2 and S4 ).
Figure 2: Repair, damage, and preceding-context effects across 22 VLMs on DCI (one shared question order across all 22 models and panels). (a) Single-question accuracy ( 1-Q ), paired co-asking accuracy shift ( Δ=Co−1-Q , percentage points), and conditional repair (among single-question errors) and damage (among single-question correct answers). (b) Per-model same-image repair gains over text-only questions and other-image questions at Positions 2–16. (c,d) Pooled repair and damage across positions under the three preceding contexts. (e) Pooled same-image repair on the 1,402 regional questions with ( IoU≥0.1 ) and without ( IoU<0.1 ) prior target-box overlap. (f) Pooled Positions 2–16 repair summary across the same 1,402 regional questions.
Figure 3: Co-asking increases prefill attention on the queried target across scenes and VLM families. (a) On one shared temple scene (Gemma-4-12B, layers 17,23,29 of 48 ), asking different questions after earlier scene questions increases final-token image attention on each queried target (white boxes; cyan dashed boxes mark earlier questions). (b) Middle-layer patch attention at the final question token on repaired questions across Ministral-3-8B (layers 12 – 22 of 34 ), Qwen3-VL-8B ( 12 – 23 of 36 ), and Qwen3-VL-32B ( 35 – 51 of 64 ). Each pair shares a display ceiling; percentages report unsmoothed in-box shares of image attention (Supp. Sec. B ).
Table 1: Target attention ratio and earlier-image block across 14 models on DCI (Positions 2–16, one shared question order). Attention ratio / alone : layer-averaged target-over-random attention ratio (adapted from [ 49 ] ) divided by asking alone ( 1.000 ) at the final question token under co-asking (Co-ask), after text-only questions (Text-only), and under earlier-image blocking (Blocked, V↛Q<k ; unnormalized values in Supp. Tab. S3 ). ΔAcc. : accuracy change (%) from blocking V→Q<k relative to the unblocked isolated-decoding baseline ( N=20,797 Positions 2–16 pairs; Sec. 4.2 ).
Figure 4: Two-step cross-question visual routing across all 14 models. (a) Two-step prefill routing across layers 0…L−1 (1: earlier questions Q<k read image patches; 2: current question Qk reads Q<k and shifts attention from background patches v1…v3 onto its queried target patch v∗ before answer decoding) and windowed blocking of earlier stems from attending to the image ( V↛Q<k ) within layers a – b while leaving Q<k→Qk and direct V→Qk open. (b) Percentage of answers lost per layer window across all 14 models on repaired questions versus questions answered right in both settings (Positions 2–16, one question order). Blue shading marks the most sensitive V↛Q<k windows (both peaks shaded on bimodal GLM-4.6V-Flash); dashed teal spans mark layers with the largest drop in Qk target attention ratio.
Figure 5: Shared question prefill and isolated answer decoding. (a) Standard co-asking encodes [V,I,Q1,…,QK] under left-to-right causal masking during prefill (blue cells), then exposes each decoded answer Ak to all other question stems Qj=k (red cells). (b) IPPD [ 12 ] masks cross-question attention in both prefill and decoding. (c) Read Together, Answer Apart (Ours) keeps one shared causal prefill pass so later questions read earlier ones, then isolates each decoded answer Ak from non-target stems Qj=k while re-indexing [A<k,Ak] consecutively after Qk . Each Decode row shows that answer’s own mask Mk ( Eq. 1 ; line-break handling per tokenizer in Supp. Sec. D ).
Position
N
Repair
Damage
Co-asking
Ours
Co-asking
Ours
1
1,384
53
24
127
61
2–16
20,797
1,408
1,494
1,901
1,625
All (1–16)
22,181
1,461
1,518
2,028
1,686
Table 2: Repair and damage counts by question position on DCI (pooled across all 14 models under one question order, N=22,181 : 1,384 at Position 1 +20,797 at Positions 2–16). Repair : wrong alone → right; Damage : right alone → wrong.
Figure 8
Benchmark
Repair (%)
1-Q
Co
Ours
Improved (max Δ )
DCI ( 16Q )
15.4–22.0
62.8
60.2
62.0
12/14 ( +4.7 )
WebUI ( 10Q )
12.9–25.9
72.6
67.3
69.6
10/14 ( +11.4 )
FinTabNet ( 20Q )
14.7–46.8
46.9
46.6
46.3
8/14 ( +5.3 )
GQA ( 5Q )
7.8–19.0
64.3
64.4
64.4
5/14 ( +1.1 )
Table 4: Cross-benchmark summary across all 14 models. Repair : single-question errors fixed under one-pass co-asking (range over the 14 models; Supp. Tabs. S2 and S8 ). 1-Q / Co / Ours : count-pooled accuracies (%); Improved : models where Ours exceeds co-asking, with max gain in points (Supp. Tab. S9 ).
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1 : Representative DCI scene, co-asked questions, and preceding-context controls (verbatim stems in a, compactly displayed in b; format suffixes in Appendix A ; benchmark examples for WebUI, FinTabNet, and GQA appear in Figs. S6 , S7 and S8 ). (a) Representative DCI harbor scene with 8 of its 16 co-asked questions spanning all eight visual ability categories under the shared prompt header. (b) Side-by-side comparison of the three preceding-context controls (main Section 4.1 and Appendix C ) evaluating the same Position-3 target question ( 4 in a) under same-image, length-matched text-only, and unseen other-image preceding questions with a shared N/A clause, alongside 22-VLM pooled Positions 2–16 repair and damage rates.
Table S1 : Single-question accuracy (%) by DCI ability category across all 22 open-weight VLMs ( 100 DCI scenes, 1,597 questions; ordered by overall single-question accuracy as in main Figure 2 a; † marks the 14 models supporting custom 4D attention masks; Global [ 199 ] denotes the semantic ability category, whereas the 195 whole-image questions in main Section 3.1 are items without a localized target box).
Table S2 : Per-model DCI co-asking results and Positions 2–16 preceding-context repair and damage rates across 22 open-weight VLMs ( 100 DCI scenes, percentages; Rank 1 – 22 matches Tab. S1 and main Figure 2 a). Left: single-question accuracy ( Single-Q ) and paired one-pass co-asking repair rate, damage rate, and net accuracy shift ( Net Δ ) across Positions 1–16 under the standard prompt. Middle and right: conditional repair and damage rates across Positions 2–16 comparing same-image questions ( Same ), synthetic text-only questions ( Text-only ), and unseen other-image questions ( Other ) under the preceding-context control prompt, which appends the N/A clause to all four arms including its matched single-question baseline ( Appendix C ).
Figure S2 : Sink-filtered and raw ( σ=0 ) patch-grid controls on Gemma-4-12B. Symmetric boundary-sink-filtered maps ( σ=0.82 display smoothing, shared per-row 99.5th-percentile ceiling) alongside unsmoothed raw patch grids across four repaired questions. Pond (main Figure 1 , top), Temple steps (main Figure 3 a), and Church benches show full uncropped 3:2 frames; Lawn group (main Figure 1 , bottom) shows its 3:2 viewport, with the full uncropped 2:3 portrait frame in Fig. S4 .
Figure S3 : Additional shared-scene multi-question example on Gemma-4-12B (Fountain scene, layers 17 , 23 , 29 ). Position 3 asks which of four listed people is farthest from the camera, and Position 5 asks which listed object or person is on the nearest woman’s own left; raw full-image target-box shares rise from 0.73% to 4.2% and from 0.56% to 3.8% .
Figure S4 : Full-frame portrait control for the lawn scene in main Figure 1 (bottom). Uncropped 2:3 raw patch grids ( σ=0 ) retain boundary sink patches while showing target-box concentration under co-asking; all quantitative target shares precede display cropping.
Table S3 : Raw unnormalized target attention ratios across all 14 models on DCI (Positions 2–16, final token of each question stem under the standard prompt). Entries report the layer-averaged ratio of mean target-box attention to mean same-size non-overlapping random-box attention; dividing by Alone yields main Table 1 .
Question position
Setting / Cohort
1
2–5
6–10
11–16
2–16
Same-image ( 22 VLMs)
11.1
20.3
21.8
23.9
22.3
Text-only ( 22 VLMs)
10.0
17.3
16.1
14.5
15.7
Other-image ( 22 VLMs)
8.3
15.7
13.2
13.1
13.7
22 VLMs ( IoU<0.1 )
—
18.6
21.2
22.5
20.9
22 VLMs ( IoU≥0.1 )
—
25.0
20.3
24.7
23.6
Appendix
Table S4 : Pooled DCI repair rate (%) across question positions by preceding context and regional target-box overlap (shared control prompt across 22 VLMs; top block covers all 1,597 questions as in main Figure 2 c; bottom block splits the 1,402 regional questions by IoU with earlier target boxes as in main Figure 2 e,f; regional text-only repair across Positions 2–16 is 15.0% ).
Table S5 : All-layer and layer-windowed blocking of earlier questions’ image attention ( V↛Q<k ) and subsequent target refocusing across all 14 models on DCI (Positions 2–16, inclusive 0 -based layer intervals a – b ). Columns report all-layer repaired-answer loss ( All R lost , underlying main Section 4.2; pooled 24.6% , 368/1,494 ), total layers L with nominal window width ( w ), per-stratum sample size N , peak V↛Q<k reading window, percentage of repaired items ( R ) and doubly correct items ( C ) lost in that window, deep-layer window range (upper third of layers) and R losses, and the Qk refocusing window (with single-layer attention-ratio peak and window-start lag).
Figure S5 : Raw all-key and image-normalized patch-attention views on a repaired Ministral-3-8B gymnasium question (Position 11, layers 9 – 18 ). (a) Raw patch weights retain the all-key softmax denominator across image and text tokens (whose total image attention mass shifts from 26.6% when asked alone to 12.7% and 14.5% in the 16 -question prompt), displayed with nearest-neighbor patches and no smoothing. (b) Dividing each map by its total image attention mass under a shared display ceiling ( σ=0.6 ) isolates spatial redistribution within the image: unsmoothed target-box share rises from 9.4% to 18.7% under co-asking and recedes to 16.2% when V→Q<k is blocked.
Table S6 : Main-order paired flip counts and three-order DCI robustness across all 14 models. Main-order N ( +/− ) gives the four-setting common valid sample size and answers newly right/wrong under Ours vs. Co-Ask underlying main Table 3 . Remaining columns pool three random question orders per scene ( Order 1 from the main paper plus two independent random permutations Orders 2–3 ; N=66,554 total, N2–16=62,385 for ΔBlk2–16=Blocked−Ours ); 1-Q reports full-set single-question accuracy as in Tab. S1 , while Co-Ask and Ours report paired fixed-prefix accuracies on the common valid set.
Table S7 : Closed-candidate log-likelihood margins across all 14 models on DCI ( N=1,046 closed-candidate questions: ground-truth candidate log-likelihood minus strongest-distractor log-likelihood; ΔOurs=Ours−Co-ask and ΔBlk=Blocked−Ours are computed prior to two-decimal rounding).
Figure S6 : Representative WebUI benchmark example ( 238 webpage screenshots, 10 co-asked questions per page). Color-coded region markers highlight the DOM elements queried by Questions 1–5 alongside verbatim question stems and gold answers.
Figure S7 : Representative FinTabNet benchmark example ( 300 financial tables, 20 co-asked questions per table). Color-coded boxes mark the cells and rows queried by Questions 1–5 alongside question stems and gold answers.
Figure S8 : Representative GQA benchmark examples ( 700 scenes, 5 entity-disjoint questions per scene rotated across 10 balanced question orders). Two scenes show the five disjoint queried entities, verbatim stems, gold answers, and balanced position rotation.
Table S9 : Cross-dataset accuracy (%) across all 14 models on WebUI, FinTabNet, and GQA. Columns report single-question querying (1-Q), fixed-prefix per-question co-asking (Co), Read Together, Answer Apart (Ours), and ΔOurs=Ours−Co on each benchmark’s common valid set, plus Positions 2–10 earlier-image blocking ( ΔBlk=Blocked−Ours ) on WebUI (boldface marks the unrounded higher of Co and Ours per model, and both on the GQA pooled tie; model-level improvement counts in main Table 4 likewise compare unrounded accuracies).
Where should the question go in a vision-language model (VLM) prompt: before the image or after it? Intuition says before: knowing what is asked should tell the model where to look. Yet across visual question answering benchmarks, question-first prompting consistently underperforms the image-first ordering recommended for frontier VLMs, a phenomenon we term the question-first paradox. We trace this paradox to a conflict between two stages of VLM computation. Logit-lens and attention probes show that question-first prompting steers perception, shifting image patch representations toward question-relevant concepts. But downstream, stranded behind hundreds of image tokens, the question is barely attended by the answer token, which instead commits to image-driven, often wrong answers. Causal attention knockout confirms that the answer reads the question only when it follows the image. This diagnosis yields a training-free fix: question echoing, restating the question on both sides of the image so one copy steers perception while the other is available at answer time. A similar division of labor appears in a fifty-year-old finding on human 'adjunct questions', where repeating a question before and after a passage improves comprehension. Echoing the image as well brings further gains by restoring the whole-image view otherwise lost by a causal decoder. The paradox holds across five open VLMs, costing up to 17.5 group-accuracy points. Echoed prompts recover most of the gap and, on NaturalBench and Winoground, surpass the best single-pass ordering by up to 19 group-accuracy points on Winoground, with no training, fine-tuning, or architecture change. The paradox reveals a tension between steering what a model sees and preserving access to what it was asked; echoing resolves this through prompt design. Project Page: https://rakshanda-cmu.github.io/ask-twice-look-twice/
Rakshanda Hassan Abhinandan, John Galeotti, Deva Ramanan +1
When do vision-language models need direct access to the image while generating an answer? We study image dependence during answer generation by examining how the visual information needed for the current question becomes available in context. We intervene on direct image access while retaining previously computed states. Across real-image and synthetic tasks, we show that, depending on the generation process, direct access can continue to support accuracy after question processing. Supplying the required attributes as text in the context weakens this dependence. On synthetic tasks, we also examine how dependence changes as the model itself states the required attributes. Before attribute expression, severing access reduces accuracy, and replacing image-side states shifts answers toward the counterfactual content. After sufficient expression, both interventions have smaller effects. Even with an identical generated prefix, dependence differs according to whether image access was available during question processing. Thus, both the visible text and the preceding image access matter. Several of these patterns hold across model families, including Qwen2.5-VL-32B and InternVL3-14B. These findings offer a view of image dependence in terms of the information needed for the current question and the history of image access, beyond generation position alone. This perspective provides a basis for deciding when to reduce visual access during an answer and which visual information to retain for subsequent questions.
Vision-language models (VLMs) are fragile under image corruption. We find that the wording of the question affects VLMs in two opposite ways. Verbose questions make VLMs substantially more robust---e.g., rephrasing "Is there a cat?" into "Please look carefully and answer: is there a cat?". Conversely, VLMs become more fragile under corruption when the question is semantically complex or finer-grained, e.g., "what colour is the cup left of the chair?" instead of "is there a cup?". Both effects stem from question-conditioned cross-modal attention, which induces a spectral filter over image patches: verbose questions broaden its frequency support, while fine-grained questions concentrate it onto fewer visual scales. The model's answer drifts most when this filter and the corruption sit on the same spatial frequencies. We test the filter view on Qwen3-VL and LLaVA-OneVision across GQA and CLEVR; verbose paraphrasing reduces drift variance by 70--81% on the 8B models. The practical recipe---pad the prompt---further yields measurable gains in accuracy, even under image corruption.
Farooq Ahmad Wani, Maria Sofia Bucarelli, Mujtaba Hussain Mirza +5
Sapienza University of Rome · Université Côte d’Azur · CNRS +3