Read Together, Answer Apart: How Earlier Questions Help Vision-Language Models Look
Organizations: South China Normal University · Tsinghua University · Beijing University of Chemical Technology · Xi’an Jiaotong University
Abstract
Vision-language models (VLMs) are often asked several questions about the same image in a single prompt (co-asking), yet existing evaluations report conflicting accuracy gains and hallucinations at the prompt level without tracking how individual answers change. By pairing each question's single-question and co-asked answers across 22 open-weight VLMs (1B-35B), we find that erratic net shifts reflect two opposing effects: on Densely Captioned Images (DCI), every model repairs 15.4%-29.8% of its single-question errors under co-asking, while the share of correct answers turned wrong falls from 30.8% on weaker models to 4.7% on stronger ones. Across all 14 models supporting custom attention masks, we show through attention readouts and layer-window interventions that earlier same-image questions read visual information into their question tokens in the early-to-middle layers during prefill, helping later questions concentrate image attention onto their queried targets. This visual help accumulates across the scene even across non-overlapping regions, whereas blocking earlier questions from attending to the image lowers both target localization and accuracy in all 14 models. While cross-question help is established during shared question reading, part of the co-asking damage enters later during answer decoding when the decoder attends to other question stems. We therefore introduce Read Together, Answer Apart, a training-free decoding mask that keeps shared prefill intact and isolates each answer from other stems, improving co-asking accuracy on 12 of 14 models on DCI (up to +4.71 percentage points, closing 70% of the pooled gap to asking alone) and on 10, 8, and 5 of the 14 models on WebUI, FinTabNet, and GQA (up to +11.4, +5.3, and +1.1 points).
Figures & tables
| Position | Repair | Damage | |||
|---|---|---|---|---|---|
| Co-asking | Ours | Co-asking | Ours | ||
| 1 | 1,384 | 53 | 24 | 127 | 61 |
| 2–16 | 20,797 | 1,408 | 1,494 | 1,901 | 1,625 |
| All (1–16) | 22,181 | 1,461 | 1,518 | 2,028 | 1,686 |
| Benchmark | Repair (%) | 1-Q | Co | Ours | Improved (max ) |
|---|---|---|---|---|---|
| DCI ( ) | 62.8 | 60.2 | 62.0 | ( ) | |
| WebUI ( ) | 72.6 | 67.3 | 69.6 | ( ) | |
| FinTabNet ( ) | 46.9 | 46.6 | 46.3 | ( ) | |
| GQA ( ) | 64.3 | 64.4 | 64.4 | ( ) |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Question position | |||||
| Setting / Cohort | 1 | 2–5 | 6–10 | 11–16 | 2–16 |
| Same-image ( VLMs) | 11.1 | 20.3 | 21.8 | 23.9 | 22.3 |
| Text-only ( VLMs) | 10.0 | 17.3 | 16.1 | 14.5 | 15.7 |
| Other-image ( VLMs) | 8.3 | 15.7 | 13.2 | 13.1 | 13.7 |
| 22 VLMs ( ) | — | 18.6 | 21.2 | 22.5 | 20.9 |
| 22 VLMs ( ) | — | 25.0 | 20.3 | 24.7 | 23.6 |