Vision-language models (VLMs) have achieved strong performance on tasks such as visual question answering, yet small image resizes can turn correct answers into errors. We investigate whether changes in visual configuration, such as image tiling and token arrangement, contribute to this instability. Across seven checkpoints and four benchmarks, equally small resizes cause more correctness flips when they switch configurations. Surprisingly, in over half of these cases, models answer the question incorrectly but can still read the correct answer when told what to read. Furthermore, attention interventions in LLaVA-NeXT suggest that configuration changes can weaken the use of readable information during answering. We therefore guide models using field cues and their own transcriptions. With annotation assistance, these forms of guidance together correct 97.2% of errors with readable information. These findings show that configuration changes can affect how models use information they can still read.
Figures & tables
Figure 1: A one-pixel height difference changes MiniCPM’s tiling from four local tiles to two and flips the answer. Keeping the visual representations that produced the error fixed, guided reading extracts the target and the model’s own field record restores the original answer. This example is from TextVQA, and all responses are generated by MiniCPM-V-4.5 using greedy decoding.
Task
Qwen3-VL-8B
MiniCPM-V-4.5
LLaVA-NeXT-7B
InternVL3-8B
Qwen2.5-VL-7B
InternVL3.5-8B
Qwen3-VL-32B
Mean
VQAv2
+1.37
+3.12 †
+3.91 †
+5.26
+3.52
+4.21 †
+5.47 †
+3.84
TextVQA
+4.69 †
+4.30 †
+6.84 †
+10.87
+7.03 †
+8.70 †
+3.91 †
+6.62
DocVQA
+6.25 †
+7.81 †
+6.25 †
+16.67
+7.03 †
+3.33
+3.91 †
+7.32
ChartQA
+2.73 †
+4.30 †
+5.08 †
+9.09
+3.52
+0.00
+7.42 †
+4.59
Mean
+3.76
+4.88
+5.52
+10.47
+5.27
+4.06
+5.18
+5.59
Table 1: Excess correctness-flip rate Δ (percentage points, pp). † indicates that the unadjusted paired 95% confidence interval excludes zero. The per-task sample size is n=512 for Qwen3-8B/MiniCPM/LLaVA and n=256 for Qwen2.5/Qwen3-32B. For both InternVL checkpoints, n=95 (VQAv2), 46 (TextVQA), 30 (DocVQA), and 22 (ChartQA).
Figure 2: Bars show four-task mean correctness-flip rates when changing only the image or configuration; error bars indicate 95% confidence intervals clustered by source.
Model
Config.
Flip Δ
Acc. Δ
Tokens Δ
Qwen3
Low
-3.32
-0.34
-3.4
High
-4.05
-0.02
+3.4
MiniCPM
Low
-4.83
-2.66
-37.6
High
-7.03
+3.03
+37.6
LLaVA
Low
-6.49
-2.37
-24.0
High
-6.74
+2.00
+24.0
Table 2: Changes in disagreement and accuracy (pp) and visual token count (%) under common Low/High configurations, relative to native processing.
Figure 3: Target readability in 300 configuration-induced errors. (a) Bars show total errors and the subset with complete target transcription through guided reading for each model. (b) The pie shows the overall distribution; remaining cases are classified by outputs under a common reading prompt.
Configuration
Original Q
Guided reading R
Correct
82.1%
71.8%
Incorrect
38.5%
74.4%
Table 3: Target-attention knockout effects in LLaVA-NeXT, after subtracting non-target effects (%).
Vision-language models (VLMs) answer visual questions by combining visual information extraction with downstream problem solving. We investigate a fundamental question: Does an incorrect answer necessarily reflect a failure in visual extraction or problem solving? A model may succeed at both abilities when tested separately yet still fail on the original multimodal question, a distinction that overall answer accuracy cannot reveal. To study this, we perform a question-level empirical analysis across multiple VLMs and visual domains. We define an exactly scorable task state (i.e., the visual information sufficient to solve a question) and use it to test whether the same model can extract the required state, solve the question from the ground-truth state, and answer the original multimodal question. We find that composition failures, where extraction and solving both succeed but direct answering fails, account for 17.7% to 75.6% of direct-answering errors across multiple VLMs and datasets. To address this failure mode, we introduce a simple yet effective method, termed State Realization Tuning (SRT). SRT fine-tunes LoRA adapters attached to the language-model layers while keeping the pretrained VLM weights frozen. It trains the model to output the ground-truth task state before the final answer in a single autoregressive response. SRT improves over standard supervised fine-tuning by 1.7 to 14.1 percentage points and repairs 92.5% to 98.1% of diagnosed composition failures. A single LoRA adapter trained with SRT also improves performance across substantially different task-state structures. Our work shows that having both visual extraction and problem-solving capabilities does not guarantee correct multimodal answering. Requiring the model to first output the visual information needed to solve the question can help bridge this gap.
Ziheng Wang, Mingxuan Xie, Yilin Liu +3
Sun Yat-sen University · Zhejiang University · The Hong Kong University of Science and Technology +2
Vision-language models (VLMs) increasingly reason over visual evidence that is cropped, segmented, retrieved, or revealed over time. Yet most VQA benchmarks present the complete image and question at once. We ask what models lose when the same information is fragmented. We introduce Layered-VQA, with 93 scenes and 300 questions. Each image is decomposed into ordered RGBA layers that exactly recompose the original scene, and each question is annotated with supporting, minimal-sufficient, and distractor layers. We evaluate eleven open-weight VLMs from 3B to 32B parameters and two proprietary models with a scale of 187,200 conversations, graded by 1.74M open-model cross-judgments. We find three consistent failures. Loss in Composition: fragmenting the question has a small effect, but fragmenting the scene substantially reduces accuracy; recomposing the same layers largely restores performance. Oracle Inversion: even oracle-selected sufficient evidence can perform worse than the complete scene. Loss in Grounding: as more evidence is required, grounding degrades much faster than answer accuracy. Together, these results show that having the right visual evidence is not enough. How that evidence is composed and presented determines whether models can use and ground it. The right evidence is not enough: VLMs need the scene it came from.
L. D. M. S. Sai Teja, Ufaq Khan, N. Siva Gopala Krishna +5
Vision-Language Models (VLMs) have demonstrated strong capability in a wide range of tasks such as visual recognition, document parsing, and visual grounding. Nevertheless, recent work shows that while VLMs often manage to capture the correct image region corresponding to the question, they do not necessarily produce the correct answers. In this work, we demonstrate that this misalignment could be attributed to suboptimal information flow within VLMs, where text tokens distribute too much attention to irrelevant visual tokens, leading to incorrect answers. Based on the observation, we show that modulating the information flow during inference can improve the perception capability of VLMs. The idea is that text tokens should only be associated with important visual tokens during decoding, eliminating the interference of irrelevant regions. To achieve this, we propose a token dynamics-based method to determine the importance of visual tokens, where visual tokens that exhibit distinct activation patterns during different decoding stages are viewed as important. We apply our approach to representative open-source VLMs and evaluate on various datasets, including visual question answering, visual grounding and counting, optical character recognition, and object hallucination. The results show that our approach significantly improves the performance of baselines. Project page: https://cxliu0.github.io/AIF/.