Vision-language models (VLMs) suffer from visual hallucination, such as saying an object that is not present in an image. We ask a more fundamental question: whether a VLM is even aware of where it is looking before understanding the semantic meaning in the image. We formulate this question as a LOOK task: identifying which parts of its visual input were available to the model. We construct the task in a self-supervised way without manually labeling the available region to the model. Our experiments show that existing VLMs fail to localize the visible region. We further show that fine-tuning VLMs on the proposed LOOK task improves a model's awareness of visual input.
Figures & tables
Figure 1: Hierarchy of visual understanding: Look → Perceive → Understand. We introduce a hierarchical visual grounding structure and argue that visual hallucination can arise from any level. This work focuses on the fundamental Look level.
Figure 2: Where does a hallucination originate? In this example, we ask what animal is grazing in a photograph of a dog on grassland, a model answers “a sheep”. The same wrong answer can arise at two levels of the hierarchy. At the Perceive level, the model uses the image but misrecognizes the dog. At the Look level, the model does not rely on the image at all, and “sheep” follows from the textual context.
Figure 3: Constructing a Look example. We select a merge-block-aligned viewport and encode only its pixels for the VLM. Importantly, we restore positional metadata from the full canvas without encoding the excluded image content. The crop’s coordinates provide the target automatically. Changed blocks indicate places where token positions are added.
Figure 4: Illustration of VisABench construction. All conditions use the same prompt, “ Does this image contain: dog? Answer with Yes or No. ”, with the only difference being the visual input. The grounded answer is Yes in (i) and No in (ii) and (iii).
Probe methods
Full image
Pixel operation
Patch embedding
Crop
Black
Grey
Zero
Mean
Local pos.
Global pos. ( Look )
IoU ↓
0.52
0.75
0.77
0.58
0.76
0.18
0.18
Table 1: Compare different probe methods for visual awareness. We report IoU of the pretrained Qwen3-VL when asked for the visible region over 1,000 views. IoU counts answers without a valid box as zero. Methods operating on pixels or patch embeddings fail to probe visual awareness because the model can exploit semantic shortcuts, often predicting the bounding box of the dominant object rather than the available visual region.
Figure 5: Four ways to probe visual awareness. The green box shows the available visual region while the red dashed box shows the prediction. (i) We input the full image, but the model boxes the dominant person rather than the whole canvas. (ii, iii) When the context pixels are greyed out or its embeddings are zeroed, the model still locates the dominant person, likely using semantic shortcuts. (iv) In Look , the model answers with the whole canvas.
Method
Base
SFT
LocPos
Ours
1:10
2:10
10:10
IoU ↑
0.181
0.111
0.371
0.609
0.665
0.742
Success@ 0.5↑
0.025
0.012
0.322
0.734
0.808
0.844
Table 2: Performance on Look task. Results are tested on the same 1,000 views of Table 1 . Success@ 0.5 is the fraction of views with an IoU of at least 0.5 .
Figure 6: Learning to look dynamics.
Figure 7: Permutation test. The grey bars mean IoU after re-pairing each prediction with the window of another image while the color lines show the IoU on the given image. The pretrained and SFT models show no positive dependence on the view.
No image
Real image
Wrong image
Yes (%) ↓
p(Yes)↓
Yes (%) ↑
p(Yes)↑
Yes (%) ↓
p(Yes)↓
Gap ↑
Base
2.5
0.041
93.3
0.927
15.6
0.158
90.7
SFT
31.2
0.409
94.9
0.915
17.0
0.196
63.7
LocPos
5.7
0.272
92.6
0.890
15.0
0.170
86.9
Ours (1:10)
2.3
0.202
94.1
0.910
15.7
0.183
91.8
Ours (2:10)
4.9
0.241
93.9
0.911
16.0
0.188
89.0
Table 3: Performance on VisABench. All models are evaluated on the same 10,752 items. For each condition, we report the Yes rate (%) and mean p(Yes) . Gap denotes the real-image Yes rate minus the no-image Yes rate. The bottom rows report results for additional pretrained VLMs ( Bai et al., 2025 ; Liu et al., 2024b ; Gemma Team, 2026 ) . † The image-placeholder embedding of this model is untrained, causing it to produce neither Yes nor No without an image. We therefore omit its no-image results.
MMStar
POPE
MMMU
MMMU-Pro
Base
66.4
88.0
55.1
39.4
SFT
63.6
88.5
56.5
40.8
LocPos
64.4
87.2
54.4
41.0
Ours (1:10)
65.4
88.6
56.0
41.0
Ours (2:10)
65.0
89.3
54.1
38.6
Ours (10:10)
63.8
88.5
56.2
39.2
Table 4: Look maintains general capabilities. We report accuracy (%) on MMStar, MMMU and MMMU-Pro and F1 score on POPE. We evaluate 500 items per benchmark.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Rows
Task
VQAv2 ( Goyal et al., 2017 )
12,000
Visual question answering
TextVQA ( Singh et al., 2019 )
6,000
Text reading and question answering
Visual7W ( Zhu et al., 2016 )
6,000
Grounded question answering
A-OKVQA ( Schwenk et al., 2022 )
5,000
Knowledge-based question answering
ChartQA ( Masry et al., 2022 )
5,000
Chart reasoning
TextCaps ( Sidorov et al., 2020 )
4,000
Captioning with text
Appendix
Table A1: The instruction mixture replay datasets. Look examples reuse its image pool.
Group
Prompt
see
Which region of the full image can you see? Answer with a bounding box.
What part of the image is visible to you? Give a bounding box.
Identify the area you are able to see; respond with a bounding box.
Report the visible region as a bounding box.
look
Where in the full image are you looking? Answer with a bounding box.
Which region are you looking at? Reply with box coordinates.
Vision-language models (VLMs) have achieved impressive performance across a wide range of multimodal tasks. However, they often fail on tasks that require fine-grained visual perception, even when the required information is still present in their internal representations. Prior work has attributed this ``hidden-in-plain-sight'' gap to the language model, but the cause remains unexplained. In this work, we demonstrate that this gap arises from the language model's lack of semantic labels for fine-grained visual details: when visual entities can be mapped to known concepts, VLMs bypass visual comparison and reason through language; when they cannot, VLMs resort to brittle and hallucinated descriptions. We verify this across semantic correspondence, synthetic shape matching, and face matching, and find that VLMs perform much better when the relevant entities are nameable than when they are unnamable. Mechanistically, Logit Lens analysis confirms that VLMs explicitly recover semantic labels for nameable entities and surface more unique tokens compared to unnameable entities. Furthermore, we show that this limitation can be addressed: teaching completely arbitrary names for unknown entities improves performance. More importantly, task-specific finetuning yields even stronger generalization without relying on language priors, i.e., through real visual perception. Our findings suggest that current VLM failures on visual tasks reflect a learned shortcut rather than a fundamental limitation of multimodal reasoning. Code and datasets are available at https://github.com/Patchwork53/VLMs-Need-Words-COLM2026.
Vision-Language Models (VLMs) still struggle on tasks requiring complex visual understanding. We argue that the core issue is not high-level reasoning, but instead failing to locate critical details in the image. Due to this shortcoming, VLMs generate often plausible but incorrect reasoning based on flawed perceptual grounding. To address this, we propose Locator-Critic (LOCI), a training-free framework that decouples visual search from evidence verification. LOCI employs a Locator agent to propose candidate visual evidence and a separate Critic agent to evaluate its relevance and sufficiency. These agents engage in an iterative refinement loop, progressively improving the evidence until it is adequate to answer the given question. This decoupled, self-correcting process yields substantial performance gains, achieving state-of-the-art results on multiple complex visual benchmarks. LOCI improves accuracy for both open-weight models like Qwen3-VL (+12.1 on V*, +5.8 on HR-Bench and +11.2 on VisualProbe-Hard) and proprietary models like Gemini 2.5 Pro (+8.9 on V*, +4.3 on HR-Bench, +4.8 on VisualProbe-Hard).
Benchmark accuracy is often implicitly assumed to reflect grounded visual understanding in vision-language models (VLMs), yet it remains unclear to what extent such scores truly reflect reliance on visual evidence. Motivated by a surprising observation that removing a substantial fraction of image tokens only degrades model performance very slightly on a widely used hallucination benchmark, we systematically investigate this mismatch in a set of open-source VLMs. Our analysis spans multiple levels of granularity, spanning global visual degradation, localized occlusion, question reformulation, answer-space expansion, and decision-level analyses beyond standard accuracy. We further complement these behavioral results with a layer-wise analysis of vision-token geometry. Throughout the experiments, we find that although VLMs do incorporate visual input, their predictions are less sensitive to the loss of fine-grained visual evidence that standard accuracy should have suggested. Even when the final prediction remains unchanged, the model's internal support for the correct answer may already be weakened. We further complement a representation-level analysis, which shows increasing similarity among visual tokens in deeper layers, providing a possible explanation for our findings. Together, these results suggest that current benchmarks are not sufficient to reliably evaluate fine-grained visual grounding in VLMs.
Zixuan Lan, Luzhe Sun, Matthew R. Walter +1
University of Chicago · Toyota Technological Institute at Chicago · Stony Brook University