Learning to Look: A Foundation for Visual Understanding

Wenshuai Zhao; Yihao Wang; Benjamin Robson; Juho Kannala; Arno Solin

cs.CVNov 1, 2026

Learning to Look: A Foundation for Visual Understanding

Authors: Wenshuai Zhao, Yihao Wang, Benjamin Robson, Juho Kannala, Arno Solin

Organizations: Aalto University · ELLIS Institute Finland · University of Oulu

Abstract

Vision-language models (VLMs) suffer from visual hallucination, such as saying an object that is not present in an image. We ask a more fundamental question: whether a VLM is even aware of where it is looking before understanding the semantic meaning in the image. We formulate this question as a LOOK task: identifying which parts of its visual input were available to the model. We construct the task in a self-supervised way without manually labeling the available region to the model. Our experiments show that existing VLMs fail to localize the visible region. We further show that fine-tuning VLMs on the proposed LOOK task improves a model's awareness of visual input.

Figures & tables

Figure 1: Hierarchy of visual understanding: Look $\rightarrow$ Perceive $\rightarrow$ Understand. We introduce a hierarchical visual grounding structure and argue that visual hallucination can arise from any level. This work focuses on the fundamental Look level.

Figure 2: Where does a hallucination originate? In this example, we ask what animal is grazing in a photograph of a dog on grassland, a model answers “a sheep”. The same wrong answer can arise at two levels of the hierarchy. At the Perceive level, the model uses the image but misrecognizes the dog. At the Look level, the model does not rely on the image at all, and “sheep” follows from the textual context.

Figure 3: Constructing a Look example. We select a merge-block-aligned viewport and encode only its pixels for the VLM. Importantly, we restore positional metadata from the full canvas without encoding the excluded image content. The crop’s coordinates provide the target automatically. Changed blocks indicate places where token positions are added.

Figure 4: Illustration of VisABench construction. All conditions use the same prompt, “ Does this image contain: dog? Answer with Yes or No. ”, with the only difference being the visual input. The grounded answer is Yes in (i) and No in (ii) and (iii).

Probe methods	Full image	Pixel operation		Patch embedding		Crop
		Black	Grey	Zero	Mean	Local pos.	Global pos. ( Look )
IoU $\downarrow$	0.52	0.75	0.77	0.58	0.76	0.18	0.18

Table 1: Compare different probe methods for visual awareness. We report IoU of the pretrained Qwen3-VL when asked for the visible region over

1{,}000

views. IoU counts answers without a valid box as zero. Methods operating on pixels or patch embeddings fail to probe visual awareness because the model can exploit semantic shortcuts, often predicting the bounding box of the dominant object rather than the available visual region.

Figure 5: Four ways to probe visual awareness. The green box shows the available visual region while the red dashed box shows the prediction. (i) We input the full image, but the model boxes the dominant person rather than the whole canvas. (ii, iii) When the context pixels are greyed out or its embeddings are zeroed, the model still locates the dominant person, likely using semantic shortcuts. (iv) In Look , the model answers with the whole canvas.

Method	Base	SFT	LocPos	Ours
				1:10	2:10	10:10
IoU $\uparrow$	0.181	0.111	0.371	0.609	0.665	0.742
Success@ $0.5$ $\uparrow$	0.025	0.012	0.322	0.734	0.808	0.844

Table 2: Performance on Look task. Results are tested on the same

1{,}000

views of Table 1 . Success@

0.5

is the fraction of views with an IoU of at least

0.5

Figure 6: Learning to look dynamics.

Figure 7: Permutation test. The grey bars mean IoU after re-pairing each prediction with the window of another image while the color lines show the IoU on the given image. The pretrained and SFT models show no positive dependence on the view.

	No image		Real image		Wrong image
	Yes (%) $\downarrow$	$p(\text{Yes})$ $\downarrow$	Yes (%) $\uparrow$	$p(\text{Yes})$ $\uparrow$	Yes (%) $\downarrow$	$p(\text{Yes})$ $\downarrow$	Gap $\uparrow$
Base	2.5	0.041	93.3	0.927	15.6	0.158	90.7
SFT	31.2	0.409	94.9	0.915	17.0	0.196	63.7
LocPos	5.7	0.272	92.6	0.890	15.0	0.170	86.9
Ours (1:10)	2.3	0.202	94.1	0.910	15.7	0.183	91.8
Ours (2:10)	4.9	0.241	93.9	0.911	16.0	0.188	89.0

Table 3: Performance on VisABench. All models are evaluated on the same

10{,}752

items. For each condition, we report the Yes rate (%) and mean

p(\text{Yes})

. Gap denotes the real-image Yes rate minus the no-image Yes rate. The bottom rows report results for additional pretrained VLMs ( Bai et al., 2025 ; Liu et al., 2024b ; Gemma Team, 2026 ) . † The image-placeholder embedding of this model is untrained, causing it to produce neither Yes nor No without an image. We therefore omit its no-image results.

	MMStar	POPE	MMMU	MMMU-Pro
Base	66.4	88.0	55.1	39.4
SFT	63.6	88.5	56.5	40.8
LocPos	64.4	87.2	54.4	41.0
Ours (1:10)	65.4	88.6	56.0	41.0
Ours (2:10)	65.0	89.3	54.1	38.6
Ours (10:10)	63.8	88.5	56.2	39.2

Table 4: Look maintains general capabilities. We report accuracy (%) on MMStar, MMMU and MMMU-Pro and F1 score on POPE. We evaluate

500

items per benchmark.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Rows	Task
VQAv2 ( Goyal et al., 2017 )	12,000	Visual question answering
TextVQA ( Singh et al., 2019 )	6,000	Text reading and question answering
Visual7W ( Zhu et al., 2016 )	6,000	Grounded question answering
A-OKVQA ( Schwenk et al., 2022 )	5,000	Knowledge-based question answering
ChartQA ( Masry et al., 2022 )	5,000	Chart reasoning
TextCaps ( Sidorov et al., 2020 )	4,000	Captioning with text

Appendix

Table A1: The instruction mixture replay datasets. Look examples reuse its image pool.

Group	Prompt
see	Which region of the full image can you see? Answer with a bounding box.
	What part of the image is visible to you? Give a bounding box.
	Identify the area you are able to see; respond with a bounding box.
	Report the visible region as a bounding box.
look	Where in the full image are you looking? Answer with a bounding box.
	Which region are you looking at? Reply with box coordinates.

Appendix

Table 13

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 751 papers

VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchors
Apr 2, 2026Haz Sameen Shahgir, Xiaofu Chen, Yu Fu +4VLM Reasoning Fine-Grained Visual Perception
University of California, Riverside 1 · MBZUAI2
LOCI: A Locator-Critic with Refinement Loop
Aug 31, 2026Walid Bousselham, Mathilde Caron, Arsha Nagrani +1Visual Question Answering Vision-Language Grounding
Google DeepMind
Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?
May 21, 2026Zixuan Lan, Luzhe Sun, Matthew R. Walter +1VLM Evaluation VLM Interpretability
University of Chicago · Toyota Technological Institute at Chicago · Stony Brook University

Apr 2, 2026J/K move · Enter open · S save

VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchors

Haz Sameen Shahgir, Xiaofu Chen, Yu Fu, Erfan Shayegani+3

University of California, Riverside 1 · MBZUAI2

VLM Reasoning Fine-Grained Visual Perception Vision-Language Grounding VLM Interpretability

Open paper AskPDF arXiv

Vision-language models (VLMs) have achieved impressive performance across a wide range of multimodal tasks. However, they often fail on tasks that require fine-grained visual perception, even when the required information is still present in their internal representations. Prior work has attributed this ``hidden-in-plain-sight'' gap to the language model, but the cause remains unexplained. In this work, we demonstrate that this gap arises from the language model's lack of semantic labels for fine-grained visual details: when visual entities can be mapped to known concepts, VLMs bypass visual comparison and reason through language; when they cannot, VLMs resort to brittle and hallucinated descriptions. We verify this across semantic correspondence, synthetic shape matching, and face matching, and find that VLMs perform much better when the relevant entities are nameable than when they are unnamable. Mechanistically, Logit Lens analysis confirms that VLMs explicitly recover semantic labels for nameable entities and surface more unique tokens compared to unnameable entities. Furthermore, we show that this limitation can be addressed: teaching completely arbitrary names for unknown entities improves performance. More importantly, task-specific finetuning yields even stronger generalization without relying on language priors, i.e., through real visual perception. Our findings suggest that current VLM failures on visual tasks reflect a learned shortcut rather than a fundamental limitation of multimodal reasoning. Code and datasets are available at https://github.com/Patchwork53/VLMs-Need-Words-COLM2026.

Abstract

Vision-Language Models

Explore similar work

VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchors

LOCI: A Locator-Critic with Refinement Loop

Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?