Retrieval Heads Meet Vision: Uncovering How VLMs Locate and Extract Visual Information

Chanho Park; Daehyeon Choi; Jihyun Lee; Minhyuk Sung

cs.CVAug 27, 2026

Retrieval Heads Meet Vision: Uncovering How VLMs Locate and Extract Visual Information

Authors: Chanho Park, Daehyeon Choi, Jihyun Lee, Minhyuk Sung

Organizations: KAIST · Independent Researcher

Abstract

Vision-language models (VLMs) can locate an image region referred to by a text prompt and route the corresponding visual evidence to the output, yet the internal mechanism behind this behavior is not understood. Inspired by retrieval heads in large language models, we ask whether VLMs contain an analogous mechanism for visual retrieval. We answer affirmatively by introducing Visual Retrieval Heads (VRHs), a small subset of attention heads (about 1.7-2.6%) that are causally responsible for grounding text descriptions to image regions. To find them, we recast existing head-scoring methods under a unified design space over query tokens, key aggregation, and cross-sample aggregation. We then show that scoring attention from output prediction tokens with a sum over the ground-truth referent region most reliably identifies causal heads. Across eleven VLMs and five referring-expression benchmarks, masking only the top 20 VRHs reduces grounding accuracy by up to 80 percentage points, while masking the same number of random heads has little effect. Beyond replicating the causal-sparse-universal triad established for text retrieval heads, VRHs exhibit several properties not previously reported: they generalize across visual reference tasks, remaining causal on attribute, spatial, counting, and visual-math benchmarks despite being discovered through bounding-box prediction; they are functionally specific, preserving output format while corrupting localization; and they are architecturally shared, transferring causally across VLMs that share an LLM backbone but differ in vision encoder, projector, and instruction tuning.

Figures & tables

Figure 1: Visual Retrieval Heads mediate visual reference resolution in VLMs. We identify a sparse set of attention heads, Visual Retrieval Heads (VRHs), that are causally necessary for resolving the image region referred to by the text prompt. On visual grounding ( left ), masking the VRHs redirects prediction-stage attention away from the ground-truth referent and causes the model to output a delocalized bounding box. The same heads, discovered only from grounding, also transfer to visual question answering ( right ): masking them removes attention to the referred object and leads to a fluent but visually unfaithful answer. Yellow boxes denote ground-truth referents; dashed yellow ellipses indicate regions where attention vanishes after VRH masking.

	Retrieval Heads [ 1 ]	QRHead [ 20 ]	OCR Heads [ 2 ]	VERA [ 3 ]	Loc. Heads [ 4 ]	VRH (Ours)
Input Source	Text	Text	Visual document	Visual document	General image	General image
Causality	✓	✓	✓	✓	✗	✓

Table 1: Comparison of prior works on retrieval-related specialized heads.

	Retrieval Heads [ 1 ]	QRHead [ 20 ]	OCR Heads [ 2 ]	VERA [ 3 ]	Loc. Head [ 4 ]
Query Token Set $\mathcal{Q}_{s}$	$\mathcal{Q}_{s}^{\texttt{out}}$	$\mathcal{Q}_{s}^{\texttt{in}}$	$\mathcal{Q}_{s}^{\texttt{out}}$	$\mathcal{Q}_{s}^{\texttt{out}}$	$\mathcal{Q}_{s}^{\texttt{in}}$
Key Aggregation $\Phi$	$\Phi^{\texttt{argmax}}$	$\Phi^{\texttt{sum}}$	$\Phi^{\texttt{argmax}}$	$\Phi^{\texttt{sum}}$	N/A †
Sample Aggregation $\Omega$	$\Omega^{\texttt{avg}}$	$\Omega^{\texttt{avg}}$	$\Omega^{\texttt{avg}}$	$\Omega^{\texttt{avg}}$	$\Omega^{\texttt{freq}}$

Table 2: Design choices in head-scoring methods. We summarize how existing methods instantiate the main components of Eq. 1 . For

\mathcal{Q}_{s}

, some methods use only a few representative tokens from the indicated input or output token set, i.e.,

\mathcal{Q}_{s}^{\texttt{in}}

\mathcal{Q}_{s}^{\texttt{out}}

; we slightly abuse notation for abstraction. † Localization Heads is a special case that does not directly use key aggregation, but instead selects heads based on the spatial entropy of attention weights.

Figure 2: Causal validation of head-scoring variants on RefCOCO [ 5 ] across four VLMs [ 6 , 7 , 8 , 9 ] . We mask the top-20 heads detected by each variant and report visual grounding accuracy; lower accuracy indicates that the detected heads are more causally important. Ours denotes our default variant

(\mathcal{Q}^{\mathrm{out}}_{s},\Phi^{\mathrm{sum}},\Omega^{\mathrm{mean}})

. The

\mathcal{Q}_{s}

\Phi

, and

\Omega

ablations each change one component of this default design at a time, while Random masks 20 randomly selected heads. The dashed line shows the unmasked model (

k=0

), and

\Delta

denotes the accuracy gap relative to Ours .

Figure 3: Comparison with existing heads. We compare VRHs with Retrieval Heads [ 1 ] , OCR Heads [ 2 ] , VERA [ 3 ] , Localization Heads [ 4 ] , and randomly selected heads on the RefCOCO visual grounding benchmark [ 5 ] . The x-axis denotes the number of masked heads (

k

). Masking the top-

k

VRHs causes substantially larger drops in grounding accuracy than masking these baselines, indicating that VRHs capture a distinct set of heads causally important for visual grounding.

Figure 4: Universality of VRHs across models and benchmarks. We report visual grounding accuracy after masking the top-

k

VRHs across four VLMs and three benchmark families: RefCOCO/+/g [ 5 , 10 ] , RefSpatialBench [ 11 ] , and Toloka [ 12 ] . The x-axis denotes the number of masked heads (

k

). Masking only a few VRHs consistently causes large accuracy drops, while masking random heads has little effect. This broad causal effect shows that VRHs are not architecture- or dataset-specific, but instead capture a shared attention-head mechanism for visual grounding.

Figure 5: Sparsity of VRHs. Distribution of head scores for VRH detection on RefCOCO [ 5 ] across four VLMs [ 6 , 7 , 8 , 9 ] . Only a small fraction of heads receive non-negligible scores.

Figure 6: VRHs are universal across VLMs. Visual grounding accuracy on RefCOCO [ 5 ] for all eleven evaluated VLMs, before masking, after masking 20 random heads, and after masking the top-20 VRHs. Masking the VRHs causes a large accuracy drop in every model, whereas masking the same number of random heads has little effect. Exact values, evaluation splits and per-model protocols are given in Sec. J .

Model	Condition	VAW	Spatial457	SpatialRGPT	CV-Bench	MMStar	CountBenchQA	MathVista
Qwen2.5-VL-7B	Baseline ( $k{=}0$ )	63.8	68.2	36.1	75.1	58.3	83.9	71.5
	Random ( $k{=}20$ )	63.7 (-0.1)	67.2 (-1.0)	39.5 (+3.4)	71.9 (-3.2)	56.8 (-1.5)	84.1 (+0.2)	72.0 (+0.5)
	VRH ( $k{=}20$ )	40.5 (-23.3)	36.0 (-32.2)	21.8 (-14.3)	53.3 (-21.8)	47.9 (-10.4)	61.5 (-22.4)	61.7 (-9.8)
Qwen3-VL-8B	Baseline ( $k{=}0$ )	74.8	78.2	50.8	87.6	63.0	88.4	66.5
	Random ( $k{=}20$ )	74.3 (-0.5)	76.4 (-1.8)	46.2 (-4.6)	87.1 (-0.5)	63.3 (+0.3)	89.6 (+1.2)	69.4 (+2.9)
	VRH ( $k{=}20$ )	51.0 (-23.8)	36.9 (-41.3)	22.9 (-27.9)	68.7 (-18.9)	47.7 (-15.3)	74.5 (-13.9)	43.0 (-23.5)

Table 3: Generalization across visual reference tasks. Masking the top-20 VRHs detected on RefCOCO [ 5 ] consistently degrades performance on seven VQA benchmarks [ 13 , 14 , 15 , 16 , 17 , 18 , 19 ] spanning attributes, spatial reasoning, counting, vision-centric understanding, and visual math, while random-head masking has little effect.

Figure 7: VRH detection consistency across same-backbone VLMs. Each block is a pair of VLMs that share a language backbone but differ in vision encoder, projector and instruction tuning: Qwen2.5-VL-7B [ 6 ] with InternVL3-8B [ 8 ] , which share a Qwen2.5-7B backbone; Qwen3-VL-8B [ 7 ] with InternVL3.5-8B [ 9 ] , which share a Qwen3-8B backbone; and Magma-8B [ 27 ] with LLaVA-NeXT-Llama3-8B [ 28 ] , which share a Llama-3 backbone. Top row: Layer-head score maps, where each cell denotes the VRH score $\bar{\Theta}_{l,h}$ at layer $l$ and head $h$ ; high-scoring heads appear at similar layer-head locations across same-backbone VLMs. Bottom row: Cross-model masking results, where the top-k VRHs identified in one model are masked at the same layer-head indices in the other model. Own VRHs are the heads detected in the panel’s own model and transferred VRHs those detected in its partner. The x-axis denotes the number of masked heads (k).

Panel of Figure 8 — Figure 8: VRHs are functionally specific and stably detectable. All experiments use Qwen2.5-VL-7B [ 6 ] on RefCOCO [ 5 ] with $k{=}20$ masked heads unless otherwise noted. (a) Visual grounding failure decomposition under VRH masking vs. random-head masking. Mislocation denotes syntactically valid but mislocalized box prediction, while parse failure denotes output that cannot be parsed as a bounding box. (b) Grounding accuracy when masking VRHs detected from a single random output token, all output tokens, or random heads. (c) Grounding accuracy when masking VRHs detected with varying numbers of scoring samples $n$ .

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 12 — Figure A1: Attention map visualizations on visual grounding tasks. From left to right, the columns show the input image and the mean attention maps over: (i) all heads, (ii) the top-20 VRHs, and (iii) all heads excluding VRHs. Green and red boxes indicate ground-truth and predicted bounding boxes, respectively.

Panel of Figure 13 — Figure A2: Attention map visualizations on visual question-answering tasks. Each panel shows the input image (upper left), and mean attention maps over: (i) all heads (lower left), (ii) the top-20 VRHs (upper right), and (iii) all heads excluding VRHs (lower right). Green boxes indicate the ground-truth target regions.

Figure A3: Grounding accuracy comparisons with DPO under varying negative policies. We compare two DPO variants that use either random-head dropping or VRH dropping to construct negative responses. Using VRH dropping as the negative policy yields stronger downstream improvements than random-head dropping, suggesting that perturbing causally relevant heads provides a more informative preference signal for optimization.

RefCOCO [ 5 ]
Model	Method	$k{=}0$	$k{=}1$	$k{=}3$	$k{=}5$	$k{=}10$	$k{=}20$
Qwen2.5-VL-7B	Random	86.2	86.2	86.4	86.2	85.9	86.1
	VRH	86.2	85.1	83.5	79.3	9.7	7.5
Qwen3-VL-8B	Random	91.3	91.3	91.2	91.3	91.1	91.1
	VRH	91.3	91.0	90.4	84.1	33.4	20.4
InternVL3-8B	Random	93.6	93.6	93.7	93.3	92.9	92.5

Appendix

Table A1: Per-dataset visual grounding accuracy.

Model	Top-20 VRH (layer, head)
Qwen2.5-VL-7B	(19,17), (16,0), (18,16), (16,20), (19,22), (19,20), (16,1), (19,23), (19,15), (17,18), (16,7), (14,11), (14,16), (16,6), (16,17), (19,26), (19,19), (14,23), (17,14), (18,15)
Qwen3-VL-8B	(20,15), (21,10), (18,19), (18,15), (21,11), (21,18), (17,27), (17,4), (19,3), (22,4), (21,8), (20,8), (18,5), (23,13), (20,17), (20,18), (23,30), (21,25), (20,16), (23,10)
InternVL3-8B	(16,20), (16,0), (19,22), (16,1), (19,23), (19,17), (19,20), (16,6), (14,11), (19,15), (16,4), (17,24), (18,16), (16,16), (16,17), (20,21), (16,7), (19,19), (14,16), (14,8)
InternVL3.5-8B	(21,11), (20,15), (21,10), (19,3), (18,15), (17,4), (21,18), (18,19), (17,27), (20,17), (24,13), (21,8), (20,18), (20,16), (21,25), (20,8), (18,5), (22,4), (18,21), (23,30)

Appendix

Table A2: Top-20 VRHs detected on RefCOCO [ 5 ] . Each entry is

(l,h)

Model	Top-20 VRH (layer, head)
Qwen2.5-VL-7B	(19,17), (16,0), (16,20), (18,16), (19,22), (19,20), (16,1), (19,15), (19,23), (17,18), (14,11), (16,7), (14,16), (16,6), (16,17), (19,19), (19,26), (17,24), (14,23), (17,14)
Qwen3-VL-8B	(20,15), (21,10), (18,19), (18,15), (17,27), (21,11), (21,18), (17,4), (19,3), (20,8), (22,4), (21,8), (20,17), (20,18), (23,13), (21,25), (18,5), (23,30), (20,16), (23,10)
InternVL3-8B	(16,0), (16,20), (19,22), (19,23), (16,1), (19,17), (19,20), (16,6), (19,15), (14,11), (16,4), (17,24), (18,16), (20,21), (16,7), (16,16), (16,17), (14,8), (14,16), (19,19)
InternVL3.5-8B	(21,11), (21,10), (20,15), (19,3), (18,15), (21,18), (17,4), (18,19), (17,27), (20,17), (21,8), (20,18), (20,16), (24,13), (20,8), (22,4), (21,25), (17,26), (23,30), (23,10)

Appendix

Table A3: Top-20 VRHs detected on RefCOCO+ [ 5 ] . Each entry is

(l,h)

Model	Top-20 VRH (layer, head)
Qwen2.5-VL-7B	(19,17), (16,0), (16,20), (19,22), (18,16), (16,1), (19,20), (19,23), (19,15), (14,11), (17,18), (16,7), (14,16), (16,6), (16,17), (19,26), (14,23), (19,19), (17,14), (18,15)
Qwen3-VL-8B	(20,15), (18,19), (21,10), (17,27), (18,15), (21,18), (21,11), (17,4), (19,3), (22,4), (21,8), (20,8), (18,5), (20,17), (20,18), (21,25), (23,30), (23,13), (20,16), (17,24)
InternVL3-8B	(16,20), (16,0), (16,1), (19,22), (19,17), (19,23), (14,11), (16,6), (19,20), (16,4), (19,15), (18,16), (16,17), (16,16), (17,24), (14,8), (14,16), (16,7), (20,21), (14,14)
InternVL3.5-8B	(21,11), (20,15), (21,10), (18,15), (19,3), (17,4), (21,18), (18,19), (17,27), (20,17), (20,18), (20,16), (21,8), (24,13), (19,13), (20,8), (18,21), (17,26), (22,4), (18,5)

Appendix

Table A4: Top-20 VRHs detected on RefCOCOg [ 10 ] . Each entry is

(l,h)

Model	Top-20 VRH (layer, head)
Qwen2.5-VL-7B	(19,17), (16,0), (19,22), (16,20), (19,15), (19,20), (19,23), (16,1), (18,16), (17,18), (19,19), (16,6), (19,26), (14,11), (19,25), (16,17), (16,16), (16,18), (16,4), (20,21)
Qwen3-VL-8B	(21,10), (20,15), (18,19), (21,11), (17,4), (18,15), (23,30), (21,8), (23,13), (22,4), (21,18), (20,17), (17,27), (23,10), (20,16), (20,8), (20,18), (19,3), (24,16), (21,31)
InternVL3-8B	(16,20), (19,17), (16,1), (19,20), (16,0), (19,15), (19,22), (16,17), (14,11), (19,23), (18,16), (16,16), (16,6), (20,21), (14,8), (19,19), (16,7), (17,24), (16,18), (21,25)
InternVL3.5-8B	(18,19), (21,11), (17,27), (21,10), (20,15), (18,15), (20,17), (17,4), (24,13), (23,30), (21,8), (19,3), (20,16), (21,18), (20,8), (23,10), (20,23), (18,21), (19,13), (21,9)

Appendix

Table A5: Top-20 VRHs detected on Toloka [ 12 ] . Each entry is

(l,h)

Model	Top-20 VRH (layer, head)
Qwen2.5-VL-7B	(19,17), (19,22), (19,20), (19,15), (16,0), (16,20), (19,23), (16,1), (19,19), (18,16), (17,18), (19,26), (20,21), (16,16), (16,6), (16,4), (14,11), (19,25), (16,7), (20,23)
Qwen3-VL-8B	(21,10), (21,11), (20,15), (21,8), (23,30), (17,4), (18,19), (18,15), (21,18), (23,13), (23,10), (22,4), (20,17), (19,3), (20,16), (17,27), (20,8), (23,14), (25,10), (21,31)
InternVL3-8B	(19,17), (16,1), (16,20), (19,22), (19,15), (19,20), (19,23), (16,0), (14,11), (20,21), (17,24), (19,19), (16,6), (18,16), (16,17), (16,7), (16,16), (21,25), (21,5), (16,4)
InternVL3.5-8B	(21,11), (21,10), (21,8), (18,15), (19,3), (18,19), (21,18), (20,17), (17,4), (20,15), (17,27), (20,8), (24,13), (23,30), (20,23), (20,16), (21,25), (23,10), (20,21), (21,9)

Appendix

Table A6: Top-20 VRHs detected on RefSpatialBench [ 11 ] . Each entry is

(l,h)

Model	# Layers $L$	# Heads/layer $H$	Total heads	Language backbone	Vision encoder	Projector
Qwen2.5-VL-7B	28	28	784	Qwen2.5-7B	Qwen ViT	MLP merger
Qwen3-VL-8B	36	32	1152	Qwen3-8B	Qwen ViT	MLP merger + DeepStack
InternVL3-8B	28	28	784	Qwen2.5-7B	InternViT-300M	2-layer MLP
InternVL3.5-8B	36	32	1152	Qwen3-8B	InternViT-300M	2-layer MLP

Appendix

Table A7: Architectural details of the four evaluated VLMs.

	Qwen2.5-VL-7B	Qwen3-VL-8B	InternVL3-8B	InternVL3.5-8B
Original model	86.11	91.27	93.59	90.48
w/ Masking VRHs ( $\mathcal{Q}^{\mathrm{in}}_{s}\cup\mathcal{Q}^{\mathrm{out}}_{s}$ )	7.14	29.85	8.98	32.94
w/ Masking VRHs ( $\mathcal{Q}^{\mathrm{out}}_{s}$ ; ours)	7.48	20.39	8.76	32.03

Appendix

Table A8: Visual grounding accuracy on RefCOCO [ 5 ] after masking the top-20 VRHs detected using

\mathcal{Q}^{\mathrm{in}}_{s}\cup\mathcal{Q}^{\mathrm{out}}_{s}

versus

\mathcal{Q}^{\mathrm{out}}_{s}

(ours) across four VLMs.

(a) Qwen2.5-VL-7B
	Random heads	VRHs (Ours)	Flipped $\mathcal{Q}_{s}$	Flipped $\Phi$	Flipped $\Omega$
$k{=}0$ (original model)	86.11	86.11	86.11	86.11	86.11
$k{=}5$	86.16	79.29	56.08	63.28	63.14
$k{=}10$	85.79	9.63	10.52	8.87	9.63
$k{=}20$	85.97	7.48	7.68	8.13	8.01
$k{=}50$	84.67	6.63	9.85	7.21	8.70

Appendix

Table A9: Scoring ablation for varying numbers of masked heads

k\in\{5,10,20,50\}

across four VLMs. We report RefCOCO [ 5 ] visual grounding accuracy after masking random heads, our detected VRHs, or VRHs obtained by flipping each scoring design choice. Bold and underline mark the largest and the second-largest causal effect among the four scoring variants in each row.

Evaluation set	Qwen2.5-VL-7B	Qwen3-VL-8B	InternVL3-8B	InternVL3.5-8B
w/ head detection samples ( $n{=}8{,}811$ )	7.46	20.44	8.76	32.14
w/o head detection samples ( $n{=}8{,}611$ )	7.48	20.39	8.76	32.03

Appendix

Table A10: Visual grounding accuracy on RefCOCO [ 5 ] after masking the top-20 VRHs, evaluated with and without the

200

head-detection samples.

Setting	# Visual tokens	Peak memory (GiB) (Eager / Fused Att.)	Time (s/sample) (Eager / Fused Att.)
Low resolution ( $448{\times}448$ )	256	15.09 / 14.92	$1.77\pm 0.47$ / $1.41\pm 0.21$
Low resolution ( $448{\times}448$ ), 2 images	512	15.55 / 15.01	$2.33\pm 0.99$ / $1.56\pm 0.34$
High resolution ( $1792{\times}1344$ )	3,328	34.59 / 16.01	$4.86\pm 0.77$ / $2.31\pm 0.29$

Appendix

Table A11: Peak memory usage and computation time of VRH detection with varying numbers of visual tokens. We use InternVL3-8B [ 8 ] with dynamic tiling as the base model and conduct all measurements on a single 48 GB A6000 GPU.

Condition	Magma-8B [ 27 ]	LLaVA-NeXT- Llama3-8B [ 28 ]	PaliGemma2- 10B [ 29 ]	DeepSeek-VL2- Small [ 30 ]	InternVL2.5- 8B [ 31 ]	GLM-4.6V- Flash [ 32 ]	Phi-4-Reasoning- Vision [ 33 ]
Language backbone	Llama-3	Llama-3	Gemma 2	DeepSeekMoE	InternLM2.5	GLM	Phi
Baseline ( $k{=}0$ )	68.8	85.8	73.6	93.0	90.0	86.4	74.2
Random ( $k{=}20$ )	64.0	77.8	73.2	91.6	86.8	85.8	73.8
VRH ( $k{=}20$ )	7.8	5.0	11.6	21.4	11.4	10.0	18.0

Appendix

Table A12: Exact values plotted in Fig. 6 for the seven VLMs introduced in addition to the four studied throughout the paper. Visual grounding accuracy on RefCOCO [ 5 ] after masking the top-20 VRHs versus the same number of random heads. GLM-4.6V-Flash and Phi-4-Reasoning-Vision were released after the main experiments of this work were completed.

	Magma-8B	LLaVA-NeXT-Llama3-8B
Original model	68.8	85.8
w/ Masking own top-20 VRHs	7.8	5.0
w/ Cross-masking VRHs	0.0	5.8
w/ Masking random heads	64.0	77.8

Appendix

Table A13: Cross-model transferability of VRHs between Magma-8B [ 27 ] and LLaVA-NeXT-Llama3-8B [ 28 ] . We report RefCOCO [ 5 ] visual grounding accuracy before and after masking the top-20 VRHs identified in the other model.

VLM Interpretability

Vision-language model interpretability explains how image and language representations shape predictions, evidence use, and multimodal reasoning.

See all 256 papers

May 26, 2026

Can Retrieval Heads See Images? Multimodal Retrieval Heads in Long-Context Vision-Language Models

Large vision-language models increasingly rely on long-context modeling to reason over documents, hour-level videos, and long-horizon agent trajectories, requiring them to locate relevant evidence across interleaved text and images. Prior work has studied this behavior using retrieval heads in large language models, but its copy-based criterion does not directly apply when evidence appears in images. We introduce a multimodal retrieval head detection method that scores attention from question tokens to textual or visual evidence. With this method, we show that multimodal retrieval heads are sparse, intrinsic, and causally important: only 4.4-10.2% of attention heads account for 50% of the positive retrieval-score mass, and masking the top-5% selected heads reduces performance on two tasks from MMLongBench by 42.5 and 62.3 percentage points, while random-head masking is far less damaging. Further analysis shows that these heads are partly shared across modalities yet remain dynamic within each modality, with image retrieval heads changing more than text retrieval heads as context length and haystack modality change. Without further training, we find that these heads can also be used directly to rank visually rich documents: on MMDocIR, Qwen3-VL-8B selected-head scoring improves Recall@1 by 7.7/7.4 macro/micro points for page retrieval and 6.3/6.8 points for layout retrieval over the strongest reported baseline.

Aaron Branson Cigres Li, Zhaowei Wang, Yu Zhao +9

1HKUST · University of Edinburgh · 4CUHK +2

Vision-Language Models VLM Interpretability Vision-Language Retrieval Visual Document Retrieval Multimodal Foundation Models Long-Context VLMs+1

Ask PDF

Aug 11, 2026

Where To Look? : Causal Tracing of Vision Encoders in VLM

Vision-language models can describe an image with remarkable accuracy, yet a more fundamental question remains unanswered: what visual information actually drives their answers? In this work, we investigate this question through causal tracing, and we observe that highly causal vision tokens often lie outside the target region. Extending the analysis to larger vision-language models reveals a similar pattern across models and corruption settings, suggesting that strong multimodal performance does not necessarily imply spatially localized causal representations. We further investigate: can these models preserve visual structure when appearance cues are removed? and find that visual cues are exploited to understand visual structures. Together, our experiments expose a gap between seeing, using, and reasoning over visual structure, and provide a causal framework for studying how visual information is transformed, preserved, and ultimately used by modern vision-language models.

Naren Kumar S, Tirth Bhatt, Mayank Singh

LINGO Research Group, Indian Institute of Technology Gandhinagar, India

Large Vision-Language Models VLM Interpretability VLM Reasoning

Ask PDF

Jun 12, 2026

Gaze Heads: How VLMs Look at What They Describe

How a vision-language model internally solves the task of describing an image is far from obvious. We find that the model develops a specific mechanism for this: a small set of attention heads in its language-model backbone, which we call gaze heads, whose attention tracks the image region the model is currently describing. We find them with a simple correlation score from a few forward passes, using comic strips as a controlled testbed where narrative order is laid out spatially. These gaze heads do not just track the image tokens being described: redirecting their attention to a chosen region forces the VLM to describe that region instead. A single attention-mask intervention on the top-100 gaze heads, fewer than 9% of all heads, steers the model's answer to any chosen comic panel at 83.1% accuracy, while the same intervention on random heads fails to redirect the answer, and intervening on all heads destroys generation. The same lever also extends to continuous control: switching the gaze target mid-generation makes the model wrap up its current panel description and move to the new one within a few tokens. Beyond comics, the same intervention redirects answers to chosen regions in natural COCO images. The mechanism further recurs across model sizes from 2B to 32B parameters and across other VLM architectures, although some frozen-encoder families show no comparable head set. More broadly, this shows that targeted edits identified through mechanistic analysis can serve as practical inference-time levers for steering multimodal model behavior, without any retraining. Our code, interactive demo, and datasets are available at https://gaze.baulab.info/

Rohit Gandikota, David Bau

Northeastern University

Image Captioning Visual Attention VLM Interpretability Vision-Language Grounding Multimodal Foundation Models Multimodal Understanding

Ask PDF

Abstract

VLM Interpretability

Explore similar work

Can Retrieval Heads See Images? Multimodal Retrieval Heads in Long-Context Vision-Language Models

Where To Look? : Causal Tracing of Vision Encoders in VLM

Gaze Heads: How VLMs Look at What They Describe