Heads That Write, Not Just Point: Image Retrieval Heads in Vision-Language Models

Junsung Park; Sangwon Yu; Donghun Kang; Yeongtak Oh; Jaewon Jeong; Gyeongtae Yoo; Han Cheol Moon; Jungbeom Lee

cs.CVNov 3, 2026

Heads That Write, Not Just Point: Image Retrieval Heads in Vision-Language Models

Authors: Junsung Park, Sangwon Yu, Donghun Kang, Yeongtak Oh, Jaewon Jeong, Gyeongtae Yoo, Han Cheol Moon, Jungbeom Lee,

Organizations: Department of Electrical and Computer Engineering, Seoul National University · Upstage AI · Interdisciplinary Program in AI, Seoul National University · Samsung Electronics ·

Abstract

Large vision-language models (LVLMs) can process interleaved text and multiple images in a single context, but how they internally identify the image relevant to a question remains poorly understood. We study this grounding step as in-context image retrieval and introduce ICIR-MCQ, a controlled probe for head-level analysis of image retrieval. Using each head's attention mass over image-token spans, we first identify sparse pointer heads that reliably point to the target image. These heads transfer beyond the probe: a single pointer head is competitive with external vision-language retrievers on a naturalistic multi-image retrieval benchmark, showing that LVLMs contain a strong internal image retrieval signal. Attention alone, however, shows where a head reads from, not what its output passes on to later layers, so we introduce a complementary output-side analysis. For each head, we train a lightweight image retriever on its value-weighted output and use it to identify writer heads, whose outputs contain enough information to identify the target image. Surprisingly, the pointer and writer criteria select substantially different head sets, with only 35 to 47 heads overlapping among the top-100 heads under each criterion. Targeted set-partition ablations show that the heads most important for standard multi-image inference are concentrated in this pointer-writer intersection, while heads selected by only one criterion show less consistent separation from random-head controls. The most consistent causal effects arise from heads selected by both the attention-based pointer and output-side writer criteria. Code and data: https://github.com/parkquasar/Image-Retrieval-Heads.

Figures & tables

Figure 1: An ICIR-MCQ sample. The model selects the caption matching the image containing the queried category ( zebra ). At the cattok , the pointer criterion ranks images by projection-based attention mass (Eq. 3 ), while the writer criterion ranks images using a retriever trained on the head output. The two criteria can disagree in the same head.

Model	Top-1	Top-2	Top-3	Top-4	Top-5	Median
InternVL3.5-8B	L21H10 (.890)	L21H11 (.850)	L21H08 (.842)	L20H17 (.825)	L21H09 (.823)	.206
Qwen3-VL-8B	L21H10 (.922)	L21H08 (.883)	L21H11 (.878)	L23H10 (.873)	L21H09 (.869)	.198
InternVL3.5-4B	L21H10 (.866)	L21H11 (.847)	L23H10 (.835)	L21H08 (.826)	L21H09 (.824)	.217
Qwen3-VL-4B	L21H10 (.905)	L23H10 (.878)	L21H11 (.870)	L21H08 (.859)	L21H09 (.856)	.206

Table 1: Top-5 pointer heads by head-level attention-mass R@1 on ICIR-MCQ. The strongest heads cluster in the same mid-block layer band, and within each LLM-backbone group (8B above; 4B below), top heads reproduce at identical (layer, head) indices. Cells show LxxHyy (R@1).

Method	Single Pointer Head				Retriever Model
Method	IV-8B	QV-8B	IV-4B	QV-4B	CLIP	SigLIP	QV-Emb
R@1	.787	.832	.794	.836	.573	.649	.751

Table 2: Image retrieval R@1 on Visual Haystacks (haystack size

N{=}10

, chance R@1 is

\nicefrac{{1}}{{10}}

). Internal results use the single pointer head L21H10 in each LVLM; external retrievers are evaluated on the same samples. IV = InternVL3.5, QV = Qwen3-VL, and QV-Emb = Qwen3-VL-Embedding-8B.

Model	Top-1	Top-2	Top-3	Top-4	Top-5	Median
InternVL3.5-8B	L24H23 (.489)	L20H28 (.435)	L20H15 (.426)	L21H31 (.421)	L18H19 (.418)	.077
Qwen3-VL-8B	L24H23 (.595)	L21H17 (.470)	L26H26 (.460)	L21H31 (.459)	L20H28 (.458)	.095
InternVL3.5-4B	L21H31 (.472)	L20H28 (.439)	L24H23 (.433)	L20H15 (.428)	L21H17 (.428)	.080
Qwen3-VL-4B	L24H23 (.509)	L20H28 (.487)	L21H31 (.486)	L21H17 (.483)	L18H19 (.463)	.101

Table 3: Top-5 writer heads by writer R@1 on the ICIR-MCQ held-out image pool. Within each LLM-backbone group (8B above; 4B below), top heads largely overlap. Writer R@1 is evaluated on a held-out pool of

1{,}200

candidate images, whereas pointer R@1 in Table 1 is evaluated on the images in each context; the two R@1 are not directly comparable in absolute scale. Cells show LxxHyy (R@1).

Figure 2: Head-level pointer R@1 versus writer R@1 on ICIR-MCQ. Each point is one attention head: colors denote pointer&writer , pointer-only , writer-only , and other heads, where membership is defined by the top-100 heads under each ranking. The vertical dotted line marks pointer-axis chance (R@1 = .183). The two criteria select substantially different head sets across all four LVLMs.

Figure 3: Set-partition ablation on Visual Haystacks. Each panel corresponds to a haystack size

N

, the number of candidate images. Curves show average accuracy drop relative to the unablated baseline as

K

heads are ablated. The pointer-writer overlap shows the most consistent effect. Shaded bands show

\pm 1

standard deviation across the four LVLMs.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Head-level attention-mass R@1 on ICIR-MCQ for all four LVLMs, shown as layers $\times$ heads with a shared color scale. Pointer ability concentrates in a narrow mid-block layer band, and the same band appears across both model families and both parameter scales. The strongest pointer head per model is L21H10 on every panel (Table 1 ).

Shared-LM-backbone pairs
	Pearson	Spearman	LayerMax	Top-3	Top-10	Top-100
InternVL3.5-8B $\leftrightarrow$ Qwen3-VL-8B	0.965	0.922	0.921	1.00	0.70	0.86
InternVL3.5-4B $\leftrightarrow$ Qwen3-VL-4B	0.966	0.927	0.933	1.00	0.60	0.80
Cross-LM-backbone pairs
InternVL3.5-8B $\leftrightarrow$ InternVL3.5-4B	0.977	0.946	0.962	0.67	0.70	0.85
Qwen3-VL-8B $\leftrightarrow$ Qwen3-VL-4B	0.971	0.939	0.980	0.67	0.60	0.85

Appendix

Table 4: Pairwise per-head R@1 similarity across the four LVLMs. Pearson and Spearman are computed on the full

1{,}152

-d head ranking; layer-max correlation is on the per-layer maximum R@1 vector (length 36); top-

K

overlap is

|\mathcal{H}^{\mathrm{ptr}}_{K}(\text{model 1})\cap\mathcal{H}^{\mathrm{ptr}}_{K}(\text{model 2})|/K

. The top-1 pointer head is L21H10 on every model.

Haystack size	2	3	5	10	20
InternVL3.5-8B	.947	.914	.860	.787	.677
Qwen3-VL-8B	.959	.931	.901	.832	.782
InternVL3.5-4B	.939	.907	.859	.794	.682
Qwen3-VL-4B	.894	.901	.895	.836	.745

Appendix

Table 5: R@1 of the shared top pointer head L21H10 on VHs across haystack sizes

\in\{2,3,5,10,20\}

(

n{=}1{,}000

per cell). Cattok-position attention argmax accuracy.

N{=}50

is not included in this readout.

	ICIR-MCQ		Visual Haystacks
Model	Head R@1	Model acc. (%)	Head R@1	Model acc. (%)
InternVL3.5-4B	.866	85.5	.794	68.1
InternVL3.5-8B	.890	88.9	.787	65.9
Qwen3-VL-4B	.905	90.1	.836	67.5
Qwen3-VL-8B	.922	91.3	.832	67.6

Appendix

Table 6: Top pointer head (L21H10) retrieval R@1 and base-model accuracy on ICIR-MCQ (6,000 samples) and Visual Haystacks (

N=10

, 1,000 samples).

Figure 5: Head-level writer R@1 on the ICIR-MCQ held-out evaluation pool (1,200 candidate images; chance R@1 $\approx.001$ ) for all four LVLMs, shown as layers $\times$ heads with a shared color scale. Writer R@1 concentrates in a narrow mid-block layer band, the same region picked out by the pointer ranking (Figure 4 ).

	Ranking agreement		Same-category retrieval
Model	object vs. full	control $\rho$ / overlap	$m$	chance	top-100 R@1	top-10 R@1
InternVL3.5-4B	0.970	0.993 / 93	12.1	0.091	0.466	0.592
InternVL3.5-8B	0.970	0.993 / 96	14.2	0.077	0.424	0.578
Qwen3-VL-4B	0.971	0.996 / 96	16.4	0.064	0.485	0.633
Qwen3-VL-8B	0.970	0.996 / 96	16.4	0.064	0.429	0.590

Appendix

Table 7: Object-region writer probes. Spearman

\rho

of the object-region head ranking against the full-image writer ranking (all 1,152 heads);

\rho

and top-100 overlap of a retrained full-image control against the same ranking; same-category retrieval with the mean number

m

of same-category distractor regions, the per-query chance

1/(m_{i}+1)

averaged over queries, and mean R@1 over the top-100 object-region heads and over the top-10.

	Agreement of the in-context ranking				Split-half stability
					shared top-100		rank 1 vs. 100 gap
Model	vs. writer	vs. pointer	$\Delta$	95% CI	in-ctx	held-out	in-ctx	held-out
InternVL3.5-4B	0.758	0.480	0.277	[0.231, 0.325]	65	85	4.3	22.0
InternVL3.5-8B	0.716	0.530	0.186	[0.143, 0.231]	71	87	4.0	24.4
Qwen3-VL-4B	0.573	0.403	0.170	[0.124, 0.217]	61	86	1.8	19.5
Qwen3-VL-8B	0.657	0.406	0.251	[0.202, 0.302]	63	84	2.2	28.9

Appendix

Table 8: In-context writer ranking versus the held-out writer and pointer rankings (Spearman

\rho

over 1,152 heads;

\Delta=\rho_{\mathrm{writer}}-\rho_{\mathrm{pointer}}

with its 95% bootstrap CI), and split-half stability of the two writer scorings: top-100 heads shared between halves and the R@1 gap between the rank-1 and rank-100 head (points).

Source intervention	InternVL3.5-4B	InternVL3.5-8B	Qwen3-VL-4B	Qwen3-VL-8B
Full head output	0.331	0.285	0.352	0.346
Target-image tokens only	0.338	0.341	0.322	0.326
Remove target-image tokens ( $\Delta$ )	$-0.175$	$-0.160$	$-0.163$	$-0.167$
95% CI	[ $-0.198$ , $-0.150$ ]	[ $-0.182$ , $-0.138$ ]	[ $-0.185$ , $-0.140$ ]	[ $-0.189$ , $-0.144$ ]
Text tokens only	0.155	0.126	0.190	0.178
Remove all text tokens ( $\Delta$ )	$-0.049$	$-0.025$	$-0.085$	$-0.075$

Appendix

Table 9: Writer R@1 of writer-only heads (mean over the set) when the head output is reconstructed from one source group or with one group removed. The two removal rows give the change relative to the full output, with its 95% bootstrap CI beneath. Text tokens are the interleaved filler passages plus the question and option tokens (including special tokens).

	cattok	$+1$	$+2$	$+3$	$+4$	mid	last
InternVL3.5-4B	–	0.960 / 79	0.946 / 77	0.945 / 75	0.948 / 76	0.957 / 78	0.683 / 52
pointer set	1.00	0.86	0.81	0.80	0.86	0.82	0.34
writer ranking $\rho$ / overlap	0.993 / –	0.850 / 72	0.812 / 67	–	0.847 / 70	0.810 / 65	0.536 / 16
writer set	1.00	0.73	0.57	–	0.68	0.51	0.32
InternVL3.5-8B	–	0.957 / 77	0.949 / 74	0.948 / 73	0.943 / 72	0.959 / 73	0.682 / 50
pointer set	1.00	0.85	0.81	0.80	0.85	0.84	0.41

Appendix

Table 10: Readout-position control. First row per model: Spearman

\rho

of the pointer ranking at each position against the cattok pointer ranking, and the top-100 overlap. Second row: mean pointer R@1 of the pointer top-100 of this sweep’s cattok ranking, at each position divided by its value at the cattok. Third row: the same correlation and overlap for the writer ranking retrained at each position; the cattok entry is the retrained cattok probe against the original cattok ranking. Fourth row: mean writer R@1 of the fixed cattok writer top-100 under the probe retrained at each position, divided by its value under the retrained cattok probe. Writer probes were not retrained at

+3

Figure 6: Top-

K

majority-vote R@1 on Visual Haystacks under the two matched ranking–metric pairs, for all four LVLMs (

n{=}1{,}000

per haystack size). Solid lines: pointer ranking, attention-score voting (A). Dashed lines: writer ranking, trained-probe voting (B). Each color is one haystack size

\in\{2,3,5,10,20\}

Model	Overlap heads (pointer-rank order)
InternVL3.5-8B (47 heads)	L21H10 (a1/r35), L21H11 (a2/r46), L21H08 (a3/r20), L20H17 (a4/r83), L21H18 (a7/r10), L20H16 (a8/r81), L20H31 (a9/r16), L20H21 (a10/r68), L21H17 (a11/r6), L20H15 (a14/r3), L19H24 (a17/r97), L20H30 (a18/r8), L22H04 (a20/r34), L23H04 (a21/r82), L20H28 (a23/r2), L21H31 (a24/r4), L20H13 (a25/r63), L24H29 (a27/r14), L24H05 (a28/r7), L18H19 (a29/r5), L20H14 (a32/r45), L19H27 (a34/r42), L24H23 (a36/r1), L19H13 (a38/r25), L28H23 (a41/r91), L19H03 (a42/r94), L17H04 (a46/r51), L22H06 (a49/r22), L23H31 (a50/r90), L28H02 (a51/r69), L13H11 (a52/r38), L19H28 (a54/r49), L19H14 (a58/r74), L18H18 (a59/r9), L24H13 (a64/r33), L19H25 (a69/r75), L31H30 (a73/r96), L23H09 (a78/r27), L29H15 (a79/r64), L19H12 (a82/r44), L29H11 (a85/r77), L15H11 (a87/r24), L23H11 (a89/r17), L23H28 (a91/r18), L20H29 (a94/r99), L18H05 (a99/r47), L24H14 (a100/r19)
Qwen3-VL-8B (35 heads)	L21H10 (a1/r55), L21H08 (a2/r21), L21H11 (a3/r60), L21H18 (a6/r18), L20H15 (a9/r24), L19H27 (a10/r16), L21H17 (a11/r2), L20H31 (a13/r28), L22H04 (a14/r86), L20H30 (a17/r6), L18H19 (a23/r7), L21H31 (a24/r4), L24H29 (a25/r10), L19H13 (a28/r96), L20H28 (a30/r5), L24H05 (a32/r11), L17H04 (a36/r45), L24H23 (a37/r1), L18H18 (a40/r17), L22H06 (a45/r80), L28H02 (a46/r26), L24H30 (a49/r58), L23H30 (a54/r68), L25H09 (a62/r81), L24H13 (a67/r15), L19H25 (a68/r91), L18H05 (a70/r49), L30H06 (a71/r25), L26H31 (a75/r29), L25H11 (a84/r76), L23H28 (a90/r62), L17H27 (a91/r79), L23H12 (a93/r97), L28H03 (a97/r43), L24H14 (a98/r8)
InternVL3.5-4B (42 heads)	L21H10 (a1/r33), L21H11 (a2/r23), L21H08 (a4/r29), L21H18 (a6/r9), L23H04 (a10/r92), L24H29 (a11/r8), L21H17 (a12/r5), L20H31 (a13/r27), L21H16 (a14/r89), L20H16 (a16/r91), L20H15 (a17/r4), L21H23 (a21/r59), L21H31 (a22/r1), L28H23 (a23/r63), L22H04 (a25/r24), L20H30 (a26/r6), L24H23 (a29/r3), L13H11 (a30/r100), L24H05 (a32/r15), L20H18 (a33/r74), L20H14 (a36/r73), L20H28 (a38/r2), L28H03 (a39/r97), L31H30 (a40/r85), L28H02 (a41/r21), L19H27 (a45/r50), L24H13 (a46/r53), L18H19 (a47/r11), L24H14 (a49/r22), L17H04 (a51/r66), L23H28 (a52/r13), L19H13 (a53/r36), L29H13 (a55/r99), L22H06 (a61/r38), L21H30 (a69/r84), L19H14 (a79/r88), L15H11 (a84/r96), L19H28 (a86/r60), L18H18 (a87/r18), L19H12 (a88/r82), L28H16 (a99/r98), L26H18 (a100/r20)
Qwen3-VL-4B (43 heads)	L21H10 (a1/r50), L21H11 (a3/r85), L21H08 (a4/r28), L21H18 (a6/r15), L22H04 (a8/r55), L21H17 (a9/r4), L20H16 (a10/r62), L20H31 (a11/r8), L20H15 (a12/r12), L20H17 (a13/r77), L19H24 (a14/r69), L19H27 (a17/r19), L20H18 (a18/r58), L21H31 (a19/r3), L21H23 (a22/r86), L18H19 (a23/r5), L19H13 (a24/r48), L20H30 (a25/r7), L24H29 (a26/r26), L28H23 (a29/r87), L19H14 (a30/r36), L19H12 (a35/r53), L21H30 (a38/r82), L20H28 (a40/r2), L30H06 (a42/r40), L22H06 (a45/r32), L18H18 (a46/r6), L24H23 (a51/r1), L18H05 (a56/r33), L17H04 (a58/r31), L24H13 (a61/r59), L24H05 (a62/r43), L19H28 (a67/r63), L19H25 (a72/r49), L28H02 (a76/r72), L24H30 (a77/r92), L13H11 (a83/r25), L17H05 (a84/r95), L24H14 (a87/r16), L17H27 (a91/r57), L23H12 (a92/r64), L23H14 (a98/r80), L26H31 (a100/r71)

Appendix

Table 11: Overlap heads (in both pointer top-100 and writer top-100, cattok query) per LVLM, listed by pointer rank, the order in which the overlap subset is ablated (the first

K

heads form the budget-

K

overlap arm). Format LxxHyy (a

x

y

) gives the head’s 1-indexed pointer rank

x

and writer rank

y

Figure 7: Set-partition dose-response per LVLM (top-100 partition, output-zero). Rows: LVLMs (top to bottom: InternVL3.5-8B, Qwen3-VL-8B, InternVL3.5-4B, Qwen3-VL-4B). Columns: haystack size

\in\{2,3,5,10,20,50\}

. Random is the mean over 3 seeds.

	overlap	pointer-only	writer-only	random (3 seeds)
InternVL3.5-4B	0.0 (0%)	0.0 (0%)	0.0 (0%)	0.0 (0%)
InternVL3.5-8B	0.0 (0%)	0.0 (0%)	0.0 (0%)	0.0 (0%)
Qwen3-VL-4B	0.0 (0%)	0.0 (0%)	0.0 (0%)	$+0.1$ / $+0.1$ / $-0.1$ (0.1–0.3%)
Qwen3-VL-8B	$-0.4$ (4.1%)	$-0.8$ (5.1%)	$-0.9$ (2.2%)	$-0.2$ / $+0.8$ / $0.0$ (0–1.7%)

Appendix

Table 12: Full-generation minus prefill-only intervention at

N=10

K=30

(VHs,

n=1{,}000

): change in accuracy (points) and fraction of samples whose parsed answer changed.

Model	LLM / vision	$H$	Top-1 R@1	Overlap	Chance	Ratio
Main-text models (4)	InternVL3.5, Qwen3-VL	1,152	0.866–0.922	35–47	8.7	4.0–5.4 $\times$
LLaVA-OneVision-7B	Qwen2 / SigLIP	784	0.962	46	12.8	3.6 $\times$
Pixtral-12B	Mistral-Nemo / Pixtral	1,280	0.874	40	7.8	5.1 $\times$
InternVL3.5-14B	Qwen3-14B / InternViT	1,600	0.844	31	6.2	5.0 $\times$

Appendix

Table 13: Pointer-writer discovery on additional LVLMs (ICIR-MCQ, 6,000 samples). Overlap counts heads in both top-100 lists; chance is

100^{2}/H

$N$	Baseline	$K{=}5$	$K{=}10$	$K{=}15$	$K{=}20$	$K{=}25$	$K{=}30$
2	0.761	$-0.123$	$-0.180$	$-0.198$	$-0.209$	$-0.211$	$-0.213$
pointer-only		$-0.011$	$-0.094$	$-0.098$	$-0.100$	$-0.106$	$-0.132$
writer-only		$-0.003$	$-0.005$	$-0.002$	$-0.007$	$-0.009$	$-0.017$
3	0.741	$-0.144$	$-0.193$	$-0.207$	$-0.213$	$-0.214$	$-0.210$
pointer-only		$-0.007$	$-0.083$	$-0.114$	$-0.113$	$-0.113$	$-0.136$
writer-only		$-0.004$	$-0.003$	$-0.007$	$-0.007$	$-0.015$	$-0.017$

Appendix

Table 14: LLaVA-OneVision-7B set-partition ablation: accuracy minus the matched random control (mean of three seeds) for the pointer-and-writer overlap (first row per

N

), pointer-only, and writer-only sets. Baseline is the unablated accuracy at each haystack size.

Head	Image rank	Text rank	Character
L21H10	1	118	Image-specific
L21H08	2	67	Image-specific
L21H11	3	66	Image-specific
L23H04	20	10	Mixed
L29H09	137	1	Text-specific
L29H11	221	2	Text-specific

Appendix

Table 15: Image vs. text retrieval rankings (Qwen3-VL-8B).

Large Vision-Language Models

Large vision-language models (LVLMs) connect image and video perception with language generation, reasoning, and interactive tasks.

See all 410 papers

Retrieval Heads Meet Vision: Uncovering How VLMs Locate and Extract Visual Information
Aug 27, 2026Chanho Park, Daehyeon Choi, Jihyun Lee +1VLM Interpretability Vision-Language Retrieval
KAIST · Independent Researcher
Can Retrieval Heads See Images? Multimodal Retrieval Heads in Long-Context Vision-Language Models
May 26, 2026Aaron Branson Cigres Li, Zhaowei Wang, Yu Zhao +9Vision-Language Models VLM Interpretability
1HKUST · University of Edinburgh · 4CUHK +2
Gaze Heads: How VLMs Look at What They Describe
Jun 12, 2026Rohit Gandikota, David BauImage Captioning Visual Attention
Northeastern University

Aug 27, 2026J/K move · Enter open · S save

Retrieval Heads Meet Vision: Uncovering How VLMs Locate and Extract Visual Information

Chanho Park, Daehyeon Choi, Jihyun Lee, Minhyuk Sung

KAIST · Independent Researcher

VLM Interpretability Vision-Language Retrieval Vision-Language Grounding

Open paper AskPDF arXiv

Vision-language models (VLMs) can locate an image region referred to by a text prompt and route the corresponding visual evidence to the output, yet the internal mechanism behind this behavior is not understood. Inspired by retrieval heads in large language models, we ask whether VLMs contain an analogous mechanism for visual retrieval. We answer affirmatively by introducing Visual Retrieval Heads (VRHs), a small subset of attention heads (about 1.7-2.6%) that are causally responsible for grounding text descriptions to image regions. To find them, we recast existing head-scoring methods under a unified design space over query tokens, key aggregation, and cross-sample aggregation. We then show that scoring attention from output prediction tokens with a sum over the ground-truth referent region most reliably identifies causal heads. Across eleven VLMs and five referring-expression benchmarks, masking only the top 20 VRHs reduces grounding accuracy by up to 80 percentage points, while masking the same number of random heads has little effect. Beyond replicating the causal-sparse-universal triad established for text retrieval heads, VRHs exhibit several properties not previously reported: they generalize across visual reference tasks, remaining causal on attribute, spatial, counting, and visual-math benchmarks despite being discovered through bounding-box prediction; they are functionally specific, preserving output format while corrupting localization; and they are architecturally shared, transferring causally across VLMs that share an LLM backbone but differ in vision encoder, projector, and instruction tuning.

Abstract

Large Vision-Language Models

Explore similar work

Retrieval Heads Meet Vision: Uncovering How VLMs Locate and Extract Visual Information

Can Retrieval Heads See Images? Multimodal Retrieval Heads in Long-Context Vision-Language Models

Gaze Heads: How VLMs Look at What They Describe