Attention heads that attend to image regions relevant to a question are increasingly used to steer, ground and interpret vision-language models (VLMs) and to reduce their hallucinations. They are mostly found by ranking heads with an attention- pattern score and validated by steering or masking them. We ask how the choices inside such a discovery score shape the heads it returns, and which of the two validations identifies them. On Qwen3-VL-8B-Instruct we vary the query type, the discovery corpus and the token positions whose attention is read (the attention source) in a full 2 x 3 x 2 design, and test all 12 head-sets for steerability (CERS: Contrastive Evidence-Region Steering score) and necessity (NCS: Necessity Con- trast Score) against layer-matched random head-sets. (1) The attention source and the query type reshape the selected heads far more than the corpus: changing the source keeps a median of 49 of the top-100 heads, the query 55 and the cor- pus 84, while two halves of the same data share 96-99. (2) Necessity follows the attention source: head-sets discovered from query-span attention are more nec- essary than last-token ones (difference 0.083, p=6 x 10^-10), and the two re- ferring + query-span sets lie 17-19 standard deviations above 250 layer-matched random sets (Holm-adjusted p=0.048). The referring + query-span effect re- plicates on several recent VLMs. (3) Steerability is not specific to the discovered heads: no discovered head-set steers better than its layer-matched random sets (rank p >= 0.14), and random sets drawn from the same or from any layer span the discovered sets' scores. Necessity therefore separates the discovery settings, while steering at this strength does not. We turn these findings into a protocol for reporting visual-head discoveries.
Figures & tables
Figure 1: Overview. (1) Discovery. Two query types (ordinal and referring; prompt shortened in the figure) × three attention sources (last prompt token, query-span tokens, generated answer tokens) × two corpora (comics, COCO) give 12 discovery scores. For each, we ask about one region p of a six-panel strip, record how much attention every head sends to p , and rank the heads; the head-set S is the top-100 heads, and each S is paired with layer-matched random controls. (2) Causal evaluation. Every S and its controls get the same two tests. Steerability ( CERS , Eq. 3 ): steer S onto the cell holding the correct answer ( S→c ) or a distractor’s cell ( S→d ) and compare P(c) . Necessity ( NCS , Eq. ): mask S ’s attention to all image tokens on a matched vs. mismatched ImageNet MCQ. The bars illustrate the related grid contrast NCSgrid (Eq. 4 ): hide a distractor’s cell ρ(d) or the answer’s cell ρ(c) from S ; P(c) for S and a random head-set on one real item.
Figure 2: Top-100 head overlap between all 12 discovery settings, Qwen3-VL-8B. The same query and attention source agree across corpora (e.g. cm-/co-ref-qsp share 85/100), while changing the attention source moves most of the set (cm-ord-last vs. cm-ref-qsp: 38/100; minimum 26/100).
Table 3
CERS (steerability)
NCS (necessity)
S
layer-matched random
any-layer random
S
layer-matched random
Setting
[95% CI]
mean [95% CI]
p
mean [95% CI]
[95% CI]
mean [95% CI]
p
cm-ord-last
− .003 [ − .033, .027]
− .039 [ − .147, .069]
.43
.268 [.171, .364]
− .009 [ − .021, .003]
− .002 [ − .003, .000]
1.0
cm-ord-out
.033 [.002, .064]
− .079 [ − .170, .011]
.24
− .028 [ − .140, .084]
.017 [ − .010, .044]
.000 [ − .002, .002]
.048
cm-ord-qsp
.139 [.111, .167]
.061 [ − .023, .146]
.33
.158 [.018, .297]
.043 [.011, .075]
− .004 [ − .006, − .001]
.048
cm-ref-last
− .303 [ − .341, − .264]
.303 [.186, .420]
1.0
.136 [.005, .267]
.002 [ − .029, .033]
.001 [ − .001, .003]
.38
Table 3: All 12 settings, Qwen3-VL-8B, flat top-100 heads, N=400 (cm/co = comics/COCO corpus; ord/ref = query; last/qsp/out = attention source). S : the discovered head-set, with a 95% t -interval over the 400 items. Random: mean over 20 random head-sets, either layer-matched or drawn from any layer, with a 95% t -interval of that mean over the draws. p = rank p of S among the individual layer-matched sets, whose spread is wider than the interval of their mean. ∗ 250 layer-matched random sets (Holm-adjusted p=0.048 ).
Figure 3: Table 3 at a glance: each discovered head-set (dot) against the mean ± SD of its 20 layer-matched random sets (bar). On NCS the query-span and output-token sets lie outside their random bands, which sit at zero; on CERS every set falls inside or below a wide random band.
Table 6
flat top-100: NCS when masked at
top-3 heads per layer (108)
Setting
all rows
question/option rows
last row
NCS
CERS / random ( p )
co-ref-qsp
.126 [.086, .165]
.140 [.100, .181]
.000 [ − .003, .003]
.046 [.013, .078]
.229 / .103 (.38)
co-ord-qsp
.036 [.002, .070]
.049 [.015, .083]
− .001 [ − .005, .002]
.031 [ − .004, .066]
.207 / − .091 (.29)
co-ref-last
− .002 [ − .035, .032]
− .001 [ − .031, .028]
− .003 [ − .007, .001]
.012 [ − .020, .045]
.190 / .071 (.33)
co-ord-last
− .002 [ − .008, .004]
.002 [ − .003, .006]
− .001 [ − .004, .002]
.005 [ − .028, .038]
.458 / .195 (.24)
Table 5: Where and with which heads necessity acts (COCO, N=400 ; 95% CIs; bold: above all 20 layer-matched random sets, whose means lie within 0.003 of zero). Left: the flat top-100 set masked from the image at every row, only at the question and option rows, or only at the last row. Right: the top-3 heads of every layer instead of the flat top-100 (Appendix A.5 ).
comics discovery
COCO discovery
Query + source
NCS [95% CI]
p
NCS [95% CI]
p
ord-last
.004 [ − .001, .009]
.010
.001 [.000, .001]
.010
ord-out
.226 [.157, .294]
.010
.268 [.195, .340]
.010
ord-qsp
.194 [.129, .259]
.010
.222 [.154, .290]
.010
ref-last
.005 [ − .004, .014]
.11
.007 [ − .003, .016]
.050
ref-out
.247 [.176, .317]
.010
.181 [.117, .244]
.010
Table 6: Necessity as a region contrast on the leak-audited grid ( N=120 ): P(c) with another cell hidden from S minus P(c) with the target cell hidden, so generic damage cancels within each item (maximum 0.716 ). p : rank p against 100 layer-matched random sets, whose means lie within 0.002 of zero (floor 0.010 ; Appendix A.8 ).
Table 9
NCS (necessity)
CERS (steerability)
query span + output
last token
query span + output
Model
S
random
S
random
S
random
Qwen3-VL-4B
.140 – .211
.006 – .032
.010
.002
−.015 – .154
−.041 – −.020
Qwen3.5-9B
.217 – .231
.025 – .043
.223
.060
−.034 – −.003
−.041 – .040
Gemma-4-E4B
.366 – .402
.070 – .097
.022
.035
−.005 – .007
−.026 – .038
Ovis1.5-Llama3-8B
.204 – .254
.005 – .023
.024
.004
.001 – .014
−.035 – −.001
Table 8: Five further models, eight out-of-domain benchmarks (heads discovered anew per model, flat top-100; one layer-matched random draw per head-set). Ranges over the five query-span and output-token sets and over their random draws; last token: cm-ord-last (Appendix C.1 ).
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
CERS
NCS
Setting
A
B
A
B
cm-ord-last
.43
.095
1.00
1.00
cm-ord-out
.24
.048
.048
.048
cm-ord-qsp
.33
.048
.048
.048
cm-ref-last
1.00
1.00
.38
.24
cm-ref-out
.62
.24
.048
.048
Appendix
Table 9: Rank p of each discovered head-set against 20 layer-matched (A) and 20 layer- and attention-matched (B) random head-sets (Qwen3-VL-8B, flat top-100, N=400 ). Scores are in Table 3 .
Setting
NCS
Control mean ± SD
p
cm-ref-qsp
0.116
0.002±.007
.004
co-ref-qsp
0.126
−0.001±.007
.004
Appendix
Table 10: The two referring + query-span settings against R=250 layer-matched random head-sets (Qwen3-VL-8B, top-100, N=400 ): control mean ± SD over the 250 draws and the rank p (floor 0.004 ).
Setting
Pmask matched
Matched drop
Pmask mism.
Mism. drop
NCS
cm-ord-last
0.996
−0.002 [-0.005,0.001]
0.212
0.005 [-0.006,0.017]
−0.007 [-0.019,0.004]
cm-ord-out
0.973
0.022 [0.009,0.036]
0.213
0.005 [-0.019,0.028]
0.017 [-0.010,0.045]
cm-ord-qsp
0.967
0.028 [0.015,0.042]
0.230
−0.013 [-0.042,0.018]
0.040 [0.009,0.070]
cm-ref-last
0.990
0.004 [-0.000,0.010]
0.217
0.000 [-0.030,0.031]
0.004 [-0.027,0.034]
cm-ref-out
0.963
0.032 [0.017,0.048]
0.216
0.002 [-0.025,0.030]
0.030 [-0.001,0.059]
cm-ref-qsp
0.885
0.110 [0.083,0.135]
0.220
−0.003 [-0.034,0.029]
0.113 [0.074,0.153]
Appendix
Table 11: NCS split into its arms, all 12 settings ( N=400 , top-100). P renormalized over the four letters; drops are baseline minus masked. 95% percentile bootstrap intervals, 2,000 resamples.
Head-set
Heads
Matched drop
Mismatched drop
NCS
NCS-v2
Text-arm drop (text masked)
Discovered ( ref-qspan )
100
0.112
−0.013
0.125
0.112
0.645
Attention sinks
100
0.002
−0.005
0.007
0.002
0.012
Text-heavy heads
100
0.002
−0.003
0.005
0.002
0.415
Whole layers
96
0.005
0.006
−0.001
0.005
0.232
Random, 200 heads
200
0.006
−0.014
0.020
0.006
0.718
Random, 400 heads
400
0.010
−0.015
0.025
0.010
0.776
Appendix
Table 12: Generic-damage controls on co-ref-qsp ( N=400 ). Matched and mismatched drops are the decrease in P(correct) when the head-set’s image attention is masked; the last column is the decrease on the text-solvable arm when the head-set’s text attention is masked.
flat top-100
top-3 per layer (108)
Setting
Masked rows
S [95% CI]
random
p
S [95% CI]
random
p
co-ref-qsp
all rows
.126 [.086, .165]
.000
.048
.046 [.013, .078]
.001
.048
image rows
− .001 [ − .023, .020]
.002
.86
.010 [ − .012, .033]
.000
.048
question/option rows
.140 [.100, .181]
− .004
.048
.056 [.025, .088]
− .001
.048
last row
.000 [ − .003, .003]
.000
.67
− .002 [ − .006, .002]
.000
1.0
co-ord-qsp
all rows
.036 [.002, .070]
− .002
.048
.031 [ − .004, .066]
.003
.048
Appendix
Table 13: NCS when the image mask is applied only at one group of attention rows (COCO settings, N=400 ). S with 95% t -interval; random: mean of 20 random head-sets with the same per-layer counts; p : rank p (floor 0.048 ).
cm-ref-qsp
co-ref-qsp
k
S [95% CI]
random
p
S [95% CI]
random
p
20
.036 [.005, .066]
− .002 ± .004
.048
.015 [ − .011, .041]
− .001 ± .004
.048
40
.045 [.014, .076]
− .001 ± .003
.048
.026 [ − .005, .056]
.000 ± .004
.048
60
.041
.001 ± .005
.048
.029
.000 ± .005
.048
100
.116 [.078, .155]
.002 ± .007
.004
.126 [.086, .165]
− .001 ± .007
.004
140
.204 [.160, .249]
.002 ± .004
.048
.092
.001 ± .006
.048
Appendix
Table 14: NCS against head-set size (Qwen3-VL-8B, N=400 ). S with 95% t -interval where the per-example outputs are available; random: mean ± SD of 20 layer-matched random sets (250 at k=100 ); p : rank p .
Figure 6: Top- k overlap between cm-ref-qsp and co-ref-qsp (Qwen3-VL-8B). Left: shared heads and each corpus’s unique heads (equal in count, not in identity). Right: overlap fraction (shared /k ).
Comparison
Source
Shared / only
Shared
A-only
B-only
corpus (cm vs co), ordinal
query span
91 / 9
+0.036 (0.01)
−0.001 (0.61)
+0.001 (0.35)
corpus (cm vs co), ordinal
last token
80 / 20
−0.003 (0.84)
−0.003 (0.84)
+0.000 (0.49)
query type (ord vs ref), comics
query span
79 / 21
+0.033 (0.015)
+0.010 (0.2)
+0.005 (0.14)
query type (ord vs ref), comics
last token
35 / 65
−0.001 (0.67)
−0.006 (0.97)
+0.005 (0.37)
source (X vs out), COCO ordinal
query span
71 / 29
+0.029 (0.026)
+0.011 (0.19)
−0.004 (0.83)
source (X vs out), COCO ordinal
last token
51 / 49
−0.001 (0.64)
+0.000 (0.42)
+0.013 (0.17)
Appendix
Table 15: NCS above the mean of 20 same-size layer-matched random controls (paired one-sided p in parentheses; bold p<0.05 ) for shared and setting-specific heads (Qwen3-VL-8B, top-100 per setting, N=400 ). “Shared / only” gives the head counts.
NCS (region contrast)
CERS (symmetric)
Setting
S [95% CI]
random
p
S [95% CI]
random
p
cm-ord-last
.004 [ − .001, .009]
.000 [.000, .000]
.010
− .010 [ − .052, .031]
− .040 [ − .078, − .003]
.40
cm-ord-out
.226 [.157, .294]
− .001 [ − .001, .000]
.010
.027 [ − .014, .068]
− .102 [ − .138, − .066]
.21
cm-ord-qsp
.194 [.129, .259]
.000 [.000, .001]
.010
.141 [.106, .176]
.067 [.030, .104]
.37
cm-ref-last
.005 [ − .004, .014]
.001 [.001, .002]
.11
− .272 [ − .325, − .220]
.271 [.218, .325]
.99
cm-ref-out
.247 [.176, .317]
.001 [.000, .001]
.010
.057 [.026, .087]
.139 [.090, .187]
.64
Appendix
Table 16: Region-contrast NCS and symmetric CERS on the leak-audited grid ( N=120 items, Qwen3-VL-8B, flat top-100). S : discovered set, with a 95% t -interval over items; random: mean [95% CI] over 100 layer-matched random sets; p : rank p (floor 0.010 ). The maximum attainable NCS is 0.716 .
mean image-attention mass
ρ (statistic, CERS ) over random sets
Setting
discovered
random
mean
median
min
max
SD
cm-ord-last
0.188
0.041 [0.032, 0.046]
− .39
− .54
− .22
− .20
− .14
cm-ord-out
0.174
0.052 [0.045, 0.062]
− .04
.12
− .01
.28
.17
cm-ord-qsp
0.187
0.046 [0.040, 0.053]
− .15
.02
− .19
− .10
− .29
cm-ref-last
0.242
0.047 [0.040, 0.061]
.39
.11
.26
.37
.40
cm-ref-out
0.188
0.051 [0.043, 0.061]
− .37
− .40
− .21
.01
− .12
Appendix
Table 17: Image-attention mass of head-sets and its relation to CERS . Mass columns: mean over the set’s heads (random: mean [min, max] over 20 layer-matched sets). ρ : Spearman correlation between the mass statistic and CERS across the 20 random sets.
cm-ref-qsp
co-ref-qsp
cm-ord-last
co-ord-last
β
S
random
S
random
S
random
S
random
1
.279
− .026 [ − .071, .009]
.176
.056 [ − .021, .146]
.132
− .040 [ − .064, − .017]
.032
.015 [ − .013, .067]
3
.869
− .103 [ − .279, .029]
.789
.204 [ − .059, .347]
.308
− .134 [ − .192, − .071]
.027
.072 [ − .076, .332]
10
.317
.004 [ − .241, .221]
.396
.270 [.087, .558]
.092
− .068 [ − .200, .088]
.182
.175 [ − .232, .461]
100
.266
− .001 [ − .214, .218]
.273
.226 [.081, .517]
.042
− .040 [ − .155, .095]
.206
.173 [ − .224, .421]
104
.179
.031 [ − .291, .291]
.172
.376 [.052, .841]
.018
− .032 [ − .192, .098]
.176
.161 [ − .202, .463]
Appendix
Table 18: CERS against steering strength β (Qwen3-VL-8B, flat top-100, N=200 ): the discovered set S and the mean [min, max] of 5 layer-matched random sets. β=30 and 103 lie between their neighbours and are omitted.
Setting (COCO)
Qwen3-VL-8B
Qwen3-VL-4B
Qwen2-VL-7B
ordinal + last token
−0.002 vs. 0.000 ( .86 )
0.006 vs. 0.003 ( .14 )
−0.002 vs. 0.001 ( 1.00 )
ordinal + query span
0.036 vs. −0.003 ( .048 )
−0.005 vs. 0.005 ( .91 )
0.001 vs. 0.002 ( .57 )
referring + last token
−0.002 vs. 0.002 ( .76 )
−0.004 vs. 0.004 ( .91 )
0.024 vs. 0.001 ( .048 )
referring + query span
0.125 vs. 0.000 ( .048 )
0.054 vs. 0.002 ( .048 )
0.029 vs. 0.002 ( .048 )
Appendix
Table 19: Second-model check: NCS of the discovered head-set vs. the mean of 20 layer-matched random controls (rank p ), COCO discovery, top-100 heads, N=400 . Bold: above all 20 controls.
What changes
Pair
Shared heads
Ceiling
Fraction
corpus
cm-ord-last vs. co-ord-last
81.2 [ 80 , 83 ]
98.4
0.82
corpus
cm-ord-qsp vs. co-ord-qsp
90.5 [ 90 , 91 ]
99.2
0.91
corpus
cm-ref-last vs. co-ref-last
83.8 [ 81 , 86 ]
96.6
0.87
corpus
cm-ref-out vs. co-ref-out
78.5 [ 76 , 80 ]
97.7
0.80
corpus
cm-ref-qsp vs. co-ref-qsp
83.8 [ 82 , 86 ]
97.8
0.86
query type
cm-ord-last vs. cm-ref-last
35.2 [ 33 , 37 ]
97.2
0.36
Appendix
Table 20: Top-100 overlap between two discovery settings at 200 strips per half (mean [ 2.5 , 97.5 percentile] over 200 half-splits), the geometric-mean noise ceiling of the two settings, and the overlap as a fraction of that ceiling.
Query + attention source
ord-last
ref-qspan
ref-out
ordinal + output tokens
8
8
9
referring + output tokens ( ref-out )
5
7
13
Appendix
Table 21: Top-20 overlap with the native head-sets for the two crossover combinations not in Table 2 ( 200 COCO strips, Qwen3-VL-8B). Bold: the combination’s own native set.
Figure 7: Top-100 overlap between the native ord-last, ref-qspan and ref-out sets and a uniform random draw of 100 heads, on eight further models (Gemma-4-E4B-it has ord-last scores only, so its other bars are marked N/A). Ord-last/ref-qspan overlap is 25 – 86 /100 across the seven scored models, ord-last/ref-out 25 – 89 /100, and ref-qspan/ref-out 40 – 92 /100; the random draw shares 3 – 76 /100 with ord-last and 1 – 78 /100 with ref-qspan.
Dataset
CERS oracle
CERS floor
NCS matched
NCS mismatched
MathVista
0.885
0.275
0.870
0.730
CountBenchQA
0.950
0.302
0.897
0.220
CVBench
0.632
0.268
0.740
0.605
MMStar
0.815
0.273
0.935
0.367
SpatialRGPT
0.282
0.230
0.492
0.470
Spatial457
0.573
0.217
0.735
0.527
Appendix
Table 22: Out-of-domain baseline audit, Qwen3-VL-8B, N=400 /dataset. Oracle: target cell named in the prompt (expect ≈1 ). Floor: blank image (expect 0.25 ). Matched/mismatched: NCS arms without intervention (expect ≈1 / 0.25 ). Bold: outside the expected range.
Head-set
CERS [95% CI]
NCS [95% CI]
cm-ref-qsp
0.288 [ 0.272 , 0.304 ]
0.225 [ 0.188 , 0.264 ]
random
−0.117 [ −0.135 , −0.100 ]
0.020 [ 0.001 , 0.039 ]
co-ref-qsp
0.092 [ 0.078 , 0.107 ]
0.211 [ 0.175 , 0.249 ]
random
0.009 [ −0.004 , 0.021 ]
0.023 [ 0.008 , 0.037 ]
co-ord-qsp
0.271 [ 0.257 , 0.286 ]
0.161 [ 0.127 , 0.196 ]
random
−0.141 [ −0.158 , −0.124 ]
0.011 [ −0.003 , 0.026 ]
Appendix
Table 23: Out-of-domain CERS and NCS, pooled over the eight datasets, Qwen3-VL-8B ( N=1280 per head-set for CERS , 563 pairs for NCS; flat top-100). Each discovered set is followed by its single layer-matched random draw. 95% intervals: t over items ( CERS ), bootstrap (NCS).
Setting
Qwen3-VL-4B
Qwen3.5-9B
Gemma-4-E4B
Ovis1.5-8B
InternVL3.5-14B
CERS
cm-ref-qsp
.098 ( −.022 )
−.003 ( −.038 )
.001 ( .035 )
.011 ( −.026 )
.246 ( .135 )
co-ref-qsp
.070 ( −.041 )
−.017 ( −.041 )
−.005 ( .004 )
.013 ( −.003 )
.341 ( −.031 )
co-ord-qsp
.154 ( −.021 )
−.017 ( .040 )
.000 ( .038 )
.005 ( −.035 )
.228 ( .071 )
cm-ord-qsp
.090 ( −.041 )
−.034 ( .028 )
.007 ( −.026 )
.014 ( −.014 )
.288 ( .102 )
co-ref-out
−.015 ( −.020 )
−.024 ( .018 )
−.002 ( −.020 )
.001 ( −.001 )
.379 ( −.176 )
Appendix
Table 24: Out-of-domain CERS / NCS of the discovered set, with its single layer-matched random draw in parentheses, five further models (flat top-100, N=1280 CERS , 563 NCS pairs).
Task
Family
What the model is shown and asked
CV-Bench_pointing
converted
Real photo; question about counting or relations; wrong answers overlaid on three copies of the same photo.
MMStar_pointing
converted
Same recipe, with MMStar’s multi-domain VQA items.
SpatialRGPT_pointing
converted
Real photo with two labeled regions and a spatial-relation question; distractor answers overlaid on copies.
Spatial457_pointing
converted
Four synthetic scene renders; the question refers to one scene’s objects; the correct cell is that scene.
CountBenchQA_pointing
converted
Real photo of counted objects; “how many X”; candidate counts as text options.
MathVista_pointing
converted
Four math diagrams; the correct item’s question and choices asked in text.
Appendix
Table 25: The 14 tasks of the pointing suite. Every task uses the same 2×2 CERS apparatus (Eq. 3 ); only how the four cells and four options are built differs.
Task
ord-last
Random
Gap
spatial457_pointing
0.427
0.459
−0.033
MathVista_pointing
0.012
0.028
−0.016
attribute_identification
0.017
0.004
0.012
general_causal_MathVista
0.069
0.069
0.000
CountBenchQA_pointing
0.019
0.013
0.006
general_causal_CVBench
0.197
0.186
0.011
Appendix
Table 26: CERS of the flat top-20 ord-last set and one layer-matched random draw, Qwen3-VL-8B, N=80 per task. Bold: the three flagged tasks. The gap is computed from unrounded values.
Model
Baseline acc.
ord-last
ref-qspan
ref-out
Random
Qwen3-VL-2B
0.492
0.161
0.033
0.022
−0.041
Qwen3-VL-4B
0.521
0.127
0.189
0.066
−0.010
Qwen2-VL-7B
0.487
0.066
0.027
0.025
−0.056
Qwen3-VL-32B (4-bit)
0.537
0.122
0.168
0.116
−0.020
InternVL3.5-8B
0.527
0.006
0.019
0.075
0.003
InternVL3.5-14B (4-bit)
0.483
0.102
0.150
0.159
−0.054
Appendix
Table 27: Pointing-suite CERS on eight further models (all 14 tasks, N=1120 per model, flat top-20, one random draw per model). Bold: p<0.05 against zero (one-sample t -test).
Figure 8: CERS vs. NCS on real example images. Left: the 2×2 MCQ grid with the correct cell (A, green) and a distractor cell (B, orange), steered via do(S,c) and do(S,d) . Right: the matched (target class visible, green) and mismatched (excluded class, orange) images used for NCS, with the attention mask indicated below each head-set box.
Result
Head-set
Evaluation data
N
R
Tables 1 , 20 (noise ceiling)
flat top-100
discovery strips, 200 half-splits
400
–
Tables 2 , 21 ; Fig. 7
top-20; top-100
200 COCO strips (discovery only)
200
–
Table 3 , Fig. 3 , Tables 9 , 11
flat top-100
ImageNet 2×2 grid ( CERS ); ImageNet MCQ (NCS)
400
20 (250)
Table 4 , 14 , Fig. 5
flat top- k , k∈{20..160}
ImageNet MCQ
400
20 (250)
Table 12
6 head-sets
ImageNet MCQ + text-solvable arm
400
–
Tables 5 , 13
flat top-100; kper_layer=3
ImageNet MCQ ( CERS : grid)
400
20
Appendix
Table 28: Protocol per result, Qwen3-VL-8B-Instruct unless stated. R : random head-sets per discovered set.
Benchmark
Source content
Recipe
CountBenchQA
counting VQA
same-image overlay (near-miss count distractors)
CV-Bench
single-image MCQ perception
same-image overlay
MathVista
MCQ diagram reasoning
same-image overlay (overlaid answer-choice text)
MMStar
single-image MCQ perception
same-image overlay
Spatial457
spatial-relation VQA
image retrieval
SpatialRGPT-Bench
spatial-relation VQA
image retrieval
Appendix
Table 29: Source and conversion recipe of the seven converted benchmarks.
Vision-language models (VLMs) can locate an image region referred to by a text prompt and route the corresponding visual evidence to the output, yet the internal mechanism behind this behavior is not understood. Inspired by retrieval heads in large language models, we ask whether VLMs contain an analogous mechanism for visual retrieval. We answer affirmatively by introducing Visual Retrieval Heads (VRHs), a small subset of attention heads (about 1.7-2.6%) that are causally responsible for grounding text descriptions to image regions. To find them, we recast existing head-scoring methods under a unified design space over query tokens, key aggregation, and cross-sample aggregation. We then show that scoring attention from output prediction tokens with a sum over the ground-truth referent region most reliably identifies causal heads. Across eleven VLMs and five referring-expression benchmarks, masking only the top 20 VRHs reduces grounding accuracy by up to 80 percentage points, while masking the same number of random heads has little effect. Beyond replicating the causal-sparse-universal triad established for text retrieval heads, VRHs exhibit several properties not previously reported: they generalize across visual reference tasks, remaining causal on attribute, spatial, counting, and visual-math benchmarks despite being discovered through bounding-box prediction; they are functionally specific, preserving output format while corrupting localization; and they are architecturally shared, transferring causally across VLMs that share an LLM backbone but differ in vision encoder, projector, and instruction tuning.
We study how to detect and mitigate visual attention sinks in vision-language models (VLMs). Visual attention sinks are image tokens that absorb a large share of attention over the image regardless of the query or task, and they mislead methods that treat attention maps as a record of where the model ``looked.'' Prior work has detected sinks through massive activation dimensions and proposed redistributing their attention, but none have addressed what makes a token become a sink. Comparing six candidate detectors on ten VLMs from four families, we find that only the centered cosine similarity between an image token's key and the key of the language model's text-sink detects visual sinks. Thus, a visual sink is an image token whose key aligns with the text-sink's key. Moving a sink's key component along this direction to the median of the image keys sends its attention back to the text-sink (Are Borrowed) while leaving model answers unchanged~(Not Seen). Building on this, we propose a training-free, entropy-triggered re-look that first cleans the attention map, then crops the most-attended region of the cleaned map and inserts it into the reasoning trace. Cleaning adds negligible computation to the forward pass, and the re-look improves accuracy with a mean gain of +4.3 points on fine-detail VQA benchmarks across model families.
Dengjia Zhang, Alexander Martin, Pengyu Yan +6
Johns Hopkins University · University of Pittsburgh · University at Buffalo +2
Whether attention weights faithfully reflect model reasoning has been actively debated in NLP, yet this question remains largely unexplored for the visual modality in Vision-Language Models (VLMs). We address this gap through causal perturbation analysis on current VLMs, evaluating both the comprehensiveness and sufficiency gap of attention-ranked visual tokens. Our analysis reveals that visual attention faithfulness is heterogeneous, manifesting in three distinct processing modes: Faithful-Sufficient, where top-k attention tokens are both necessary and sufficient for prediction; Faithful-Distributed, where they are necessary but broader visual context remains required; and Non-Focal, where no localized attention region is individually necessary while visual information remains an essential trigger for prediction. Furthermore, human-annotated ground-truth regions satisfy comprehensiveness in only ∼60% of cases compared with model attention rankings, revealing systematic divergence between model visual reliance and human intuition. We demonstrate these patterns across both general VQA on VQAv2 and document tasks on VRDU and ChartQA, showing that visual attention faithfulness varies systematically with processing demands and model architectures rather than being uniformly faithful or unfaithful.
Xurui Song, Weishi Wang, Zhongqi Yue +5
SAP · Microsoft · College of Computing and Data Science, Nanyang Technological University, Singapore