Attention Sinks in VLMs Are Borrowed, Not Seen

Dengjia Zhang; Alexander Martin; Pengyu Yan; Xihan Sun; Jiaqi Li; Benjamin Van Durme; Andrew Yates; Reno Kriz

cs.AIOct 11, 2026

Attention Sinks in VLMs Are Borrowed, Not Seen

Authors: Dengjia Zhang, Alexander Martin, Pengyu Yan, Xihan Sun, Jiaqi Li, Benjamin Van Durme, Andrew Yates, Reno Kriz,

Organizations: Johns Hopkins University · University of Pittsburgh · University at Buffalo · Vanderbilt University ·

Abstract

We study how to detect and mitigate visual attention sinks in vision-language models (VLMs). Visual attention sinks are image tokens that absorb a large share of attention over the image regardless of the query or task, and they mislead methods that treat attention maps as a record of where the model ``looked.'' Prior work has detected sinks through massive activation dimensions and proposed redistributing their attention, but none have addressed what makes a token become a sink. Comparing six candidate detectors on ten VLMs from four families, we find that only the centered cosine similarity between an image token's key and the key of the language model's text-sink detects visual sinks. Thus, a visual sink is an image token whose key aligns with the text-sink's key. Moving a sink's key component along this direction to the median of the image keys sends its attention back to the text-sink (Are Borrowed) while leaving model answers unchanged~(Not Seen). Building on this, we propose a training-free, entropy-triggered re-look that first cleans the attention map, then crops the most-attended region of the cleaned map and inserts it into the reasoning trace. Cleaning adds negligible computation to the forward pass, and the re-look improves accuracy with a mean gain of $+4.3$ points on fine-detail VQA benchmarks across model families.

Figures & tables

Figure 1: Visual attention sinks point the same way as the text-sink. Left: visual attention sinks (red, $\geq 100\times$ uniform attention outside the target) and the target (green) in a V ∗ image. Right: key direction (angle) vs. attention (radius) of every image token

Figure 2: Image tokens on V ∗ at each model’s peak layer. Angle: centered cosine of a token’s key with the text-sink key ( Equation 2 ). Radius: attention received relative to uniform, log scale. Sinks: tokens above $100\times$ uniform (dotted ring) outside the target box. Dashed ray: gate threshold $.5$ . Red numbers: median cosine of the sinks; ∗ : sink-free

Model	LM	per image	$\times$ uniform	share
QVL-8B	Qwen3	3.9	1259	.66
QVL-2B	Qwen3	2.7	163	.12
IVL3.5-8B	Qwen3	1.5	342	.24
IVL3.5-2B	Qwen3	1.0	151	.08
IVL3-8B	Qwen2.5	0.4	93	.02
Pix-12B	Mistral	1.0	2093	.76

Table 1: Visual attention sinks in VLMs on V ∗ at each model’s peak layer. key1: sinks per image; key 2: their attention relative to uniform; key 3: their share of the in-image attention.

Figure 3: Crop selection in Qwen3-VL-8B-Thinking on two V ∗ images. Green: target box; red/blue: raw/edited window. Right: question attention over the whole prompt (one cell per $1\%$ ). The edit moves the visual attention sinks’ share to the text-sink.

	Map			Model
Model	$\max_{j}r_{j}$	top-1 $\in\mathcal{B}$	text-sink	accuracy	agreement	KL	caption
Qwen3-VL-8B	$.25\to.04$	$.00\to.47$	$.57\to.67$	$.73\to.73$	.99	.004	.83
Qwen3-VL-4B	$.24\to.05$	$.00\to.41$	$.54\to.68$	$.73\to.73$	.98	.003	.82
Qwen3-VL-2B	$.04\to.03$	$.28\to.23$	$.63\to.63$	$.68\to.61$	.81	.039	.57
InternVL3.5-8B	$.34\to.05$	$.00\to.72$	$.44\to.65$	$.71\to.68$	.94	.001	.85
InternVL3.5-2B	$.04\to.04$	$.60\to.59$	$.65\to.65$	$.64\to.61$	.93	.002	.80

Table 2: Effects of the key edit on V ∗ .

L

: number of language-model layers. text-sink: share of question attention on the text-sink, averaged over the probed layers

Feature	mean	min
hidden norm	80.1	25.0
$-\\|v\\|$	72.0	8.0
$-\\|k\\|$	77.4	49.0
$-$ logit std	78.4	65.0
raw key cos	99.0	94.0
centered key cos	99.9	99.0

Table 3: AUC of each feature at separating visual attention sinks from other image tokens, at the peak layer on V ∗ . Mean and min over sink-bearing models.

Figure 4: Overview of the entropy-triggered re-look. (1) The key edit removes the visual attention sinks from the prefill attention map. (2) The selector keeps the $\frac{1}{3}$ -grid window with the most attention on the cleaned map. (3) When the next-token entropy first exceeds $H_{0}$ after $c$ tokens of thought, the crop of that window is inserted into the chain of thought.

V ∗

HR-4K

HR-8K

MME-RW-Lite

Model

Base

Raw

\Delta

Clean

\Delta

Base

Raw

\Delta

Clean

\Delta

Base

Raw

\Delta

Clean

\Delta

Base

Raw

\Delta

Clean

\Delta

Qwen3-VL-Thinking

79.1

+2.1

+3.7

72.6

+0.8

+4.1

†

68.9

+2.0

+4.4

†

46.5

+1.0

+2.0

†

82.2

-2.1

+2.6

74.1

-0.5

+0.9

67.4

+1.5

+5.1

†

45.1

+0.8

+3.4

†

70.2

+3.1

+1.6

69.5

+3.0

+3.0

†

63.9

+4.1

+3.1

†

47.2

+2.1

+2.0

†

InternVL3.5 (thinking)

Table 4: Base accuracy (%) and paired re-look gain. Clean

\Delta

: crop selected on the sink-cleaned map. Raw

\Delta

: on the raw attention map. Shading scales with gain (green) or loss (red); † :

p<.05

(exact McNemar, clean)

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Model	LM	position	share	$\times$ uniform	per image	share	layer	at peak	best layer
		Text sink		Visual sink (peak layer)				Target top-1 $\in\mathcal{B}$
Qwen3-VL-8B	Qwen3	0	.74	1259	3.9	.66	30/36	.00	.80
Qwen3-VL-4B	Qwen3	0	.69	1049	4.9	.73	29/36	.00	.82
Qwen3-VL-2B	Qwen3	0	.75	163	2.7	.12	27/28	.00	.75
InternVL3.5-8B	Qwen3	0	.65	342	1.5	.24	30/36	.14	.62
InternVL3.5-2B	Qwen3	0	.77	151	1.0	.08	7/28	.00	.69

Appendix

Table 5: Text and visual sinks in ten models on V ∗ , extending Table 1 . LM is the language-model backbone. Text sink: the token index of the text sink and its share of the question’s attention. Visual sink (peak layer): attention relative to uniform, count per image, share of the in-image attention and the peak layer out of the model’s layer count. Target top-1

\in\mathcal{B}

: the fraction of images whose most-attended image token lies in the target box, at the peak layer and at the best layer.

Figure 5: Sink attention across depth on V ∗ for all ten models. Red: visual attention sink share of in-image attention. Gray dashed: text-sink share of question attention. Dotted: peak layer. Gemma-4 reaches the text sink only in its full-attention layers (triangles). Title color marks the language-model backbone (Qwen3, Gemma-4, Qwen2.5 and Mistral).

Model	centered key cos	raw key cos	$-\\|v\\|$	hidden norm	$\\|k\\|$	$-\\|k\\|$	$-$ logit std
Qwen3-VL-8B	.99/.98/1.00	.99/.96/1.00	.99/.94/1.00	.99/.93/1.00	.00/.00/.01	.23/.00/.99	.00/.00/.69
Qwen3-VL-4B	.99/.95/1.00	.99/.93/1.00	.99/.68/1.00	.98/.80/1.00	.00/.00/.28	.01/.00/.72	.00/.00/.71
InternVL3.5-8B	.95/.70/1.00	.95/.66/1.00	.95/.73/1.00	.95/.61/.99	.00/.00/.01	.74/.28/.99	.00/.00/.90
Gemma-4-12B	.61/.00/1.00	.61/.00/1.00	.00/.00/.40	.00/.00/.93	.01/.00/.33	.18/.21/.68	.02/.00/.85
Gemma-4-E4B	.18/.00/1.00	.04/.00/.99	.00/.00/.56	.00/.00/.44	.00/.00/.50	.00/.00/.49	.00/.00/.65
Gemma-4-E2B	.07/.00/.99	.00/.00/.94	.00/.00/.08	.00/.00/.25	.06/.00/.46	.15/.12/.55	.00/.00/.69

Appendix

Table 6: Token features as detectors of visual sinks at each model’s peak layer on V ∗ . Cells: precision at

k

/ top-1 / AUC, where

k

is the number of sinks in the image. The three sink-free models are below the rule. There the few tokens above

100\times

uniform (0.4 to 2.7 per image) stand in for the sinks. The first two columns are the centered cosine between the token’s key and the text-sink key ( Equation 2 ) and the same cosine without centering. The next five are the negated value norm, the hidden-state norm, the key norm with both signs and the negated standard deviation of the token’s attention logits. Apart from the key norm, every feature is signed so that a larger value is more sink-like. Hidden norm is the feature of Kang et al. (2025a) .

		Remove			Align
Model	Layer	before	$-\hat{u}$	$-$ random	before	$:=k_{t^{\star}}$	random key
Qwen3-VL-8B	30	.68	.00	.66	.00	.97	.04
Qwen3-VL-4B	29	.75	.00	.74	.00	.95	.00
InternVL3.5-8B	30	.36	.00	.35	.00	.96	.09
Pixtral-12B	27	.49	.25	.49	.00	.91	.64
median target	27	.49	.00	.49	.00	.91	.01

Appendix

Table 7: Key interventions at the peak layer on 20 V ∗ images. Remove : in-image share of the gated tokens before and after projecting out

\hat{u}

, with a random unit direction as the control. Align : share of

k{=}8

random image tokens before and after setting their keys to

k_{t^{\star}}

, with a random key of the same norm as the control. Median target : the

\hat{u}

-component of the edited keys, in the Remove edit and in the Align random-key control, is moved to the population median instead of zero.

Model	text before image	image	top-1, absolute	top-1, share of image
Gemma-4-E2B	$.00\to.38$	$.55\to.19$	$.21\to.07$	$.37\to.35$
Gemma-4-E4B	$.00\to.37$	$.51\to.16$	$.16\to.03$	$.30\to.18$
Gemma-4-12B	$.00\to.49$	$.60\to.14$	$.10\to.03$	$.17\to.19$

Appendix

Table 8: Gemma-4 sliding-window layers at 1,120 versus 280 soft tokens on 100 V ∗ images, averaged over heads, layers and images. Each cell reads released budget

\to

reduced budget. The text before image and image columns are shares of the question’s attention. The strongest image token’s share of the question’s attention is top-1, absolute , and its share of the in-image attention is top-1, share of image .

Figure 6: (a) Sink share vs. target attention per head at the peak layer on V ∗ . Marker size is the head’s share of image attention. (b) Precision and recall of the centered-cosine gate at threshold

\tau_{c}

against the attention-defined sinks, one curve per sink-bearing model on V ∗ . Shaded band: thresholds that give the same edit. The paper uses 0.5.

Model	attention	readout: sinks	readout: $t^{\star}$	cos
Qwen3-VL-8B	.70	.11	.66	.994
Qwen3-VL-4B	.75	.13	.52	.991
InternVL3.5-8B	.29	.03	.38	.999
Pixtral-12B	.77	.16	.49	.983
Gemma-4-12B	.62	1.20	.00	.67
Gemma-4-E4B	.65	1.00	.00	.41

Appendix

Table 9: Visual-sink contributions to the attention output at the peak layer on V ∗ . The attention column is the sinks’ share of the head’s in-image attention at the peak layer, averaged over heads. It therefore differs slightly from the share in Table 5 , which is read from the head-averaged map. The readout columns divide the contribution norm by the full output norm, for the visual sinks and for the text sink

t^{\star}

. A readout above 1 means the other contributions partly cancel the sinks’. The cos column is the cosine between the output with and without the sink contribution. The three sink-free models are below the rule, with their tokens above

100\times

uniform in place of the sinks.

Model	raw	masked $>100\times$	masked $>10\times$	key edit	in box
Qwen3-VL-8B	.00	.20	.10	.45	5%
Qwen3-VL-4B	.00	.17	.10	.39	3%
InternVL3.5-8B	.00	.18	.28	.66	2%
Pixtral-12B	.00	.09	.03	.14	0%
Qwen3-VL-2B	.22	.18	.05	.17	25%
InternVL3.5-2B	.57	.47	.05	.56	86%

Appendix

Table 10: Share of V ∗ images whose most-attended image token lies in the target box. The columns give the raw map, the map after masking (every token above

100\times

or above

10\times

uniform zeroed) and the map after the key edit. In box is the share of the

>100\times

tokens inside the target. Numbers average the upper-half layers on 100 images, so the key-edit column is not the value of Table 2 . The Gemma-4 rows average all upper-half layers, sliding-window layers included.

Figure 7: Per-position KL along the base model’s chain-of-thought trace with the key edit applied at every step (solid) against a kernel-only control (dotted). InternVL3.5-8B and Pixtral-12B on 60 V ∗ questions, greedy chains of up to 192 tokens.

Model	$n$	$t^{\star}$ share	$\times$ unif.	per image	share	$n$	$t^{\star}$ share	$\times$ unif.	per image	share
	HR-4K					HR-8K
Qwen3-VL-8B	12194	.72	4626	6.1	.69	12416	.71	4659	6.1	.71
Qwen3-VL-4B	12194	.67	3468	7.6	.76	12416	.67	3675	8.1	.78
Qwen3-VL-2B	12194	.74	1096	5.5	.17	12416	.75	1366	6.4	.20
InternVL3.5-8B	2555	.64	652	2.5	.41	1987	.65	314	1.8	.28
InternVL3.5-2B	2555	.76	143	1.6	.10	1987	.77	110	0.9	.07

Appendix

Table 11: Text and visual sinks on the first 50 distinct images of HR-4K and of HR-8K. Columns as in Table 5 without the LM, position, layer and target columns. The visual-sink columns are read at the peak layer.

n

is the number of image tokens per prompt.

	HR-4K					HR-8K
Model	$\max_{j}r_{j}$	text sink	accuracy	agreement	KL	$\max_{j}r_{j}$	text sink	accuracy	agreement	KL
Qwen3-VL-8B	$.25\to.02$	$.54\to.64$	$.72\to.72$	.99	.001	$.24\to.02$	$.54\to.65$	$.68\to.67$	.99	.001
Qwen3-VL-4B	$.21\to.03$	$.51\to.65$	$.74\to.74$	.97	.000	$.22\to.03$	$.51\to.65$	$.69\to.69$	.99	.000
Qwen3-VL-2B	$.02\to.02$	$.60\to.59$	$.71\to.57$	.74	.017	$.02\to.02$	$.61\to.59$	$.72\to.59$	.78	.024
InternVL3.5-8B	$.31\to.04$	$.42\to.63$	$.73\to.72$	.98	.000	$.33\to.04$	$.44\to.64$	$.69\to.70$	.99	.000
InternVL3.5-2B	$.04\to.04$	$.62\to.61$	$.76\to.76$	.87	.002	$.04\to.04$	$.63\to.63$	$.62\to.63$	.90	.008

Appendix

Table 12: The key edit on the first 50 distinct images of HR-4K and of HR-8K. Columns as in Table 2 without the box and caption columns. Gate off means the gate admits under one token per image, so the edit is an identity.

Model	layers / total
Qwen3-VL-8B	18, 21 / 36
Qwen3-VL-4B	18, 21 / 36
InternVL3.5-8B	18, 21 / 36
Qwen3-VL-2B	14, 16 / 28
InternVL3.5-2B	14, 16 / 28
InternVL3-8B	14, 16 / 28

Appendix

Table 13: Selector layers per model. The selector averages the head-averaged question-to-image attention of these two layers (§ 4.1 ). The number after the slash is the count of language-model layers. The other full-attention models read the cleaned map at the same relative depths as Qwen3-VL-8B. The Gemma-4 rows list full-attention layers, which the selector reads on the raw map.

Figure 8: Per-layer top-1 localization on V ∗ for the raw map (dashed) and the cleaned map (solid). Here top-1 localization is the fraction of images whose most-attended image token lies in the target box. The shaded area between the curves is the gain from the key edit. Dotted lines mark the selector layers. The gray band is the first quarter of depth, which the peak-layer rule skips. Gemma-4 shows full-attention layers only (triangles).

Qwen3-VL-Thinking
Model	V ∗	HR-4K	HR-8K	MME-RW-Lite
8B	59.7	60.1	73.1	90.9
4B	65.4	65.4	74.9	94.2
2B	72.3	71.8	78.2	92.5
InternVL3.5 (thinking)
8B	79.6	79.9	83.9	91.0

Appendix

Table 14: Fraction of questions (%) on which the re-look is triggered, per model and benchmark. V ∗ has 191 questions, HR-4K and HR-8K 800 each, MME-RealWorld-Lite 1,919. Same runs as Table 4 .

Qwen3-VL-Thinking
	V ∗		HR-4K		HR-8K		MME-RW-Lite		V ∗ crop hit
Model	vs. base	vs. raw	vs. base	vs. raw	vs. base	vs. raw	vs. base	vs. raw	cleaned	raw
8B	9/2	7/4	51/18 †	40/13 †	65/30 †	41/22 †	114/75 †	83/64	.90	.22
4B	11/6	13/4 †	30/23	32/21	62/21 †	59/30 †	144/79 †	118/68 †	.89	.08
2B	14/11	4/7	58/34 †	12/12	71/46 †	16/24	159/120 †	38/40	.76	.82
InternVL3.5 (thinking)

Appendix

Table 15: Paired comparisons behind Table 4 . For every model and benchmark, wins/losses counts the questions the re-look with the cleaned map answers correctly and the reference answers wrongly, against the reverse. The reference is the base run ( base ) or the same re-look with the crop selected on the raw map ( raw ). † marks

p<.05

under the exact McNemar test. The last two columns give the V ∗ crop-hit rate on triggered questions for the cleaned and for the raw map. A crop hits when it contains the center of a target box. Gemma-4’s selector reads the raw map, so its one hit rate is listed under cleaned and it has no raw columns. V ∗ has 191 questions, HR-4K and HR-8K 800 each, MME-RealWorld-Lite 1,919. Same runs as Table 4 .

Attention Intervention in VLMs

See all 15 papers

When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models
Apr 1, 2026Jiho Choi, Jaemin Kim, Sanghwan Kim +2Large Vision-Language Models VLM Interpretability
KAIST, South Korea · Chung-Ang University, South Korea · Technical University of Munich, Germany
Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosis
Sep 1, 2026Minsik Choi, Geewook Kim, Young Geun KimVision-Language Models VLM Adaptation
Korea University · NAVER Cloud AI
SAGE: Sink-Aware Guided Emphasis for Visual Grounding in Vision-Language Decoders
Oct 8, 2026Jeonghyo Song, YoungJoon YooAttention Intervention in VLMs Vision-Language Grounding
Department of Artificial Intelligence, Chung-Ang University

Apr 1, 2026J/K move · Enter open · S save

When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models

Jiho Choi, Jaemin Kim, Sanghwan Kim, Seunghoon Hong+1

KAIST, South Korea · Chung-Ang University, South Korea · Technical University of Munich, Germany

Large Vision-Language Models VLM Interpretability Attention Sinks Attention Mechanisms

Open paper AskPDF arXiv

Attention sinks are defined as tokens that attract disproportionate attention. While these have been studied in single modality transformers, their cross-modal impact in Large Vision-Language Models (LVLM) remains largely unexplored: are they redundant artifacts or essential global priors? This paper first categorizes visual sinks into two distinct categories: ViT-emerged sinks (V-sinks), which propagate from the vision encoder, and LLM-emerged sinks (L-sinks), which arise within deep LLM layers. Based on the new definition, our analysis reveals a fundamental performance trade-off: while sinks effectively encode global scene-level priors, their dominance can suppress the fine-grained visual evidence required for local perception. Furthermore, we identify specific functional layers where modulating these sinks most significantly impacts downstream performance. To leverage these insights, we propose Layer-wise Sink Gating (LSG), a lightweight, plug-and-play module that dynamically scales the attention contributions of V-sink and the rest visual tokens. LSG is trained via standard next-token prediction, requiring no task-specific supervision while keeping the LVLM backbone frozen. In most layers, LSG yields improvements on representative multimodal benchmarks, effectively balancing global reasoning and precise local evidence.

Abstract

Attention Intervention in VLMs

Explore similar work

When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models

Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosis

SAGE: Sink-Aware Guided Emphasis for Visual Grounding in Vision-Language Decoders