Attention Sinks in VLMs Are Borrowed, Not Seen

Dengjia Zhang; Alexander Martin; Pengyu Yan; Xihan Sun; Jiaqi Li; Benjamin Van Durme; Andrew Yates; Reno Kriz

cs.AIOct 11, 2026

Attention Sinks in VLMs Are Borrowed, Not Seen

Authors: Dengjia Zhang, Alexander Martin, Pengyu Yan, Xihan Sun, Jiaqi Li, Benjamin Van Durme, Andrew Yates, Reno Kriz,

Organizations: Johns Hopkins University · University of Pittsburgh · University at Buffalo · Vanderbilt University ·

Abstract

We study how to detect and mitigate visual attention sinks in vision-language models (VLMs). Visual attention sinks are image tokens that absorb a large share of attention over the image regardless of the query or task, and they mislead methods that treat attention maps as a record of where the model ``looked.'' Prior work has detected sinks through massive activation dimensions and proposed redistributing their attention, but none have addressed what makes a token become a sink. Comparing six candidate detectors on ten VLMs from four families, we find that only the centered cosine similarity between an image token's key and the key of the language model's text-sink detects visual sinks. Thus, a visual sink is an image token whose key aligns with the text-sink's key. Moving a sink's key component along this direction to the median of the image keys sends its attention back to the text-sink (Are Borrowed) while leaving model answers unchanged~(Not Seen). Building on this, we propose a training-free, entropy-triggered re-look that first cleans the attention map, then crops the most-attended region of the cleaned map and inserts it into the reasoning trace. Cleaning adds negligible computation to the forward pass, and the re-look improves accuracy with a mean gain of $+4.3$ points on fine-detail VQA benchmarks across model families.

Figures & tables

Figure 1: Visual attention sinks point the same way as the text-sink. Left: visual attention sinks (red, $\geq 100\times$ uniform attention outside the target) and the target (green) in a V ∗ image. Right: key direction (angle) vs. attention (radius) of every image token

Figure 2: Image tokens on V ∗ at each model’s peak layer. Angle: centered cosine of a token’s key with the text-sink key ( Equation 2 ). Radius: attention received relative to uniform, log scale. Sinks: tokens above $100\times$ uniform (dotted ring) outside the target box. Dashed ray: gate threshold $.5$ . Red numbers: median cosine of the sinks; ∗ : sink-free

Model	LM	per image	$\times$ uniform	share
QVL-8B	Qwen3	3.9	1259	.66
QVL-2B	Qwen3	2.7	163	.12
IVL3.5-8B	Qwen3	1.5	342	.24
IVL3.5-2B	Qwen3	1.0	151	.08
IVL3-8B	Qwen2.5	0.4	93	.02
Pix-12B	Mistral	1.0	2093	.76

Table 1: Visual attention sinks in VLMs on V ∗ at each model’s peak layer. key1: sinks per image; key 2: their attention relative to uniform; key 3: their share of the in-image attention.

Figure 3: Crop selection in Qwen3-VL-8B-Thinking on two V ∗ images. Green: target box; red/blue: raw/edited window. Right: question attention over the whole prompt (one cell per $1\%$ ). The edit moves the visual attention sinks’ share to the text-sink.

	Map			Model
Model	$\max_{j}r_{j}$	top-1 $\in\mathcal{B}$	text-sink	accuracy	agreement	KL	caption
Qwen3-VL-8B	$.25\to.04$	$.00\to.47$	$.57\to.67$	$.73\to.73$	.99	.004	.83
Qwen3-VL-4B	$.24\to.05$	$.00\to.41$	$.54\to.68$	$.73\to.73$	.98	.003	.82
Qwen3-VL-2B	$.04\to.03$	$.28\to.23$	$.63\to.63$	$.68\to.61$	.81	.039	.57
InternVL3.5-8B	$.34\to.05$	$.00\to.72$	$.44\to.65$	$.71\to.68$	.94	.001	.85
InternVL3.5-2B	$.04\to.04$	$.60\to.59$	$.65\to.65$	$.64\to.61$	.93	.002	.80

Table 2: Effects of the key edit on V ∗ .

L

: number of language-model layers. text-sink: share of question attention on the text-sink, averaged over the probed layers

Feature	mean	min
hidden norm	80.1	25.0
$-\\|v\\|$	72.0	8.0
$-\\|k\\|$	77.4	49.0
$-$ logit std	78.4	65.0
raw key cos	99.0	94.0
centered key cos	99.9	99.0

Table 3: AUC of each feature at separating visual attention sinks from other image tokens, at the peak layer on V ∗ . Mean and min over sink-bearing models.

Figure 4: Overview of the entropy-triggered re-look. (1) The key edit removes the visual attention sinks from the prefill attention map. (2) The selector keeps the $\frac{1}{3}$ -grid window with the most attention on the cleaned map. (3) When the next-token entropy first exceeds $H_{0}$ after $c$ tokens of thought, the crop of that window is inserted into the chain of thought.

V ∗

HR-4K

HR-8K

MME-RW-Lite

Model

Base

Raw

\Delta

Clean

\Delta

Base

Raw

\Delta

Clean

\Delta

Base

Raw

\Delta

Clean

\Delta

Base

Raw

\Delta

Clean

\Delta

Qwen3-VL-Thinking

79.1

+2.1

+3.7

72.6

+0.8

+4.1

†

68.9

+2.0

+4.4

†

46.5

+1.0

+2.0

†

82.2

-2.1

+2.6

74.1

-0.5

+0.9

67.4

+1.5

+5.1

†

45.1

+0.8

+3.4

†

70.2

+3.1

+1.6

69.5

+3.0

+3.0

†

63.9

+4.1

+3.1

†

47.2

+2.1

+2.0

†

InternVL3.5 (thinking)

Table 4: Base accuracy (%) and paired re-look gain. Clean

\Delta

: crop selected on the sink-cleaned map. Raw

\Delta

: on the raw attention map. Shading scales with gain (green) or loss (red); † :

p<.05

(exact McNemar, clean)

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Model	LM	position	share	$\times$ uniform	per image	share	layer	at peak	best layer
		Text sink		Visual sink (peak layer)				Target top-1 $\in\mathcal{B}$
Qwen3-VL-8B	Qwen3	0	.74	1259	3.9	.66	30/36	.00	.80
Qwen3-VL-4B	Qwen3	0	.69	1049	4.9	.73	29/36	.00	.82
Qwen3-VL-2B	Qwen3	0	.75	163	2.7	.12	27/28	.00	.75
InternVL3.5-8B	Qwen3	0	.65	342	1.5	.24	30/36	.14	.62
InternVL3.5-2B	Qwen3	0	.77	151	1.0	.08	7/28	.00	.69

Appendix

Table 5: Text and visual sinks in ten models on V ∗ , extending Table 1 . LM is the language-model backbone. Text sink: the token index of the text sink and its share of the question’s attention. Visual sink (peak layer): attention relative to uniform, count per image, share of the in-image attention and the peak layer out of the model’s layer count. Target top-1

\in\mathcal{B}

: the fraction of images whose most-attended image token lies in the target box, at the peak layer and at the best layer.

Figure 5: Sink attention across depth on V ∗ for all ten models. Red: visual attention sink share of in-image attention. Gray dashed: text-sink share of question attention. Dotted: peak layer. Gemma-4 reaches the text sink only in its full-attention layers (triangles). Title color marks the language-model backbone (Qwen3, Gemma-4, Qwen2.5 and Mistral).

Model	centered key cos	raw key cos	$-\\|v\\|$	hidden norm	$\\|k\\|$	$-\\|k\\|$	$-$ logit std
Qwen3-VL-8B	.99/.98/1.00	.99/.96/1.00	.99/.94/1.00	.99/.93/1.00	.00/.00/.01	.23/.00/.99	.00/.00/.69
Qwen3-VL-4B	.99/.95/1.00	.99/.93/1.00	.99/.68/1.00	.98/.80/1.00	.00/.00/.28	.01/.00/.72	.00/.00/.71
InternVL3.5-8B	.95/.70/1.00	.95/.66/1.00	.95/.73/1.00	.95/.61/.99	.00/.00/.01	.74/.28/.99	.00/.00/.90
Gemma-4-12B	.61/.00/1.00	.61/.00/1.00	.00/.00/.40	.00/.00/.93	.01/.00/.33	.18/.21/.68	.02/.00/.85
Gemma-4-E4B	.18/.00/1.00	.04/.00/.99	.00/.00/.56	.00/.00/.44	.00/.00/.50	.00/.00/.49	.00/.00/.65
Gemma-4-E2B	.07/.00/.99	.00/.00/.94	.00/.00/.08	.00/.00/.25	.06/.00/.46	.15/.12/.55	.00/.00/.69

Appendix

Table 6: Token features as detectors of visual sinks at each model’s peak layer on V ∗ . Cells: precision at

k

/ top-1 / AUC, where

k

is the number of sinks in the image. The three sink-free models are below the rule. There the few tokens above

100\times

uniform (0.4 to 2.7 per image) stand in for the sinks. The first two columns are the centered cosine between the token’s key and the text-sink key ( Equation 2 ) and the same cosine without centering. The next five are the negated value norm, the hidden-state norm, the key norm with both signs and the negated standard deviation of the token’s attention logits. Apart from the key norm, every feature is signed so that a larger value is more sink-like. Hidden norm is the feature of Kang et al. (2025a) .

		Remove			Align
Model	Layer	before	$-\hat{u}$	$-$ random	before	$:=k_{t^{\star}}$	random key
Qwen3-VL-8B	30	.68	.00	.66	.00	.97	.04
Qwen3-VL-4B	29	.75	.00	.74	.00	.95	.00
InternVL3.5-8B	30	.36	.00	.35	.00	.96	.09
Pixtral-12B	27	.49	.25	.49	.00	.91	.64
median target	27	.49	.00	.49	.00	.91	.01

Appendix

Table 7: Key interventions at the peak layer on 20 V ∗ images. Remove : in-image share of the gated tokens before and after projecting out

\hat{u}

, with a random unit direction as the control. Align : share of

k{=}8

random image tokens before and after setting their keys to

k_{t^{\star}}

, with a random key of the same norm as the control. Median target : the

\hat{u}

-component of the edited keys, in the Remove edit and in the Align random-key control, is moved to the population median instead of zero.

Model	text before image	image	top-1, absolute	top-1, share of image
Gemma-4-E2B	$.00\to.38$	$.55\to.19$	$.21\to.07$	$.37\to.35$
Gemma-4-E4B	$.00\to.37$	$.51\to.16$	$.16\to.03$	$.30\to.18$
Gemma-4-12B	$.00\to.49$	$.60\to.14$	$.10\to.03$	$.17\to.19$

Appendix

Table 8: Gemma-4 sliding-window layers at 1,120 versus 280 soft tokens on 100 V ∗ images, averaged over heads, layers and images. Each cell reads released budget

\to

reduced budget. The text before image and image columns are shares of the question’s attention. The strongest image token’s share of the question’s attention is top-1, absolute , and its share of the in-image attention is top-1, share of image .

Figure 6: (a) Sink share vs. target attention per head at the peak layer on V ∗ . Marker size is the head’s share of image attention. (b) Precision and recall of the centered-cosine gate at threshold

\tau_{c}

against the attention-defined sinks, one curve per sink-bearing model on V ∗ . Shaded band: thresholds that give the same edit. The paper uses 0.5.

Model	attention	readout: sinks	readout: $t^{\star}$	cos
Qwen3-VL-8B	.70	.11	.66	.994
Qwen3-VL-4B	.75	.13	.52	.991
InternVL3.5-8B	.29	.03	.38	.999
Pixtral-12B	.77	.16	.49	.983
Gemma-4-12B	.62	1.20	.00	.67
Gemma-4-E4B	.65	1.00	.00	.41

Appendix

Table 9: Visual-sink contributions to the attention output at the peak layer on V ∗ . The attention column is the sinks’ share of the head’s in-image attention at the peak layer, averaged over heads. It therefore differs slightly from the share in Table 5 , which is read from the head-averaged map. The readout columns divide the contribution norm by the full output norm, for the visual sinks and for the text sink

t^{\star}

. A readout above 1 means the other contributions partly cancel the sinks’. The cos column is the cosine between the output with and without the sink contribution. The three sink-free models are below the rule, with their tokens above

100\times

uniform in place of the sinks.

Model	raw	masked $>100\times$	masked $>10\times$	key edit	in box
Qwen3-VL-8B	.00	.20	.10	.45	5%
Qwen3-VL-4B	.00	.17	.10	.39	3%
InternVL3.5-8B	.00	.18	.28	.66	2%
Pixtral-12B	.00	.09	.03	.14	0%
Qwen3-VL-2B	.22	.18	.05	.17	25%
InternVL3.5-2B	.57	.47	.05	.56	86%

Appendix

Table 10: Share of V ∗ images whose most-attended image token lies in the target box. The columns give the raw map, the map after masking (every token above

100\times

or above

10\times

uniform zeroed) and the map after the key edit. In box is the share of the

>100\times

tokens inside the target. Numbers average the upper-half layers on 100 images, so the key-edit column is not the value of Table 2 . The Gemma-4 rows average all upper-half layers, sliding-window layers included.

Figure 7: Per-position KL along the base model’s chain-of-thought trace with the key edit applied at every step (solid) against a kernel-only control (dotted). InternVL3.5-8B and Pixtral-12B on 60 V ∗ questions, greedy chains of up to 192 tokens.

Model	$n$	$t^{\star}$ share	$\times$ unif.	per image	share	$n$	$t^{\star}$ share	$\times$ unif.	per image	share
	HR-4K					HR-8K
Qwen3-VL-8B	12194	.72	4626	6.1	.69	12416	.71	4659	6.1	.71
Qwen3-VL-4B	12194	.67	3468	7.6	.76	12416	.67	3675	8.1	.78
Qwen3-VL-2B	12194	.74	1096	5.5	.17	12416	.75	1366	6.4	.20
InternVL3.5-8B	2555	.64	652	2.5	.41	1987	.65	314	1.8	.28
InternVL3.5-2B	2555	.76	143	1.6	.10	1987	.77	110	0.9	.07

Appendix

Table 11: Text and visual sinks on the first 50 distinct images of HR-4K and of HR-8K. Columns as in Table 5 without the LM, position, layer and target columns. The visual-sink columns are read at the peak layer.

n

is the number of image tokens per prompt.

	HR-4K					HR-8K
Model	$\max_{j}r_{j}$	text sink	accuracy	agreement	KL	$\max_{j}r_{j}$	text sink	accuracy	agreement	KL
Qwen3-VL-8B	$.25\to.02$	$.54\to.64$	$.72\to.72$	.99	.001	$.24\to.02$	$.54\to.65$	$.68\to.67$	.99	.001
Qwen3-VL-4B	$.21\to.03$	$.51\to.65$	$.74\to.74$	.97	.000	$.22\to.03$	$.51\to.65$	$.69\to.69$	.99	.000
Qwen3-VL-2B	$.02\to.02$	$.60\to.59$	$.71\to.57$	.74	.017	$.02\to.02$	$.61\to.59$	$.72\to.59$	.78	.024
InternVL3.5-8B	$.31\to.04$	$.42\to.63$	$.73\to.72$	.98	.000	$.33\to.04$	$.44\to.64$	$.69\to.70$	.99	.000
InternVL3.5-2B	$.04\to.04$	$.62\to.61$	$.76\to.76$	.87	.002	$.04\to.04$	$.63\to.63$	$.62\to.63$	.90	.008

Appendix

Table 12: The key edit on the first 50 distinct images of HR-4K and of HR-8K. Columns as in Table 2 without the box and caption columns. Gate off means the gate admits under one token per image, so the edit is an identity.

Model	layers / total
Qwen3-VL-8B	18, 21 / 36
Qwen3-VL-4B	18, 21 / 36
InternVL3.5-8B	18, 21 / 36
Qwen3-VL-2B	14, 16 / 28
InternVL3.5-2B	14, 16 / 28
InternVL3-8B	14, 16 / 28

Appendix

Table 13: Selector layers per model. The selector averages the head-averaged question-to-image attention of these two layers (§ 4.1 ). The number after the slash is the count of language-model layers. The other full-attention models read the cleaned map at the same relative depths as Qwen3-VL-8B. The Gemma-4 rows list full-attention layers, which the selector reads on the raw map.

Figure 8: Per-layer top-1 localization on V ∗ for the raw map (dashed) and the cleaned map (solid). Here top-1 localization is the fraction of images whose most-attended image token lies in the target box. The shaded area between the curves is the gain from the key edit. Dotted lines mark the selector layers. The gray band is the first quarter of depth, which the peak-layer rule skips. Gemma-4 shows full-attention layers only (triangles).

Qwen3-VL-Thinking
Model	V ∗	HR-4K	HR-8K	MME-RW-Lite
8B	59.7	60.1	73.1	90.9
4B	65.4	65.4	74.9	94.2
2B	72.3	71.8	78.2	92.5
InternVL3.5 (thinking)
8B	79.6	79.9	83.9	91.0

Appendix

Table 14: Fraction of questions (%) on which the re-look is triggered, per model and benchmark. V ∗ has 191 questions, HR-4K and HR-8K 800 each, MME-RealWorld-Lite 1,919. Same runs as Table 4 .

Qwen3-VL-Thinking
	V ∗		HR-4K		HR-8K		MME-RW-Lite		V ∗ crop hit
Model	vs. base	vs. raw	vs. base	vs. raw	vs. base	vs. raw	vs. base	vs. raw	cleaned	raw
8B	9/2	7/4	51/18 †	40/13 †	65/30 †	41/22 †	114/75 †	83/64	.90	.22
4B	11/6	13/4 †	30/23	32/21	62/21 †	59/30 †	144/79 †	118/68 †	.89	.08
2B	14/11	4/7	58/34 †	12/12	71/46 †	16/24	159/120 †	38/40	.76	.82
InternVL3.5 (thinking)

Appendix

Table 15: Paired comparisons behind Table 4 . For every model and benchmark, wins/losses counts the questions the re-look with the cleaned map answers correctly and the reference answers wrongly, against the reverse. The reference is the base run ( base ) or the same re-look with the crop selected on the raw map ( raw ). † marks

p<.05

under the exact McNemar test. The last two columns give the V ∗ crop-hit rate on triggered questions for the cleaned and for the raw map. A crop hits when it contains the center of a target box. Gemma-4’s selector reads the raw map, so its one hit rate is listed under cleaned and it has no raw columns. V ∗ has 191 questions, HR-4K and HR-8K 800 each, MME-RealWorld-Lite 1,919. Same runs as Table 4 .

Attention Intervention in VLMs

See all 16 papers

Apr 1, 2026

When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models

Attention sinks are defined as tokens that attract disproportionate attention. While these have been studied in single modality transformers, their cross-modal impact in Large Vision-Language Models (LVLM) remains largely unexplored: are they redundant artifacts or essential global priors? This paper first categorizes visual sinks into two distinct categories: ViT-emerged sinks (V-sinks), which propagate from the vision encoder, and LLM-emerged sinks (L-sinks), which arise within deep LLM layers. Based on the new definition, our analysis reveals a fundamental performance trade-off: while sinks effectively encode global scene-level priors, their dominance can suppress the fine-grained visual evidence required for local perception. Furthermore, we identify specific functional layers where modulating these sinks most significantly impacts downstream performance. To leverage these insights, we propose Layer-wise Sink Gating (LSG), a lightweight, plug-and-play module that dynamically scales the attention contributions of V-sink and the rest visual tokens. LSG is trained via standard next-token prediction, requiring no task-specific supervision while keeping the LVLM backbone frozen. In most layers, LSG yields improvements on representative multimodal benchmarks, effectively balancing global reasoning and precise local evidence.

Jiho Choi, Jaemin Kim, Sanghwan Kim +2

KAIST, South Korea · Chung-Ang University, South Korea · Technical University of Munich, Germany

Large Vision-Language Models VLM Interpretability Attention Sinks Attention Mechanisms

Ask PDF

Sep 1, 2026

Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosis

Fine-tuning a pretrained LLM into a vision-language model (VLM) can erode the backbone's text capability, with the damage concentrated on tasks that require following exact output rules, such as instruction following, chain-of-thought reasoning graded on a strictly parsed final answer, and similar evaluations with strict graders. We trace this gap to attention-sink corruption: VL fine-tuning perturbs the early sink position that anchors a large fraction of attention probability, and how well the base LLM preserves its sink tracks how much of the affected capability survives adaptation. Building on this view, we introduce Sink Strength, a single scalar computed on the base LLM in a few seconds on a single GPU that predicts post-VL degradation without any VL training. It consistently tracks relative degradation across the six VLM-LLM pairs and multiple format-sensitive tasks. Complementing this diagnostic, we find that post-pretraining QK-RMSNorm injection fails to reproduce the protection of native QK-RMSNorm, while several off-the-shelf weight-merging settings fail to recover the lost capability after VL training. These negative results underscore the value of screening backbones with Sink Strength before VL training and narrow the intervention space toward head-selective training-time protection.

Minsik Choi, Geewook Kim, Young Geun Kim

Korea University · NAVER Cloud AI

Vision-Language Models VLM Adaptation VLM Reasoning Attention Sinks

Ask PDF

Oct 8, 2026

SAGE: Sink-Aware Guided Emphasis for Visual Grounding in Vision-Language Decoders

Recent large vision-language models (VLMs) pair a visual encoder with a large language model (LLM) and perform well on diverse image-text tasks, yet their reliability is often limited by decoder attention pathologies that suppress visual evidence and exacerbate hallucinations. In this paper, we revisit visual attention sinks and uncover a structured, layer-dependent behavior: across prompts, early and late decoder layers exhibit prompt-invariant attention collapse onto the same few image regions, which we term PIS (Prompt-Invariant Sinks), whereas mid layers become prompt-conditioned and drive vision-language alignment. This split suggests that treating sinks as a uniform effect is incomplete. Building on this insight, we propose SAGE (Sink-Aware Guided Emphasis), a lightweight intervention that steers decoder attention away from PIS and toward query-dependent regions of interest (ROIs) using token-aligned ROI masks derived from standard vision backbones such as CLIP, ViT, and DINOv3. Evaluated on diverse vision-encoder + decoder-only LLM VLM families, SAGE improves visual grounding, reduces hallucinations, and yields consistent gains across public downstream vision-language benchmarks, including fine-grained visual discrimination settings where localized evidence is crucial, when instantiated with backbone-derived ROI masks.

Jeonghyo Song, YoungJoon Yoo

Department of Artificial Intelligence, Chung-Ang University

Attention Intervention in VLMs Vision-Language Grounding Large Vision-Language Models VLM Hallucination Mitigation

Ask PDF

Abstract

Attention Intervention in VLMs

Explore similar work

When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models

Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosis

SAGE: Sink-Aware Guided Emphasis for Visual Grounding in Vision-Language Decoders