On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMs

Suho Yoo; Youngjoon Jang; Joon Son Chung

cs.CVMar 15, 2026

On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMs

Authors: Suho Yoo, Youngjoon Jang, Joon Son Chung

Organizations: KAIST · VGG, University of Oxford

Abstract

The goal of this paper is to strengthen the reasoning of Omnimodal Large Language Models (Omni-LLMs) at inference time, without additional training. These models jointly process video, audio, and text, and given the large number of tokens they consume, how attention is routed across them is central to their behaviour. We focus specifically on attention sinks, tokens that absorb a disproportionate share of attention mass regardless of their semantic content, to understand how this routing unfolds. To this end, we conduct a systematic analysis of sink behaviour in Omni-LLMs. Our analysis yields two key findings: (i) high sink attention does not solely indicate head redundancy, suggesting that sink value representations play additional functional roles; (ii) the sink value vector acts as a shared bias added to every token's output, serving as a global signal that organises the representation as a whole. Building on this, we propose OutRo, which correspondingly aligns non-sink token representations with the sink in feature space, and relaxes the causal mask for sink tokens at an early layer to sharpen this bias before the rest of decoding proceeds. This design enhances the reasoning process without requiring additional forward passes or access to attention maps. Based on extensive experiments, OutRo consistently improves performance on seven video QA benchmarks and demonstrates strong generalisation, while incurring only a 1.1x decoding overhead.

Figures & tables

Panel of Figure 1 — Figure 1: $\Phi_{\text{VLM}}$ incorrectly identifies semantic tokens as sinks. (a) $\Phi_{\text{VLM}}$ rapidly over-identifies sink tokens in deeper layers, unlike the sparse behaviour of $\Phi_{\text{LLM}}$ . (b) For the query “Is the helicopter visible?”, highly attended object tokens are still identified as sinks under $\Phi_{\text{VLM}}$ .

	Dims
Sink	458, 2570, …
Outlier	458 , 2570 , 3206, 3281, 1427, …

Table 1: Sink/outlier dimensions. Sink dimensions strongly overlap with outlier feature patterns.

Panel of Figure 2 — Figure 2: High sink attention does not imply head redundancy. (a) Performance change after removing individual heads, distinguishing improvement and decrease heads. (b) High sink scores appear in both improvement and decrease heads, indicating sink-heavy heads are not always redundant.

Head	Sink score	$A\to V$	$V\to A$	Overall
L15H3	0.7553	+ 0.88%	+ 0.48%	+ 1.36%
L18H12	0.6633	+ 0.26%	+ 0.87%	+ 1.14%
L11H6	0.7003	+ 0.62%	+ 0.09%	+ 0.70%
L24H10	0.7755	-3.43%	-0.48%	-3.91%
L16H14	0.8773	-2.55%	-0.13%	-2.68%
L14H12	0.7497	-1.32%	-0.44%	-1.76%

Table 2: Head ablation performance. Heads with similar sink scores can yield opposite performance changes.

Panel of Figure 3 — Figure 3: Geometric analysis of sink token representations. (a) Sink keys and values have substantially smaller norms than other tokens, yet (b) their activations are not uniformly small across dimensions. (c) In t-SNE, sink keys cluster near the query distribution, and (d) their cosine similarity with queries remains high, yielding large attention scores.

Figure 4: Sink token attention score. Compared to the baseline, zeroing the top-1 dimension in the sink key representations substantially reduces attention to sink tokens across layers.

Figure 5: Zeroing the top-1 sink key dimension (Zero-K). Attention maps before (Baseline) and after (Zero-K Top-1) the intervention show that suppressing the dominant sink key dimension effectively redirects attention away from sink positions, as confirmed by the difference map.

Panel of Figure 6 — Figure 6: Sink token head outputs before and after injection. The outputs remain nearly identical, with only minimal differences observed at the last layer.

Table 9

Figure 7: Attention to sink tokens before and after rotation. Rotation consistently reduces the total attention allocated to sink tokens.

Figure 8: Overall OutRo pipeline. (a) Gated head output rotation. Non-sink outputs are rotated toward the sink value direction via adaptive gating. (b) Sink enhancement. The causal mask is removed for sink positions, sharpening the shared bias direction.

Models	Audio-Visual QA				Visual-Only QA
Models	OmniBench	AVUT	AVHBench	DailyOmni	Holmes	MME	ActivityNet
Qwen2.5-Omni 3B	41.33	61.79	70.91	55.38	43.54	58.00	41.82
+ OutRo	42.03	62.87	71.67	55.56	43.54	58.11	41.86
Qwen2.5-Omni 7B	47.37	65.79	71.60	54.14	47.46	62.44	43.59
+ OutRo	48.25	66.57	73.78	54.64	47.63	62.96	44.01
video-SALMONN2 + 3B	36.69	56.73	62.07	51.71	42.68	73.67	44.31

Table 6: Performance comparison of OutRo . OutRo consistently improves performance across models and benchmarks, demonstrating its generality.

Table 13

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

		Outlier			Sink $\Phi_{\text{VLM}}$			Sink $\Phi_{\text{LLM}}$
Dim	Activation	Layer(%)	Token(%)	Outlier	$\Phi_{\text{VLM}}$	Token(%)	Sink	$\Phi_{\text{LLM}}$	Index	Sink
458	8768.00	100.0	86.89	✓	46.00	59.83	✓	19072	2	✓
2570	6272.00	100.0	89.80	✓	45.75	44.00	✓	13376	2	✓
2718	2656.00	92.9	2.68		46.00	59.83	✓	6208	2	✓
2730	2128.00	100.0	2.75		46.00	59.83	✓	4960	2	✓
3206	716.00	100.0	57.34	✓	46.00	72.68	✓	1640	2	✓

Appendix

Table A.1 : Dimensions identified as outliers or sinks. Under the

\Phi_{\text{VLM}}

criterion, sink dimensions ( i.e., 458 and 2570) cause many tokens to be classified as sinks.

Method	$A\to V$	$V\to A$	Matching	Overall
Baseline	82.31	82.58	51.71	71.60
All tokens	0.00	0.00	0.00	0.00
Sink token ( $\Phi_{\text{LLM}}$ )	0.00	0.00	0.00	0.00
Random token ( $\Phi_{\text{VLM}}$ )	80.02	78.95	52.67	69.87
Sink tokens ( $\Phi_{\text{VLM}}$ )	78.96	82.14	52.77	71.06

Appendix

Table A.2 : Effect of removing sink dimensions under

\Phi_{\text{LLM}}

and

\Phi_{\text{VLM}}

on AVHBench. Removing the

\Phi_{\text{LLM}}

sink token collapses performance, while removing tokens identified by

\Phi_{\text{VLM}}

has little effect.

Figure A.1 : Additional qualitative examples. Each example visualises attention scores for query tokens ( e.g., kid, fly, ball, name) on AVHBench videos.

Figure A.2 : Number of sink tokens across layers under $\Phi_{\text{VLM}}$ and $\Phi_{\text{LLM}}$ on video-SALMONN2+. $\Phi_{\text{VLM}}$ rapidly over-identifies sink tokens as depth increases, eventually classifying nearly all tokens ( $N=11140$ ) as sinks in the final layers. In contrast, $\Phi_{\text{LLM}}$ consistently identifies only a sparse and stable set of sink tokens throughout all layers.

Models	$\tau$	Sink Token Proportion (%)
Models	$\tau$	OmniBench	AVUT	AVHBench	DailyOmni
Qwen2.5-Omni 3B	20	15.33	13.79	10.02	11.29
	25	2.50	1.91	1.30	1.38
	30	0.13	0.03	0.06	0.03
Qwen2.5-Omni 7B	20	33.32	26.16	26.46	24.53
	25	16.41	11.83	12.09	11.44

Appendix

Table A.3 : Sensitivity of

\Phi_{\mathrm{VLM}}

to the sink threshold. We report the mean proportion of input tokens identified as sinks. Increasing

\tau

reduces the proportion of detected sinks, but the same threshold yields substantially different proportions across model scales.

Table 20

Figure B.1 : Sink score vs. cosine similarity. Attention to sink tokens is strongly correlated with the cosine similarity between head outputs and the sink value direction.

Model	Attn. to video / tok	Attn. to audio / tok	$\Delta$ Sink	$\Delta$ Video	$\Delta$ Audio
Qwen2.5-Omni [ 58 ]	2.20e-04	2.87e-04	-0.0176	+0.0127	+0.0022
video-SALMONN2+ [ 48 ]	6.45e-05	3.64e-04	-0.0095	+0.0088	+0.0005
VideoLLaMA2 [ 8 ]	3.22e-04	2.15e-04	-0.0234	+0.0103	+0.0123

Appendix

Table B.4: Attention redistribution after rotation. Attention to video/audio values are baseline attention mass normalised by the number of modality tokens. Deltas show the change in total attention mass after rotation. Rotation reduces sink attention and preferentially increases attention toward the modality that was originally attended to less.

Mode	$A\to V$	$V\to A$	Matching	Overall
Baseline	80.11	75.41	59.86	70.91
Internal (Ours)	80.37	75.81	60.18	71.25
Encoder	79.93	74.93	60.55	70.91
Encoder w/o text	80.11	75.33	59.59	70.78

Appendix

Table B.5 : Effect of different sink attention sources. Using internal features achieves the best performance, whereas encoder inputs do not improve performance.

Models	Holmes	MME	ActivityNet
Qwen2.5-VL 3B [ 4 ]	41.26	57.89	47.02
+ OutRo	42.41	58.33	46.87
Qwen3-VL 8B [ 3 ]	46.65	67.44	46.65
+ OutRo	46.76	68.33	46.85
VideoLLaMA3 7B [ 66 ]	41.15	61.78	49.32
+ OutRo	42.62	62.56	49.92

Appendix

Table B.6 : Performance of OutRo on VLMs.

Panel of Figure 25 — Figure B.2 : Effect of different $\gamma$ values. Optimal $\gamma$ values vary across models: $3.0$ for Qwen2.5, $0.5$ for Qwen3-VL and video-SALMONN2 + , and $1.5$ for VideoLLaMA3.

Models	Method	Audio-Visual QA
Models	Method	OmniBench	AVUT	AVHBench	DailyOmni
Qwen2.5-Omni 3B	Baseline	$40.19\pm 0.57$	$60.23\pm 0.56$	$64.39\pm 0.66$	$55.03\pm 0.70$
	OutRo ( $\Phi_{\mathrm{LLM}}$ )	$\mathbf{40.90\pm 0.68}$	$\mathbf{61.11\pm 0.47}$	$\mathbf{64.78\pm 0.74}$	$\mathbf{55.77\pm 0.64}$
	$\Phi_{\mathrm{VLM}}$ top 1%	$34.20\pm 0.58$	$48.69\pm 0.50$	$60.97\pm 0.59$	$33.02\pm 0.49$
	$\Phi_{\mathrm{VLM}}$ top 1	$38.87\pm 0.84$	$56.38\pm 1.01$	$61.90\pm 0.52$	$43.97\pm 0.54$
Qwen2.5-Omni 7B	Baseline	$48.13\pm 0.88$	$\mathbf{64.53\pm 0.53}$	$71.13\pm 0.32$	$\mathbf{52.89\pm 1.07}$

Appendix

Table B.7 : Rotation with different sink identification criteria. We compare rotation toward sinks identified by

\Phi_{\mathrm{LLM}}

with rotation toward the top 1% or top 1 token under

\Phi_{\mathrm{VLM}}

. All variants use rotation only, without mask relaxation. We report mean accuracy and standard deviation over five seeds. Restricting

\Phi_{\mathrm{VLM}}

to fewer tokens does not recover an effective rotation direction.

Models	Image–Text	Text-Only
Models	POPE	GSM8K
Qwen2.5-Omni 3B	87.67	73.92
+ OutRo	88.06	74.45
Qwen2.5-7B-Instruct	-	$59.03\pm 0.73$
+ OutRo	-	$\mathbf{61.41\pm 0.85}$
Llama-3.1-8B-Instruct	-	$76.00\pm 0.46$

Appendix

Table B.8 : Generalisation to image–text and text-only settings. OutRo improves performance on POPE with image–text inputs and GSM8K with text-only inputs. Results on Qwen2.5-Omni use greedy decoding over the full evaluation sets. For text-only LLMs, we report mean accuracy and standard deviation over five seeds.

Method	Audio-Visual QA
Method	OmniBench	AVUT	AVHBench	DailyOmni
Baseline	$40.19\pm 0.57$	$60.23\pm 0.56$	$64.39\pm 0.66$	$55.03\pm 0.70$
OutRo	$\mathbf{40.90\pm 0.68}$	$\mathbf{61.11\pm 0.47}$	$\mathbf{64.78\pm 0.74}$	$\mathbf{55.77\pm 0.64}$
Random video	$36.38\pm 1.28$	$56.52\pm 0.95$	$61.04\pm 0.39$	$46.59\pm 1.34$
Random audio	$37.14\pm 0.70$	$56.44\pm 0.90$	$60.57\pm 0.92$	$45.32\pm 0.91$
Random text	$36.59\pm 0.73$	$53.42\pm 0.99$	$61.75\pm 0.42$	$44.49\pm 1.55$

Appendix

Table B.9 : Rotation toward alternative directions. We compare the sink value direction with random video, audio, and text token directions and a head-mean direction on Qwen2.5-Omni 3B. All variants use gated rotation without mask relaxation and preserve the original head-output norm. We report mean accuracy and standard deviation over five seeds.

\mathcal{S}^{(\ell)}\leftarrow\Phi_{\mathrm{LLM}}(\mathbf{X}^{(\ell)})

Appendix

Algorithm 1 OutRo in a Decoder Layer

Panel of Figure 30 — Figure E.1 : Qualitative results on video-QA on AVHBench. Compared with the baseline, OutRo produces answers that better align with visual and audio-visual evidence in the video, leading to more accurate multimodal reasoning.

Panel of Figure 31 — Figure E.2 : Qualitative results for video captioning on AVHBench. OutRo produces captions grounded in both visual and audio-visual cues, whereas the baseline often fails to capture such multimodal details.

Attention Sinks

Attention sinks are tokens that attract disproportionate Transformer attention, shaping information flow, long-context behavior, and model efficiency.

See all 55 papers

Apr 1, 2026

When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models

Attention sinks are defined as tokens that attract disproportionate attention. While these have been studied in single modality transformers, their cross-modal impact in Large Vision-Language Models (LVLM) remains largely unexplored: are they redundant artifacts or essential global priors? This paper first categorizes visual sinks into two distinct categories: ViT-emerged sinks (V-sinks), which propagate from the vision encoder, and LLM-emerged sinks (L-sinks), which arise within deep LLM layers. Based on the new definition, our analysis reveals a fundamental performance trade-off: while sinks effectively encode global scene-level priors, their dominance can suppress the fine-grained visual evidence required for local perception. Furthermore, we identify specific functional layers where modulating these sinks most significantly impacts downstream performance. To leverage these insights, we propose Layer-wise Sink Gating (LSG), a lightweight, plug-and-play module that dynamically scales the attention contributions of V-sink and the rest visual tokens. LSG is trained via standard next-token prediction, requiring no task-specific supervision while keeping the LVLM backbone frozen. In most layers, LSG yields improvements on representative multimodal benchmarks, effectively balancing global reasoning and precise local evidence.

Jiho Choi, Jaemin Kim, Sanghwan Kim +2

KAIST, South Korea · Chung-Ang University, South Korea · Technical University of Munich, Germany

Large Vision-Language Models VLM Interpretability Attention Sinks Attention Mechanisms

Ask PDF

Jul 25, 2026

OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models

Existing token compression methods for omnimodal large language models typically rely on one modality to determine what to retain in the other. We show that this assumption often breaks down: for the same query, audio and video relevance often peaks at different moments. This cross-modal salience mismatch makes unidirectional guidance prone to discarding answer-critical cues under aggressive compression. We propose OmniScope, a training-free token compression framework that uses the query as a shared semantic anchor while estimating relevance separately for audio and video. OmniScope allocates modality-specific token budgets, prunes visual tokens with an anchor-delta strategy that preserves both global context and temporal changes, and merges audio tokens within each second to reduce redundancy while maintaining temporal continuity. Across four audio-video benchmarks and two Qwen2.5-Omni model scales, OmniScope achieves the best average accuracy across all compression settings. At 25% overall token retention, it delivers up to 3.53x prefill speedup and more than 15% GPU memory reduction, with only a 0.35-point drop in average accuracy. These results suggest a simple design principle for OmniLLM inference: share the query across modalities, but not the salience estimates. The code is available at https://github.com/MAC-AutoML/OmniScope.

Jinsen Su, Yongdong Luo, Yuexiao Ma +3

Media Analytics and Computing Lab, Xiamen University, Xiamen, China · Institute of Artificial Intelligence, Xiamen University, Xiamen, China · School of Informatics, Xiamen University, Xiamen, China +1

Efficient Multimodal Inference Audio Token Compression Multimodal Large Language Models Multimodal Token Compression Visual Token Pruning

Ask PDF

May 19, 2026

Stage-adaptive Token Selection for Efficient Omni-modal LLMs

The unified audio-visual understanding capability of omni-modal large language models (om-LLMs) comes at the high cost of processing tens of thousands of visual and audio tokens throughout the LLM. To reduce the cost, training-free methods for om-LLM token selection are being actively developed, operating either pre-LLM or inner-LLM. Recognizing that the non-textual token redundancy varies across stages and LLM layers, we propose in this paper a novel stage-adaptive token selection (SEATS) method that operates in both pre-LLM and inner-LLM stages. Given an overall token budget measured by the token retention rate (TRR), SEATS performs top-down token budget allocation in an input-dependent manner. Specifically, a larger TRR is used for pre-LLM token selection, whilst within the LLM, all tokens are retained in shallow layers, increasingly decreased TRRs are applied in middle layers, and the non-textual tokens are fully removed in late layers. Extensive experiments on three representative om-LLMs and six benchmarks verify the viability of the proposed method. By retaining only 10% of visual and audio tokens, SEATS preserves 94.9% of the full-token performance on Qwen2.5-Omni while achieving a 9.3x FLOPs reduction and a 4.9x prefill speedup.

Zijie Xin, Jie Yang, Ruixiang Zhao +4

Renmin University of China · WeChat Vision, Tencent Inc.

Efficient Multimodal Inference Multimodal Large Language Models Token Pruning Multimodal Token Compression

Ask PDF

Abstract

Attention Sinks

Explore similar work

When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models

OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models

Stage-adaptive Token Selection for Efficient Omni-modal LLMs