SAGE: Sink-Aware Guided Emphasis for Visual Grounding in Vision-Language Decoders

Jeonghyo Song; YoungJoon Yoo

cs.CVOct 8, 2026

SAGE: Sink-Aware Guided Emphasis for Visual Grounding in Vision-Language Decoders

Authors: Jeonghyo Song, YoungJoon Yoo

Organizations: Department of Artificial Intelligence, Chung-Ang University

Abstract

Recent large vision-language models (VLMs) pair a visual encoder with a large language model (LLM) and perform well on diverse image-text tasks, yet their reliability is often limited by decoder attention pathologies that suppress visual evidence and exacerbate hallucinations. In this paper, we revisit visual attention sinks and uncover a structured, layer-dependent behavior: across prompts, early and late decoder layers exhibit prompt-invariant attention collapse onto the same few image regions, which we term PIS (Prompt-Invariant Sinks), whereas mid layers become prompt-conditioned and drive vision-language alignment. This split suggests that treating sinks as a uniform effect is incomplete. Building on this insight, we propose SAGE (Sink-Aware Guided Emphasis), a lightweight intervention that steers decoder attention away from PIS and toward query-dependent regions of interest (ROIs) using token-aligned ROI masks derived from standard vision backbones such as CLIP, ViT, and DINOv3. Evaluated on diverse vision-encoder + decoder-only LLM VLM families, SAGE improves visual grounding, reduces hallucinations, and yields consistent gains across public downstream vision-language benchmarks, including fine-grained visual discrimination settings where localized evidence is crucial, when instantiated with backbone-derived ROI masks.

Figures & tables

Figure 1: Prompt-Invariant Sinks and prompt-conditioned mid-layer attention in a VLM decoder. We visualize decoder attention over visual tokens for three different prompts (rows) at representative early, mid, and final decoder layers (columns). Early and final layers exhibit prompt-invariant attention collapse onto nearly identical image regions across prompts, forming Prompt-Invariant Sinks. In contrast, mid-layer attention is prompt-conditioned: it shifts to the menu board when asking for the espresso price (Prompt 1), to the wall clock when asking for the current time (Prompt 2), and becomes broader for holistic scene description (Prompt 3). Warmer colors indicate higher attention.

Figure 2: Overview of SAGE. Given an image $I$ and a query $Q$ , SAGE operates in two stages. Step 1 (ROI construction from standard vision backbones): a token-aligned ROI mask over visual tokens is derived from patch-level relevance maps obtained from CLIP, ViT, or DINOv3. Step 2 (Training-free attention steering and layer selection): using the ROI mask, SAGE applies an additive bias to visual-token attention logits (Eq. ( 1 )) at selected decoder layer(s), redistributing attention away from sink regions and toward query-relevant visual evidence. Layer selection can be performed via a layer sweep and top- $k$ selection strategy (Section 3.4 ). The resulting attention redistribution improves visual grounding during autoregressive generation.

Figure 3: Evidence for sink structure, spectral dominance, and attention redistribution. (a) Prompt-invariant sink structure. For each layer, we form a sink set as the top- $5\%$ visual tokens by attention weight and report the IoU of sink sets across different prompts for the same image (error bars: standard deviation). (b) Spectral dominance. We stack visual-key attention logits across prompts and decoding steps to form $\mathbf{A}^{(\ell,h)}$ (Eq. ( 3 )) and report the top-1 energy ratio $\rho_{1}$ (Eq. ( 5 )); SAGE reduces spectral concentration, indicating weaker low-rank sink dominance. (c) Attention redistribution. we report attention mass assigned to sink tokens (top- $5\%$ tokens) and ROI tokens; SAGE shifts attention budget away from sink tokens and toward ROI, consistent with the odds-ratio effect in Eq. ( 6 ).

Model	TextVQA	POPE		GQA	ScienceQA	MME	MMVP		MMVP Original		MMVet
Model	TextVQA	Acc	F1	GQA	ScienceQA	MME	Image	Pair	Image	Pair	MMVet
LLaVA 1.5-13B	55.50	82.75	81.55	61.65	63.98	1783.11	61.48	11.85	33.58	10.65	31.20
	61.94	86.84	86.58	62.65	68.63	1842.58	62.96	37.78	50.00	22.00	42.20
+ SAGE (CLIP)	( +6.44 )	( +4.09 )	( +5.03 )	( +1.00 )	( +4.65 )	( +59.47 )	( +1.48 )	( +25.93 )	( +16.42 )	( +11.35 )	( +11.00 )
	61.85	85.81	84.82	65.50	69.15	1895.55	68.89	38.52	58.58	21.27	34.50
+ SAGE (ViT)	( +6.35 )	( +3.06 )	( +3.27 )	( +3.85 )	( +5.17 )	( +112.44 )	( +7.41 )	( +26.67 )	( +25.00 )	( +10.62 )	( +3.30 )

Table 1: Main results of SAGE across VLM families. For each model, benchmark, and ROI source, the reported SAGE score corresponds to the best single-layer result obtained from the layer sweep. Exact selected layer indices are listed in Appendix Table 6 .

Figure 4: Layer-wise averaged normalized improvement on MMVP-family benchmarks. For each ROI source (CLIP, ViT, DINOv3), we apply SAGE to one decoder layer and evaluate on four fine-grained sub-benchmarks: MMVP image, MMVP pair, MMVP-Original image, and MMVP-Original pair. For each sub-benchmark, we compute the normalized improvement over the no-intervention baseline as

(s-b)/b\times 100

, where

s

is the score with SAGE and

b

is the baseline score. The plotted value is the average of these four normalized improvements at each intervention layer. The x-axis shows the LLaVA-1.5 decoder layer index, and values above zero indicate improvement over the baseline.

Figure 5: Layer-wise single-layer sweep of SAGE interventions. We report relative performance improvement over the no-intervention baseline when applying SAGE to a single decoder layer at a time. The x-axis is normalized decoder depth $d(\ell)=\ell/(L-1)$ , allowing comparison across models with different decoder depths. The horizontal reference at $0\%$ indicates the no-intervention baseline. Bars aggregate results across all benchmarks evaluated in our experiments, and error bars show the min–max range. Orange bars indicate the top-performing normalized depth positions.

Depth composition	Mean Acc.
2-layer sets
Very-early only	19.19
Early-mixed	49.85
Mid–mid	61.96
Mid–final	62.60
Final–final	62.42

Table 2: Depth-grouped summaries of multi-layer SAGE sets on TextVQA. Each value reports the mean accuracy within a coarse depth-composition group.

		SAGE
Benchmark	Base	Best	L24	Rand.
TextVQA	44.0	55.4	54.6	43.1
GQA	57.1	62.8	62.3	57.0
POPE ${}_{\text{acc}}$	72.7	73.6	73.1	72.1
POPE ${}_{\text{F1}}$	69.3	73.0	69.9	69.1
ScienceQA	68.1	68.6	68.6	68.0

Table 3: Fixed-layer result and ROI content control . Best selects the layer on the benchmark; L24 uses one fixed layer chosen by label-free diagnostics. Rand. replaces the guided ROI with a random token set of the same size, at the same layer and bias strength.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Value / Description
Local VLM	Vision-encoder + decoder-only LLM VLMs (e.g., LLaVA-1.5 Liu et al. (2024b) /Qwen2-VL Wang et al. (2024) /InternVL3 Zhu et al. (2025) /NVILA Liu et al. (2025) ) with standard preprocessing and decoding settings.
ROI sources	Patch-level ROI masks derived from standard pretrained vision backbones (CLIP/ViT/DINOv3).
Patch relevance scoring	Backbone-specific patch relevance scores are thresholded to form a binary ROI mask; the same scoring rule is used consistently within each source.
Patch-to-token mapping	The patch-level mask is aligned to the visual-token grid used by the evaluated VLM; the mapping is fixed per model and preprocessing configuration.
Patch grid	Model-specific patch grid size determined by the vision encoder and preprocessing; grid configuration is fixed per model and recorded in the experiment scripts.
Intervention layers	Layer set $\mathcal{L}_{\mathrm{int}}$ specified as a list or range (single-layer or multi-layer), fixed per experiment.

Appendix

Table 4: Key hyperparameters and settings for SAGE. We use

(\alpha,\beta)=(5,2)

as the default bias strengths unless otherwise stated.

Variant	$(\alpha,\beta)$	MMVP (%)
No re-weighting	$(0,0)$	61.48
Pull-only	$(0,2)$	54.07
Push-only	$(5,0)$	57.49
Weak	$(3,1)$	61.91
Default	$(5,2)$	62.96
Strong	$(7,3)$	59.39

Appendix

Table 5: Sensitivity of SAGE to logit-level intervention strength

(\alpha,\beta)

. We report MMVP (%).

Model	TextVQA	POPE	GQA	SQA	MME	MMVet	MMVP Img	MMVP Pair	MMVP-O Img	MMVP-O Pair
LLaVA 1.5-13B
+ SAGE (CLIP)	L17	L3	L20	L18	L24	L20	L26	L26	L18	L6
+ SAGE (ViT)	L29	L10	L19	L20	L22	L18	L18	L18	L2	L18
+ SAGE (DINOv3)	L30	L10	L30	L25	L16	L20	L26	L26	L18	L18
Qwen2-VL-7B
+ SAGE (CLIP)	L11	L18	L21	L15	L25	L14	L18	L18	L15	L15

Appendix

Table 6: Exact model-specific best-layer indices used for the Best results in Table 1 . Indices are raw layer numbers within each model and should not be compared directly across architectures; cross-model discussion in the main text uses normalized decoder depth.

Layer set	Depth composition	Accuracy (%)
(14,16)	Mid–mid	62.43
(17,21)	Mid–final	62.72
(28,31)	Final–final	62.58

Appendix

Table 7: Illustrative 2-layer SAGE sets on TextVQA (LLaVA-1.5). These examples are selected to represent stronger non-early depth compositions.

Layer set	Depth composition	Accuracy (%)
(14,18,21)	Mid $\times 3$	62.33
(17,24,30)	Mid–final–final	62.63
(24,27,30)	Final $\times 3$	62.53

Appendix

Table 8: Illustrative 3-layer SAGE sets on TextVQA (LLaVA-1.5). These examples are selected to represent stronger multi-layer compositions away from the earliest decoder layers.

Layer set	Depth composition	Accuracy (%)
(0,2)	Very-early only	12.07
(0,2,4)	Very-early $\times 3$	7.12
(1,2,3)	Very-early $\times 3$	1.12

Appendix

Table 9: Illustrative failure-prone early-layer sets on TextVQA (LLaVA-1.5). These examples show that sets concentrated near the earliest decoder layers can still lead to substantial degradation.

Figure 6: Layer-skip ablation on LLaVA-1.5. For each decoder layer, we skip only that layer at inference and evaluate performance. The top row reports TextVQA and POPE accuracy versus the skipped depth, and the bottom row shows the change relative to the no-skip baseline.

Figure 7: Qualitative comparison of baseline vs. SAGE. For each example, we show the input image and query, baseline and SAGE predictions, and attention visualizations. SAGE encourages attention to concentrate on query-relevant visual evidence, improving grounding in fine-grained cases.

Figure 8: Failure case under unstable local ROI estimates. Different local ROI sources can produce fragmented or partially misaligned masks, which may steer attention away from the true evidence and lead SAGE to fail.

Figure 9: Single-layer sweep with a CLIP-derived ROI mask. We report per-benchmark performance as a function of the intervened layer using a CLIP-based patch-level ROI mask.

Figure 10: Single-layer sweep with a ViT-derived ROI mask. We report per-benchmark performance as a function of the intervened layer using a ViT-based patch-level ROI mask.

Figure 11: Single-layer sweep with a DINOv3-derived ROI mask. We report per-benchmark performance as a function of the intervened layer using a DINOv3-based patch-level ROI mask.

Explore similar work

When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models
Apr 1, 2026Jiho Choi, Jaemin Kim, Sanghwan Kim +2Large Vision-Language Models VLM Interpretability
KAIST, South Korea · Chung-Ang University, South Korea · Technical University of Munich, Germany
See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs
Jun 29, 2026Yuqing Lei, Wenbo Lyu, Yingjun Du +3Visual Attention Large Vision-Language Models
University of Chinese Academy of Sciences · University of Amsterdam · United Imaging Healthcare Co., Ltd.
Instruction-Evidence Contrastive Dual-Stream Decoding for Grounded Vision-Language Reasoning
Apr 28, 2026Yashwant Pravinrao Bangde, Debaditya RoyObject Hallucination in VLMs Vision-Language Grounding
Department of Computer Science and Engineering Indian Institute of Technology Kharagpur Kharagpur, India 721302

Apr 1, 2026, cs.CVJ/K move · Enter open · S save

When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models

Jiho Choi, Jaemin Kim, Sanghwan Kim, Seunghoon Hong+1

KAIST, South Korea · Chung-Ang University, South Korea · Technical University of Munich, Germany

Large Vision-Language Models VLM Interpretability Attention Sinks Attention Mechanisms

Open paper AskPDF arXiv

Attention sinks are defined as tokens that attract disproportionate attention. While these have been studied in single modality transformers, their cross-modal impact in Large Vision-Language Models (LVLM) remains largely unexplored: are they redundant artifacts or essential global priors? This paper first categorizes visual sinks into two distinct categories: ViT-emerged sinks (V-sinks), which propagate from the vision encoder, and LLM-emerged sinks (L-sinks), which arise within deep LLM layers. Based on the new definition, our analysis reveals a fundamental performance trade-off: while sinks effectively encode global scene-level priors, their dominance can suppress the fine-grained visual evidence required for local perception. Furthermore, we identify specific functional layers where modulating these sinks most significantly impacts downstream performance. To leverage these insights, we propose Layer-wise Sink Gating (LSG), a lightweight, plug-and-play module that dynamically scales the attention contributions of V-sink and the rest visual tokens. LSG is trained via standard next-token prediction, requiring no task-specific supervision while keeping the LVLM backbone frozen. In most layers, LSG yields improvements on representative multimodal benchmarks, effectively balancing global reasoning and precise local evidence.

Abstract

Explore similar work

When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models

See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs

Instruction-Evidence Contrastive Dual-Stream Decoding for Grounded Vision-Language Reasoning