SAGE: Sink-Aware Grounded Decoding for Multimodal Hallucination Mitigation

Tripti Shukla; Zsolt Kira

cs.CVMar 29, 2026

SAGE: Sink-Aware Grounded Decoding for Multimodal Hallucination Mitigation

Authors: Tripti Shukla, Zsolt Kira

Organizations: Georgia Institute of Technology

Abstract

Large vision-language models (VLMs) frequently suffer from hallucinations, generating content that is inconsistent with visual inputs. Existing methods typically address this problem through post-hoc filtering, additional training objectives, or external verification, but they do not intervene during the decoding process when hallucinations arise. In this work, we introduce SAGE, a Sink-Aware Grounded Decoding framework that mitigates hallucinations by dynamically modulating self-attention during generation. Hallucinations are strongly correlated with attention sink tokens - punctuation or function tokens that accumulate disproportionate attention despite carrying limited semantic content. SAGE leverages these tokens as anchors to monitor grounding reliability in real time. At each sink trigger, the method extracts semantic concepts from the generated sequence, estimates their visual grounding using both self-attention maps and gradient-based attribution, and measures their spatial agreement. Based on this signal, self-attention distributions are adaptively sharpened or broadened to reinforce grounded regions or suppress unreliable ones. Extensive experiments across diverse hallucination benchmarks demonstrate that SAGE consistently outperforms existing decoding strategies, achieving substantial reductions in hallucination while preserving descriptive coverage, without requiring model retraining or architectural modifications. Our method achieves an average relative improvement of 10.65% on MSCOCO and 7.19% on AMBER across diverse VLM architectures, demonstrating consistent gains in hallucination mitigation.

Figures & tables

Figure 1: We propose SAGE , a sink-aware grounded decoding strategy for mitigating hallucinations in VLMs. SAGE detects sink tokens during generation and dynamically recalibrates attention towards image tokens to produce visually grounded descriptions.

Figure 2: Overview of our inference-time decoding strategy, SAGE . Stage 1: SAGE identifies attention sink tokens and extracts key semantic concepts $C$ from the partially generated output. Stage 2: We project the sink-token attention onto the image to obtain an attention-based grounding map $i_{1}$ , and compute concept-level visual grounding $i_{2}$ using Grad-CAM from the final layer of the vision encoder. An overlap score $o$ is then calculated between the two maps. Stage 3: Based on a threshold $\tau$ , SAGE adaptively modulates attention: when $o<\tau$ , attention is diffused to explore alternative image regions; otherwise, attention is reinforced to strengthen grounding in the same regions.

Panel of Figure 3 — Figure 3: (a) Layer-wise analysis showing the average IoU between spatial self-attention maps of ground-truth object tokens and their corresponding bounding boxes, alongside attention entropy. Results are averaged over $500$ images from MSCOCO. (b) Grad-CAM visualizations from the final layer of the vision encoder for individual concepts and their union.

Figure 4: Qualitative results of SAGE, including sink-token spatial attention, GradCAM-derived visual grounding maps, the computed overlap score $o$ , and the modulated spatial attention maps. The sink token is highlighted in red.

Method	LLaVA-1.5		Shikra		MiniGPT-4		InstructBLIP		Qwen2-VL-7B
Method	$C_{S}\downarrow$	$C_{I}\downarrow$	$C_{S}\downarrow$	$C_{I}\downarrow$	$C_{S}\downarrow$	$C_{I}\downarrow$	$C_{S}\downarrow$	$C_{I}\downarrow$	$C_{S}\downarrow$	$C_{I}\downarrow$
OPERA	45.2	12.4	37.2	12.3	26.0	9.1	46.7	14.6	23.7	7.1
HALC	36.9	9.8	38.3	15.9	28.5	10.3	49.4	14.2	22.5	6.9
EAH	37.8	10.4	47.5	13.2	30.7	9.9	56.4	9.9	22.4	6.5
DML-LVLM	25.7	7.2	24.6	9.9	23.5	8.8	32.5	13.8	25.7	7.2
GIFT	37.4	9.6	33.5	11.7	29.3	10.2	38.8	11.9	22.5	8.1

Table 1: CHAIR

\downarrow

results on MSCOCO for LVLMs with different decoding baselines aimed at hallucination mitigation.

C_{S}

and

C_{I}

denote

CHAIR_{S}

and

CHAIR_{I}

, respectively. Blue text indicates the relative reduction (%) achieved by SAGE over the best competing baseline (lower is better). All reported scores are averaged over generations with maximum lengths of 256 and 512 tokens.

Method	LLaVA-1.5		Shikra		MiniGPT-4		InstructBLIP		Qwen2-VL-7B
Method	$C_{I}\downarrow$	$C_{R}\uparrow$	$C_{I}\downarrow$	$C_{R}\uparrow$	$C_{I}\downarrow$	$C_{R}\uparrow$	$C_{I}\downarrow$	$C_{R}\uparrow$	$C_{I}\downarrow$	$C_{R}\uparrow$
OPERA	12.1	45.3	11.2	42.6	10.3	50.4	12.2	48.7	8.8	52.4
HALC	9.7	51.7	16.2	39.6	11.1	53.6	13.1	49.3	7.8	54.7
EAH	11.3	48.4	12.6	52.3	10.6	54.5	10.7	49.1	7.1	55.2
DML-LVLM	7.8	52.6	9.2	46.7	8.9	47.4	13.6	44.3	8.1	51.6
GIFT	8.9	50.6	10.2	47.2	9.6	41.8	11.2	39.4	6.3	58.7

Table 2: CHAIR

\downarrow

and Cover

\uparrow

results on AMBER for LVLMs with different decoding baselines aimed at hallucination mitigation.

C_{I}

and

C_{R}

denote

CHAIR_{I}

and Cover, respectively. Blue text indicates the relative improvement (%) achieved by SAGE over the best competing baseline (lower is better for

C_{I}

, higher is better for

C_{R}

). All reported scores are averaged over generations with maximum lengths of 256 and 512 tokens.

Method	LLaVA-1.5		Qwen2-vl-7b
Method	$A$	$D$	$A$	$D$
GIFT [ 33 ]	$4.3$	$4.7$	$6.5$	$6.8$
LVLMs-Saliency [ 55 ]	$5.6$	$5.9$	$6.1$	$5.8$
SAGE(Ours)	6.9	6.6	8.2	8.1

Table 3: Gemini 3 Pro-based evaluation results.

A

stands for accuracy and

D

stands for detailedness .

Method	LLaVA-1.5		Shikra		MiniGPT-4		InstructBLIP		Qwen2-VL-7B
Method	$C_{S}\downarrow$	$C_{I}\downarrow$	$C_{S}\downarrow$	$C_{I}\downarrow$	$C_{S}\downarrow$	$C_{I}\downarrow$	$C_{S}\downarrow$	$C_{I}\downarrow$	$C_{S}\downarrow$	$C_{I}\downarrow$
SAGE w/o Grad-CAM	47.4	13.2	50.8	13.9	49.6	14.6	54.2	13.7	38.3	11.2
SAGE (Ours)	21.3	5.2	24.3	8.7	23.8	8.2	25.7	9.4	21.1	5.9

Table 4: CHAIR

\downarrow

results on MSCOCO for SAGE with and without Grad-CAM.

C_{S}

and

C_{I}

denote

CHAIR_{S}

and

CHAIR_{I}

, respectively. All reported scores are averaged over generations with maximum lengths of 256 and 512 tokens.

Reinforce only ( $\alpha_{1}=0$ )
Hyperparameters			LLaVA-1.5			Qwen2-VL-7B
$\alpha_{1}$	$\alpha_{2}$	$\tau$	$C_{S}\downarrow$	$C_{I}\downarrow$	$F1\uparrow$	$C_{S}\downarrow$	$C_{I}\downarrow$	$F1\uparrow$
0.0	1.2	0.5	56.7	14.7	50.6	49.9	13.8	58.9
0.0	1.5	0.5	57.2	14.8	47.7	50.3	13.9	56.7
0.0	1.8	0.5	57.9	15.3	48.0	53.7	14.2	55.2
0.0	2.0	0.5	59.3	15.2	48.7	54.2	14.3	53.2

Table 5: Ablation study of SAGE hyperparameters

\alpha_{1}

\alpha_{2}

, and

\tau

on MSCOCO. We report

C_{S}

C_{I}

, and

F1

for LLaVA-1.5b and Qwen2-vl-7b. Lower values are better for

C_{S}

and

C_{I}

, while higher is better for

F1

. All reported scores are generated with maximum lengths of 512 tokens.

Method	LLaVA-1.5		Shikra		MiniGPT-4		InstructBLIP		Qwen2-VL-7B
Method	$C_{S}\downarrow$	$C_{I}\downarrow$	$C_{S}\downarrow$	$C_{I}\downarrow$	$C_{S}\downarrow$	$C_{I}\downarrow$	$C_{S}\downarrow$	$C_{I}\downarrow$	$C_{S}\downarrow$	$C_{I}\downarrow$
Periodic Modulation (every 10 tokens)	39.4	12.5	41.3	12.9	39.9	13.1	38.2	11.7	34.6	10.8
Sink-triggered Modulation (SAGE)	21.3	5.2	24.3	8.7	23.8	8.2	25.7	9.4	21.1	5.9

Table 6: Ablation study of the modulation strategy on MSCOCO. We compare periodic modulation applied every 10 decoding steps with our sink-triggered modulation strategy.

C_{S}

and

C_{I}

denote

CHAIR_{S}

and

CHAIR_{I}

, respectively. All reported scores are averaged over generations with maximum lengths of 256 and 512 tokens.

Figure 5: Qualitative results of SAGE, including sink-token spatial attention, GradCAM-derived visual grounding maps, the computed overlap score $o$ , and the modulated spatial attention maps. The sink token is highlighted in red.

Component	Avg. Additional Cost (%)	Description
Concept Extraction (POS Tagging)	1.2	Extract key concepts from decoded text
Attention Map Extraction + IoU	1.7	Compute spatial overlap between maps
Grad-CAM Attribution	7.6	Compute visual attribution from vision encoder
Attention Modulation	0.3	Rescale attention weights
Total SAGE Overhead	10.8	Relative to standard decoding

Table 7: Average runtime overhead introduced by different components of SAGE. Grad-CAM computation dominates the additional cost, while concept extraction and attention modulation incur negligible overhead.

Figure 6: SAGE’s performance on mitigating hallucinations of LLaVA-1.5.

VLM Hallucination Mitigation

Vision-language model hallucination mitigation keeps generated claims grounded in visual evidence.

See all 228 papers

VGS-Decoding: Visual Grounding Score Guided Decoding for Hallucination Mitigation in Medical VLMs
Mar 19, 2026Govinda Kolli, Adinath Madhavrao Dukre, Behzad Bozorgtabar +2VLM Hallucination Mitigation Medical VLMs
MBZUAI, Abu Dhabi, UAE · Rajiv Gandhi University of Knowledge Technologies, India · Aarhus University, Aarhus, Denmark +1
Look Clearly Before Answering: Mitigating Hallucinations in LVLMs via Saliency-Driven Perceptual Realignment
Jul 18, 2026Pengxu Chen, Yao Zhu, Guangming Zhu +4Large Vision-Language Models VLM Hallucination Mitigation
Xidian University Xi’an, China · Tsinghua University Beijing, China · Shanghai Road Transport Development Center Shanghai, China +1
Hallucinations Leave a Grounding Signature:Verifier-Guided Decoding for Selective Object Correction
Jul 30, 2026Lei Yang, Xinze Liu, Dayan Wu +7Hallucination Detection in VLMs LLM Hallucination Mitigation
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China · Department of Applied Mathematics and Statistics, Johns Hopkins University, Baltimore, MD, USA

Mar 19, 2026J/K move · Enter open · S save

VGS-Decoding: Visual Grounding Score Guided Decoding for Hallucination Mitigation in Medical VLMs

Govinda Kolli, Adinath Madhavrao Dukre, Behzad Bozorgtabar, Dwarikanath Mahapatra+1

MBZUAI, Abu Dhabi, UAE · Rajiv Gandhi University of Knowledge Technologies, India · Aarhus University, Aarhus, Denmark+1

VLM Hallucination Mitigation Medical VLMs Medical VQA

Open paper AskPDF arXiv

Medical Vision-Language Models (VLMs) often hallucinate by generating responses based on language priors rather than visual evidence, posing risks in clinical applications. We propose Visual Grounding Score Guided Decoding (VGS-Decoding), a training-free method to mitigate hallucinations during inference. Our key insight is that hallucinated tokens maintain or increase their probability when visual information is degraded, while visually grounded tokens decrease in probability. We introduce the Visual Grounding Score (VGS), which measures each token's visual dependency by comparing distributions from original and distorted images. During decoding, we reweight probabilities by amplifying visually grounded tokens while suppressing hallucinations. Unlike fixed-weight contrastive methods, VGS-Decoding provides per-token adaptive control. Experiments on MIMIC-Diff-VQA and VQA-RAD across LLaVA-Med, CheXagent, and MedGemma demonstrate consistent improvements, with up to +9.12% overall gain and

+8.98\%

in open-ended recall, while introducing only

2\times

inference overhead and no additional training, making it practical for clinical deployment. Upon acceptance, code will be released publicly to facilitate reproducibility.

Abstract

VLM Hallucination Mitigation

Explore similar work

VGS-Decoding: Visual Grounding Score Guided Decoding for Hallucination Mitigation in Medical VLMs

Look Clearly Before Answering: Mitigating Hallucinations in LVLMs via Saliency-Driven Perceptual Realignment

Hallucinations Leave a Grounding Signature:Verifier-Guided Decoding for Selective Object Correction