Spotlight and Shadow: Attention-Guided Dual-Anchor Introspective Decoding for MLLM Hallucination Mitigation

Yebo Wu; Han Jin; Zhijiang Guo; Li Li

cs.CVApr 11, 2026

Spotlight and Shadow: Attention-Guided Dual-Anchor Introspective Decoding for MLLM Hallucination Mitigation

Authors: Yebo Wu, Han Jin, Zhijiang Guo, Li Li

Organizations: State Key Laboratory of IOTSC, University of Macau · Hong Kong University of Science and Technology (Guangzhou)

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated remarkable reasoning capabilities yet continue to suffer from hallucination, where generated text contradicts visual content. In this paper, we introduce Dual-Anchor Introspective Decoding (DaID), a novel contrastive decoding framework that dynamically calibrates each token generation by mining the model's internal perceptual discrepancies. Specifically, DaID identifies a Spotlight layer to amplify visual factual signals and a Shadow layer to suppress textual inertia. By leveraging visual attention distributions to guide this dual-anchor selection process, our method ensures precise, token-specific adaptation. Experimental results across multiple benchmarks and MLLMs demonstrate that DaID significantly mitigates hallucination while enhancing general reasoning capabilities.

Figures & tables

Figure 1: A failure case of Visual Contrastive Decoding (VCD) caused by external perturbations.

Panel of Figure 2 — Figure 2: Layer-wise diagnosis of MLLM decoding dynamics on the POPE benchmark, revealing the evolution of hallucination rates, visual perception capability, and attention distribution across layers.

Figure 3: Framework of DaID . DaID dynamically identifies dual-anchor layers guided by the visual attention and calibrates the final output by leveraging both the Spotlight and Shadow anchors to ensure visual grounding.

LLaVA-1.5-7B
Method	POPE		CHAIR		MME
Method	Acc. $\uparrow$	F1 $\uparrow$	CHAIR S $\downarrow$	CHAIR I $\downarrow$	Existence $\uparrow$	Count $\uparrow$	Position $\uparrow$	Color $\uparrow$	Total $\uparrow$
Greedy	81.38	82.20	49.6	14.4	173.45	122.72	113.39	149.92	559.48
Beam Search	84.66	84.60	46.3	12.9	175.67	124.67	114.00	151.00	565.34
DoLa Chuang et al. (2023)	84.06	84.62	47.1	13.8	180.10	127.40	119.30	154.60	581.40
VCD Leng et al. (2024)	84.66	84.52	49.2	14.8	184.66	137.33	128.67	153.00	603.66

Table 1: Performance comparison of DaID against state-of-the-art methods on POPE, CHAIR, and MME benchmarks. The best results are highlighted in bold , and the second-best results are underlined .

Panel of Figure 4 — Figure 4: Performance evaluation of LLaVA-1.5-7B/13B across five general vision-language benchmarks.

Panel of Figure 5 — Figure 5: Impact of visual perception reinforcement intensity $\alpha$ on LLaVA-1.5 and LLaVA-NeXT.

Panel of Figure 6 — Figure 6: Impact of language prior suppression intensity $\beta$ on LLaVA-1.5 and LLaVA-NeXT.

Panel of Figure 7 — Figure 7: Generalization analysis of DaID across diverse MLLM architectures on CHAIR and POPE benchmarks.

LLaVA-1.5
Method	POPE		CHAIR
Method	Acc. $\uparrow$	F1 $\uparrow$	CHAIR S $\downarrow$	CHAIR I $\downarrow$
+ DaID	85.08	85.92	35.9	11.3
w/o PA	84.57	84.99	38.1	12.0
w/o NA	82.79	82.85	48.9	14.3
LLaVA-NeXT

Table 2: Ablation study of DaID . PA and NA denote the positive and negative anchors, respectively.

Panel of Figure 8 — Figure 8: Qualitative case study of DaID on a representative instance (input image and query detailed in Figure 3 ).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure 9: Analysis of the topological constraint ( $L_{shad.}<L_{spot.}$ ) on LLaVA-1.5-7B. We compare the performance between the constrained and unconstrained settings.

Panel of Figure 10 — Figure 10: Inference latency comparison (ms/token) on NVIDIA RTX 5070 Ti. We evaluate the decoding speed of normal decoding (Baseline), our method, and VCD on LLaVA-1.5-7B and LLaVA-NeXT-7B.

Panel of Figure 11 — Figure 11: Visualization of token-specific dynamic anchor selection on LLaVA-1.5. We track the layer indices selected for the Spotlight Anchor and Shadow Anchor across 20 decoding steps.

VLM Hallucination Mitigation

Vision-language model hallucination mitigation keeps generated claims grounded in visual evidence.

See all 228 papers

Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation
Feb 27, 2026Xingyu Zhu, Kesen Zhao, Liang Yi +4VLM Hallucination Mitigation Multimodal Large Language Models
MoE Key Lab of BIPC, University of Science and Technology of China · Nanyang Technological University
Decoding by Perturbation: Mitigating MLLM Hallucinations via Dynamic Textual Perturbation
Apr 14, 2026Sihang Jia, Shuliang Liu, Songbo Yang +3VLM Hallucination Mitigation Multimodal Large Language Models
The Hong Kong University of Science and Technology (Guangzhou)
Seeing to Ground: Visual Attention for Hallucination-Resilient MDLLMs
Mar 26, 2026Vishal Narnaware, Animesh Gupta, Kevin Zhai +2Language Model Decoding Multimodal Grounding
Institute of Artificial Intelligence, University of Central Florida

Feb 27, 2026J/K move · Enter open · S save

Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation

Xingyu Zhu, Kesen Zhao, Liang Yi, Shuo Wang+3

MoE Key Lab of BIPC, University of Science and Technology of China · Nanyang Technological University

VLM Hallucination Mitigation Multimodal Large Language Models

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) have achieved remarkable progress in vision-language reasoning, yet they remain vulnerable to hallucination, where generated content deviates from visual evidence. Existing mitigation strategies either require costly supervision during training or introduce additional latency at inference time. Recent vision enhancement methods attempt to address this issue by reinforcing visual tokens during decoding, but they typically inject all tokens indiscriminately, which causes interference from background regions and distracts the model from critical cues. To overcome this challenge, we propose Adaptive Visual Reinforcement (AIR), a training-free framework for MLLMs. AIR consists of two components. Prototype-based token reduction condenses the large pool of visual tokens into a compact subset to suppress redundancy. OT-guided patch reinforcement quantifies the alignment between hidden states and patch embeddings to selectively integrate the most consistent patches into feed-forward layers. As a result, AIR enhances the model's reliance on salient visual information and effectively mitigates hallucination. Extensive experiments across representative MLLMs demonstrate that AIR substantially reduces hallucination while preserving general capabilities, establishing it as an effective solution for building reliable MLLMs.

Abstract

VLM Hallucination Mitigation

Explore similar work

Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation

Decoding by Perturbation: Mitigating MLLM Hallucinations via Dynamic Textual Perturbation

Seeing to Ground: Visual Attention for Hallucination-Resilient MDLLMs