Multimodal Large Language Models (MLLMs) have demonstrated remarkable reasoning capabilities yet continue to suffer from hallucination, where generated text contradicts visual content. In this paper, we introduce Dual-Anchor Introspective Decoding (DaID), a novel contrastive decoding framework that dynamically calibrates each token generation by mining the model's internal perceptual discrepancies. Specifically, DaID identifies a Spotlight layer to amplify visual factual signals and a Shadow layer to suppress textual inertia. By leveraging visual attention distributions to guide this dual-anchor selection process, our method ensures precise, token-specific adaptation. Experimental results across multiple benchmarks and MLLMs demonstrate that DaID significantly mitigates hallucination while enhancing general reasoning capabilities.
Figures & tables
Figure 1: A failure case of Visual Contrastive Decoding (VCD) caused by external perturbations.
Figure 2: Layer-wise diagnosis of MLLM decoding dynamics on the POPE benchmark, revealing the evolution of hallucination rates, visual perception capability, and attention distribution across layers.
Figure 3: Framework of DaID . DaID dynamically identifies dual-anchor layers guided by the visual attention and calibrates the final output by leveraging both the Spotlight and Shadow anchors to ensure visual grounding.
Method
POPE
CHAIR
MME
Acc. ↑
F1 ↑
CHAIR S ↓
CHAIR I ↓
Existence ↑
Count ↑
Position ↑
Color ↑
Total ↑
LLaVA-1.5-7B
Greedy
81.38
82.20
49.6
14.4
173.45
122.72
113.39
149.92
559.48
Beam Search
84.66
84.60
46.3
12.9
175.67
124.67
114.00
151.00
565.34
DoLa Chuang et al. (2023)
84.06
84.62
47.1
13.8
180.10
127.40
119.30
154.60
581.40
VCD Leng et al. (2024)
84.66
84.52
49.2
14.8
184.66
137.33
128.67
153.00
603.66
Table 1: Performance comparison of DaID against state-of-the-art methods on POPE, CHAIR, and MME benchmarks. The best results are highlighted in bold , and the second-best results are underlined .
Figure 4: Performance evaluation of LLaVA-1.5-7B/13B across five general vision-language benchmarks.
Figure 5: Impact of visual perception reinforcement intensity α on LLaVA-1.5 and LLaVA-NeXT.
Figure 6: Impact of language prior suppression intensity β on LLaVA-1.5 and LLaVA-NeXT.
Figure 7: Generalization analysis of DaID across diverse MLLM architectures on CHAIR and POPE benchmarks.
Method
POPE
CHAIR
Acc. ↑
F1 ↑
CHAIR S ↓
CHAIR I ↓
LLaVA-1.5
+ DaID
85.08
85.92
35.9
11.3
w/o PA
84.57
84.99
38.1
12.0
w/o NA
82.79
82.85
48.9
14.3
LLaVA-NeXT
Table 2: Ablation study of DaID . PA and NA denote the positive and negative anchors, respectively.
Figure 8: Qualitative case study of DaID on a representative instance (input image and query detailed in Figure 3 ).
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Analysis of the topological constraint ( Lshad.<Lspot. ) on LLaVA-1.5-7B. We compare the performance between the constrained and unconstrained settings.
Figure 10: Inference latency comparison (ms/token) on NVIDIA RTX 5070 Ti. We evaluate the decoding speed of normal decoding (Baseline), our method, and VCD on LLaVA-1.5-7B and LLaVA-NeXT-7B.
Figure 11: Visualization of token-specific dynamic anchor selection on LLaVA-1.5. We track the layer indices selected for the Spotlight Anchor and Shadow Anchor across 20 decoding steps.
Multimodal large language models (MLLMs) have achieved remarkable progress in vision-language reasoning, yet they remain vulnerable to hallucination, where generated content deviates from visual evidence. Existing mitigation strategies either require costly supervision during training or introduce additional latency at inference time. Recent vision enhancement methods attempt to address this issue by reinforcing visual tokens during decoding, but they typically inject all tokens indiscriminately, which causes interference from background regions and distracts the model from critical cues. To overcome this challenge, we propose Adaptive Visual Reinforcement (AIR), a training-free framework for MLLMs. AIR consists of two components. Prototype-based token reduction condenses the large pool of visual tokens into a compact subset to suppress redundancy. OT-guided patch reinforcement quantifies the alignment between hidden states and patch embeddings to selectively integrate the most consistent patches into feed-forward layers. As a result, AIR enhances the model's reliance on salient visual information and effectively mitigates hallucination. Extensive experiments across representative MLLMs demonstrate that AIR substantially reduces hallucination while preserving general capabilities, establishing it as an effective solution for building reliable MLLMs.
Xingyu Zhu, Kesen Zhao, Liang Yi +4
MoE Key Lab of BIPC, University of Science and Technology of China · Nanyang Technological University
Multimodal Large Language Models frequently suffer from inference hallucinations, partially stemming from language priors dominating visual evidence. Existing training-free mitigation methods either perturb the visual representation and deviate from the natural image distribution, or enforce intrusive manipulations that compromise the model's inherent generative fluency. We introduce a novel perspective that multimodal hallucination manifests as the hypersensitivity of visual grounding to textual phrasing during the decoding phase. Building on this insight, we propose Decoding by Perturbation (DeP), a training-free framework mitigating prior-induced hallucinations via controlled textual interventions. DeP employs a dynamic probe applying multi-level textual perturbations to elicit latent language priors. Leveraging attention variance, it enhances stable evidence regions while suppressing suspicious noise in the feature space. Furthermore, it constructs an interpretable prior drift direction using logits statistics to counteract probability biases from textual co-occurrences. Extensive experiments confirm DeP effectively reduces hallucinations and achieves superior performance across multiple benchmarks.
Sihang Jia, Shuliang Liu, Songbo Yang +3
The Hong Kong University of Science and Technology (Guangzhou)
Multimodal Diffusion Large Language Models (MDLLMs) achieve high-concurrency generation through parallel masked decoding, yet the architectures remain prone to multimodal hallucinations. This structural vulnerability stems from an algorithmic flaw: the decoder ranks candidate tokens based on textual likelihood without verifying localized visual support. We establish that this language-only ranking induces an objective mismatch, where language probability mass acts as a misspecified proxy for the intended multimodal task. Consequently, we reinterpret hallucination as a localized optimization error, a phenomenon where the decoder exploits language shortcuts to maximize a proxy score at the expense of visual grounding. To address this objective mismatch, we introduce VISAGE, a training-free decoding framework that calibrates the objective at inference time. VISAGE estimates the proxy discrepancy by quantifying the spatial entropy of cross-attention distributions. By enforcing a localization consensus across attention heads, the method penalizes spatially uniform distributions and re-ranks token commitments to favor visually grounded outcomes. We provide an analytical stability guarantee establishing that VISAGE maintains a bounded objective loss under estimation error. Evaluations across hallucination-sensitive and general-purpose benchmarks demonstrate the robustness of the framework, yielding relative gains of 8.59% on MMMU-val and 7.75% on HallusionBench.
Vishal Narnaware, Animesh Gupta, Kevin Zhai +2
Institute of Artificial Intelligence, University of Central Florida