cs.CLApr 14, 2026

Decoding by Perturbation: Mitigating MLLM Hallucinations via Dynamic Textual Perturbation

Authors: Sihang Jia, Shuliang Liu, Songbo Yang, Yibo Yan, Xin Zou, Xuming Hu

Organizations: The Hong Kong University of Science and Technology (Guangzhou)

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Multimodal Large Language Models frequently suffer from inference hallucinations, partially stemming from language priors dominating visual evidence. Existing training-free mitigation methods either perturb the visual representation and deviate from the natural image distribution, or enforce intrusive manipulations that compromise the model's inherent generative fluency. We introduce a novel perspective that multimodal hallucination manifests as the hypersensitivity of visual grounding to textual phrasing during the decoding phase. Building on this insight, we propose Decoding by Perturbation (DeP), a training-free framework mitigating prior-induced hallucinations via controlled textual interventions. DeP employs a dynamic probe applying multi-level textual perturbations to elicit latent language priors. Leveraging attention variance, it enhances stable evidence regions while suppressing suspicious noise in the feature space. Furthermore, it constructs an interpretable prior drift direction using logits statistics to counteract probability biases from textual co-occurrences. Extensive experiments confirm DeP effectively reduces hallucinations and achieves superior performance across multiple benchmarks.

Figures & tables

Figure 1: Illustration of multimodal hallucination driven by text hypersensitivity. Although the modified prompts preserve the core semantics of the original query, minor variations in their surface structures lead to drastically different hallucinated outputs (e.g., “fake” or “red”). As revealed by the attention maps on the right, these textual perturbations cause a severe drift in visual grounding. Notably, the generation of “red” indicates that the model has established a spurious correlation, erroneously shifting its visual attention (bottom map) to the red hoodie rather than grounding it on the glasses.
Figure 2: Overview of the DeP framework. DeP mitigates language prior-driven hallucinations during inference based on text perturbations. First, DeP utilizes attention consistency statistics across perturbations to decouple visual evidence and suspicious regions for hidden state calibration. Then, it estimates the prior drift direction from perturbed logits, applying it as a penalty to yield the final prediction.
SettingMethodLLaVA-1.5InstructBLIP
Acc ↑\uparrowPrec. ↑\uparrowRec. ↑\uparrowF1 ↑\uparrowAcc ↑\uparrowPrec. ↑\uparrowRec. ↑\uparrowF1 ↑\uparrow
AdversarialOrigin71.7 ∼\sim 0065.7 ∼\sim 0090.7 ∼\sim 0076.2 ∼\sim 0068.5 ∼\sim 0062.0 ∼\sim 0081.5 ∼\sim 0070.4 ∼\sim 00
VCD70.4 ↓\downarrow 1.31.364.5 ↓\downarrow 1.21.292.5 ↑\uparrow 1.81.876.9 ↑\uparrow 0.70.770.7 ↑\uparrow 2.22.264.9 ↑\uparrow 2.92.984.1 ↑\uparrow 2.62.673.2 ↑\uparrow 2.82.8
CICD72.0 ↑\uparrow 0.30.366.1 ↑\uparrow 0.40.492.0 ↑\uparrow 1.31.377.0 ↑\uparrow 0.80.871.2 ↑\uparrow 2.72.765.7 ↑\uparrow 3.73.785.5 ↑\uparrow 4474.3 ↑\uparrow 3.93.9
ClearSight74.3 ↑\uparrow 2.62.668.8 ↑\uparrow 3.13.191.1 ↑\uparrow 0.40.478.3 ↑\uparrow 2.12.171.6 ↑\uparrow 3.13.167.4 ↑\uparrow 5.45.483.5 ↑\uparrow 2274.6 ↑\uparrow 4.24.2
VTI74.4 ↑\uparrow 2.72.768.9 ↑\uparrow 3.23.290.9 ↑\uparrow 0.20.278.4 ↑\uparrow 2.22.273.8 ↑\uparrow 5.35.368.7 ↑\uparrow 6.76.785.4 ↑\uparrow 3.93.976.1 ↑\uparrow 5.75.7
Table 1: Performance evaluation on POPE (%). Delta denotes the improvement compared to Origin. Best results are highlighted in bold .
Table 2: Performance evaluation on MMHal-Bench. (a) General results with delta improvements compared to Origin. (b) Detailed scores on the eight categories, where “Overall” indicates the averaged performance.
Table 3: Ablation study of DeP components. (a) Results of core components on POPE and MMHal benchmarks. (b) Effectiveness of our NeffN_{\mathrm{eff}} strategy against random perturbation ratios pp on MMHal. Delta denotes the improvement compared to Origin.
Figure 3: Sensitivity analysis of perturbation count MM . (a) Score variance decreases as MM increases, indicating improved stability in visual decoupling. (b) Inference latency scales linearly with MM .
Figure 4: Sensitivity of DeP performance to β\beta across different α\alpha values with γ\gamma fixed to 0.30.3 . We report overall score (a) and hallucination rate (b) . The best-performing configuration is marked with red star . The dashed line indicates the results of the current best method (VTI).
More in this topic

VLM Hallucination Mitigation

Vision-language model hallucination mitigation keeps generated claims grounded in visual evidence.

See all 228 papers

Explore similar work

CardsList

  1. Spotlight and Shadow: Attention-Guided Dual-Anchor Introspective Decoding for MLLM Hallucination Mitigation

    Apr 11, 2026Yebo Wu, Han Jin, Zhijiang Guo +1VLM Hallucination MitigationLanguage Model Decoding

    State Key Laboratory of IOTSC, University of Macau · Hong Kong University of Science and Technology (Guangzhou)

  2. Mitigating Manifold Departure: Uncertainty-Aware Subspace Rectification for Trustworthy MLLM Decoding

    May 31, 2026Yingxuan Zhuang, Jingxiao Yang, Miao Pan +7Object Hallucination in VLMsVLM Hallucination Mitigation

    Ningbo Global Innovation Center, Zhejiang University

  3. Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation

    Feb 27, 2026Xingyu Zhu, Kesen Zhao, Liang Yi +4VLM Hallucination MitigationMultimodal Large Language Models

    MoE Key Lab of BIPC, University of Science and Technology of China · Nanyang Technological University