Multimodal Diffusion Large Language Models (MDLLMs) achieve high-concurrency generation through parallel masked decoding, yet the architectures remain prone to multimodal hallucinations. This structural vulnerability stems from an algorithmic flaw: the decoder ranks candidate tokens based on textual likelihood without verifying localized visual support. We establish that this language-only ranking induces an objective mismatch, where language probability mass acts as a misspecified proxy for the intended multimodal task. Consequently, we reinterpret hallucination as a localized optimization error, a phenomenon where the decoder exploits language shortcuts to maximize a proxy score at the expense of visual grounding. To address this objective mismatch, we introduce VISAGE, a training-free decoding framework that calibrates the objective at inference time. VISAGE estimates the proxy discrepancy by quantifying the spatial entropy of cross-attention distributions. By enforcing a localization consensus across attention heads, the method penalizes spatially uniform distributions and re-ranks token commitments to favor visually grounded outcomes. We provide an analytical stability guarantee establishing that VISAGE maintains a bounded objective loss under estimation error. Evaluations across hallucination-sensitive and general-purpose benchmarks demonstrate the robustness of the framework, yielding relative gains of 8.59% on MMMU-val and 7.75% on HallusionBench.
Figures & tables
Figure 1: VISAGE resolves language shortcuts by correcting the objective mismatch during parallel unmasking. Given the query “ Is there a cup in the image? ”, the standard decoder (left) assigns a high language-only confidence ( 0.9 ) to the statistically plausible token “ no .” Because this proxy score lacks visual verification, the token is finalized prematurely and induces a hallucination. In contrast, VISAGE (right) calculates a visually corrected confidence ( 0.6 ) by estimating the proxy discrepancy bi . This modification to the ranking score prevents the early commitment of the ungrounded token, allowing the model to recover the correct multimodal outcome: “ There is a cup on the deck. ”
Figure 2: The disparity between visual and linguistic probability mass across decoding steps suggests objective misspecification. We visualize the normalized peak probability mass for visual (orange) and language (blue) components across the refinement process. (a) In Visually Grounded Generation , peak visual probability mass exceeds the language attention prior before the commitment step (vertical dotted line), consistent with a localized, low-entropy distribution over image tokens. (b) In a Language Shortcut Hallucination , the visual component retains a uniform spatial distribution, resulting in a suppressed peak that falls below the language attention prior. This structural disparity demonstrates that the decoder optimizes for textual likelihood while bypassing localized visual probability mass, suggesting that the spatial Shannon entropy of cross-attention provides a detectable signal for re-ranking ungrounded commitments.
Figure 3: Overview of VISAGE during parallel masked decoding. At each decoding step, the frozen MDLLM generates candidate tokens alongside their initial confidence c . To verify visual support, we extract each candidate token’s last-layer cross-attention weights over the image and compute Shannon entropy for each attention head. We then aggregate these values across heads via a β -quantile operator, yielding Robust Grounding Entropy H , which quantifies the concentration of visual support. A penalty multiplier g=1/(1+H) is then computed. Tokens are subsequently re-ranked using the linear ranking score u=c⋅gα ( α=0.5 ). As illustrated, the ungrounded token ( “a” ) is penalized by high entropy, ensuring the visually supported token ( “no” ) attains a high linear ranking score and is successfully committed to the sequence.
Method
MMMU-val (Acc % ↑ )
HallusionBench (Acc % ↑ )
POPE (F1 ↑ )
MME (Score ↑ )
MMaDA (Base)
27.11
34.18
75.97
1383.29
MMaDA + VCD
28.44
34.80
75.85
1342.21
MMaDA + VISAGE (Ours)
29.44
36.83
76.17
1372.05
Relative Gain % (Ours vs. Base)
+8.59%
+7.75%
+0.26%
-0.81%
Table 1: VISAGE improves visual reasoning across subject-specific and diagnostic benchmarks. . MMaDA + VISAGE (Ours) consistently outperforms the baseline and VCD across multiple benchmarks. We observe substantial gains over the base model, notably +2.65 on HallusionBench and +2.33% on MMMU-val.
Figure 4: HallusionBench Category Analysis. Radar chart comparing VISAGE against the Baseline and VCD. Our method achieves robust improvements across illusion and spatial-reasoning ( map , figure ) categories.
Figure 5: Qualitative comparison of visually grounded yes/no questions. We compare MMaDA and our method on two examples where hallucination arises from language shortcuts. Top: MMaDA incorrectly answers “ No ” by over-relying on the textual prior that a spoon rests outside a bowl. Our method correctly grounds its “ Yes ” decision in localized visual evidence. Bottom: While both methods correctly answer “ No ”, MMaDA hallucinates irrelevant objects in its intermediate thinking trace. In contrast our method maintains strictly visually consistent descriptions.
Penalty ( α )
MME (Score ↑ )
0.5
1320.77
0.3
1372.05
0.1
1362.68
Table 2: Effect of α . Sensitivity analysis of the grounding penalty on MME scores.
Aggregation
MMMU-val (Acc% ↑ )
Min over Heads
28.56
Mean over Heads
28.78
qβ (Ours)
29.44
Table 3: Ablative Study on head aggregation. Quantile aggregation improves robustness over mean pooling.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1 : Qualitative comparison on HallusionBench. We present MMaDA (baseline) and VISAGE generations across three deceptive visual contexts. Top row: A perspective illusion where MMaDA hallucinates a height difference, while VISAGE correctly anchors its prediction on the actual silhouettes. Middle row: A color illusion where the baseline misinterprets uniform circles as distinct colors due to the background patterns. Bottom row: A geometric deception where MMaDA relies on a linguistic shortcut about checkerboards, whereas VISAGE correctly grounds its answer in the localized visual arrangement. In all cases, VISAGE successfully aligns with the ground truth by suppressing ungrounded language priors.
Figure A.2 : Qualitative comparison on MMMU-val. We present MMaDA (baseline) and VISAGE generations on MMMU-val Benchmark. Top row: A visual motif recognition task where MMaDA incorrectly classifies a Lord of the Rings poster as a generic “Fable,” while VISAGE correctly identifies it as a “Quest Story”. Middle row: A reading comprehension task where the baseline selects a loosely related thematic option for a Langston Hughes poem, whereas VISAGE maintains strict alignment with the text to extract the direct instructional message. Bottom row: A historical reasoning task where MMaDA hallucinates a bench and defaults to a broad regional prior (patriarchal society) rather than analyzing the specific visual and contextual cues that indicate Japanese women exercised societal influence during this period. In all three examples, VISAGE successfully anchors its reasoning in the provided evidence, avoiding the shallow linguistic shortcuts that mislead the baseline.
Figure A.3 : Ground-truth label noise in POPE. We present examples where VISAGE correctly grounds its generation in the visual evidence but is penalized as incorrect due to flawed or missing annotations inherited from the MS COCO dataset. Top row: The ground truth falsely asserts a toothbrush is present. MMaDA hallucinates the object to satisfy the prompt, while VISAGE correctly refutes its existence. Middle row: The ground truth loosely labels a white van as a “truck.” VISAGE precisely identifies the vehicle as a van and rejects the generic truck label, whereas the baseline complies. Bottom row: The ground truth fails to account for unannotated bottles resting on the shower shelf. VISAGE successfully discovers them through rigorous spatial grounding. These instances demonstrate issues with POPE’s annotation quality, thereby capping the measurable relative gains on the benchmark.
Figure A.4 : Temporal failure cases. We present examples where both the baseline (MMaDA) and VISAGE struggle to evaluate out-of-order video frames. Top row: Given a shuffled sequence of a fighter moving around in the fighting ring, both models erroneously confirm the chronological order, failing to capture the true motion dynamics. However, the baseline changes it’s answer to “Yes” in the end. Bottom row: For a shuffled sequence of baseball players hugging, VISAGE again fails to identify the temporal inconsistency. Notably, while the baseline outputs the correct final answer (“NO”), its intermediate reasoning reveals hallucinations. It falsely claims a non-existent text overlay that provided the answer. These cases demonstrate that without explicit spatio-temporal modeling, both methods remain vulnerable to temporal reasoning failures, even when the baseline occasionally outputs the correct label by chance.
Multimodal large language models (MLLMs) have become a key interface for visual reasoning and grounded question answering, yet they remain vulnerable to visual hallucinations, where generated responses contradict image content or mention nonexistent objects. A central challenge is that hallucination is not always caused by a simple lack of visual attention: the model may still assign substantial attention mass to image tokens while internally drifting toward an incorrect answer. In this paper, we show that the high-frequency structure of visual attention, measured by layer-wise Laplacian energy, reveals both the layer where hallucinated preferences emerge and the layer where the ground-truth answer transiently recovers. Building on this finding, we propose LaSCD (Laplacian-Spectral Contrastive Decoding), a training-free decoding strategy that selects informative layers via Laplacian energy and remaps next-token logits in closed form. Experiments on hallucination and general multimodal benchmarks show that LaSCD consistently reduces hallucination while preserving general capabilities, highlighting its potential as a faithful decoding paradigm. The code is available at https://github.com/macovaseas/LaSCD.
Fanpu Cao, Xin Zou, Xuming Hu +1
Thrust of Artificial Intelligence, HKUST (Guangzhou), China · Department of Computer Science and Engineering, HKUST, Hong Kong SAR, China
Large vision-language models (VLMs) frequently suffer from hallucinations, generating content that is inconsistent with visual inputs. Existing methods typically address this problem through post-hoc filtering, additional training objectives, or external verification, but they do not intervene during the decoding process when hallucinations arise. In this work, we introduce SAGE, a Sink-Aware Grounded Decoding framework that mitigates hallucinations by dynamically modulating self-attention during generation. Hallucinations are strongly correlated with attention sink tokens - punctuation or function tokens that accumulate disproportionate attention despite carrying limited semantic content. SAGE leverages these tokens as anchors to monitor grounding reliability in real time. At each sink trigger, the method extracts semantic concepts from the generated sequence, estimates their visual grounding using both self-attention maps and gradient-based attribution, and measures their spatial agreement. Based on this signal, self-attention distributions are adaptively sharpened or broadened to reinforce grounded regions or suppress unreliable ones. Extensive experiments across diverse hallucination benchmarks demonstrate that SAGE consistently outperforms existing decoding strategies, achieving substantial reductions in hallucination while preserving descriptive coverage, without requiring model retraining or architectural modifications. Our method achieves an average relative improvement of 10.65% on MSCOCO and 7.19% on AMBER across diverse VLM architectures, demonstrating consistent gains in hallucination mitigation.
Human visual reasoning typically follows a coarse-to-fine attention process, starting from global scene understanding and gradually focusing on question-relevant regions. However, multimodal large language models may deviate from this pattern due to attention drift and the underutilization of visual evidence, which can lead to hallucinations. To mitigate these issues, this study proposes a Dual-Indicator Guided Contrastive Alignment (DICA), which tracks two information-theoretic indicators during inference: Visual Attention Entropy (VAE), which reflects the concentration of visual attention, and Output Image Correlation (OIC), which measures the dependence of generated outputs on the visual input. An abnormal increase in VAE or a decrease in OIC corresponds to different failure modes, which trigger targeted contrastive alignment to restore visual grounding. Experimental results across multiple benchmarks demonstrate that DICA consistently outperforms existing approaches and substantially reduces hallucinations, highlighting the effectiveness of indicator-driven intervention in improving multimodal inference reliability. The code is publicly available at https://github.com/BGWH123/DICA/.
Hao Yang, Jin Wang, Xuejie Zhang
School of Information Science and Engineering Yunnan University Kunming, China