cs.CVMar 26, 2026
SaveSeeing to Ground: Visual Attention for Hallucination-Resilient MDLLMs
Organizations: Institute of Artificial Intelligence, University of Central Florida
Abstract
Multimodal Diffusion Large Language Models (MDLLMs) achieve high-concurrency generation through parallel masked decoding, yet the architectures remain prone to multimodal hallucinations. This structural vulnerability stems from an algorithmic flaw: the decoder ranks candidate tokens based on textual likelihood without verifying localized visual support. We establish that this language-only ranking induces an objective mismatch, where language probability mass acts as a misspecified proxy for the intended multimodal task. Consequently, we reinterpret hallucination as a localized optimization error, a phenomenon where the decoder exploits language shortcuts to maximize a proxy score at the expense of visual grounding. To address this objective mismatch, we introduce VISAGE, a training-free decoding framework that calibrates the objective at inference time. VISAGE estimates the proxy discrepancy by quantifying the spatial entropy of cross-attention distributions. By enforcing a localization consensus across attention heads, the method penalizes spatially uniform distributions and re-ranks token commitments to favor visually grounded outcomes. We provide an analytical stability guarantee establishing that VISAGE maintains a bounded objective loss under estimation error. Evaluations across hallucination-sensitive and general-purpose benchmarks demonstrate the robustness of the framework, yielding relative gains of 8.59% on MMMU-val and 7.75% on HallusionBench.
Figures & tables
Figure 1: VISAGE resolves language shortcuts by correcting the objective mismatch during parallel unmasking. Given the query “ Is there a cup in the image? ”, the standard decoder (left) assigns a high language-only confidence ( ) to the statistically plausible token “ no .” Because this proxy score lacks visual verification, the token is finalized prematurely and induces a hallucination. In contrast, VISAGE (right) calculates a visually corrected confidence ( ) by estimating the proxy discrepancy . This modification to the ranking score prevents the early commitment of the ungrounded token, allowing the model to recover the correct multimodal outcome: “ There is a cup on the deck. ”
Figure 2: The disparity between visual and linguistic probability mass across decoding steps suggests objective misspecification. We visualize the normalized peak probability mass for visual (orange) and language (blue) components across the refinement process. (a) In Visually Grounded Generation , peak visual probability mass exceeds the language attention prior before the commitment step (vertical dotted line), consistent with a localized, low-entropy distribution over image tokens. (b) In a Language Shortcut Hallucination , the visual component retains a uniform spatial distribution, resulting in a suppressed peak that falls below the language attention prior. This structural disparity demonstrates that the decoder optimizes for textual likelihood while bypassing localized visual probability mass, suggesting that the spatial Shannon entropy of cross-attention provides a detectable signal for re-ranking ungrounded commitments.
Figure 3: Overview of VISAGE during parallel masked decoding. At each decoding step, the frozen MDLLM generates candidate tokens alongside their initial confidence . To verify visual support, we extract each candidate token’s last-layer cross-attention weights over the image and compute Shannon entropy for each attention head. We then aggregate these values across heads via a -quantile operator, yielding Robust Grounding Entropy , which quantifies the concentration of visual support. A penalty multiplier is then computed. Tokens are subsequently re-ranked using the linear ranking score ( ). As illustrated, the ungrounded token ( “a” ) is penalized by high entropy, ensuring the visually supported token ( “no” ) attains a high linear ranking score and is successfully committed to the sequence.
| Method | MMMU-val (Acc % ) | HallusionBench (Acc % ) | POPE (F1 ) | MME (Score ) |
|---|---|---|---|---|
| MMaDA (Base) | 27.11 | 34.18 | 75.97 | 1383.29 |
| MMaDA + VCD | 28.44 | 34.80 | 75.85 | 1342.21 |
| MMaDA + VISAGE (Ours) | 29.44 | 36.83 | 76.17 | 1372.05 |
| Relative Gain % (Ours vs. Base) | +8.59% | +7.75% | +0.26% | -0.81% |
Table 1: VISAGE improves visual reasoning across subject-specific and diagnostic benchmarks. . MMaDA + VISAGE (Ours) consistently outperforms the baseline and VCD across multiple benchmarks. We observe substantial gains over the base model, notably on HallusionBench and on MMMU-val.
Figure 4: HallusionBench Category Analysis. Radar chart comparing VISAGE against the Baseline and VCD. Our method achieves robust improvements across illusion and spatial-reasoning ( map , figure ) categories.
Figure 5: Qualitative comparison of visually grounded yes/no questions. We compare MMaDA and our method on two examples where hallucination arises from language shortcuts. Top: MMaDA incorrectly answers “ No ” by over-relying on the textual prior that a spoon rests outside a bowl. Our method correctly grounds its “ Yes ” decision in localized visual evidence. Bottom: While both methods correctly answer “ No ”, MMaDA hallucinates irrelevant objects in its intermediate thinking trace. In contrast our method maintains strictly visually consistent descriptions.
| Penalty ( ) | MME (Score ) |
|---|---|
| 0.5 | 1320.77 |
| 0.3 | 1372.05 |
| 0.1 | 1362.68 |
Table 2: Effect of . Sensitivity analysis of the grounding penalty on MME scores.
| Aggregation | MMMU-val (Acc% ) |
|---|---|
| Min over Heads | 28.56 |
| Mean over Heads | 28.78 |
| (Ours) | 29.44 |
Table 3: Ablative Study on head aggregation. Quantile aggregation improves robustness over mean pooling.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1 : Qualitative comparison on HallusionBench. We present MMaDA (baseline) and VISAGE generations across three deceptive visual contexts. Top row: A perspective illusion where MMaDA hallucinates a height difference, while VISAGE correctly anchors its prediction on the actual silhouettes. Middle row: A color illusion where the baseline misinterprets uniform circles as distinct colors due to the background patterns. Bottom row: A geometric deception where MMaDA relies on a linguistic shortcut about checkerboards, whereas VISAGE correctly grounds its answer in the localized visual arrangement. In all cases, VISAGE successfully aligns with the ground truth by suppressing ungrounded language priors.
Figure A.2 : Qualitative comparison on MMMU-val. We present MMaDA (baseline) and VISAGE generations on MMMU-val Benchmark. Top row: A visual motif recognition task where MMaDA incorrectly classifies a Lord of the Rings poster as a generic “Fable,” while VISAGE correctly identifies it as a “Quest Story”. Middle row: A reading comprehension task where the baseline selects a loosely related thematic option for a Langston Hughes poem, whereas VISAGE maintains strict alignment with the text to extract the direct instructional message. Bottom row: A historical reasoning task where MMaDA hallucinates a bench and defaults to a broad regional prior (patriarchal society) rather than analyzing the specific visual and contextual cues that indicate Japanese women exercised societal influence during this period. In all three examples, VISAGE successfully anchors its reasoning in the provided evidence, avoiding the shallow linguistic shortcuts that mislead the baseline.
Figure A.3 : Ground-truth label noise in POPE. We present examples where VISAGE correctly grounds its generation in the visual evidence but is penalized as incorrect due to flawed or missing annotations inherited from the MS COCO dataset. Top row: The ground truth falsely asserts a toothbrush is present. MMaDA hallucinates the object to satisfy the prompt, while VISAGE correctly refutes its existence. Middle row: The ground truth loosely labels a white van as a “truck.” VISAGE precisely identifies the vehicle as a van and rejects the generic truck label, whereas the baseline complies. Bottom row: The ground truth fails to account for unannotated bottles resting on the shower shelf. VISAGE successfully discovers them through rigorous spatial grounding. These instances demonstrate issues with POPE’s annotation quality, thereby capping the measurable relative gains on the benchmark.
Figure A.4 : Temporal failure cases. We present examples where both the baseline (MMaDA) and VISAGE struggle to evaluate out-of-order video frames. Top row: Given a shuffled sequence of a fighter moving around in the fighting ring, both models erroneously confirm the chronological order, failing to capture the true motion dynamics. However, the baseline changes it’s answer to “Yes” in the end. Bottom row: For a shuffled sequence of baseball players hugging, VISAGE again fails to identify the temporal inconsistency. Notably, while the baseline outputs the correct final answer (“NO”), its intermediate reasoning reveals hallucinations. It falsely claims a non-existent text overlay that provided the answer. These cases demonstrate that without explicit spatio-temporal modeling, both methods remain vulnerable to temporal reasoning failures, even when the baseline occasionally outputs the correct label by chance.
More in this topic
Language Model Decoding
Language model decoding studies how models choose and generate outputs, balancing quality, reliability, diversity, and inference cost.
See all 168 papers