Large Vision-Language Models (LVLMs) have achieved impressive progress in multimodal reasoning, yet they remain prone to object hallucinations, generating descriptions of objects that are not present in the input image. Recent approaches attempt to mitigate hallucinations by suppressing unreliable visual signals in the vision encoder, but many rely on iterative optimization for each input, resulting in substantial inference latency. In this work, we investigate the internal attention dynamics of vision encoders in LVLMs and identify a consistent three-phase structure of visual information processing: diffusion, focus, and rediffusion. Our analysis reveals that hallucination behavior is particularly sensitive to tokens receiving low attention during the focus phase. Motivated by this observation, we propose a lightweight inference-time intervention that selectively suppresses such tokens during the focus phase. The method operates in a training-free manner using statistics from a single forward pass and employs a Determinantal Point Process (DPP) to preserve diverse visual cues while filtering redundant tokens. Extensive experiments across multiple LVLM backbones and decoding strategies demonstrate that the proposed approach consistently reduces hallucination metrics while maintaining competitive caption quality. Moreover, compared to adversarial uncertainty estimation methods, our approach achieves comparable hallucination mitigation with negligible additional inference latency.
Figures & tables
Figure 1: Runtime comparison and hallucination mitigation performance (CHAIR) across recent LVLMs.
Figure 2: Layer-wise attention dynamics across various LVLM backbones. (a) The progression of the maximum attention score to entropy ratio ( R(l) ) across vision encoder layers. (b) Visualization of attention maps, demonstrating a consistent three-phase visual processing structure: diffusion, focus, and rediffusion.
Figure 3: Impact of masking strategies across different processing phases on hallucination metrics. Masking visual tokens during the focus phase (Mask 2) effectively reduces hallucinations (indicated by lower CHAIRS (CS) and CHAIRI (CI) scores) while preserving object recognition capabilities (F1 score), highlighting the focus phase as the effective intervention point.
Figure 4: Qualitative examples of hallucination behavior across masking phases. GT captions are shown for reference, and hallucinated statements are highlighted in red. While no masking or masking in the diffusion (Phase 1) and rediffusion (Phase 3) phases produces inconsistent results, masking in the focus phase (Phase 2) consistently reduces hallucinations and yields captions more consistent with the image content.
Figure 5: Visual Attention Ratio analysis under different masking conditions. Left: Distribution of image-level mean VAR across masking settings. Masking in the focus phase(Phase 2) yields a significantly higher VAR compared to the baseline (No masking) ( p<0.001 ). Right: Layer-head VAR heatmaps showing increased visual attention in intermediate layers when DPP masking is applied.
Greedy
OPERA
VCD
PAI
Devils
Model
Bench
Metric
Orig.
AUE
Ours ( Δ )
Orig.
AUE
Ours ( Δ )
Orig.
AUE
Ours ( Δ )
Orig.
AUE
Ours ( Δ )
Orig.
AUE
Ours ( Δ )
LLaVA-1.5-7b
CHAIR
CHAIRS↓
45.0
30.2
28.8( -1.4 )
44.2
29.8
31.6( +1.8 )
46.2
34.2
35.6( +1.4 )
32.4
19.4
17.4( -2.0 )
23.2
13.0
12.8( -0.2 )
CHAIRI↓
13.3
10.4
10.2( -0.2 )
13.5
11.2
11.2( 0.0 )
14.9
11.8
12.5( +0.7 )
9.6
7.6
6.5( -1.1 )
7.9
7.2
6.3( -0.9 )
F1↑
74.2
71.8
72.0( +0.2 )
74.4
72.3
71.4( -0.9 )
71.3
70.5
70.4( -0.1 )
74.6
67.7
68.0( +0.3 )
72.0
63.5
65.1( +1.6 )
POPE
ran.↑
87.4
85.6
86.4( +0.8 )
86.3
84.5
85.2( +0.7 )
83.7
78.9
80.6( +1.7 )
87.4
85.8
86.2( +0.4 )
87.7
84.5
85.4( +0.9 )
pop.↑
84.3
83.0
83.1( +0.1 )
83.3
81.9
81.5( -0.4 )
80.9
77.1
77.5( +0.4 )
84.5
83.2
83.0( -0.2 )
85.3
83.0
83.2( +0.2 )
Table 1: Results on CHAIR and POPE benchmark. Comparison of Origin (baseline), AUE, and Ours across decoding strategies. Numbers in parentheses in the ours columns denote the AUE-relative delta Δ=ours−AUE . For CHAIRS/CHAIRI (lower is better), negative Δ indicates improvement and is highlighted in blue. For F1 and POPE (higher is better), positive Δ indicates improvement and is highlighted in blue.
Figure 6: Per-sample latency (seconds). Vision encoder time vs total time for Orig./AUE/Ours.
Qwen-2.5-VL
Intern-VL-2.5
Bench
Metric
Orig.
AUE
Ours
Orig.
AUE
Ours
CHAIR
CHAIRS↓
31.4
27.8
27.8
28.6
28.4
27.0
CHAIRI↓
7.6
7.4
7.3
7.1
7.0
6.8
F1↑
75.5
75.8
74.7
76.3
76.6
76.1
POPE
ran.↑
81.3
79.6
80.3
94.27
94.17
94.13
pop.↑
80.8
79.2
79.9
88.68
88.53
88.65
Table 2: Greedy decoding results (CHAIR/POPE).
Figure 7: Qualitative examples on the CHAIR dataset. Our method reduces hallucinated object mentions compared to vanilla LVLMs and AUE.
Figure 8: Comparison of masking patterns between AUE and our DPP-based method.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Layer-wise attention dynamics across evaluated LVLM backbones. The solid lines indicate the ratio of maximum attention score to entropy ( R(l) ). The shaded regions represent the identified Diffusion (yellow), Focus (red), and Rediffusion (purple) phases based on our gradient-based boundary formulation.
Figure 10: Qualitative examples of continuous token modulation in the focus phase. Increasing the influence of low-attention tokens introduces hallucinated objects in the generated descriptions (highlighted in red), while suppressing their influence reduces hallucination and produces captions more consistent with the visual content.
Figure 11: Additional qualitative examples demonstrating the effect of token influence modulation across diverse scenes. Strengthening the influence of low-attention tokens tends to introduce hallucinated objects or relationships, whereas suppressing their influence produces descriptions that better align with the visual evidence.
Figure 12: VAR analysis on LLaVA-1.5-13B. Left: Distribution of image-level mean VAR across masking strategies. Focus-phase masking (Phase 2) produces a significantly higher VAR than the baseline (no masking), indicating increased reliance on visual tokens during generation. Right: Layer-head VAR heatmaps comparing DPP masking and no masking. Applying focus-phase masking increases visual attention across intermediate decoder layers.
Figure 13: VAR analysis on Shikra-7B. Left: Distribution of image-level mean VAR under different masking phases. Focus-phase masking consistently yields higher VAR compared to the baseline. Right: Layer-head VAR heatmaps illustrating the increase in visual attention when DPP masking is applied during the focus phase.
Origin
AUE [ 20 ]
Ours
Model
Enc. Time
Total
Enc. Time
Total
Enc. Time
Total
LLaVA-1.5
0.008
3.207
4.588 ( +4.580 )
7.307 ( +4.100 )
0.198 ( +0.190 )
3.182 ( -0.025 )
Shikra
0.006
3.267
4.586 ( +4.580 )
7.677 ( +4.410 )
0.188 ( +0.182 )
3.267 ( +0.000 )
Qwen-2.5-VL
0.074
2.598
31.244 ( +31.170 )
33.456 ( +30.858 )
0.286 ( +0.212 )
2.813 ( +0.215 )
InternVL2.5
0.047
2.299
38.768 ( +38.721 )
41.291 ( +38.992 )
0.259 ( +0.212 )
2.722 ( +0.423 )
Appendix
Table 3: Inference time per sample (seconds). Values in parentheses indicate the change relative to the origin baseline. Our DPP-based masking introduces only a small overhead, while AUE significantly increases latency.
Figure 14: Sentence-level qualitative analysis using Ground-Truth (GT) captions. Focus-phase masking (Phase 2) consistently mitigates hallucinated statements compared to No masking or interventions in Phase 1 and 3.
Figure 15: Additional qualitative examples using Ground-Truth (GT) captions. The comparisons highlight that applying the intervention strictly to the focus phase is crucial for ensuring factual alignment without degrading the overall sentence structure.
Figure 16: Extended qualitative results on the CHAIR benchmark using LLaVA-1.5-7B.
Figure 17: Extended qualitative results on the CHAIR benchmark using LLaVA-1.5-13B.
Figure 18: Extended qualitative results on the CHAIR benchmark using Shikra-7B.
Figure 19: Extended qualitative results on the CHAIR benchmark using Qwen-2.5-VL.
Figure 20: Extended qualitative results on the CHAIR benchmark using InternVL-2.5.
Figure 21: Extended qualitative results on the POPE benchmark using LLaVA-1.5-7B.
Figure 22: Extended qualitative results on the POPE benchmark using LLaVA-1.5-13B.
Figure 23: Extended qualitative results on the POPE benchmark using Shikra-7B.
Figure 24: Extended qualitative results on the POPE benchmark using Qwen-2.5-VL.
Figure 25: Extended qualitative results on the POPE benchmark using InternVL-2.5.
Figure 26: Spatial mask comparisons between our DPP-based method and the AUE method under similar masked-token budgets. Our method produces a more spatially dispersed pattern, effectively preserving diverse visual cues while filtering redundancy.
LLaVA-1.5-7B
Shikra-7B
Bench
Metric
Orig.
Ours
Orig.
Ours
Gen.
CHAIR ↓
8.2
6.2
12.2
11.5
Cover ↑
48.5
46.2
48.9
47.5
Hal ↓
34.9
24.7
49.7
44.8
Cog ↓
3.5
2.4
5.6
4.6
Disc.
Acc. ↑
70.0
71.1
73.5
73.3
Appendix
Table 4: Evaluation results on the AMBER dataset. The best result in each row is highlighted in bold.
Despite the impressive capabilities of Large Vision-Language Models (LVLMs), they remain susceptible to hallucination, generating content inconsistent with the input image. Recent studies attribute this to the dominance of language priors over visual inputs and employ contrastive decoding methods to mitigate this dominance, but the mechanistic origin remains unexplored. We investigate the information flow through each transformer layer and find that attention modules consistently aggregate visual evidence, while FFN modules at critical layers act as the source of language priors. These priors can override visual evidence, causing correct predictions in intermediate layers to drift toward incorrect outputs. Based on this insight, we propose FADE (FFN Attenuation for DEcoding), a training-free method that attenuates FFN outputs to reduce language-prior dominance. Evaluations on POPE, CHAIR, and MME benchmarks across LLaVA-1.5, mPLUG-Owl2, and InstructBLIP show that FADE effectively mitigates hallucinations while preserving inference efficiency.
Yichen Guo, Kai Tang, Fenglai Lin +5
1Nanyang Technological University · 2State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · 3Tsinghua University
The generation of factually incorrect objects, commonly known as object hallucination, remains a persistent challenge in Large Vision-Language Models (LVLMs). Current approaches to address this issue - ranging from expensive data-driven fine-tuning and high-latency contrastive decoding to rigid attention head truncation - frequently compromise either computational efficiency or the continuity of the model's feature space. To overcome these limitations, we introduce a novel, training-free inference strategy that operates as a region-aware adaptive weighting mechanism to dynamically correct semantic drift without relying on abrupt heuristic truncations. By computing an outlier-resistant statistical midpoint across various attention heads, we establish a stable anchor for reliable visual representations. We then utilize the inter-head disagreement mapped across regions to dynamically determine intervention budgets, gently suppressing hallucination-inducing attention paths through a continuous penalty modulation. This recalibration process effectively rectifies visual-semantic misalignments while fully preserving generative fluency and language priors. Comprehensive evaluations on standard multimodal benchmarks, including CHAIR, POPE, and MME, reveal that our strategy substantially curtails both instance- and sentence-level hallucinations. The results demonstrate state-of-the-art performance against contemporary baselines, confirming our method's efficiency and algorithmic robustness. Our code will be public.
Yuanzhi Xu, Qian Gao, Jun Fan +4
Qilu University of Technology (Shandong Academy of Sciences) Jinan, China · China Telecom Digital Intelligence Technology Co, Ltd Jinan, China · Shenyang Aerospace University Shenyang, China +1
Large Vision-Language Models (LVLMs) excel at multimodal tasks but remain prone to object hallucinations. Prior training-free remedies often uniformly strengthen visual signals, which may also amplify irrelevant regions and introduce spurious evidence, harming fluency. We propose Context-aware Attention Intervention (CAI), a training-free inference-time mechanism that enforces a see only when needed principle via two-axis selectivity: where to look and when to intervene. At each decoding step, CAI derives token-specific visual relevance from early-layer representations to localize semantically aligned regions, and applies a conservative, entropy- and depth-gated attention tilt only for uncertainty-spiking tokens in deeper layers where visual grounding degrades, leaving confident tokens and irrelevant regions largely unchanged. This targeted intervention strengthens visual grounding while preserving linguistic fluency, and it yields consistent improvements even without contrastive decoding, which remains optional as an auxiliary bias-suppression module. Extensive experiments across multiple LVLM backbones and benchmarks show that CAI achieves state-of-the-art hallucination mitigation, and our analysis characterizes CAI as a KL-minimal attention reweighting with bounded interference under inactive gates or small tilts. Code is available at https://github.com/Iris1946/CAI.
Yuqing Lei, Wenbo Lyu, Yingjun Du +3
University of Chinese Academy of Sciences · University of Amsterdam · United Imaging Healthcare Co., Ltd.