Focus Matters: Phase-Aware Suppression for Hallucination in Vision-Language Models

Sohyeon Kim; Sang Yeon Yoon; Kyeongbo Kong

cs.CVApr 4, 2026

Focus Matters: Phase-Aware Suppression for Hallucination in Vision-Language Models

Authors: Sohyeon Kim, Sang Yeon Yoon, Kyeongbo Kong

Organizations: Pusan National University · Pukyong National University

Abstract

Large Vision-Language Models (LVLMs) have achieved impressive progress in multimodal reasoning, yet they remain prone to object hallucinations, generating descriptions of objects that are not present in the input image. Recent approaches attempt to mitigate hallucinations by suppressing unreliable visual signals in the vision encoder, but many rely on iterative optimization for each input, resulting in substantial inference latency. In this work, we investigate the internal attention dynamics of vision encoders in LVLMs and identify a consistent three-phase structure of visual information processing: diffusion, focus, and rediffusion. Our analysis reveals that hallucination behavior is particularly sensitive to tokens receiving low attention during the focus phase. Motivated by this observation, we propose a lightweight inference-time intervention that selectively suppresses such tokens during the focus phase. The method operates in a training-free manner using statistics from a single forward pass and employs a Determinantal Point Process (DPP) to preserve diverse visual cues while filtering redundant tokens. Extensive experiments across multiple LVLM backbones and decoding strategies demonstrate that the proposed approach consistently reduces hallucination metrics while maintaining competitive caption quality. Moreover, compared to adversarial uncertainty estimation methods, our approach achieves comparable hallucination mitigation with negligible additional inference latency.

Figures & tables

Figure 1: Runtime comparison and hallucination mitigation performance (CHAIR) across recent LVLMs.

Figure 2: Layer-wise attention dynamics across various LVLM backbones. (a) The progression of the maximum attention score to entropy ratio ( $R^{(l)}$ ) across vision encoder layers. (b) Visualization of attention maps, demonstrating a consistent three-phase visual processing structure: diffusion, focus, and rediffusion.

Figure 3: Impact of masking strategies across different processing phases on hallucination metrics. Masking visual tokens during the focus phase (Mask 2) effectively reduces hallucinations (indicated by lower $\mathrm{CHAIR}_{S}$ (CS) and $\mathrm{CHAIR}_{I}$ (CI) scores) while preserving object recognition capabilities (F1 score), highlighting the focus phase as the effective intervention point.

Figure 4: Qualitative examples of hallucination behavior across masking phases. GT captions are shown for reference, and hallucinated statements are highlighted in red. While no masking or masking in the diffusion (Phase 1) and rediffusion (Phase 3) phases produces inconsistent results, masking in the focus phase (Phase 2) consistently reduces hallucinations and yields captions more consistent with the image content.

Figure 5: Visual Attention Ratio analysis under different masking conditions. Left: Distribution of image-level mean VAR across masking settings. Masking in the focus phase(Phase 2) yields a significantly higher VAR compared to the baseline (No masking) ( $p<0.001$ ). Right: Layer-head VAR heatmaps showing increased visual attention in intermediate layers when DPP masking is applied.

			Greedy			OPERA			VCD			PAI			Devils
Model	Bench	Metric	Orig.	AUE	Ours ( $\Delta$ )	Orig.	AUE	Ours ( $\Delta$ )	Orig.	AUE	Ours ( $\Delta$ )	Orig.	AUE	Ours ( $\Delta$ )	Orig.	AUE	Ours ( $\Delta$ )
LLaVA-1.5-7b	CHAIR	$\mathrm{CHAIR}_{S}\downarrow$	45.0	30.2	28.8( -1.4 )	44.2	29.8	31.6( +1.8 )	46.2	34.2	35.6( +1.4 )	32.4	19.4	17.4( -2.0 )	23.2	13.0	12.8( -0.2 )
		$\mathrm{CHAIR}_{I}\downarrow$	13.3	10.4	10.2( -0.2 )	13.5	11.2	11.2( 0.0 )	14.9	11.8	12.5( +0.7 )	9.6	7.6	6.5( -1.1 )	7.9	7.2	6.3( -0.9 )
		$\mathrm{F1}\uparrow$	74.2	71.8	72.0( +0.2 )	74.4	72.3	71.4( -0.9 )	71.3	70.5	70.4( -0.1 )	74.6	67.7	68.0( +0.3 )	72.0	63.5	65.1( +1.6 )
	POPE	$\mathrm{ran.}\uparrow$	87.4	85.6	86.4( +0.8 )	86.3	84.5	85.2( +0.7 )	83.7	78.9	80.6( +1.7 )	87.4	85.8	86.2( +0.4 )	87.7	84.5	85.4( +0.9 )
		$\mathrm{pop.}\uparrow$	84.3	83.0	83.1( +0.1 )	83.3	81.9	81.5( -0.4 )	80.9	77.1	77.5( +0.4 )	84.5	83.2	83.0( -0.2 )	85.3	83.0	83.2( +0.2 )

Table 1: Results on CHAIR and POPE benchmark. Comparison of Origin (baseline), AUE, and Ours across decoding strategies. Numbers in parentheses in the ours columns denote the AUE-relative delta

\Delta=\text{ours}-\text{AUE}

. For

\mathrm{CHAIR}_{S}/\mathrm{CHAIR}_{I}

(lower is better), negative

\Delta

indicates improvement and is highlighted in blue. For F1 and POPE (higher is better), positive

\Delta

indicates improvement and is highlighted in blue.

Figure 6: Per-sample latency (seconds). Vision encoder time vs total time for Orig./AUE/Ours.

		Qwen-2.5-VL			Intern-VL-2.5
Bench	Metric	Orig.	AUE	Ours	Orig.	AUE	Ours
CHAIR	$\mathrm{CHAIR}_{S}\downarrow$	31.4	27.8	27.8	28.6	28.4	27.0
	$\mathrm{CHAIR}_{I}\downarrow$	7.6	7.4	7.3	7.1	7.0	6.8
	$\mathrm{F1}\uparrow$	75.5	75.8	74.7	76.3	76.6	76.1
POPE	$\mathrm{ran.}\uparrow$	81.3	79.6	80.3	94.27	94.17	94.13
	$\mathrm{pop.}\uparrow$	80.8	79.2	79.9	88.68	88.53	88.65

Table 2: Greedy decoding results (CHAIR/POPE).

Figure 7: Qualitative examples on the CHAIR dataset. Our method reduces hallucinated object mentions compared to vanilla LVLMs and AUE.

Figure 8: Comparison of masking patterns between AUE and our DPP-based method.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Layer-wise attention dynamics across evaluated LVLM backbones. The solid lines indicate the ratio of maximum attention score to entropy ( $R^{(l)}$ ). The shaded regions represent the identified Diffusion (yellow), Focus (red), and Rediffusion (purple) phases based on our gradient-based boundary formulation.

Figure 10: Qualitative examples of continuous token modulation in the focus phase. Increasing the influence of low-attention tokens introduces hallucinated objects in the generated descriptions (highlighted in red), while suppressing their influence reduces hallucination and produces captions more consistent with the visual content.

Figure 11: Additional qualitative examples demonstrating the effect of token influence modulation across diverse scenes. Strengthening the influence of low-attention tokens tends to introduce hallucinated objects or relationships, whereas suppressing their influence produces descriptions that better align with the visual evidence.

Figure 12: VAR analysis on LLaVA-1.5-13B. Left: Distribution of image-level mean VAR across masking strategies. Focus-phase masking (Phase 2) produces a significantly higher VAR than the baseline (no masking), indicating increased reliance on visual tokens during generation. Right: Layer-head VAR heatmaps comparing DPP masking and no masking. Applying focus-phase masking increases visual attention across intermediate decoder layers.

Figure 13: VAR analysis on Shikra-7B. Left: Distribution of image-level mean VAR under different masking phases. Focus-phase masking consistently yields higher VAR compared to the baseline. Right: Layer-head VAR heatmaps illustrating the increase in visual attention when DPP masking is applied during the focus phase.

	Origin		AUE [ 20 ]		Ours
Model	Enc. Time	Total	Enc. Time	Total	Enc. Time	Total
LLaVA-1.5	0.008	3.207	4.588 ( +4.580 )	7.307 ( +4.100 )	0.198 ( +0.190 )	3.182 ( -0.025 )
Shikra	0.006	3.267	4.586 ( +4.580 )	7.677 ( +4.410 )	0.188 ( +0.182 )	3.267 ( +0.000 )
Qwen-2.5-VL	0.074	2.598	31.244 ( +31.170 )	33.456 ( +30.858 )	0.286 ( +0.212 )	2.813 ( +0.215 )
InternVL2.5	0.047	2.299	38.768 ( +38.721 )	41.291 ( +38.992 )	0.259 ( +0.212 )	2.722 ( +0.423 )

Appendix

Table 3: Inference time per sample (seconds). Values in parentheses indicate the change relative to the origin baseline. Our DPP-based masking introduces only a small overhead, while AUE significantly increases latency.

Figure 14: Sentence-level qualitative analysis using Ground-Truth (GT) captions. Focus-phase masking (Phase 2) consistently mitigates hallucinated statements compared to No masking or interventions in Phase 1 and 3.

Figure 15: Additional qualitative examples using Ground-Truth (GT) captions. The comparisons highlight that applying the intervention strictly to the focus phase is crucial for ensuring factual alignment without degrading the overall sentence structure.

Figure 16: Extended qualitative results on the CHAIR benchmark using LLaVA-1.5-7B.

Figure 17: Extended qualitative results on the CHAIR benchmark using LLaVA-1.5-13B.

Figure 18: Extended qualitative results on the CHAIR benchmark using Shikra-7B.

Figure 19: Extended qualitative results on the CHAIR benchmark using Qwen-2.5-VL.

Figure 20: Extended qualitative results on the CHAIR benchmark using InternVL-2.5.

Figure 21: Extended qualitative results on the POPE benchmark using LLaVA-1.5-7B.

Figure 22: Extended qualitative results on the POPE benchmark using LLaVA-1.5-13B.

Figure 23: Extended qualitative results on the POPE benchmark using Shikra-7B.

Figure 24: Extended qualitative results on the POPE benchmark using Qwen-2.5-VL.

Figure 25: Extended qualitative results on the POPE benchmark using InternVL-2.5.

Figure 26: Spatial mask comparisons between our DPP-based method and the AUE method under similar masked-token budgets. Our method produces a more spatially dispersed pattern, effectively preserving diverse visual cues while filtering redundancy.

		LLaVA-1.5-7B		Shikra-7B
Bench	Metric	Orig.	Ours	Orig.	Ours
Gen.	CHAIR $\downarrow$	8.2	6.2	12.2	11.5
	Cover $\uparrow$	48.5	46.2	48.9	47.5
	Hal $\downarrow$	34.9	24.7	49.7	44.8
	Cog $\downarrow$	3.5	2.4	5.6	4.6
Disc.	Acc. $\uparrow$	70.0	71.1	73.5	73.3

Appendix

Table 4: Evaluation results on the AMBER dataset. The best result in each row is highlighted in bold.

Object Hallucination in VLMs

Object hallucination in vision-language models is the production of object claims unsupported by visual evidence.

See all 128 papers

FADE: Mitigating Hallucinations by Reducing Language-Prior Dominance in Large Vision-Language Models
Jun 28, 2026Yichen Guo, Kai Tang, Fenglai Lin +5Vision-Language Models Large Vision-Language Models
1Nanyang Technological University · 2State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · 3Tsinghua University
Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration
May 24, 2026Yuanzhi Xu, Qian Gao, Jun Fan +4VLM Robustness Vision-Language Models
Qilu University of Technology (Shandong Academy of Sciences) Jinan, China · China Telecom Digital Intelligence Technology Co, Ltd Jinan, China · Shenyang Aerospace University Shenyang, China +1
See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs
Jun 29, 2026Yuqing Lei, Wenbo Lyu, Yingjun Du +3Visual Attention Large Vision-Language Models
University of Chinese Academy of Sciences · University of Amsterdam · United Imaging Healthcare Co., Ltd.

Jun 28, 2026J/K move · Enter open · S save

FADE: Mitigating Hallucinations by Reducing Language-Prior Dominance in Large Vision-Language Models

Yichen Guo, Kai Tang, Fenglai Lin, Yiding Sun+4

1Nanyang Technological University · 2State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · 3Tsinghua University

Vision-Language Models Large Vision-Language Models Object Hallucination in VLMs VLM Interpretability Faithfulness in VLMs VLM Hallucination Mitigation

Open paper AskPDF arXiv

Despite the impressive capabilities of Large Vision-Language Models (LVLMs), they remain susceptible to hallucination, generating content inconsistent with the input image. Recent studies attribute this to the dominance of language priors over visual inputs and employ contrastive decoding methods to mitigate this dominance, but the mechanistic origin remains unexplored. We investigate the information flow through each transformer layer and find that attention modules consistently aggregate visual evidence, while FFN modules at critical layers act as the source of language priors. These priors can override visual evidence, causing correct predictions in intermediate layers to drift toward incorrect outputs. Based on this insight, we propose FADE (FFN Attenuation for DEcoding), a training-free method that attenuates FFN outputs to reduce language-prior dominance. Evaluations on POPE, CHAIR, and MME benchmarks across LLaVA-1.5, mPLUG-Owl2, and InstructBLIP show that FADE effectively mitigates hallucinations while preserving inference efficiency.

Abstract

Object Hallucination in VLMs

Explore similar work

FADE: Mitigating Hallucinations by Reducing Language-Prior Dominance in Large Vision-Language Models

Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration

See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs