Segmentation-Based Attention Entropy: Detecting and Mitigating Object Hallucinations in Large Vision-Language Models

Jiale Song; Jiaxin Luo; Xue-song Tang; Kuangrong Hao; Mingbo Zhao

cs.CVMar 17, 2026

Segmentation-Based Attention Entropy: Detecting and Mitigating Object Hallucinations in Large Vision-Language Models

Authors: Jiale Song, Jiaxin Luo, Xue-song Tang, Kuangrong Hao, Mingbo Zhao

Organizations: School of Information and Intelligent Science, Donghua University, Shanghai, China

Abstract

Large Vision-Language Models (LVLMs) achieve strong performance on many multimodal tasks, but object hallucinations severely undermine their reliability. Most existing studies focus on the text modality, attributing hallucinations to overly strong language priors and insufficient visual grounding. In contrast, we observe that abnormal attention patterns within the visual modality can also give rise to hallucinated objects. Building on this observation, we propose Segmentation-based Attention Entropy (SAE), which leverages semantic segmentation to quantify visual attention uncertainty in a semantic space. Based on SAE, we further design a reliability score for hallucination detection and an SAE-guided attention adjustment method that modifies visual attention at inference time to mitigate hallucinations. We evaluate our approach on public benchmarks and in real embodied multimodal scenarios with quadruped robots. Experiments show that SAE reduces hallucinations without additional training, improving LVLM reliability.

Figures & tables

Figure 1. Object hallucinations in LVLMs and SAE. (a) Example of real (green) and hallucinated (red) objects in an LVLM description. (b) SAE pipeline: real objects correspond to low SAE, hallucinated objects to high SAE.

Figure 2. Head-level attention distributions for real objects (green) and hallucinated objects (red). Real objects have low SAE with attention concentrated on the corresponding image tokens, whereas hallucinated objects have high SAE with attention spread over many irrelevant image tokens.

Figure 3. Layer-head SAE statistics. The horizontal axis denotes the head index and the vertical axis the layer index, and the difference between real and hallucinated objects is most pronounced in the middle layers.

Figure 4. Qualitative example of SAE-guided attention intervention during inference. SAE-guided modulation reduces hallucinated objects (red) such as a spoon, a dining table, and a dog, while preserving or even completing real objects (green).

Figure 5. SAE-Guided Navigation Pipeline: The robot takes a first-person RGB-D image and language instruction as input to the LVLM, with the output being an action path in the physical world.

Figure 6. ROC and PR curves of different hallucination detection methods on LLaVA-1.5-7B and LLaVA-1.5-13B.

Method	LLaVA-1.5-7B					LLaVA-1.5-13B					Avg
Method	$C_{S}\!\downarrow$	$C_{I}\!\downarrow$	R $\uparrow$	P $\uparrow$	F1 $\uparrow$	$C_{S}\!\downarrow$	$C_{I}\!\downarrow$	R $\uparrow$	P $\uparrow$	F1 $\uparrow$	$C_{S}\!\downarrow$	$C_{I}\!\downarrow$
Nucleus	56.0	16.4	73.4	71.0	72.2	54.0	15.0	74.6	73.0	73.8	55.0	15.7
Greedy	47.9	13.6	78.1	75.8	77.0	44.3	11.7	77.6	78.7	78.2	46.1	12.6
Beam	52.1	14.1	78.5	75.2	76.8	49.7	12.7	79.1	77.3	78.2	50.9	13.4
OPERA	48.5	13.7	78.1	75.5	76.8	52.6	14.1	68.1	73.1	70.5	50.6	13.9
[1pt/2pt] VCD	55.4	15.7	75.6	71.8	73.7	52.3	13.9	75.5	74.6	75.1	53.8	14.8

Table 1. CHAIR and P/R/F1 results of different methods on LLaVA-1.5-7B and LLaVA-1.5-13B (max length = 512).

Method	64			128			256			Avg
Method	$C_{S}\!\downarrow$	$C_{I}\!\downarrow$	F1 $\uparrow$	$C_{S}\!\downarrow$	$C_{I}\!\downarrow$	F1 $\uparrow$	$C_{S}\!\downarrow$	$C_{I}\!\downarrow$	F1 $\uparrow$	$C_{S}\!\downarrow$	$C_{I}\!\downarrow$	F1 $\uparrow$
Nucleus	25.6	9.2	68.7	54.1	16.1	72.0	56.0	16.4	72.2	45.2	13.9	71.0
Greedy	21.3	6.6	74.5	46.9	13.3	77.0	47.9	13.6	77.0	38.7	11.2	76.2
Beam	18.7	6.1	73.3	50.2	13.7	76.9	52.1	14.1	76.8	40.3	11.3	75.7
OPERA	17.0	5.4	74.3	47.2	13.4	76.8	48.5	13.7	76.8	37.6	10.8	76.0
[1pt/2pt] VCD	23.2	8.1	71.9	53.9	15.0	74.6	55.4	15.7	73.7	44.2	12.9	73.4

Table 2. Effect of maximum generation length on hallucination (CHAIR) and F1 on LLaVA-1.5-7B.

Method	POPE Acc. $\uparrow$	POPE F1 $\uparrow$	MMHal Score $\uparrow$	MMHal Hall. Rate $\downarrow$
Greedy	84.7	85.2	2.58	55.4
VCD	86.0	86.4	2.71	52.1
PAI	87.1	87.5	2.87	49.1
VAR	87.6	88.0	2.95	47.4
Ours	88.8	89.1	3.11	43.2

Table 3. POPE and MMHal-Bench results on LLaVA-1.5-7B.

Method	Qwen2-VL-7B			InternVL2-8B
Method	$C_{S}\!\downarrow$	$C_{I}\!\downarrow$	F1 $\uparrow$	$C_{S}\!\downarrow$	$C_{I}\!\downarrow$	F1 $\uparrow$
Nucleus	36.8	10.4	72.9	38.4	10.9	73.4
Greedy	31.6	9.1	77.4	33.8	9.6	77.1
Beam	33.0	9.3	77.1	35.1	9.8	76.8
OPERA	32.8	9.1	76.6	34.6	9.7	76.2
[1pt/2pt] VCD	35.9	10.1	74.6	37.2	10.4	74.2

Table 4. CHAIR results on Qwen2-VL-7B and InternVL2-8B.

Figure 7. LVLM navigation in single- and multi-target scenarios with and without SAE.

Scenario	Method	Success Rate $\uparrow$	Time (s)	Replans $\downarrow$	Unsup. $\downarrow$
Single-target	Vanilla	66.7	96.4 $\pm$ 18.7	2.3 $\pm$ 1.0	0.92 $\pm$ 0.51
Single-target	Ours	91.7	84.1 $\pm$ 16.2	1.1 $\pm$ 0.6	0.25 $\pm$ 0.22
Multi-target	Vanilla	41.7	72.6 $\pm$ 13.9	4.0 $\pm$ 1.4	1.58 $\pm$ 0.67
Multi-target	Ours	75.0	99.8 $\pm$ 15.6	2.1 $\pm$ 1.0	0.63 $\pm$ 0.41

Table 5. Real-robot study on LVLM-based navigation.

$\lambda$ (Ours)	$C_{S}$ $\downarrow$	$C_{I}$ $\downarrow$	R $\uparrow$	P $\uparrow$	F1 $\uparrow$
0.3	40.4	11.8	75.5	77.3	76.4
0.4	40.2	12.1	75.1	76.1	75.6
0.5	24.2	5.9	68.2	87.3	76.6
0.6	2.6	2.0	37.8	97.8	54.6
0.7	1.4	1.8	8.9	98.0	16.2
Greedy	47.9	13.6	78.1	75.8	77.0

Table 6. Sensitivity to

\lambda

on LLaVA-1.5-7B (CHAIR, P/R/F1).

Partition setting	$C_{S}$ $\downarrow$	$C_{I}$ $\downarrow$	R $\uparrow$	P $\uparrow$	F1 $\uparrow$
Greedy	47.9	13.6	78.1	75.8	77.0
Ours (clean seg.)	24.2	5.9	68.2	87.3	76.6
Boundary noise (mild)	25.1	6.2	67.9	86.8	76.1
Boundary noise (moderate)	26.4	6.7	67.1	86.1	75.4
Token reassignment ( $\rho=0.1$ )	25.7	6.3	67.6	86.5	75.9
Token reassignment ( $\rho=0.2$ )	27.3	7.1	66.7	85.4	74.9

Table 7. Robustness to segmentation errors on LLaVA-1.5-7B.

Setting	$C_{S}$ $\downarrow$	$C_{I}$ $\downarrow$	F1 $\uparrow$	Time / image (s) $\downarrow$
Greedy (no partition)	47.9	13.6	77.0	0.82
Block entropy	37.4	11.2	76.2	0.93
Mask2Former-Swin-T	27.8	7.2	75.0	1.01
Mask2Former-Swin-L	24.2	5.9	76.6	1.17

Table 8. Ablation on region partitioning on LLaVA-1.5-7B.

Method	Tokens/sec. $\uparrow$	Relative overhead $\downarrow$	GPU memory (GB)
Greedy	17.3	1.00 $\times$	21.4
Ours (precomputed masks)	16.5	1.05 $\times$	21.8
Ours + block entropy	16.2	1.07 $\times$	21.9
Ours + online Swin-T	15.4	1.12 $\times$	22.4
Ours + online Swin-L	14.4	1.20 $\times$	23.1

Table 9. Runtime overhead analysis on LLaVA-1.5-7B.

Layer range	$C_{S}$ $\downarrow$	$C_{I}$ $\downarrow$	R $\uparrow$	P $\uparrow$	F1 $\uparrow$
Early layers (1–10)	31.8	8.7	70.5	82.1	75.9
Middle layers (11–22)	24.2	5.9	68.2	87.3	76.6
Late layers (23–32)	29.7	7.6	66.1	84.8	74.3
All layers	27.9	7.1	71.0	84.1	76.1

Table 10. Ablation on layer selection on LLaVA-1.5-7B.

Object Hallucination in VLMs

Object hallucination in vision-language models is the production of object claims unsupported by visual evidence.

See all 128 papers

Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination
Aug 7, 2026Zichuan Wang, Songlin Yang, Bo Peng +4Hallucination Detection in VLMs Vision-Language Models
School of Artificial Intelligence, University of Chinese Academy of Sciences · 3Hong Kong University of Science and Technology · 2New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Science
Mitigating Object Hallucinations in LVLMs via Attention Imbalance Rectification
Mar 25, 2026Han Sun, Qin Li, Peixin Wang +1Object Hallucination in VLMs VLM Hallucination Mitigation
Shanghai Key Laboratory of Trustworthy Computing, East China Normal University
Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration
May 24, 2026Yuanzhi Xu, Qian Gao, Jun Fan +4VLM Robustness Vision-Language Models
Qilu University of Technology (Shandong Academy of Sciences) Jinan, China · China Telecom Digital Intelligence Technology Co, Ltd Jinan, China · Shenyang Aerospace University Shenyang, China +1

Aug 7, 2026J/K move · Enter open · S save

Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination

Zichuan Wang, Songlin Yang, Bo Peng, Zhenchen Tang+3

School of Artificial Intelligence, University of Chinese Academy of Sciences · 3Hong Kong University of Science and Technology · 2New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Science

Hallucination Detection in VLMs Vision-Language Models VLM Hallucination Large Vision-Language Models Object Hallucination in VLMs VLM Hallucination Mitigation

Open paper AskPDF arXiv

Large Vision-Language Models (LVLMs) often suffer from object hallucination, generating objects that are absent from the image. Prior work largely attributes this to insufficient visual attention. However, we find that both real and hallucinated objects receive equally strong visual attention in the model's mid-to-late layers, suggesting that the key issue may not be how much the model attends, but what it attends to and why. To this end, we decode the visual features of high-attention regions using Logit Lens, and observe that regions corresponding to real objects can be correctly decoded to the target object tokens, whereas those for hallucinated objects cannot. Building on this, we identify two hallucination mechanisms: (i) visual uncertainty, triggered by semantically similar or confusable regions; masking these regions eliminates the hallucination. (ii) contextual prior, triggered by strong co-occurrence priors; even when the initially attended region is masked, the hallucination persists and attention drifts to other regions. Based on these findings, we propose a simple yet effective training-free Detect-Mitigate framework comprising a Logit-Lens Consistency Check to detect hallucination and targeted remedies: High-Attention Regions Masking (HARM) for visual uncertainty hallucination, and Visual Evidence Enhanced Decoding (VEED) for contextual prior hallucination. Our approach achieves state-of-the-art results on multiple hallucination benchmarks. Code will be available.

Abstract

Object Hallucination in VLMs

Explore similar work

Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination

Mitigating Object Hallucinations in LVLMs via Attention Imbalance Rectification

Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration