Object hallucination in Large Vision-Language Models (LVLMs) severely compromises their reliability in real-world applications, posing a critical barrier to their deployment in high-stakes scenarios such as autonomous driving and medical image analysis. Through systematic empirical investigation, we identify that the imbalanced attention allocation, both across modalities (i.e., vision and language) and within modalities (among individual tokens), exhibits a strong causal correlation with the occurrence of object hallucination. Leveraging this insight, we introduce a novel concept termed attention imbalance, which not only quantifies the degree of attention disparity but also visually delineates the underlying patterns (e.g., over-attentiveness to irrelevant language tokens or under-attentiveness to discriminative visual features) that drive object hallucination. To mitigate object hallucination, we further propose Attention Imbalance Rectification (AIR), a lightweight decoding-time intervention method that reallocates attention weights and adjusts attention distributions to rectify modality-wise and token-wise imbalances. Extensive evaluations on four mainstream LVLMs and three benchmarks (CHAIR, POPE, and MM-Vet) with seven baselines demonstrate that AIR consistently reduces object hallucination rates, achieving up to a 35.1% reduction compared to the baselines, while improving up to 15.9% of LVLMs' general capability across diverse vision-language tasks.
Figures & tables
Figure 1: Attention imbalance causes LLaVA-1.5 to hallucinate (up) while hallucination disappears once the attention is balanced by our AIR approach (down).
Figure 2: The attention map of output tokens. Imbalanced token is highlighted with a red box, and the hallucinated tokens are indicated by red arrows ↓ .
Figure 3: Co-occurrence between imbalanced tokens and object hallucinations across LVLMs.
Figure 4: (a) The distribution of attention heads. We highlight the top twenty hallucination-sensitive heads with red boxes and the bottom twenty hallucination-insensitive heads with blue boxes. (b) Comparison of the average modality-token attention weights among hallucination-sensitive heads, average heads, and hallucination-insensitive heads. (c) Comparison of the total modality-level attention weights of three head groups. (d) Comparison of attention map similarities between the LVLM and its base language model.
Table 1: CHAIR hallucination evaluation results on four LVLMs across baselines. Smaller CS and CI indicate fewer hallucinations. Results are reported with maximum new tokens set to 256 and 64. Our method AIR achieves the best hallucination mitigation performance across different max new token settings compared to the baselines. Δ% represents the relative improvement in performance over the second-best method.
Figure 5: POPE hallucination evaluation results of four LVLMs across different baselines. The four filled regions denote Adversarial, Popular, Random subsets and their average. AIR consistently achieves the highest average performance across all four LVLMs.
Method
Rec
OCR
Know
Gen
Spat
Math
Overall ↑
Greedy
32.7
17.6
14.5
15.3
21.9
7.7
27.6
FarSight
32.3
20.6
14.4
16.6
23.5
11.5
28.3 (+2.5%)
VCD
32.5
21.5
15.0
17.6
24.8
3.8
28.3 (+2.5%)
DoLA
32.1
18.3
13.8
14.6
22.0
7.7
27.6 (+0.0%)
HALC
32.4
17.7
14.2
15.5
19.9
7.7
27.2 (-1.4%)
OPERA
32.9
18.4
14.4
15.5
22.3
7.7
27.9 (+1.1%)
Table 2: MM-Vet evaluation results on LLaVA-1.5 . The best scores are highlighted in bold. AIR demonstrates stronger general capability compared with the baseline methods.
Method
Rec
OCR
Know
Gen
Spat
Math
Overall ↑
Greedy
22.3
14.7
13.6
10.0
21.7
7.7
20.0
FarSight
26.3
12.3
14.5
14.6
14.7
3.5
20.3 (+1.5%)
VCD
17.1
10.0
7.9
7.8
11.7
7.3
14.2 (-29.0%)
DoLA
20.7
14.4
13.6
10.0
21.7
7.3
18.8 (-6.0%)
HALC
19.4
11.9
11.3
8.6
16.5
6.5
16.6 (-17.0%)
OPERA
20.9
15.1
10.5
10.5
19.5
12.7
18.5 (-7.5%)
Table 3: MM-Vet evaluation results on MiniGPT-4 . The best scores are highlighted in bold. AIR preserves the model’s general capability without degradation.
Method
CS↓
CI↓
F1↑
MM-Vet Overall ↑
None
51.8
13.7
0.83
27.6
R-only
32.1
9.9
0.85
30.5 (+10.5%)
P-only
48.6
12.1
0.84
28.0 (+1.4%)
Full (AIR)
28.8
8.6
0.86
32.0 ( +15.9% )
Table 4: Effect of the two core mechanisms in AIR on LLaVA-1.5. R-only applies only the attention reallocation, while P-only applies only the projection regularization. Full enables both mechanisms.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Sensitivity of the attention reallocation mechanism to hyperparameters. (a) shows the impact of the visual reallocation factor λ . (b) illustrates the effect of the textual reallocation factor γ . (c) demonstrates the influence of the attention reallocation threshold τtext .
CS↓
CI↓
F1↑
MM-Vet Overall ↑
β=0
32.1
9.9
0.85
30.5
β=0.3
28.8
8.6
0.86
32.0 ( +4.9% )
β=0.6
30.4
9.0
0.86
31.9 (+4.6%)
β=0.9
32.0
9.2
0.85
27.4 (-10.2%)
Appendix
Table 5: Effect of the regularization coefficient β .
Figure 7: Comparison of inference time across baselines. Except for HALC and OPERA, most methods maintain an average decoding time of under 6 seconds per sample.
Max New Tokens: 256
Methods
LLaVA-NeXT-7B [ 37 ]
Qwen-2.5‑VL-7B [ 2 ]
InternVL-3.5-8B [ 17 ]
DeepSeek-VL2-4.5B [ 63 ]
CS↓
CI↓
CS↓
CI↓
CS↓
CI↓
CS↓
CI↓
Greedy
31.2
8.6
38.0
9.4
32.9
9.0
42.4
11.6
AIR (Ours)
22.5
6.1
29.3
8.8
25.1
8.3
25.8
6.9
Δ%
↓ 27.9%
↓ 29.1%
↓ 22.9%
↓ 6.4%
↓ 23.7%
↓ 7.8%
↓ 39.2%
↓ 40.5%
Appendix
Table 6: CHAIR hallucination evaluation results on additional four LVLMs across baselines. Smaller CS and CI indicate fewer hallucinations. Results are reported with maximum new tokens set to 256. Δ% represents the relative improvement over the baseline. Our method AIR consistently enhances hallucination mitigation across all additional LVLMs.
Max New Tokens: 256
Methods
LLaVA-1.5-7B
LLaVA-1.5-13B
LLaVA-NeXT-34B
CS↓
CI↓
CS↓
CI↓
CS↓
CI↓
Greedy
51.8
13.7
46.3
12.5
25.7
6.1
AIR (Ours)
28.8
8.6
31.0
9.0
23.2
6.0
Δ%
↓ 44.4%
↓ 37.2%
↓ 33.0%
↓ 28.0%
↓ 9.7%
↓ 1.6%
Appendix
Table 7: CHAIR hallucination evaluation results across LVLMs of different scales. Smaller CS and CI indicate fewer hallucinations. All results are reported with the maximum new token limit set to 256. Δ% denotes the relative improvement over the baseline. Our method AIR consistently improves hallucination mitigation across models of varying scales.
Figure 8: Attention maps across layers and heads for Case One.
Figure 9: Attention maps across layers and heads for Case Two.
Figure 10: Attention maps across layers and heads for Case Three.
The generation of factually incorrect objects, commonly known as object hallucination, remains a persistent challenge in Large Vision-Language Models (LVLMs). Current approaches to address this issue - ranging from expensive data-driven fine-tuning and high-latency contrastive decoding to rigid attention head truncation - frequently compromise either computational efficiency or the continuity of the model's feature space. To overcome these limitations, we introduce a novel, training-free inference strategy that operates as a region-aware adaptive weighting mechanism to dynamically correct semantic drift without relying on abrupt heuristic truncations. By computing an outlier-resistant statistical midpoint across various attention heads, we establish a stable anchor for reliable visual representations. We then utilize the inter-head disagreement mapped across regions to dynamically determine intervention budgets, gently suppressing hallucination-inducing attention paths through a continuous penalty modulation. This recalibration process effectively rectifies visual-semantic misalignments while fully preserving generative fluency and language priors. Comprehensive evaluations on standard multimodal benchmarks, including CHAIR, POPE, and MME, reveal that our strategy substantially curtails both instance- and sentence-level hallucinations. The results demonstrate state-of-the-art performance against contemporary baselines, confirming our method's efficiency and algorithmic robustness. Our code will be public.
Yuanzhi Xu, Qian Gao, Jun Fan +4
Qilu University of Technology (Shandong Academy of Sciences) Jinan, China · China Telecom Digital Intelligence Technology Co, Ltd Jinan, China · Shenyang Aerospace University Shenyang, China +1
Large Vision-Language Models (LVLMs) often suffer from object hallucination, generating objects that are absent from the image. Prior work largely attributes this to insufficient visual attention. However, we find that both real and hallucinated objects receive equally strong visual attention in the model's mid-to-late layers, suggesting that the key issue may not be how much the model attends, but what it attends to and why. To this end, we decode the visual features of high-attention regions using Logit Lens, and observe that regions corresponding to real objects can be correctly decoded to the target object tokens, whereas those for hallucinated objects cannot. Building on this, we identify two hallucination mechanisms: (i) visual uncertainty, triggered by semantically similar or confusable regions; masking these regions eliminates the hallucination. (ii) contextual prior, triggered by strong co-occurrence priors; even when the initially attended region is masked, the hallucination persists and attention drifts to other regions. Based on these findings, we propose a simple yet effective training-free Detect-Mitigate framework comprising a Logit-Lens Consistency Check to detect hallucination and targeted remedies: High-Attention Regions Masking (HARM) for visual uncertainty hallucination, and Visual Evidence Enhanced Decoding (VEED) for contextual prior hallucination. Our approach achieves state-of-the-art results on multiple hallucination benchmarks. Code will be available.
Zichuan Wang, Songlin Yang, Bo Peng +4
School of Artificial Intelligence, University of Chinese Academy of Sciences · 3Hong Kong University of Science and Technology · 2New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Science
Although Large Vision-Language Models (LVLMs) have demonstrated remarkable performance on downstream tasks, they frequently produce contents that deviate from visual information, leading to object hallucination. To tackle this, recent works mostly depend on expensive manual annotations and training cost, or decoding strategies which significantly increase inference time. In this work, we observe that LVLMs' attention to visual information is significantly enhanced when answering caption queries compared to non-caption queries. Inspired by this phenomenon, we propose Caption-guided Visual Attention Steering (CAST), a training-free, plug-and-play hallucination mitigation method that leverages the attention activation pattern corresponding to caption queries to enhance LVLMs' visual perception capability. Specifically, we use probing techniques to identify attention heads that are highly sensitive to caption queries and estimate optimized steering directions for their outputs. This steering strengthens LVLM's fine-grained visual perception capabilities, thereby effectively mitigating object hallucination. CAST reduced object hallucination by an average of 6.03% across five widely used LVLMs and five benchmarks including both discriminative and generative tasks, demonstrating state-of-the-art performance while adding little inference cost and preserving other foundational capabilities.
Qiming Li, Zekai Ye, Xiaocheng Feng +9
1Harbin Institute of Technology · 2Peng Cheng Laboratory · 3Harbin Institute of Technology, Shenzhen