Mitigating Object Hallucinations in LVLMs via Attention Imbalance Rectification

Han Sun; Qin Li; Peixin Wang; Min Zhang

cs.CVMar 25, 2026

Mitigating Object Hallucinations in LVLMs via Attention Imbalance Rectification

Authors: Han Sun, Qin Li, Peixin Wang, Min Zhang

Organizations: Shanghai Key Laboratory of Trustworthy Computing, East China Normal University

Abstract

Object hallucination in Large Vision-Language Models (LVLMs) severely compromises their reliability in real-world applications, posing a critical barrier to their deployment in high-stakes scenarios such as autonomous driving and medical image analysis. Through systematic empirical investigation, we identify that the imbalanced attention allocation, both across modalities (i.e., vision and language) and within modalities (among individual tokens), exhibits a strong causal correlation with the occurrence of object hallucination. Leveraging this insight, we introduce a novel concept termed attention imbalance, which not only quantifies the degree of attention disparity but also visually delineates the underlying patterns (e.g., over-attentiveness to irrelevant language tokens or under-attentiveness to discriminative visual features) that drive object hallucination. To mitigate object hallucination, we further propose Attention Imbalance Rectification (AIR), a lightweight decoding-time intervention method that reallocates attention weights and adjusts attention distributions to rectify modality-wise and token-wise imbalances. Extensive evaluations on four mainstream LVLMs and three benchmarks (CHAIR, POPE, and MM-Vet) with seven baselines demonstrate that AIR consistently reduces object hallucination rates, achieving up to a 35.1% reduction compared to the baselines, while improving up to 15.9% of LVLMs' general capability across diverse vision-language tasks.

Figures & tables

Figure 1: Attention imbalance causes LLaVA-1.5 to hallucinate (up) while hallucination disappears once the attention is balanced by our AIR approach (down).

Figure 2: The attention map of output tokens. Imbalanced token is highlighted with a red box, and the hallucinated tokens are indicated by red arrows $\downarrow$ .

Figure 3: Co-occurrence between imbalanced tokens and object hallucinations across LVLMs.

Figure 4: (a) The distribution of attention heads. We highlight the top twenty hallucination-sensitive heads with red boxes and the bottom twenty hallucination-insensitive heads with blue boxes. (b) Comparison of the average modality-token attention weights among hallucination-sensitive heads, average heads, and hallucination-insensitive heads. (c) Comparison of the total modality-level attention weights of three head groups. (d) Comparison of attention map similarities between the LVLM and its base language model.

A^{\prime(\ell,h)}_{ij}\leftarrow\begin{cases}{\lambda}A^{(\ell,h)}_{ij},&j\in\mathcal{T},\\[2.0pt] {\gamma}A^{(\ell,h)}_{ij},&j\in\mathcal{V},\end{cases}

Algorithm 1 AIR: Attention Imbalance Rectification

	Max New Tokens: 256								Max New Tokens: 64
Methods	LLaVA-1.5		MiniGPT4		InstructBLIP		Shikra		LLaVA-1.5		MiniGPT4		InstructBLIP		Shikra
Methods	$C_{S}\!\downarrow$	$C_{I}\!\downarrow$	$C_{S}\!\downarrow$	$C_{I}\!\downarrow$	$C_{S}\!\downarrow$	$C_{I}\!\downarrow$	$C_{S}\!\downarrow$	$C_{I}\!\downarrow$	$C_{S}\!\downarrow$	$C_{I}\!\downarrow$	$C_{S}\!\downarrow$	$C_{I}\!\downarrow$	$C_{S}\!\downarrow$	$C_{I}\!\downarrow$	$C_{S}\!\downarrow$	$C_{I}\!\downarrow$
Greedy	51.8	13.7	43.0	13.4	54.7	20.2	55.4	15.2	20.8	6.2	28.8	11.4	29.8	14.2	22.3	8.2
FarSight	43.9	12.5	39.2	12.4	48.3	18.7	51.6	12.2	18.6	5.9	22.5	8.9	20.2	9.6	15.4	7.6
VCD	59.4	16.0	40.0	14.6	50.1	19.4	54.9	15.1	22.8	7.1	28.0	11.3	34.1	15.8	23.5	8.3
DoLA	54.0	14.2	42.0	13.5	52.8	19.1	55.7	14.8	22.2	6.7	28.6	11.5	28.1	14.2	20.2	9.6

Table 1: CHAIR hallucination evaluation results on four LVLMs across baselines. Smaller

C_{S}

and

C_{I}

indicate fewer hallucinations. Results are reported with maximum new tokens set to 256 and 64. Our method AIR achieves the best hallucination mitigation performance across different max new token settings compared to the baselines.

\Delta\%

represents the relative improvement in performance over the second-best method.

Figure 5: POPE hallucination evaluation results of four LVLMs across different baselines. The four filled regions denote Adversarial, Popular, Random subsets and their average. AIR consistently achieves the highest average performance across all four LVLMs.

Method	Rec	OCR	Know	Gen	Spat	Math	Overall $\uparrow$
Greedy	32.7	17.6	14.5	15.3	21.9	7.7	27.6
FarSight	32.3	20.6	14.4	16.6	23.5	11.5	28.3 (+2.5%)
VCD	32.5	21.5	15.0	17.6	24.8	3.8	28.3 (+2.5%)
DoLA	32.1	18.3	13.8	14.6	22.0	7.7	27.6 (+0.0%)
HALC	32.4	17.7	14.2	15.5	19.9	7.7	27.2 (-1.4%)
OPERA	32.9	18.4	14.4	15.5	22.3	7.7	27.9 (+1.1%)

Table 2: MM-Vet evaluation results on LLaVA-1.5 . The best scores are highlighted in bold. AIR demonstrates stronger general capability compared with the baseline methods.

Method	Rec	OCR	Know	Gen	Spat	Math	Overall $\uparrow$
Greedy	22.3	14.7	13.6	10.0	21.7	7.7	20.0
FarSight	26.3	12.3	14.5	14.6	14.7	3.5	20.3 (+1.5%)
VCD	17.1	10.0	7.9	7.8	11.7	7.3	14.2 (-29.0%)
DoLA	20.7	14.4	13.6	10.0	21.7	7.3	18.8 (-6.0%)
HALC	19.4	11.9	11.3	8.6	16.5	6.5	16.6 (-17.0%)
OPERA	20.9	15.1	10.5	10.5	19.5	12.7	18.5 (-7.5%)

Table 3: MM-Vet evaluation results on MiniGPT-4 . The best scores are highlighted in bold. AIR preserves the model’s general capability without degradation.

Method	$C_{S}\!\downarrow$	$C_{I}\!\downarrow$	$F_{1}\!\uparrow$	MM-Vet Overall $\uparrow$
None	51.8	13.7	0.83	27.6
R-only	32.1	9.9	0.85	30.5 (+10.5%)
P-only	48.6	12.1	0.84	28.0 (+1.4%)
Full (AIR)	28.8	8.6	0.86	32.0 ( +15.9% )

Table 4: Effect of the two core mechanisms in AIR on LLaVA-1.5. R-only applies only the attention reallocation, while P-only applies only the projection regularization. Full enables both mechanisms.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Sensitivity of the attention reallocation mechanism to hyperparameters. (a) shows the impact of the visual reallocation factor $\lambda$ . (b) illustrates the effect of the textual reallocation factor $\gamma$ . (c) demonstrates the influence of the attention reallocation threshold $\tau_{\mathrm{text}}$ .

	$C_{S}\!\downarrow$	$C_{I}\!\downarrow$	$F_{1}\!\uparrow$	MM-Vet Overall $\uparrow$
$\beta=0$	32.1	9.9	0.85	30.5
$\beta=0.3$	28.8	8.6	0.86	32.0 ( +4.9% )
$\beta=0.6$	30.4	9.0	0.86	31.9 (+4.6%)
$\beta=0.9$	32.0	9.2	0.85	27.4 (-10.2%)

Appendix

Table 5: Effect of the regularization coefficient

\beta

Figure 7: Comparison of inference time across baselines. Except for HALC and OPERA, most methods maintain an average decoding time of under 6 seconds per sample.

	Max New Tokens: 256
Methods	LLaVA-NeXT-7B [ 37 ]		Qwen-2.5‑VL-7B [ 2 ]		InternVL-3.5-8B [ 17 ]		DeepSeek-VL2-4.5B [ 63 ]
Methods	$C_{S}\!\downarrow$	$C_{I}\!\downarrow$	$C_{S}\!\downarrow$	$C_{I}\!\downarrow$	$C_{S}\!\downarrow$	$C_{I}\!\downarrow$	$C_{S}\!\downarrow$	$C_{I}\!\downarrow$
Greedy	31.2	8.6	38.0	9.4	32.9	9.0	42.4	11.6
AIR (Ours)	22.5	6.1	29.3	8.8	25.1	8.3	25.8	6.9
$\Delta\%$	$\downarrow$ 27.9%	$\downarrow$ 29.1%	$\downarrow$ 22.9%	$\downarrow$ 6.4%	$\downarrow$ 23.7%	$\downarrow$ 7.8%	$\downarrow$ 39.2%	$\downarrow$ 40.5%

Appendix

Table 6: CHAIR hallucination evaluation results on additional four LVLMs across baselines. Smaller

C_{S}

and

C_{I}

indicate fewer hallucinations. Results are reported with maximum new tokens set to 256.

\Delta\%

represents the relative improvement over the baseline. Our method AIR consistently enhances hallucination mitigation across all additional LVLMs.

	Max New Tokens: 256
Methods	LLaVA-1.5-7B		LLaVA-1.5-13B		LLaVA-NeXT-34B
Methods	$C_{S}\!\downarrow$	$C_{I}\!\downarrow$	$C_{S}\!\downarrow$	$C_{I}\!\downarrow$	$C_{S}\!\downarrow$	$C_{I}\!\downarrow$
Greedy	51.8	13.7	46.3	12.5	25.7	6.1
AIR (Ours)	28.8	8.6	31.0	9.0	23.2	6.0
$\Delta\%$	$\downarrow$ 44.4%	$\downarrow$ 37.2%	$\downarrow$ 33.0%	$\downarrow$ 28.0%	$\downarrow$ 9.7%	$\downarrow$ 1.6%

Appendix

Table 7: CHAIR hallucination evaluation results across LVLMs of different scales. Smaller

C_{S}

and

C_{I}

indicate fewer hallucinations. All results are reported with the maximum new token limit set to 256.

\Delta\%

denotes the relative improvement over the baseline. Our method AIR consistently improves hallucination mitigation across models of varying scales.

Figure 8: Attention maps across layers and heads for Case One.

Figure 9: Attention maps across layers and heads for Case Two.

Figure 10: Attention maps across layers and heads for Case Three.

Object Hallucination in VLMs

Object hallucination in vision-language models is the production of object claims unsupported by visual evidence.

See all 128 papers

Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration
May 24, 2026Yuanzhi Xu, Qian Gao, Jun Fan +4VLM Robustness Vision-Language Models
Qilu University of Technology (Shandong Academy of Sciences) Jinan, China · China Telecom Digital Intelligence Technology Co, Ltd Jinan, China · Shenyang Aerospace University Shenyang, China +1
Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination
Aug 7, 2026Zichuan Wang, Songlin Yang, Bo Peng +4Hallucination Detection in VLMs Vision-Language Models
School of Artificial Intelligence, University of Chinese Academy of Sciences · 3Hong Kong University of Science and Technology · 2New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Science
CAST: Mitigating Object Hallucination in Large Vision-Language Models via Caption-Guided Visual Attention Steering
May 6, 2026Qiming Li, Zekai Ye, Xiaocheng Feng +9Hallucination in Language Models Large Vision-Language Models
1Harbin Institute of Technology · 2Peng Cheng Laboratory · 3Harbin Institute of Technology, Shenzhen

May 24, 2026J/K move · Enter open · S save

Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration

Yuanzhi Xu, Qian Gao, Jun Fan, Guohui Ding+3

Qilu University of Technology (Shandong Academy of Sciences) Jinan, China · China Telecom Digital Intelligence Technology Co, Ltd Jinan, China · Shenyang Aerospace University Shenyang, China+1

VLM Robustness Vision-Language Models Hallucination in Language Models Object Hallucination in VLMs VLM Hallucination Mitigation Attention Intervention in VLMs+1

Open paper AskPDF arXiv

The generation of factually incorrect objects, commonly known as object hallucination, remains a persistent challenge in Large Vision-Language Models (LVLMs). Current approaches to address this issue - ranging from expensive data-driven fine-tuning and high-latency contrastive decoding to rigid attention head truncation - frequently compromise either computational efficiency or the continuity of the model's feature space. To overcome these limitations, we introduce a novel, training-free inference strategy that operates as a region-aware adaptive weighting mechanism to dynamically correct semantic drift without relying on abrupt heuristic truncations. By computing an outlier-resistant statistical midpoint across various attention heads, we establish a stable anchor for reliable visual representations. We then utilize the inter-head disagreement mapped across regions to dynamically determine intervention budgets, gently suppressing hallucination-inducing attention paths through a continuous penalty modulation. This recalibration process effectively rectifies visual-semantic misalignments while fully preserving generative fluency and language priors. Comprehensive evaluations on standard multimodal benchmarks, including CHAIR, POPE, and MME, reveal that our strategy substantially curtails both instance- and sentence-level hallucinations. The results demonstrate state-of-the-art performance against contemporary baselines, confirming our method's efficiency and algorithmic robustness. Our code will be public.

Abstract

Object Hallucination in VLMs

Explore similar work

Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibration

Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination

CAST: Mitigating Object Hallucination in Large Vision-Language Models via Caption-Guided Visual Attention Steering