Self-Correction Inside the Model: Leveraging Layer Attention to Mitigate Hallucinations in Large Vision Language Models

April Fu

cs.CVFeb 28, 2026

Self-Correction Inside the Model: Leveraging Layer Attention to Mitigate Hallucinations in Large Vision Language Models

Authors: April Fu

Organizations: Independent Researcher

Abstract

Although Large Vision-Language Models (LVLMs) have made substantial progress, hallucination, where generated text is not grounded in the visual input, remains a challenge. As LVLMs become stronger, previously reported hallucination patterns, such as linguistic bias and overthinking phenomenon, become far less consistent, making the corresponding mitigation techniques substantially less effective. In this paper, we introduce an Internal self-Correction mechanism utilizing Layer Attention (ICLA) that operates directly on hidden states during generation. Each layer selectively retrieves information from all preceding layers through a diagonal cross-layer attention mechanism, enabling self-refinement without any external correction signals. With introducing and training only 0.2M and 0.1M additional parameters on LLaVA1.5-7B and Qwen2.5-VL-7B, \ours consistently improves visual grounding across multiple hallucination benchmarks, demonstrating its effectiveness for more advanced LVLMs.

Figures & tables

Figure 1: Performance degradation of existing methods on Qwen2.5-VL-7B, where DoLA exhibits a significant drop, while the performance drop of MME exceeds the plot range and is not shown in the figure.

Figure 2: Overall architecture of ICLA .

Model	LLaVA1.5-7B			Qwen2.5-VL-7B
Method	MME	LLaVA-Bench	MMMU	MME	LLaVA-Bench	MMMU
Vanilla	1484	59.6	35.3	1689	87.0	67.5
DoLA	1485	60.5	35.7	1403	66.2	60.8
VCD	1469	60.6	35.8	1689	88.7	68.3
DeCo	1456	57.0	33.9	1681	86.8	62.5
POVID	1483	60.2	35.3	-	-	-

Table 1: Experimental results on MME (total perception score), LLaVA-Bench (overall accuracy), and MMMU (accuracy) for LLaVA1.5-7B and Qwen2.5-VL-7B. The best results are highlighted in bold with light blue shading.

Setting	Method	LLaVA1.5-7B				Qwen2.5-VL-7B
		MSCOCO		A-OKVQA		MSCOCO		A-OKVQA
		F1	Acc	F1	Acc	F1	Acc	F1	Acc
Adversarial	Vanilla	81.76	79.77	76.12	69.37	80.72	83.40	80.73	79.53
	DoLA	81.69	79.73	76.22	69.53	77.44	80.63	80.91	76.57
	VCD	80.50	78.33	75.05	67.97	81.13	77.77	80.73	72.03
	DeCo	81.14	78.37	74.93	67.03	80.86	83.47	80.93	80.10

Table 2: Results on the POPE benchmark. Comparison between LLaVA1.5-7B and Qwen2.5-VL-7B . We report F1 Score (%) and Accuracy (%) on the MSCOCO and A-OKVQA datasets under Adversarial, Popular, and Random settings. The best results are highlighted in bold blue .

Method	Vanilla	DoLA	VCD	DeCo	VDD	DAMO	ICLA (Ours)
Complex Reasoning (%)	95.8	66.5	98.6	95.2	94.8	96.3	93.3
Conversation (%)	82.4	65.0	86.1	83.7	85.0	82.0	89.4
Detail Description (%)	76.0	66.9	74.4	75.8	76.4	77.7	85.5
All (%)	87.0	66.2	88.7	86.8	87.2	87.4	90.2

Table 3: Comparison of LLaVA-Bench performance for various baselines with backbone Qwen2.5-VL-7B. The best results are highlighted in bold blue.

Figure 3: (a) Ablation results for starting layer

k_{0}

on POPE benchmark. (b) Ablation results for reduction ratio

r

on MMMU and MME benchmarks; (c) Ablation results on scaling factor

\alpha

on MMMU and MME benchmarks.

Figure 4: Case study comparing Vanilla and ICLA based on Qwen2.5-VL-7B. The example is sampled from LLaVA-Bench.

Method	Random		Adversarial
Method	F1	Acc	F1	Acc
Vanilla	88.52	87.33	76.12	69.37
ICLA (Last)	88.29	87.40	76.18	69.47
Random Agg.	88.55	87.37	76.18	69.47
ICLA	88.80	87.70	76.32	69.73

Table 4: Comparison of different ICLA variants on the POPE benchmark (A-OKVQA dataset) using LLaVA1.5-7B. Random Add. denotes the Random Aggregation.

Figure 5: Visualization of average attention weights in ICLA using Qwen for selected samples

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Method	OCR	Artwork	Celebrity	Color	Count	Existence	Landmark	Position	Posters	Scene	Total
Vanilla	181	138	157	194	154	199	184	159	169	154	1689
DoLA	165	115	87	184	118	174	121	154	146	139	1403
VCD	181	138	157	194	154	199	184	159	169	154	1689
DeCo	181	138	153	194	154	199	181	159	168	154	1681
VDD	181	138	157	194	154	199	184	159	169	154	1689
DAMO	181	138	153	194	154	199	181	159	168	154	1681

Appendix

Table 5: Comparison on the MME benchmark (Perception) across various baselines with backbone Qwen2.5-VL-7B. The best results are highlighted in bold .

Method	Vanilla	DoLa	VCD	DeCo	VDD	DAMO	ICLA (Ours)
Overall Accuracy(%)	67.5	60.8	68.3	62.5	65.8	65.8	69.2

Appendix

Table 6: Comparison on overall accuracy of the MMMU benchmark across various baselines with backbone Qwen2.5-VL-7B. The best results are highlighted in bold .

Method	Vanilla	DoLa	VCD	DeCo	POVID	VDD	DAMO	ICLA (Ours)
Complex reasoning (%)	72.1	70.5	70.7	66.8	69.4	71.9	65.1	70.4
Conversation (%)	50.6	53.7	55.0	52.7	55.0	52.3	55.0	59.6
Detail description (%)	48.8	50.0	49.3	45.0	50.4	45.7	48.0	50.0
All (%)	59.6	60.5	60.6	57.0	60.2	59.4	57.7	61.9

Appendix

Table 7: Comparison of LLaVA-Bench performance for various baselines. The best results are highlighted in bold .

Method	MME	LLaVA-Bench	MMMU
Vanilla	1726	98.8	57.1
DoLA	1705	99.8	58.0
VCD	1726	106.7	57.1
DeCo	1734	60.6	57.2
VDD	1720	106.3	56.5
DAMO	1704	102.8	56.9

Appendix

Table 8: Experimental results on MME (total perception score), LLaVA-Bench (overall accuracy), and MMMU (accuracy) for Qwen3-VL-8B. The best results are highlighted in bold .

Token Length	Total FLOPs (TFLOPs)	ICLA FLOPs (GFLOPs)	Overhead (%)	Params Added
128	5.12	18.8	0.37	277K
256	10.29	37.8	0.37
512	13.86	50.6	0.37

Appendix

Table 9: Computation overhead of ICLA compared to vanilla LLaVA1.5-7B under different token lengths.

Token Length	Total FLOPs (TFLOPs)	ICLA FLOPs (GFLOPs)	Overhead (%)	Params Added
128	2.12	1.48	0.07	105K
256	4.36	2.98	0.07
512	7.99	5.94	0.07

Appendix

Table 10: Computation overhead of ICLA compared to vanilla Qwen2.5-VL-7B under different token lengths.

Figure 6: Visualization of average attention weights in ICLA using LLaVA1.5-7B for selected samples

Explore similar work

Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy
May 20, 2026Yutong Xie, Zhenglin Hua, Ran Wang +3Vision-Language Models Hallucination in Language Models
School of Computer Science and Engineering, Southeast University, Nanjing, China · School of Artificial Intelligence, Shenzhen University, Shenzhen, China · National Engineering Laboratory for Big Data Systems Computing Technology, Shenzhen University, Shenzhen, China +3
Mitigating Hallucinations via Inter-Layer Consistency Aggregation in Large Vision-Language Models
May 18, 2025Kai Tang, Jinhao You, Yichen Guo +8Vision-Language Models Hallucination in Language Models
Peking University · University of Pennsylvania · Nanyang Technological University +4
See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs
Jun 29, 2026Yuqing Lei, Wenbo Lyu, Yingjun Du +3Visual Attention Large Vision-Language Models
University of Chinese Academy of Sciences · University of Amsterdam · United Imaging Healthcare Co., Ltd.

May 20, 2026J/K move · Enter open · S save

Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy

Yutong Xie, Zhenglin Hua, Ran Wang, Wing W. Y. Ng+2

School of Computer Science and Engineering, Southeast University, Nanjing, China · School of Artificial Intelligence, Shenzhen University, Shenzhen, China · National Engineering Laboratory for Big Data Systems Computing Technology, Shenzhen University, Shenzhen, China+3

Vision-Language Models Hallucination in Language Models VLM Hallucination Large Vision-Language Models VLM Hallucination Mitigation

Open paper AskPDF arXiv

Large Vision-Language Models (LVLMs) have shown remarkable performance on a wide range of vision-language tasks. Despite this progress, they are still prone to hallucination, generating responses that are inconsistent with visual content. In this work, we find that LVLMs tend to hallucinate when they pay insufficient attention to the correct visual evidence and gradually forget it during the generation process. We empirically find that although LVLMs overall attend insufficiently to visual evidence, they exhibit sensitivity to the correct visual evidence in specific layers, with notable inter-layer discrepancy. Motivated by this observation, we propose a novel hallucination mitigation method that enhances visual evidence based on Inter-Layer Visual Attention Discrepancy (ILVAD). Specifically, we obtain the attention weights from early generated tokens to visual tokens across layers and identify the tokens that are repeatedly activated as visual evidence, forming a saliency map. We then enhance attention to visual evidence during generation through the saliency map to reduce visual forgetting. In addition, we leverage the saliency map to obtain attention scores of generated text to visual evidence, in order to select and emphasize text tokens that are strongly grounded in visual evidence. Our method is training-free and plug-and-play. Multiple benchmark evaluations conducted on five recently released models show that our method can consistently mitigate hallucinations in different LVLMs over various architectures. Code is available at https://github.com/ytx-ML/ILVAD.

Abstract

Explore similar work

Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy

Mitigating Hallucinations via Inter-Layer Consistency Aggregation in Large Vision-Language Models

See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs