Visual Attention Drifts,but Anchors Hold:Mitigating Hallucination in Multimodal Large Language Models via Cross-Layer Visual Anchors

Chengxu Yang; Jingling Yuan; Chuang Hu; Jiawei Jiang

cs.CVMar 26, 2026

Visual Attention Drifts,but Anchors Hold:Mitigating Hallucination in Multimodal Large Language Models via Cross-Layer Visual Anchors

Authors: Chengxu Yang, Jingling Yuan, Chuang Hu, Jiawei Jiang

Organizations: School of Computer Science and Artificial Intelligence, Wuhan University of Technology, China · School of Computer Science, Wuhan University, China

Abstract

Multimodal Large Language Models often suffer from object hallucination. While existing research utilizes attention enhancement and visual retracing, we find these works lack sufficient interpretability regarding attention drift in final model stages. In this paper, we investigate the layer wise evolution of visual features and discover that hallucination stems from deep layer attention regressing toward initial visual noise from early layers. We observe that output reliability depends on acquiring visual anchors at intermediate layers rather than final layers. Based on these insights, we propose CLVA, which stands for Cross-Layer Visual Anchors, a training free method that reinforces critical mid layer features while suppressing regressive noise. This approach effectively pulls deep layer attention back to correct visual regions by utilizing essential anchors captured from attention dynamics. We evaluate our method across diverse architectures and benchmarks, demonstrating outstanding performance without significant increase in computational time and GPU memory.

Figures & tables

Figure 1: Matrix visualization of visual attention intensity across 32 layers and 32 heads in LLaVA. Colors represent the average attention weights to visual tokens $T_{vis}$ , while numbers denote the average attention weights to system and text prompt tokens.

Figure 2: Attention analysis on LLaVA-1.5-7B. (a) Heatmap visualization of attention patterns across different layers. (b) Statistical metrics where the blue line represents the average attention entropy of vision sensitive heads. The green and red lines denote the Pearson correlation coefficients between the attention distributions of each layer and the mean attention of insensitive heads in initial layers and sensitive heads in intermediate layers, respectively.

Figure 3: Architecture of CLVA. Based on our attention analysis, we propose a two-step process to anchor drifting attention, where positive anchoring anchors core visual semantic features while negative anchoring excludes background noise using insensitive heads.

Evaluation	Methods	Random		Popular		Adversarial		Average
Evaluation	Methods	F1-score $\uparrow$	Accuracy $\uparrow$	F1-score $\uparrow$	Accuracy $\uparrow$	F1-score $\uparrow$	Accuracy $\uparrow$	F1-score $\uparrow$	Accuracy $\uparrow$
LLaVA-1.5	Greedy	87.18	88.26	84.48	85.02	81.24	81.11	84.30	84.79
	Beam Search	87.26	88.35	84.50	85.03	81.21	81.09	84.32	84.82
	MemVR	87.83	88.40	83.20	84.37	80.61	80.53	83.88	84.43
	ClearSight	88.16	88.77	85.13	85.02	81.44	80.39	84.91	84.72
	IMCCD	88.22	88.82	85.45	86.00	82.07	81.66	85.24	85.49

Table 1: Results on the POPE benchmark. Results are averaged across the MS-COCO, A-OKVQA, and GQA datasets. The best and second best results are bolded and underlined, respectively. “-” indicates the result is not supported by the released implementation.

Methods	LLaVA-1.5			InstructBLIP			Qwen-VL
Methods	CHAIR i $\downarrow$	CHAIR s $\downarrow$	Recall $\uparrow$	CHAIR i $\downarrow$	CHAIR s $\downarrow$	Recall $\uparrow$	CHAIR i $\downarrow$	CHAIR s $\downarrow$	Recall $\uparrow$
Greedy	53.0	15.2	76.7	57.0	16.7	70.4	6.6	7.0	45.5
Beam Search	55.6	16.0	76.7	56.2	14.7	71.4	6.4	5.7	45.7
MemVR	49.2	12.9	78.4	-	-	-	4.7	4.0	45.3
ClearSight	50.0	13.6	78.8	-	-	-	4.2	3.8	44.0
IMCCD	52.2	13.5	76.6	55.3	17.0	73.0	6.6	5.3	41.7

Table 2: CHAIR hallucination evaluation results. The best and second best results are bolded and underlined, respectively.

Model	Decoding	Object-level		Attribute-level		Total $\uparrow$
Model	Decoding	Existence $\uparrow$	Count $\uparrow$	Position $\uparrow$	Color $\uparrow$	Total $\uparrow$
LLaVA-1.5	Greedy	170.00	130.00	93.33	121.66	514.99
	Beam Search	170.00	130.00	93.33	121.66	514.99
	MemVR	190.00	155.00	133.33	170.00	648.33
	ClearSight	190.00	160.00	133.33	160.00	643.33
	IMCCD	195.00	138.33	123.33	155.00	611.66

Table 3: Results on the hallucination subset of MME. The best performances within each setting are bolded.

Figure 4: Results of LLaVA1.5 on MME-Fullset.

POS	NEG	MME	MME	POPE(Random & COCO)
POS	NEG	(Hallucination)	(Overall)	F1-score	Accuracy
		514.99	1558.76	84.60	86.76
+		654.66	1843.92	85.39	87.00
	+	643.66	1819.95	85.50	87.06
+	+	666.66	1851.23	86.99	88.26
-	-	505.00	1531.26	83.73	86.20

Table 4: Ablation study of different components in CLVA for LLaVA-1.5. POS and NEG denote positive and negative anchors, while + and - represent the positive and negative usage of anchors, respectively.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Method	Accuracy	Total Time	GPU-Memory	Latency/Example
LLaVA-1.5	Greedy	84.79	0:7:48	15.7G	0.155s
	IMCCD	85.49	0:16:38	18.7G	0.353s
	CLVA (ours)	85.98	0:7:54	15.8G	0.157s
InstructBLIP	Greedy	77.02	0:5:07	16.6G	0.102s
	IMCCD	81.60	0:10:30	16.7G	0.207s
	CLVA (ours)	82.46	0:5:22	16.6G	0.105s

Appendix

Table 5: Comparison of inference speed and GPU memory usage for different models and methods on the POPE dataset

Figure 5: Ablation study of

\alpha

and

\beta

	Method	LLaVA-1.5		InstructBLIP		Qwen-VL
	Method	F1	Accuracy	F1	Accuracy	F1	Accuracy
random	sample	81.94	83.76	81.12	81.50	83.72	85.73
random	ours	82.97	84.33	84.06	84.43	84.87	86.56
popular	sample	80.86	82.56	78.74	78.50	83.03	84.96
popular	ours	83.52	84.96	82.54	81.55	83.54	85.86
adversarial	sample	78.43	79.73	77.91	77.43	81.43	83.13

Appendix

Table 6: Results of different models using nucleus sampling on the POPE dataset

Figure 6: Is there a backpack in the image?

Figure 7: Is there a bicycle in the image?

Figure 8: An example of the LLaVA-1.5 model on LLaVA-Bench; red text indicates incorrect answers.

Figure 9: An example of the InstructBLIP model on LLaVA-Bench; red text indicates incorrect answers.

Figure 10: An example of the Qwen-VL model on LLaVA-Bench; red text indicates incorrect answers.

Object Hallucination in VLMs

Object hallucination in vision-language models is the production of object claims unsupported by visual evidence.

See all 128 papers

MHSA: A Lightweight Framework for Mitigating Hallucinations via Steered Attention in LVLMs
May 14, 2026Wei Ding, Yilin Li, Yudong Zhang +4Vision-Language Models LLM Hallucination Mitigation
Tsinghua University Beijing, China · Tsinghua University, Tencent Beijing, China · Tencent Beijing, China +2
When Looking Is Not Enough: Visual Attention Structure Reveals Hallucination in MLLMs
May 12, 2026Fanpu Cao, Xin Zou, Xuming Hu +1Hallucination Detection in VLMs Hallucination in Language Models
Thrust of Artificial Intelligence, HKUST (Guangzhou), China · Department of Computer Science and Engineering, HKUST, Hong Kong SAR, China
Detecting Object Hallucinations in Large Vision-Language Models via Cross-Modal Attention Drifts and Mask-Based Verification
Sep 2, 2026Xuanbing Wen, Boxu Chen, Le Yang +4Hallucination Detection in VLMs Hallucination Detection
Xi’an Jiaotong University, Xi’an 710049, China · Zhongguancun Laboratory, Beijing 100194, China

May 14, 2026J/K move · Enter open · S save

MHSA: A Lightweight Framework for Mitigating Hallucinations via Steered Attention in LVLMs

Wei Ding, Yilin Li, Yudong Zhang, Ruobing Xie+3

Tsinghua University Beijing, China · Tsinghua University, Tencent Beijing, China · Tencent Beijing, China+2

Vision-Language Models LLM Hallucination Mitigation Hallucination in Language Models VLM Hallucination Large Vision-Language Models Cross-Modal Attention+1

Open paper AskPDF arXiv

Large vision-language models (LVLMs) have achieved remarkable performance across diverse multimodal tasks, yet they continue to suffer from hallucinations, generating content that is inconsistent with the visual input. Prior work DHCP (Detecting Hallucinations by Cross-modal Attention Pattern) has explored hallucination detection from the perspective of cross-modal attention, but does not address hallucination mitigation. In this paper, we propose MHSA (Mitigating Hallucinations via Steered Attention), a lightweight framework that mitigates hallucinations by learning to correct cross-modal attention patterns in LVLMs. MHSA trains a simple three-layer MLP generator to produce corrected attention, guided by supervisory signals from the DHCP discriminator and the LVLM itself. During inference, MHSA mitigates both discriminative and generative hallucinations across various datasets and LVLMs by simply replacing the original cross-modal attention with the corrected one, without modifying any LVLM parameters. By extending cross-modal attention mechanisms from hallucination detection to hallucination mitigation, MHSA offers a novel perspective on hallucination research in LVLMs and helps enhance their reliability.

Abstract

Object Hallucination in VLMs

Explore similar work

MHSA: A Lightweight Framework for Mitigating Hallucinations via Steered Attention in LVLMs

When Looking Is Not Enough: Visual Attention Structure Reveals Hallucination in MLLMs

Detecting Object Hallucinations in Large Vision-Language Models via Cross-Modal Attention Drifts and Mask-Based Verification