MLLMs Hallucinate when Information Distribution Drifts in Synergy Heads

Meng'en Qin; Junye Chen; Jucheng Liu; Yinchen Liu; Youlu Xing; Song Wang; Ruize Han

cs.CVSep 5, 2026

MLLMs Hallucinate when Information Distribution Drifts in Synergy Heads

Authors: Meng'en Qin, Junye Chen, Jucheng Liu, Yinchen Liu, Youlu Xing, Song Wang, Ruize Han

Organizations: Shenzhen University of Advanced Technology, Shenzhen, China · University of Electronic Science and Technology of China, Chengdu, China

Abstract

Multimodal Large Language Models (MLLMs) often struggle with hallucinations, thus hindering their reliable practical applications. Existing attention-based mitigation methods mainly rely on indirect signals (e.g., attention weights) that fail to accurately reflect the actual information shift underlying hallucination generation. In this paper, we propose HEAL, Head-lEvel information disentAnglement and caLibration for identifying and mitigating hallucinations. HEAL first employs causal noise intervention on multi-head outputs to filter out causally redundant heads. Subsequently, it disentangles information distribution within the remaining heads via the counterfactual Difference-in-Differences, categorizing heads into four types. Through analysis, we observe: hallucinations happen when information distribution drifts away from a healthy equilibrium in synergy heads, not strongly correlated with the quantity or strength of modality-specific heads. Motivated by this insight, HEAL injects dynamic information calibration factors into the value vectors of synergy heads and actively regulates visual-language dependencies, steering the output distribution towards factual evidence. Extensive experiments demonstrate that HEAL effectively reduces hallucinations across multiple MLLMs, offering a simple and interpretable pathway to enhance model trustworthiness.

Figures & tables

Figure 1: (a) Mean number of heads of different types when generating hallucinated versus correct tokens; (b) Mean information content of visual and language heads in hallucinated and correct token generation; (c) Mean information proportion within synergy heads for hallucinated and correct tokens.

Figure 2: (a) Head distribution in the LLaVA-NeXT-7B model when generating different tokens, where green, blue, yellow and gray denote visual, synergy, language and redundant heads, respectively; (b) The evolution of the number of heads in the autoregressive generation. (c) 3D scatter plot of attention heads of different types, where the axes correspond to visual, language and synergy contributions. (d) Illustration of the internal information structure within a single attention head. The visualization in (a) and (b) is based on the same VQA example as in Figure 1 . From the figure, we observe that the number of visual heads remains nearly constant throughout the generation.

Figure 3: Illustration of the proposed HEAL. It employs the counterfactual difference-in-differences mechanism to disentangle the information distribution within attention heads and dynamically calibrates the information drift in synergy heads.

Method	POPE		CHAIR				MME
Method	F1 $\uparrow$	Acc $\uparrow$	C S $\downarrow$	C I $\downarrow$	Recall $\uparrow$	Length	Exist. $\uparrow$	Count $\uparrow$	Pos. $\uparrow$	Color $\uparrow$	Total $\uparrow$
Beam Search	85.4	84.0	51.0	15.2	75.2	102.2	175.67	124.67	114.00	151.00	565.34
DoLa ( Chuang et al., 2023 )	80.2	83.1	57.0	15.2	78.2	97.5	180.10	127.40	119.30	154.60	594.10
VCD ( Leng et al., 2024 )	85.3	85.0	51.0	14.9	77.2	101.9	184.66	137.33	128.67	153.00	603.66
OPERA ( Huang et al., 2024 )	84.2	85.2	47.0	14.6	78.5	95.3	180.67	133.33	111.67	123.33	549.00
DOPRA ( Wei and Zhang, 2024 )	84.6	84.3	46.3	13.8	78.2	96.1	185.67	138.33	120.67	133.00	577.67

Table 1: Comparison of HEAL with other SOTA methods on POPE, CHAIR, and MME datasets. The best performances are bolded and baseline model is LLaVA-1.5-7B.

	Comprehensive Benchmark	Hallucination Benchmark
Method	LLaVA-Bench $\uparrow$	CHAIR S $\downarrow$	CHAIR I $\downarrow$	POPE-R $\uparrow$	POPE-F1 $\uparrow$	POPE-A $\uparrow$
LLaVA-1.5-7B	72.5	51.0	15.2	87.0	85.4	84.0
+ HEAL	75.2	36.9	10.7	89.1	87.8	88.5
LLaVA-NeXT-7B	81.6	29.9	9.2	87.4	86.5	84.7
+ HEAL	82.2	24.6	7.9	89.9	88.1	89.0
Qwen2.5-VL-7B	76.8	27.2	9.0	80.4	87.4	88.4

Table 2: MLLM performance with and without HEAL on the POPE, CHAIR, and LLaVA-Bench.

Figure 5: (a) An example of HEAL eliminating the hallucinated content. (b) The generated content under varying settings of the equilibrium factor.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Ratio	0.3	base (no intervening)	0.4	0.5	0.6	0.7
LLaVA-1.5-7B	72.1	72.5	73.6	74.6	75.2	74.8
Qwen2.5-VL-7B	75.6	76.8	78.3	78.5	78.2	77.9

Appendix

Table 4: Bidirectional causal intervention in LLaVA-1.5-7B and Qwen2.5-VL-7B by manipulating the visual-language ratio. Performance scores are evaluated on the LLaVA-Bench.

Figure 6: Bidirectional causal intervention in LLaVA-1.5-7B and Qwen2.5-VL-7B replies by manipulating the visual-language information distribution (equilibrium factor). base means no intervention.

Masking strategy	Gaussian masking (base)	one-point (zero) masking	uniform masking	swapping actual tokens
Head Assignment Agreement (%)	100.00	92.13	93.21	95.36

Appendix

Table 5: Head type assignment agreement under different masking strategies.

Masking strategy	Gaussian masking (base)	one-point (zero) masking	uniform masking	swapping actual tokens
F1 score $\uparrow$	87.84	87.12	86.93	87.53
correct tokens	0.43:0.54	0.41:0.53	0.39:0.48	0.41:0.56
hallucinated tokens	0.28:0.62	0.21:0.52	0.23:0.57	0.31:0.66

Appendix

Table 6: Robustness to different masking strategies. We report the visual-language information ratios of correct and hallucinated tokens in synergy heads and LLaVA-1.5-7B performance on POPE.

Table 7: Robustness to threshold selection. We report the visual-language information ratios of correct and hallucinated tokens in synergy heads and LLaVA-1.5-7B performance on POPE.

Metric	normalized cosine similarity	$L_{2}$ norm + Sigmoid	cosine similarity + $L_{2}$ norm + Sigmoid
Head Assignment Agreement (%)	100.00 (base)	95.73	97.98

Appendix

Table 8: Head assignment agreement under different measurement metrics.

	no intervention	Gaussian (base)	one-point (zero)	uniform	Cauchy (extreme outliers)	swapping actual outputs
Head Assignment Agreement (%)	-	100.00	95.25	96.31	97.46	96.97
F1 score $\uparrow$	86.89	87.84	87.73	87.69	87.81	87.75

Appendix

Table 9: Robustness to different replacement distributions in Section 3.1 . We report the Head Assignment Agreement metric and LLaVA-1.5-7B performance on POPE.

Metric	before Yeo-Johnson transformation (base)	after Yeo-Johnson transformation
Head Assignment Agreement (%)	100.00	98.25

Appendix

Table 10: Redundant head assignment agreement before and after Yeo-Johnson transformation.

$\sigma(I^{(t,l,:)})$ coefficient	1	2	3	$\mathbf{\infty}$
F1 score $\uparrow$	86.97	87.21	87.84	87.89

Appendix

Table 11: Robustness analysis on the

\sigma(I^{(t,l,:)})

coefficient. We report LLaVA-1.5-7B performance on POPE.

Figure 7: Head distribution in an update interval of 10 steps. We observe that the macroscopic distribution of head attributes shows negligible variation, justifying our periodic update strategy.

Equilibrium Factor	LLaVA-Bench (LLaVA-1.5-7B) $\uparrow$	LLaVA-Bench (Qwen2.5-VL-7B) $\uparrow$	$C_{S}$ (LLaVA-1.5-7B) $\downarrow$	$C_{S}$ (Qwen2.5-VL-7B) $\downarrow$
0.3	72.1	75.6	39.0	23.8
0.4	73.6	78.3	37.1	23.3
0.5	74.6	78.5	36.9	23.5
0.6	75.2	78.2	37.4	24.1
0.7	74.8	77.9	38.1	24.9

Appendix

Table 12: LLaVA-Bench score and

C_{S}

(CHAIR) under different equilibrium factors.

Update Interval	LLaVA-Bench (LLaVA-1.5-7B) $\uparrow$	LLaVA-Bench (Qwen2.5-VL-7B) $\uparrow$	POPE-F1 (LLaVA-1.5-7B) $\uparrow$	POPE-F1 (Qwen2.5-VL-7B) $\uparrow$
2	75.5	78.7	87.9	88.8
5	75.3	78.7	87.8	88.6
10	75.2	78.5	87.8	88.4
15	75.0	78.2	87.5	88.3
20	74.7	77.8	87.1	88.0

Appendix

Table 13: LLaVA-Bench score and POPE-F1 under different update intervals.

Method	LLaVA-Bench $\uparrow$	CHAIR ${}_{S}\downarrow$	CHAIR ${}_{I}\downarrow$	POPE-R $\uparrow$	POPE-F1 $\uparrow$	POPE-A $\uparrow$
InternVL-7B	51.6	46.6	12.4	80.0	85.3	86.2
+LocoRE ( Zhang et al., 2026 )	52.8	40.2	10.5	85.8	87.2	87.3
+HEAL	53.4	39.2	9.6	86.5	87.8	87.9

Appendix

Table 14: Intern-VL performance comparison with our and other methods.

Method	MMHal-Bench		BLINK-Twice
Method	Halluc. Rate $\downarrow$	Score $\uparrow$	No-Acc	Yes-Acc	Q-Acc	I-Acc	G-Acc
Qwen3-VL-8B	17.5	4.82	0.476	0.653	0.493	0.336	0.140
+HEAL	16.6	4.91	0.557	0.714	0.536	0.353	0.213
InternVL3.5-8B	19.4	4.53	0.355	0.588	0.475	0.264	0.109
+HEAL	18.3	4.71	0.463	0.624	0.583	0.334	0.158

Appendix

Table 15: HEAL performance on recent benchmarks and MLLMs.

Multimodal Hallucination

Multimodal hallucination occurs when AI systems produce claims unsupported by, or conflicting with, image, audio, video, or other input evidence.

See all 51 papers

Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation
Feb 27, 2026Xingyu Zhu, Kesen Zhao, Liang Yi +4VLM Hallucination Mitigation Multimodal Large Language Models
MoE Key Lab of BIPC, University of Science and Technology of China · Nanyang Technological University
HalluScope: Fine-grained Hallucination Diagnosis for Multimodal Large Language Models
Jul 23, 2026Weilin Jin, Mingyu Wang, Wenbo Li +5Hallucination Detection in VLMs Multimodal Hallucination
Peking University Beijing, China · Joy Future Academy Beijing, China
Spotlight and Shadow: Attention-Guided Dual-Anchor Introspective Decoding for MLLM Hallucination Mitigation
Apr 11, 2026Yebo Wu, Han Jin, Zhijiang Guo +1VLM Hallucination Mitigation Language Model Decoding
State Key Laboratory of IOTSC, University of Macau · Hong Kong University of Science and Technology (Guangzhou)

Feb 27, 2026J/K move · Enter open · S save

Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation

Xingyu Zhu, Kesen Zhao, Liang Yi, Shuo Wang+3

MoE Key Lab of BIPC, University of Science and Technology of China · Nanyang Technological University

VLM Hallucination Mitigation Multimodal Large Language Models

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) have achieved remarkable progress in vision-language reasoning, yet they remain vulnerable to hallucination, where generated content deviates from visual evidence. Existing mitigation strategies either require costly supervision during training or introduce additional latency at inference time. Recent vision enhancement methods attempt to address this issue by reinforcing visual tokens during decoding, but they typically inject all tokens indiscriminately, which causes interference from background regions and distracts the model from critical cues. To overcome this challenge, we propose Adaptive Visual Reinforcement (AIR), a training-free framework for MLLMs. AIR consists of two components. Prototype-based token reduction condenses the large pool of visual tokens into a compact subset to suppress redundancy. OT-guided patch reinforcement quantifies the alignment between hidden states and patch embeddings to selectively integrate the most consistent patches into feed-forward layers. As a result, AIR enhances the model's reliance on salient visual information and effectively mitigates hallucination. Extensive experiments across representative MLLMs demonstrate that AIR substantially reduces hallucination while preserving general capabilities, establishing it as an effective solution for building reliable MLLMs.

Abstract

Multimodal Hallucination

Explore similar work

Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation

HalluScope: Fine-grained Hallucination Diagnosis for Multimodal Large Language Models

Spotlight and Shadow: Attention-Guided Dual-Anchor Introspective Decoding for MLLM Hallucination Mitigation