Multimodal Large Language Models (MLLMs) often struggle with hallucinations, thus hindering their reliable practical applications. Existing attention-based mitigation methods mainly rely on indirect signals (e.g., attention weights) that fail to accurately reflect the actual information shift underlying hallucination generation. In this paper, we propose HEAL, Head-lEvel information disentAnglement and caLibration for identifying and mitigating hallucinations. HEAL first employs causal noise intervention on multi-head outputs to filter out causally redundant heads. Subsequently, it disentangles information distribution within the remaining heads via the counterfactual Difference-in-Differences, categorizing heads into four types. Through analysis, we observe: hallucinations happen when information distribution drifts away from a healthy equilibrium in synergy heads, not strongly correlated with the quantity or strength of modality-specific heads. Motivated by this insight, HEAL injects dynamic information calibration factors into the value vectors of synergy heads and actively regulates visual-language dependencies, steering the output distribution towards factual evidence. Extensive experiments demonstrate that HEAL effectively reduces hallucinations across multiple MLLMs, offering a simple and interpretable pathway to enhance model trustworthiness.
Figures & tables
Figure 1: (a) Mean number of heads of different types when generating hallucinated versus correct tokens; (b) Mean information content of visual and language heads in hallucinated and correct token generation; (c) Mean information proportion within synergy heads for hallucinated and correct tokens.
Figure 2: (a) Head distribution in the LLaVA-NeXT-7B model when generating different tokens, where green, blue, yellow and gray denote visual, synergy, language and redundant heads, respectively; (b) The evolution of the number of heads in the autoregressive generation. (c) 3D scatter plot of attention heads of different types, where the axes correspond to visual, language and synergy contributions. (d) Illustration of the internal information structure within a single attention head. The visualization in (a) and (b) is based on the same VQA example as in Figure 1 . From the figure, we observe that the number of visual heads remains nearly constant throughout the generation.
Figure 3: Illustration of the proposed HEAL. It employs the counterfactual difference-in-differences mechanism to disentangle the information distribution within attention heads and dynamically calibrates the information drift in synergy heads.
Method
POPE
CHAIR
MME
F1 ↑
Acc ↑
C S ↓
C I ↓
Recall ↑
Length
Exist. ↑
Count ↑
Pos. ↑
Color ↑
Total ↑
Beam Search
85.4
84.0
51.0
15.2
75.2
102.2
175.67
124.67
114.00
151.00
565.34
DoLa ( Chuang et al., 2023 )
80.2
83.1
57.0
15.2
78.2
97.5
180.10
127.40
119.30
154.60
594.10
VCD ( Leng et al., 2024 )
85.3
85.0
51.0
14.9
77.2
101.9
184.66
137.33
128.67
153.00
603.66
OPERA ( Huang et al., 2024 )
84.2
85.2
47.0
14.6
78.5
95.3
180.67
133.33
111.67
123.33
549.00
DOPRA ( Wei and Zhang, 2024 )
84.6
84.3
46.3
13.8
78.2
96.1
185.67
138.33
120.67
133.00
577.67
Table 1: Comparison of HEAL with other SOTA methods on POPE, CHAIR, and MME datasets. The best performances are bolded and baseline model is LLaVA-1.5-7B.
Comprehensive Benchmark
Hallucination Benchmark
Method
LLaVA-Bench ↑
CHAIR S ↓
CHAIR I ↓
POPE-R ↑
POPE-F1 ↑
POPE-A ↑
LLaVA-1.5-7B
72.5
51.0
15.2
87.0
85.4
84.0
+ HEAL
75.2
36.9
10.7
89.1
87.8
88.5
LLaVA-NeXT-7B
81.6
29.9
9.2
87.4
86.5
84.7
+ HEAL
82.2
24.6
7.9
89.9
88.1
89.0
Qwen2.5-VL-7B
76.8
27.2
9.0
80.4
87.4
88.4
Table 2: MLLM performance with and without HEAL on the POPE, CHAIR, and LLaVA-Bench.
Figure 6
Figure 5: (a) An example of HEAL eliminating the hallucinated content. (b) The generated content under varying settings of the equilibrium factor.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Ratio
0.3
base (no intervening)
0.4
0.5
0.6
0.7
LLaVA-1.5-7B
72.1
72.5
73.6
74.6
75.2
74.8
Qwen2.5-VL-7B
75.6
76.8
78.3
78.5
78.2
77.9
Appendix
Table 4: Bidirectional causal intervention in LLaVA-1.5-7B and Qwen2.5-VL-7B by manipulating the visual-language ratio. Performance scores are evaluated on the LLaVA-Bench.
Figure 6: Bidirectional causal intervention in LLaVA-1.5-7B and Qwen2.5-VL-7B replies by manipulating the visual-language information distribution (equilibrium factor). base means no intervention.
Masking strategy
Gaussian masking (base)
one-point (zero) masking
uniform masking
swapping actual tokens
Head Assignment Agreement (%)
100.00
92.13
93.21
95.36
Appendix
Table 5: Head type assignment agreement under different masking strategies.
Masking strategy
Gaussian masking (base)
one-point (zero) masking
uniform masking
swapping actual tokens
F1 score ↑
87.84
87.12
86.93
87.53
correct tokens
0.43:0.54
0.41:0.53
0.39:0.48
0.41:0.56
hallucinated tokens
0.28:0.62
0.21:0.52
0.23:0.57
0.31:0.66
Appendix
Table 6: Robustness to different masking strategies. We report the visual-language information ratios of correct and hallucinated tokens in synergy heads and LLaVA-1.5-7B performance on POPE.
Table 7: Robustness to threshold selection. We report the visual-language information ratios of correct and hallucinated tokens in synergy heads and LLaVA-1.5-7B performance on POPE.
Metric
normalized cosine similarity
L2 norm + Sigmoid
cosine similarity + L2 norm + Sigmoid
Head Assignment Agreement (%)
100.00 (base)
95.73
97.98
Appendix
Table 8: Head assignment agreement under different measurement metrics.
no intervention
Gaussian (base)
one-point (zero)
uniform
Cauchy (extreme outliers)
swapping actual outputs
Head Assignment Agreement (%)
-
100.00
95.25
96.31
97.46
96.97
F1 score ↑
86.89
87.84
87.73
87.69
87.81
87.75
Appendix
Table 9: Robustness to different replacement distributions in Section 3.1 . We report the Head Assignment Agreement metric and LLaVA-1.5-7B performance on POPE.
Metric
before Yeo-Johnson transformation (base)
after Yeo-Johnson transformation
Head Assignment Agreement (%)
100.00
98.25
Appendix
Table 10: Redundant head assignment agreement before and after Yeo-Johnson transformation.
σ(I(t,l,:)) coefficient
1
2
3
∞
F1 score ↑
86.97
87.21
87.84
87.89
Appendix
Table 11: Robustness analysis on the σ(I(t,l,:)) coefficient. We report LLaVA-1.5-7B performance on POPE.
Figure 7: Head distribution in an update interval of 10 steps. We observe that the macroscopic distribution of head attributes shows negligible variation, justifying our periodic update strategy.
Equilibrium Factor
LLaVA-Bench (LLaVA-1.5-7B) ↑
LLaVA-Bench (Qwen2.5-VL-7B) ↑
CS (LLaVA-1.5-7B) ↓
CS (Qwen2.5-VL-7B) ↓
0.3
72.1
75.6
39.0
23.8
0.4
73.6
78.3
37.1
23.3
0.5
74.6
78.5
36.9
23.5
0.6
75.2
78.2
37.4
24.1
0.7
74.8
77.9
38.1
24.9
Appendix
Table 12: LLaVA-Bench score and CS (CHAIR) under different equilibrium factors.
Update Interval
LLaVA-Bench (LLaVA-1.5-7B) ↑
LLaVA-Bench (Qwen2.5-VL-7B) ↑
POPE-F1 (LLaVA-1.5-7B) ↑
POPE-F1 (Qwen2.5-VL-7B) ↑
2
75.5
78.7
87.9
88.8
5
75.3
78.7
87.8
88.6
10
75.2
78.5
87.8
88.4
15
75.0
78.2
87.5
88.3
20
74.7
77.8
87.1
88.0
Appendix
Table 13: LLaVA-Bench score and POPE-F1 under different update intervals.
Method
LLaVA-Bench ↑
CHAIR S↓
CHAIR I↓
POPE-R ↑
POPE-F1 ↑
POPE-A ↑
InternVL-7B
51.6
46.6
12.4
80.0
85.3
86.2
+LocoRE ( Zhang et al., 2026 )
52.8
40.2
10.5
85.8
87.2
87.3
+HEAL
53.4
39.2
9.6
86.5
87.8
87.9
Appendix
Table 14: Intern-VL performance comparison with our and other methods.
Method
MMHal-Bench
BLINK-Twice
Halluc. Rate ↓
Score ↑
No-Acc
Yes-Acc
Q-Acc
I-Acc
G-Acc
Qwen3-VL-8B
17.5
4.82
0.476
0.653
0.493
0.336
0.140
+HEAL
16.6
4.91
0.557
0.714
0.536
0.353
0.213
InternVL3.5-8B
19.4
4.53
0.355
0.588
0.475
0.264
0.109
+HEAL
18.3
4.71
0.463
0.624
0.583
0.334
0.158
Appendix
Table 15: HEAL performance on recent benchmarks and MLLMs.
Multimodal large language models (MLLMs) have achieved remarkable progress in vision-language reasoning, yet they remain vulnerable to hallucination, where generated content deviates from visual evidence. Existing mitigation strategies either require costly supervision during training or introduce additional latency at inference time. Recent vision enhancement methods attempt to address this issue by reinforcing visual tokens during decoding, but they typically inject all tokens indiscriminately, which causes interference from background regions and distracts the model from critical cues. To overcome this challenge, we propose Adaptive Visual Reinforcement (AIR), a training-free framework for MLLMs. AIR consists of two components. Prototype-based token reduction condenses the large pool of visual tokens into a compact subset to suppress redundancy. OT-guided patch reinforcement quantifies the alignment between hidden states and patch embeddings to selectively integrate the most consistent patches into feed-forward layers. As a result, AIR enhances the model's reliance on salient visual information and effectively mitigates hallucination. Extensive experiments across representative MLLMs demonstrate that AIR substantially reduces hallucination while preserving general capabilities, establishing it as an effective solution for building reliable MLLMs.
Xingyu Zhu, Kesen Zhao, Liang Yi +4
MoE Key Lab of BIPC, University of Science and Technology of China · Nanyang Technological University
Although Multimodal Large Language Models have achieved strong performance across a wide range of vision-language tasks, they still suffer from hallucinations, where model outputs become inconsistent with the visual content, textual context, or commonsense knowledge. Existing studies primarily address this problem through coarse-grained detection. However, these approaches often provide insufficient diagnostic information for understanding hallucination types and supporting downstream hallucination mitigation. To bridge this gap, we propose fine-grained hallucination diagnosis for MLLMs, a new unified task that jointly performs hallucination detection, classification, and interpretable explanation generation. We develop an automated data generation pipeline and construct HalluScope-30K, a large-scale diagnostic dataset covering eight sources and five task categories. Based on this dataset, we design a multi-granular joint reward function and train two diagnosis models, HalluScope-4B and HalluScope-8B, which achieve state-of-the-art performance on both the MHALO benchmark and our fine-grained hallucination classification benchmark. Notably, detection and classification are mutually beneficial under joint optimization. Furthermore, diagnosis-driven feedback experiments show that the fine-grained diagnostic explanations produced by our model effectively guide target models to correct their hallucinations, with full diagnosis substantially outperforming all baselines on both Qwen3-VL-8B-Instruct and LLaVA-1.5-7B. Our code, data, and models are available at https://github.com/wkinglin/HalluScope.
Weilin Jin, Mingyu Wang, Wenbo Li +5
Peking University Beijing, China · Joy Future Academy Beijing, China
Multimodal Large Language Models (MLLMs) have demonstrated remarkable reasoning capabilities yet continue to suffer from hallucination, where generated text contradicts visual content. In this paper, we introduce Dual-Anchor Introspective Decoding (DaID), a novel contrastive decoding framework that dynamically calibrates each token generation by mining the model's internal perceptual discrepancies. Specifically, DaID identifies a Spotlight layer to amplify visual factual signals and a Shadow layer to suppress textual inertia. By leveraging visual attention distributions to guide this dual-anchor selection process, our method ensures precise, token-specific adaptation. Experimental results across multiple benchmarks and MLLMs demonstrate that DaID significantly mitigates hallucination while enhancing general reasoning capabilities.
Yebo Wu, Han Jin, Zhijiang Guo +1
State Key Laboratory of IOTSC, University of Macau · Hong Kong University of Science and Technology (Guangzhou)