Understanding and Mitigating Hallucinations in Multimodal Chain-of-Thought Models
Authors: Ji Ma, Wei Suo, Peng Wang, Yanning Zhang
Organizations: School of Computer Science and Ningbo Institute, Northwestern Polytechnical University, China. · National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology, China.
Multimodal Chain-of-Thought (MCoT) models have demonstrated impressive capability in complex visual reasoning tasks. Unfortunately, recent studies reveal that they suffer from severe hallucination problems due to diminished visual attention during the generation process. However, visual attention decay is a well-studied problem in Large Vision-Language Models (LVLMs). Considering the fundamental differences in reasoning processes between MCoT models and traditional LVLMs, we raise a basic question: Whether MCoT models have unique causes of hallucinations? To answer this question, we systematically investigate the hallucination patterns of MCoT models and find that fabricated texts are primarily generated in associative reasoning steps, which we term divergent thinking. Leveraging these insights, we introduce a simple yet effective strategy that can effectively localize divergent thinking steps and intervene in the decoding process to mitigate hallucinations. Extensive experiments show that our method outperforms existing methods by a large margin. More importantly, our proposed method can be conveniently integrated with other hallucination mitigation methods and further boost their performance. The code is publicly available at https://github.com/ASGO-MM/MCoT-hallucination.
Figures & tables
Figure 1: Visualization of different reasoning paradigms. (a) Traditional LVLMs apply an implicit reasoning paradigm. (b) MCoT models reason explicitly and answers are influenced by the thinking process.
Figure 2: (a) Co-occurrence relationship of hallucinations in the MCoT model’s thinking and answering. We find that hallucinations in thinking and answering are positively correlated. (b) Further attention visualization reveals that this phenomenon is caused by the MCoT model’s attention bias towards its thinking process when generating final answers.
Figure 3: Hallucination patterns in the MCoT model’s thinking process. We find that models are prone to hallucinations during associative reasoning steps, termed as divergent thinking . In this thinking mode, MCoT models exhibit ∼ 5 times more hallucinations compared to normal thinking.
Figure 4: (a) Visual entropy across different thinking modes. The results show that models exhibit higher values when engaging in divergent thinking. (b) Predicting the divergent thinking mode using visual entropy . The logistic curve demonstrates that high visual entropy reliably predicts the divergent thinking steps.
Figure 5: Visualization of visual entropy . We randomly choose ten image tokens for clarity, and dark colors denote low entropy values. (a) Compared to normal thinking, entropy values are significantly higher in divergent thinking steps. (b) After applying our method, visual entropy shows an evident decline.
Figure 6: Overview of our method: Given the image and user’s query, we first leverage MCoT model’s internal visual entropy to detect divergent thinking steps. When the model is identified as thinking divergently, we further use this score to dynamically intervene in the decoding process and mitigate hallucinations.
Methods
Object HalBench
MMhalBench
POPE Random
POPE Adversarial
POPE Popular
CHAIR S ↓
CHAIR I ↓
HalRate ↓
Score ↑
Acc ↑
F1 ↑
Acc ↑
F1 ↑
Acc ↑
F1 ↑ 0
GRIT-3B [ 8 ]
23.8
10.5
0.51
1.94
78.1
72.2
77.5
71.8
78.6
73.1
+ DoLa [ 6 ]
21.6
8.0
0.49
1.98
79.0
73.6
77.6
72.2
78.3
72.7
+ VCD [ 21 ]
20.4
7.9
0.50
1.97
78.9
73.4
78.1
73.0
79.0
73.7
+ MemVR [ 65 ]
19.4
8.1
0.46
2.01
79.5
74.3
78.7
73.8
78.9
73.6
+ FlexAC [ 61 ]
19.2
7.4
0.48
2.04
79.8
74.9
79.0
74.1
79.6
74.7
Table 1: Performance comparison with state-of-the-art methods on several hallucination benchmarks. The best-performing method is highlighted in bold . ↑ indicates that higher is better, while ↓ indicates that lower is better.
Method
MathVista
MMStar
VSR
GRIT-3B [ 8 ]
59.8
50.7
72.9
+ Ours
60.9
51.5
76.1
PixelReasoner-7B [ 53 ]
62.6
58.7
82.4
+ Ours
64.0
59.4
84.3
R1-Onevision-7B [ 60 ]
64.1
52.8
73.4
+ Ours
65.0
54.2
74.9
Table 2: Evaluation on perception and reasoning benchmarks.
Methods
Object HalBench
MathVista
CHAIR S ↓
CHAIR I ↓
Acc ↑
PixelReasoner-7B
22.0
7.8
62.6
Intervention Location:
All Thinking
16.1
5.7
61.4
Normal-only
21.8
7.7
62.7
Divergent-only (Ours)
15.4
5.3
64.0
Table 3: The effects of different settings for our method. “All thinking” denotes intervening in the MCoT model’s entire thinking process, while “Normal-only” denotes only modulating the normal thinking steps.
Methods
Object HalBench
MathVista
CHAIR S ↓
CHAIR I ↓
Acc ↑
PixelReasoner-7B
22.0
7.8
62.6
+ DoLa [ 6 ]
20.6
7.1
62.2
+ DoLa + Ours
17.9
6.0
63.4
+ VCD [ 21 ]
19.8
7.2
61.9
+ VCD + Ours
16.1
5.3
63.5
Table 4: Compatibility with other mitigation strategies.
Methods
Object HalBench
MathVista
CHAIR S ↓
CHAIR I ↓
Acc ↑
Greedy Decoding:
Original
22.0
7.8
63.1
+ Ours
15.4
5.3
63.6
Nucleus Sampling:
Original
25.8
9.4
62.6
Table 5: Compatibility with different decoding strategies.
Figure 7: (a) Performance balance between reasoning and hallucinations. (b) Trade-off between effectiveness and efficiency.
Figure 8: (a) Effects of different threshold γ . (b) Effects of different intervention degree α .
Figure 9: Qualitative analyses. The original MCoT models produce hallucinated contents. Our method effectively mitigates hallucinations and performs faithful reasoning and answering.