Organizations: University of North Carolina at Chapel Hill · University of Science and Technology of China · Arizona State University · Honda Research Institute, USA
The recent success of reinforcement learning (RL) in large reasoning models has inspired the growing adoption of RL for post-training Multimodal Large Language Models (MLLMs) to enhance their visual reasoning capabilities. Although many studies have reported improved performance, it remains unclear whether RL training truly enables models to learn from visual information. In this work, we propose the Hallucination-as-Cue Framework, an analytical framework designed to investigate the effects of RL-based post-training on multimodal reasoning models from the perspective of model hallucination. Specifically, we introduce hallucination-inductive, modality-specific corruptions that remove or replace essential information required to derive correct answers, thereby forcing the model to reason by hallucination. By applying these corruptions during both training and evaluation, our framework provides a unique perspective for diagnosing RL training dynamics and understanding the intrinsic properties of datasets. Through extensive experiments and analyses across multiple multimodal reasoning benchmarks, we reveal that the role of model hallucination for RL-training is more significant than previously recognized. For instance, we find that RL post-training under purely hallucination-inductive settings can still significantly improve models' reasoning performance, and in some cases even outperform standard training. These findings challenge prevailing assumptions about MLLM reasoning training and motivate the development of more modality-aware RL-based training designs.
Figures & tables
Figure 2: Hallucination-as-Cue Framework. (a) Modality-Specific Corruptions : We define three types of data corruptions: Blank Image Replacement , Random Image Replacement , and Textual Information Removal . (b) Hallucination-Inductive Training : We apply these types of modality-specific corruptions to the training data to obtain three models. Since the input information is corrupted, the model learns to hallucinate the corrupted information for inference. We refer to this process as hallucination-inductive training. (c) Hallucination-Inductive Inference and Analysis : We then analyze these three models under the three types of data corruptions to compare task accuracy and examine model behavior.
Figure 3: Accuracy of different training regimes on the normal training and test sets.
Figure 4: Accuracy of different training regimes on corrupted training and test sets.
Model
MathVision (%)
MathVerse (%)
MathVista (%)
WeMath (%)
AVG (%)
Qwen2.5-VL-3B
Qwen2.5-VL-3B
18.19
34.82
51.40
54.48
39.72
Qwen2.5-VL-3B + GRPO
22.73
37.72
58.40
60.11
44.74
Qwen2.5-VL-3B + GRPO-BI
20.95
35.10
56.40
56.55
42.25
Qwen2.5-VL-3B + GRPO-RI
20.86
35.76
58.00
55.17
42.45
Qwen2.5-VL-3B + GRPO-TR
19.97
35.53
58.50
55.34
42.34
Table 1: Benchmark results comparing different training regimes with Qwen2.5-VL [ 2 ] baselines. All RL-based training is performed on the Geometry3K [ 22 ] dataset. For the 3B model, standard GRPO yields the best average performance. Strikingly, for the 7B model, hallucination-inductive training (GRPO-RI) achieves the highest average score, even outperforming standard GRPO trained on clean data.
Method
Post-training Dataset
MathVision (%)
MathVerse (%)
MathVista (%)
WeMath (%)
AVG (%)
Qwen2.5-VL-3B
Qwen2.5-VL-3B
–
18.19
34.82
51.40
54.48
39.72
GRPO
Geometry3K
22.73
37.72
58.40
60.11
44.74
GRPO-BI
Geometry3K
20.95
35.10
56.40
56.55
42.25
GRPO
MMR1-V0
26.18
39.26
65.00
62.47
48.23
GRPO-BI
MMR1-V0
24.28
40.03
61.20
61.61
46.78
Table 2: Benchmark results analyzing the effect of different post-training datasets. We compare the performance of the base Qwen2.5-VL model, the standard GRPO-trained model, and the BI-corrupted GRPO-trained model across three post-training datasets: Geometry3K [ 22 ] , MMR1-V0 [ 14 ] , and CLEVR [ 11 ] .
Model
Inference Corruption?
MathVision (%)
MathVerse (%)
MathVista (%)
WeMath (%)
AVG (%)
Qwen2.5-VL-3B
Qwen2.5-VL-3B
no
18.19
34.82
51.40
54.48
39.72
Qwen2.5-VL-3B
yes
18.91( +0.72 )
19.14( -15.68 )
28.10( -23.30 )
29.14( -25.34 )
23.82( -15.90 )
Qwen2.5-VL-3B+GRPO
no
22.73
37.72
58.40
60.11
44.74
Qwen2.5-VL-3B+GRPO
yes
20.82( -1.91 )
21.65( -16.07 )
28.80( -29.60 )
33.22( -26.89 )
26.12( -18.62 )
Qwen2.5-VL-3B+GRPO-BI
no
20.95
35.10
56.40
56.55
42.25
Table 3: Benchmark results of different models with BI corruption applied to the evaluation data. Performance differences relative to normal inference are indicated in the table.
Model
Inference Corruption?
Text Dominant
Text Lite
Vision Intensive
Vision Dominant
Qwen2.5-VL-7B
no
54.82
48.73
45.94
45.81
Qwen2.5-VL-7B
yes
50.51
28.17
20.05
23.86
Δ
-
-4.31
-20.56
-25.89
-21.95
Qwen2.5-VL-7B + GRPO
no
59.77
50.63
46.70
47.46
Qwen2.5-VL-7B + GRPO
yes
52.28
30.58
21.57
24.49
Δ
-
-7.49
-20.05
-25.13
-22.97
Table 4: Fine-grained analysis of different visual reasoning problem types using the MathVerse [ 41 ] benchmark. BI corruption is applied as the study case for inference corruption.