Reinforcement Learning with Verifiable Rewards (RLVR) and Group Relative Policy Optimization (GRPO) have significantly advanced the reasoning capabilities of large language models. Extending these methods to multimodal settings, however, faces a critical challenge: the instability of std-based normalization, which is easily distorted by extreme samples with nearly positive or negative rewards. Unlike pure-text LLMs, multimodal models are particularly sensitive to such distortions, as both perceptual and reasoning errors influence their responses. To address this, we characterize each sample by its difficulty, defined through perceptual complexity (measured via visual entropy) and reasoning uncertainty (captured by model confidence). Building on this characterization, we propose difficulty-aware group normalization (Durian), which re-groups samples by difficulty levels and shares the std within each group. Our approach preserves GRPO's intra-group distinctions while eliminating sensitivity to extreme cases, yielding significant performance gains across multiple multimodal reasoning benchmarks.
Figures & tables
Figure 1: The advantage distribution after the normalization of reward varies among samples. Extreme samples like easy and hard ones are amplified after std -normalization, whereas medium samples exhibit more balanced advantages.
Figure 2: Overview of two difficulty-based regrouping strategies of Durian. Upper: For perceptual difficulty, we extract image patch features through the visual encoder and compute patch covariance matrices, whose eigenvalue entropy characterizes visual complexity. Bottom: For reasoning difficulty, model confidence is estimated from normalized sequence-level log probabilities across multiple rollouts. In both strategies, samples in the same group share the same std .
Model
Data Size
MathVerse
MathVision
MathVista
WeMath
HallusionBench
Average
Close-source models
GPT-4o
-
50.8
30.4
63.8
69.0
71.4
-
Claude-3.5-Sonnet
-
26.5
38.0
67.7
-
71.6
Open-source models
InternVL-2.5-8B-Instruct ( Chen et al., 2024 )
-
39.5
19.7
64.4
-
67.3
-
LLaVA-OneVision-7B ( Li et al., 2024 )
-
26.2
-
63.2
-
48.4
-
Table 1: Performance comparison of Multi-modal LLMs with over 5 benchmarks. Accuracy scores (%) are reported for all benchmarks for clarity. Data sizes used for SFT and RL are annotated in blue and red , respectively. The best value in each column is shown in bold , and the second-best is underlined .
Figure 3: Acc Improvements of two re-grouping strategies over Qwen2.5-VL. We take DAPO as our backbone.
Figure 4: The distribution of pre-calculated entropy on the Geometry3K. x axis represents entropy, y axis is the probability density, Q25 and Q75 denotes 25th and 75th percentiles, respectively .
Groups b
MathVerse
MathVision
MathVista
WeMath
HallusionBench
4
49.4
28.1
68.6
66.7
66.8
6
50.6
27.1
69.7
70.2
70.0
10
50.6
27.5
71.3
70.8
68.9
12
52.3
28.5
70.9
71.4
69.6
16
50.4
27.5
72.4
70.7
69.9
20
51.5
28.1
71.3
70.4
68.8
Table 2: The accuracy performance of different numbers of groups b on five benchmarks.
αOri
αPercep
αReason
MathVerse
MathVision
MathVista
WeMath
HallusionBench
0.1
0.2
0.7
50.7
28.6
71.6
71.8
71.2
0.15
0.25
0.6
50.8
29.0
71.6
70.6
70.8
0.2
0.1
0.7
51.2
28.4
71.5
70.4
71.0
0.3
0.1
0.6
51.7
27.8
70.7
70.6
70.8
0.4
0.3
0.3
51.9
27.9
71.4
70.5
71.1
0.6
0.2
0.2
50.4
28.8
72.2
71.0
71.4
Table 3: The effects of different weighting coefficients (built upon DAPO) on 5 benchmarks
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Training steps
1
10
20
30
40
50
60
Effective samples (participating in training)
323
327
324
322
297
314
306
Extreme success (7 correct & 1 wrong)
41
39
48
66
78
60
82
Extreme failure (7 wrong & 1 correct)
78
89
74
51
54
54
51
Total Extreme Ratio
36.8%
39.1%
37.7%
36.3%
44.4%
36.31%
43.5%
Appendix
Table 4: The statistics of rewards about extreme samples within the batch across 60 training steps.
low-entropy
medium-entropy
high-entropy
Wemath
23.6%
31.3%
45.1%
HallusionBench
21.2%
29.6%
49.2%
Appendix
Table 5: The error rate of perceptual difficulty groups in perceptual errors on two benchmarks.
group 1 (low confidence)
group 2
group 3
group 4
group 5
group 6
group 7
group 8
group 9
group 10 (high confidence)
Wemath
13.4%
12.6%
13.2%
12.7%
11.6%
9.1%
9.7%
7.2%
6.7%
4.2%
HallusionBench
12.7%
11.9%
11.2%
11.7%
10.0%
9.8%
9.0%
8.2%
8.3%
7.1%
Appendix
Table 6: The error rate of reasoning difficulty groups in reasoning errors on two benchmarks.
rollout
MathVerse
MathVision
MathVista
WeMath
HallusionBench
2
48.7
27.1
70.1
69.7
67.3
4
50.1
28.4
71.5
70.2
68.9
8
51.9
29.0
72.2
71.8
71.4
16
52.1
29.2
72.1
71.2
71.2
24
51.7
29.0
71.9
71.9
71.5
32
51.9
28.9
72.2
71.0
71.3
Appendix
Table 7: The effects of rollout size (built upon DAPO) on 5 benchmarks.
Figure 5: Illustrative examples of different levels of entropy
Figure 6: Case Study 1 showing improved reasoning capability on HallusionBench over vanilla GRPO.
Figure 7: Case Study 2 showing improved reasoning capability on Mathvision over vanilla GRPO.
Figure 8: Case Study 2 showing improved reasoning capability on Mathverse over vanilla GRPO.
Figure 9: Case Study 3 showing improved reasoning capability on Mathvista over vanilla GRPO.
Figure 10: Case Study 4 showing improved reasoning capability on Wemath over vanilla GRPO.
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Kuaishou Technology, Beijing, China