Organizations: State Key Laboratory of AI Safety, Institute of Computing Technology, CAS, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Kuaishou Technology, Beijing, China
Reinforcement learning (RL) with group relative policy optimization (GRPO) has become a widely adopted approach for enhancing the reasoning capabilities of multimodal large language models (MLLMs). While GRPO enables long-chain reasoning without a critic, it often suffers from sparse rewards on difficult problems and advantage vanishing when group-level rewards are too consistent for overly easy or hard problems. Existing solutions (sample expansion, selective utilization, and indirect reward design) often fail to maintain enough variance in within-group reward distributions to yield clear optimization signals. To address this, we propose DIVA-GRPO, a difficulty-adaptive variant advantage method that adjusts variant difficulty distributions from a global perspective. DIVA-GRPO dynamically assesses problem difficulty, samples variants with appropriate difficulty levels, and calculates advantages across local and global groups using difficulty-weighted and normalized scaling. This alleviates reward sparsity and advantage vanishing while improving training stability. Extensive experiments on six reasoning benchmarks demonstrate that DIVA-GRPO outperforms existing approaches in training efficiency and reasoning performance. Code: https://github.com/Siaaaaaa1/DIVA-GRPO
Figures & tables
Figure 1: (a) Selective sample utilization relies on only a subset of data, leading to underuse. (b) Sample enhancement expands data without difficulty awareness, causing even severe advantage sparsity. (c) Our method adaptively expands the sample space by problem difficulty, ensuring a stable difficulty distribution.
Figure 2: Overview of the proposed DIVA-GRPO method. For a given question, we dynamically assess its difficulty based on past rollout rewards and adaptively sample variants of different difficulty levels. As shown, when the original question is hard, easier variants are sampled to ensure reward diversity. We then compute local (the question itself) and global (the question with its variants) advantages, and obtain the final advantage through difficulty-aware reweighting and reward-range rescaling to update the policy model.
Model
MathVista
MathVerse
MathVision
OlympiadBench
WeMath
MMK12test
Avg.
Claude3.7-Sonnet
66.8
52.0
41.3
48.9
72.6
55.3
56.15
GPT-4o
63.8
50.2
30.4
35.0
68.8
49.9
49.68
o1
73.9
57.0
60.3
68.0
98.7
73.9
72.30
Gemini2-flash
70.4
59.3
41.3
51.0
71.4
65.2
59.77
InternVL2.5-VL-8B
64.4
39.5
19.7
12.3
53.5
45.6
39.17
Qwen-2.5-VL-7B
68.2
47.9
25.4
20.2
62.1
53.6
46.23
Table 1: Performance comparison across multimodal mathematical benchmarks. Bold denotes the best performance among 7B models, and underline marks the best overall performance. Evaluation is conducted with VLMEvalKit ( Duan et al., 2024 ) , while results for other models are taken from Meng et al. (2025) and Yao et al. (2025) . For each entry, the score before “/” is our re-evaluation using the officially released checkpoints, and the score after “/” is reported in the original paper.
Figure 3: Effect of Training Steps on Model Performance on the Validation Set.
Figure 4: Statistics of time and training speed during training and testing, showing that our method only leads to a slight increase in training time per step, primarily due to the increased output length.
Method
MathVista
MathVerse
MMK12test
Avg.
w/o Variant Generation
70.0
53.7
61.1
61.6
w/o Difficulty-Weighting
69.9
55.7
66.5
64.0
w/o RRB-Rescaling
71.5
55.2
64.7
63.8
w/o G-L Balance
70.8
55.4
66.0
64.1
Full DIVA-GRPO
73.2
56.3
68.8
66.1
Table 2: Ablation study of DIVA-GRPO , showing that each component provides gains and the full model achieves the best performance (accuracy, %).
Method
MathVista
MathVerse
MMK12test
Avg.
GRPO (base)
69.3
52.8
58.6
60.23
+ RRB
70.0
53.6
62.9
62.17
Table 3: Evaluation of RRB-Rescaling on standard GRPO.
Figure 5: 3D kernel density estimation (KDE) surfaces of the joint distribution between problem difficulty and global advantage across different training stages. The surface height reflects the sample density, illustrating how the model’s learning dynamics evolve with respect to difficulty and advantage.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Variant
Description
1
Gaussian (intensity = 0.45) and Rotate (multiples of 1°) + Variant text / Add text to image
2
Gaussian (intensity = 0.45) or Rotate (multiples of 30°) + Variant text / Add text to image
3
Gaussian (intensity = 0.3) or Rotate (multiples of 45°) + Variant text / Add text to image
4
Gaussian (intensity = 0.3) or Rotate (multiples of 90°)
5
Variant text
6
Original query / Variant text
Appendix
Table C.1: Variants from Hard to Easy.
Model
Mathematics
Physics
Chemistry
Biology
Avg.
Qwen3-VL-4B
Qwen3-VL-4B-Instruct
65.5
55.5
58.5
56.1
58.9
+GRPO
75.5
68.5
71.0
68.6
70.9
+DIVA-GRPO (Ours)
78.3
70.7
72.7
71.9
73.4
improv.
+2.8
+2.2
+1.7
+3.3
+2.5
Qwen3-VL-8B
Appendix
Table C.2: Performance comparison across different base model.
Figure D.1: Effect of different hyperparameters on model performance on the validation set.
Model
Mathematics
Physics
Chemistry
Biology
Avg.
Qwen-2.5-VL-7B
+ Qwen-Plus
78.1
61.5
69.1
70.9
69.8
+ GPT-o3
78.3
62.2
69.6
70.7
70.2
improv.
+0.2
+0.7
+0.5
-0.2
+0.3
Appendix
Table D.3: Performance comparison across different external models on Qwen-2.5-VL-7B.
Figure D.2: Analysis of the Effectiveness of DIVA-GRPO.
Model
Mathematics
Physics
Chemistry
Biology
Avg.
Closed-Source Models
Claude3.7-Sonnet
57.4
53.4
55.4
55.0
55.3
GPT-4o
55.8
41.2
47.0
55.4
49.9
o1
81.6
68.8
71.4
74.0
73.9
Gemini2-flash
76.8
53.6
64.6
66.0
65.2
Open-Source General Models
Appendix
Table F.4: Performance comparison across different disciplines in MMK12. Bold denotes the best performance among 7B models, and underline marks the best overall performance.
Figure H.4: Original Problem
Figure H.5: Vision-Dominant Problem
Figure H.6: Image Variants: Salt Noise, Blur, Gauss Noise, and Rotate.
Figure H.7: Original Problem
Figure H.8: Vision-Dominant Problem
Figure H.9: Image Variants: Salt Noise, Blur, Gauss Noise, and Rotate.
National University of Defense Technology, Changsha, Hunan, China · Intelligent Game and Decision Lab, Beijing, China · The Chinese University of Hong Kong, Shenzhen, Guangdong, China