Recent studies have demonstrated that Reinforcement Learning (RL), notably Group Relative Policy Optimization (GRPO), can intrinsically elicit and enhance the reasoning capabilities of Vision-Language Models (VLMs). However, despite the promise, the underlying mechanisms that drive the effectiveness of RL models as well as their limitations remain underexplored. In this paper, we highlight a fundamental behavioral distinction between RL and base models, where the former engages in deeper yet narrow reasoning, while base models, despite less refined along individual path, exhibit broader and more diverse thinking patterns. Through further analysis of training dynamics, we show that GRPO is prone to diversity collapse, causing models to prematurely converge to a limited subset of reasoning strategies while discarding the majority of potential alternatives, leading to local optima and poor scalability. To address this, we propose Multi-Group Policy Optimization (MUPO), a simple yet effective approach designed to incentivize divergent thinking across multiple solutions, and demonstrate its effectiveness on established benchmarks. Project page: https://xytian1008.github.io/MUPO/
Figures & tables
Figure 1: The failure cases of RL models. We use Vision-R1 [ 13 ] as a representative RL model, with its corresponding base model being Qwen2.5-VL-7B [ 1 ] . The examples are selected from MathVerse [ 54 ] and MathVista [ 19 ] , respectively. For each question, we set the sampling temperature to 1.0 and generate multiple responses, each of which is displayed in a gray box. Main differences in the proposed reasoning strategies are annotated in blue and pink , while correct and incorrect answers are highlighted in green and red , respectively.
Figure 2: The impact of reasoning diversity on model performance. In (A) and (B), we report acc@k for both RL and base models on established benchmarks, with color intensity decreasing as k=1,2,4 . In (C) and (D), we plot relationship between reasoning diversity and the corresponding acc@4 scores. Each point is based on a set of 4 responses, and a regression line is fitted to capture the overall trend.
Figure 3: The diversity collapse of GRPO. In (A), we plot the evolution of reasoning diversity across training steps. In (B), we present an illustration of policy distribution over training to highlight the contrasting dynamics of convergent and divergent thinking. The gray region denotes rewards associated with different reasoning trajectories, while the blue curve indicates corresponding sampling probabilities.
Figure 4: The t-SNE projection of reasoning embeddings. We analyze a successful case where RL models produce correct answers, and a failure case where they cannot despite multiple samplings.
Figure 5: The overview of MUPO. The upper part illustrates the high-level pipeline, where responses are partitioned into multiple groups, and the overall optimization objective is formulated as a composition of multiple GRPO objectives, each corresponding to a group. In the lower part, we present the advantage computation for a group, in which we introduce diversity reward to encourage inter-group separation.
Figure 6: The t-SNE projection of our reasoning embeddings. The selected examples here correspond to the ones presented in Fig. 4 .
Figure 7: The qualitative analysis. We consider Vision-R1 [ 13 ] and MUPO-Thinker as typical GRPO and MUPO models and evaluate on an example from MMStar [ 5 ] with greedy decoding. We reorganize responses with numbering and omit special tags for better visualization.
Figure 8: The learning curves of accuracy and diversity reward. We use exponential moving average (EMA) for smoothing.
K
MathVerse
MathVista
MathVision
MMStar
HallBench
Average
1
46.9
69.1
24.1
64.8
54.7
51.9
2
49.4
72.3
28.0
65.2
56.7
54.3
3
51.2
74.1
29.3
65.8
56.5
55.4
4
50.9
74.6
29.4
65.1
56.3
55.3
5
50.6
74.8
29.1
64.5
55.9
55.0
Table 4: The benchmark accuracy varying number of groups K .
Figure 9: The ablation study of initial and final values of diversity reward weight λmax and λmin on average benchmark accuracy.
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Kuaishou Technology, Beijing, China