Reinforcement Learning with verifiable rewards (RLVR) has emerged as a primary learning paradigm for enhancing the reasoning capabilities of multi-modal large language models (MLLMs). However, during RL training, the enormous state space of MLLM and sparse rewards often leads to entropy collapse, policy degradation, or over-exploitation of suboptimal behaviors. This necessitates an exploration strategy that maintains productive stochasticity while avoiding the drawbacks of uncontrolled random sampling, yielding inefficient exploration. In this paper, we propose CalibRL, a hybrid-policy RLVR framework that supports controllable exploration with expert guidance, enabled by two key mechanisms. First, a distribution-aware advantage weighting scales updates by group rareness to calibrate the distribution, therefore preserving exploration. Meanwhile, the asymmetric activation function (LeakyReLU) leverages the expert knowledge as a calibration baseline to moderate overconfident updates while preserving their corrective direction. CalibRL increases policy entropy in a guided manner and clarifies the target distribution by estimating the on-policy distribution through online sampling. Updates are driven by these informative behaviors, avoiding convergence to erroneous patterns. Importantly, these designs help alleviate the distributional mismatch between the model's policy and expert trajectories, thereby achieving a more stable balance between exploration and exploitation. Extensive experiments across eight benchmarks, including both in-domain and out-of-domain settings, demonstrate consistent improvements, validating the effectiveness of our controllable hybrid-policy RLVR training. Code is available at https://github.com/zhh6425/CalibRL.
Figures & tables
Figure 1: Entropy, reward, and accuracy curves of different methods. We split the entropy comparison into two panels for clarity.
Method
in-distribution
out-of-distribution
Avg.
GeoEval
Geo3K
GeoQA
GRPO
26.15
39.77
52.52
39.48
SFT+GRPO
6.00
18.64
40.98
21.87
DAPO
25.19
40.93
52.52
39.55
LUFFY ( Yan et al., 2025 )
25.62
39.10
51.19
38.64
RL-PLUS ( Dong et al., 2025 )
20.69
36.94
46.42
34.68
Table 1: Performance comparison on in-domain geometry benchmarks.
Method
Sci
Sp.
MathVerse
MathVision
MathVista
MMMU
ScienceQA
Avg.
GRPO
61.99
48.02
49.01
27.89
70.00
55.44
88.34
57.24
SFT+GRPO
54.33
38.19
35.10
23.36
54.70
54.67
86.36
49.53
DAPO
61.61
50.21
47.82
27.07
70.40
54.67
87.95
57.10
LUFFY ( Yan et al., 2025 )
63.11
48.18
47.97
26.64
70.30
55.22
87.95
57.05
RL-PLUS ( Dong et al., 2025 )
61.21
47.55
46.40
27.14
69.90
55.88
87.50
56.51
CalibRL
65.12
53.64
51.35
27.93
71.90
56.55
89.04
59.36
Table 2: Performance comparison on out-of-domain benchmarks. We present the Science benchmark as ‘Sci.’ and the Spatial Reasoning benchmark as ‘Sp.’.
Figure 2: Performance comparison showing relative changes from GRPO baseline across in-domain geometry and out-of-domain reasoning tasks.
Method
GeoEval
Geo3K
GeoQA
MathVision
MathVista
Avg.
Qwen2.5VL-3B ( Bai et al., 2025 )
GRPO
15.11
28.95
40.05
22.99
65.1
34.44
LUFFY ( Yan et al., 2025 )
12.54
28.95
33.55
23.42
62.9
32.27 -2.17
RL-PLUS ( Dong et al., 2025 )
12.00
27.79
35.15
24.98
63.4
32.66 -1.78
CalibRL
19.08
30.28
46.15
24.16
65.8
37.09 +2.65
IntrenVL3-8B ( Zhu et al., 2025 )
Table 3: Performance comparison on different base models.
Setting
in-domain
out-of-domain
Avg.
GeoEval
Geo3K
GeoQA
Sci.
Sp.
MathVerse
MathVision
MathVista
w/o ∣A^i∣
25.19
28.95
57.56
61.76
47.71
43.63
26.71
70.90
45.30
w/ ∣A^i∣
α =0.3
31.30
42.43
59.81
64.52
50.88
50.46
28.88
71.00
49.91
α =0.5
33.44
40.60
60.74
65.12
53.64
51.35
27.93
71.90
50.59
α =0.8
30.44
43.59
58.75
65.27
50.62
50.46
28.71
71.00
49.85
Table 4: Ablation studies on the controllable exploration objective. We present the Science benchmark as ‘Sci.’ and the Spatial Reasoning benchmark as ‘Sp.’. The highlighted row represents our optimal results. Bold and underlined values denote the best and second-best results, respectively.
Figure 3: Entropy evolution during training for different α values in our framework. We split the comparison into two panels for clarity. The curves demonstrate how α controls exploration strength.
λ
in-domain
out-of-domain
Avg.
GeoEval
Geo3K
GeoQA
Sci.
Sp.
MathVerse
MathVision
MathVista
0.01
29.47
38.10
58.36
64.49
50.10
49.21
28.72
70.80
48.66
0.03
30.44
42.10
61.41
63.68
51.14
50.69
27.86
72.10
49.93
0.05
28.83
41.10
57.82
64.23
49.01
49.21
27.80
71.10
48.64
0.1
33.44
40.60
60.74
65.12
53.64
51.35
27.93
71.90
50.59
0.3
23.58
26.96
58.36
61.70
48.02
42.06
25.82
70.10
44.58
Table 5: Ablation studies on the balance weight. We present the Science benchmark as ‘Sci.’ and the Spatial Reasoning benchmark as ‘Sp.’. The highlighted row represents our optimal results. Bold and underlined values denote the best and second-best results, respectively.
Method
in-distribution
out-of-distribution
Avg.
GeoEval
Geo3K
GeoQA
GRPO
26.15
39.77
52.52
39.48
+ entropy coefficient = 0.01
22.62
40.27
47.48
36.79
+ KL-Cov ( Cui et al., 2025 )
26.47
41.60
53.58
40.55
+ Clip-Cov ( Cui et al., 2025 )
25.40
39.43
52.12
38.98
CalibRL
33.44
40.60
60.74
44.93
Table 6: Performance comparison of different entropy control.
Method
in-distribution
out-of-distribution
Avg.
GeoEval
Geo3K
GeoQA
GRPO
26.15
39.77
52.52
39.48
CalibRL
w/ ReLU
26.47
33.11
57.96
39.18
w/ Sigmoid
26.69
32.61
60.08
39.79
w/ Huber
24.22
27.62
55.70
35.85
Table 7: Ablations on activation functions.
Method
in-distribution
out-of-distribution
Avg.
GeoEval
Geo3K
GeoQA
CalibRL w/ expert baseline
33.44
40.60
60.74
44.93
CalibRL w/ ref policy baseline
27.87
40.77
55.57
42.74
Table 8: Ablations on reference baselines.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Table 9: Prompt template.
Method
MATH-500 (in-distribution)
AMC (out-of-distribution)
Qwen2.5-VL-7B
64.8
31.97
GRPO
68.5
31.59
CalibRL (ours)
70.2
32.08
Appendix
Table 10: Performance of the Qwen2.5-VL-7B model on math reasoning tasks.
Method
in-distribution
out-of-distribution
Avg.
GeoEval
Geo3K
GeoQA
Qwen2.5-VL-32B
37.94
48.24
60.87
48.60
GRPO
44.48
51.75
63.53
53.25
CalibRL(ours)
48.77
52.58
67.90
56.42
Appendix
Table 11: Performance of the Qwen2.5-VL-32B model.
Method
in-distribution
out-of-distribution
Avg.
GeoEval
Geo3K
GeoQA
CalibRL
w/o length norm
33.44
40.60
60.74
44.93
w/ length norm
29.26
39.10
60.34
42.90
Appendix
Table 12: Performance on length norm.
Method
in-distribution
out-of-distribution
Avg.
GeoEval
Geo3K
GeoQA
On Qwen data
GRPO
23.26
41.10
49.87
38.08
CalibRL
26.90
40.43
56.90
41.41 +3.33
On GPT data
GRPO
26.15
39.77
52.52
39.48
Appendix
Table 13: Trained on different expert data.
Component
GRPO
CalibRL
Additional Cost
Rollout sampling
G
G
0
Forward passes
G
G + 1
+1
Reward model queries
G
G
0
Backward passes
G
G
0
Appendix
Table 14: Per-prompt computational breakdown for a typical group size of G.
Figure 4: Relationship between ∣A^i∣ and reward frequency in a group of 10 samples.
Method
Δℓi
policy entropy
reward
response length
(expert length 382.19)
Qwen2.5-VL-7B
0.1459
0.4401
0.2029
437.76
GRPO
0.4256
0.2508
0.5381
431.79
LUFFY
0.0881
0.4688
0.4980
414.58
RL-PLUS
0.3452
0.2580
0.4975
425.08
CalibRL
-0.8025
1.4968
0.5667
271.71
Appendix
Table 15: Statistical data points from our trained checkpoints.
Figure 5: A case of CalibRL compared with baselines GRPO and SFT+GRPO.
Figure 6: A case of CalibRL compared with LUFFY and RL-PLUS.
Reinforcement Learning with Verifiable Rewards (RLVR) has become a widely adopted technique for enhancing the reasoning ability of Large Language Models (LLMs). However, the effectiveness of RLVR strongly depends on the capability of base models. This issue arises because it requires the model to have sufficient capability to perform high-quality exploration, which involves both effectiveness and diversity. Unfortunately, existing methods address this issue by imitating expert trajectories, which improve effectiveness but neglect diversity. To address this, we argue that the expert only needs to provide guidance only at critical decision points rather than the entire reasoning path. Based on this insight, we propose MENTOR: Mixed-policy Expert Navigation for Token-level Optimization of Reasoning, a framework that provides expert guidance only at critical decision points to perform effective and diverse exploration in RLVR. Extensive experiments show that MENTOR enables models capture the essence of expert strategies rather than surface imitation, thereby performing high-quality exploration and achieving superior overall performance. Our code is available online.
Zishang Jiang, Jinyi Han, Tingyun Li +7
School of Data Science, Fudan University · Shanghai Institute of Artificial Intelligence for Education, East China Normal University · College of Computer Science and Artificial Intelligence, Fudan University +1
Reinforcement Learning with Verifiable Rewards (RLVR) significantly improves LLM reasoning but often causes a drop in policy entropy, leading to narrowed reasoning coverage and degraded pass@k for large k. While existing methods mitigate this entropy collapse through algorithmic regularizations, cross-model non-parametric perturbation is also neglected. In this work, we propose a simple yet effective approach to preserve the generative diversity of LLMs during RLVR. Instead of relying solely on internal exploration, we force the target model to generate answers based on partial reasoning trajectories generated by a smaller, weaker language models. These unfamiliar prefixes effectively disrupt over-confidence and encourage the exploration of distinct reasoning paths. We empirically study the potential of outer prefixes, revealing the mechanism of the impact of distributional discrepancy to the exploration dynamics in RLVR training. Experiments across multiple mathematical benchmarks show that our method consistently outperforms vanilla RLVR. Notably, the performance gain becomes increasingly pronounced as k scales up, demonstrating a substantial expansion of reasoning coverage. Furthermore, our approach efficiently mitigates entropy collapse without requiring additional SFT, intricate reward designs, or complex prompting.
Xingyu Shen, Huishuai Zhang, Peng Li +2
Wangxuan Institute of Computer Technology, Peking University · National Engineering Research Center of New Electronic Publishing Technologies
Recent advances in reinforcement learning (RL) have improved the reasoning capabilities of large language models (LLMs) and vision-language models (VLMs). However, the widely used Group Relative Policy Optimization (GRPO) consistently suffers from entropy collapse, causing the policy to converge prematurely and lose diversity. Existing exploration methods introduce additional bias or variance during exploration, making it difficult to maintain optimization stability. We propose Unified Entropy Control for Reinforcement Learning (UEC-RL), a framework that provides targeted mechanisms for exploration and stabilization. UEC-RL activates more exploration on difficult prompts to search for potential and valuable reasoning trajectories. In parallel, a stabilizer prevents entropy from growing uncontrollably, thereby keeping training stable as the model consolidates reliable behaviors. Together, these components expand the search space when needed while maintaining robust optimization throughout training. Experiments on both LLM and VLM reasoning tasks show consistent gains over RL baselines on both Pass@1 and Pass@k. On Geometry3K, UEC-RL achieves a 37.9% relative improvement over GRPO, indicating that it sustains effective exploration without compromising convergence and underscoring UEC-RL as a key for scaling RL-based reasoning in large models. Our code is available at https://github.com/597358816/UEC-RL.
Chen Wang, Lai Wei, Yanzhi Zhang +5
College of Software, Nankai University · Zhongguancun Academy · Shanghai Jiao Tong University +3