Controllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoning

Zhuoxu Huang; Mengxi Jia; Hao Sun; Xuelong Li; Jungong Han

cs.LGFeb 22, 2026

Controllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoning

Authors: Zhuoxu Huang, Mengxi Jia, Hao Sun, Xuelong Li, Jungong Han

Organizations: Aberystwyth University · Institute of Artificial Intelligence (TeleAl), China Telecom · Tsinghua University

Abstract

Reinforcement Learning with verifiable rewards (RLVR) has emerged as a primary learning paradigm for enhancing the reasoning capabilities of multi-modal large language models (MLLMs). However, during RL training, the enormous state space of MLLM and sparse rewards often leads to entropy collapse, policy degradation, or over-exploitation of suboptimal behaviors. This necessitates an exploration strategy that maintains productive stochasticity while avoiding the drawbacks of uncontrolled random sampling, yielding inefficient exploration. In this paper, we propose CalibRL, a hybrid-policy RLVR framework that supports controllable exploration with expert guidance, enabled by two key mechanisms. First, a distribution-aware advantage weighting scales updates by group rareness to calibrate the distribution, therefore preserving exploration. Meanwhile, the asymmetric activation function (LeakyReLU) leverages the expert knowledge as a calibration baseline to moderate overconfident updates while preserving their corrective direction. CalibRL increases policy entropy in a guided manner and clarifies the target distribution by estimating the on-policy distribution through online sampling. Updates are driven by these informative behaviors, avoiding convergence to erroneous patterns. Importantly, these designs help alleviate the distributional mismatch between the model's policy and expert trajectories, thereby achieving a more stable balance between exploration and exploitation. Extensive experiments across eight benchmarks, including both in-domain and out-of-domain settings, demonstrate consistent improvements, validating the effectiveness of our controllable hybrid-policy RLVR training. Code is available at https://github.com/zhh6425/CalibRL.

Figures & tables

Figure 1: Entropy, reward, and accuracy curves of different methods. We split the entropy comparison into two panels for clarity.

Method	in-distribution	out-of-distribution		Avg.
Method	GeoEval	Geo3K	GeoQA	Avg.
GRPO	26.15	39.77	52.52	39.48
SFT+GRPO	6.00	18.64	40.98	21.87
DAPO	25.19	40.93	52.52	39.55
LUFFY ( Yan et al., 2025 )	25.62	39.10	51.19	38.64
RL-PLUS ( Dong et al., 2025 )	20.69	36.94	46.42	34.68

Table 1: Performance comparison on in-domain geometry benchmarks.

Method	Sci	Sp.	MathVerse	MathVision	MathVista	MMMU	ScienceQA	Avg.
GRPO	61.99	48.02	49.01	27.89	70.00	55.44	88.34	57.24
SFT+GRPO	54.33	38.19	35.10	23.36	54.70	54.67	86.36	49.53
DAPO	61.61	50.21	47.82	27.07	70.40	54.67	87.95	57.10
LUFFY ( Yan et al., 2025 )	63.11	48.18	47.97	26.64	70.30	55.22	87.95	57.05
RL-PLUS ( Dong et al., 2025 )	61.21	47.55	46.40	27.14	69.90	55.88	87.50	56.51
CalibRL	65.12	53.64	51.35	27.93	71.90	56.55	89.04	59.36

Table 2: Performance comparison on out-of-domain benchmarks. We present the Science benchmark as ‘Sci.’ and the Spatial Reasoning benchmark as ‘Sp.’.

Figure 2: Performance comparison showing relative changes from GRPO baseline across in-domain geometry and out-of-domain reasoning tasks.

Method	GeoEval	Geo3K	GeoQA	MathVision	MathVista	Avg.
Qwen2.5VL-3B ( Bai et al., 2025 )
GRPO	15.11	28.95	40.05	22.99	65.1	34.44
LUFFY ( Yan et al., 2025 )	12.54	28.95	33.55	23.42	62.9	32.27 -2.17
RL-PLUS ( Dong et al., 2025 )	12.00	27.79	35.15	24.98	63.4	32.66 -1.78
CalibRL	19.08	30.28	46.15	24.16	65.8	37.09 +2.65
IntrenVL3-8B ( Zhu et al., 2025 )

Table 3: Performance comparison on different base models.

Setting	in-domain			out-of-domain					Avg.
Setting	GeoEval	Geo3K	GeoQA	Sci.	Sp.	MathVerse	MathVision	MathVista	Avg.
w/o $\|\hat{A}_{i}\|$	25.19	28.95	57.56	61.76	47.71	43.63	26.71	70.90	45.30
w/ $\|\hat{A}_{i}\|$
$\alpha$ =0.3	31.30	42.43	59.81	64.52	50.88	50.46	28.88	71.00	49.91
$\alpha$ =0.5	33.44	40.60	60.74	65.12	53.64	51.35	27.93	71.90	50.59
$\alpha$ =0.8	30.44	43.59	58.75	65.27	50.62	50.46	28.71	71.00	49.85

Table 4: Ablation studies on the controllable exploration objective. We present the Science benchmark as ‘Sci.’ and the Spatial Reasoning benchmark as ‘Sp.’. The highlighted row represents our optimal results. Bold and underlined values denote the best and second-best results, respectively.

Figure 3: Entropy evolution during training for different $\alpha$ values in our framework. We split the comparison into two panels for clarity. The curves demonstrate how $\alpha$ controls exploration strength.

$\lambda$	in-domain			out-of-domain					Avg.
$\lambda$	GeoEval	Geo3K	GeoQA	Sci.	Sp.	MathVerse	MathVision	MathVista	Avg.
0.01	29.47	38.10	58.36	64.49	50.10	49.21	28.72	70.80	48.66
0.03	30.44	42.10	61.41	63.68	51.14	50.69	27.86	72.10	49.93
0.05	28.83	41.10	57.82	64.23	49.01	49.21	27.80	71.10	48.64
0.1	33.44	40.60	60.74	65.12	53.64	51.35	27.93	71.90	50.59
0.3	23.58	26.96	58.36	61.70	48.02	42.06	25.82	70.10	44.58

Table 5: Ablation studies on the balance weight. We present the Science benchmark as ‘Sci.’ and the Spatial Reasoning benchmark as ‘Sp.’. The highlighted row represents our optimal results. Bold and underlined values denote the best and second-best results, respectively.

Method	in-distribution	out-of-distribution		Avg.
Method	GeoEval	Geo3K	GeoQA	Avg.
GRPO	26.15	39.77	52.52	39.48
+ entropy coefficient = 0.01	22.62	40.27	47.48	36.79
+ KL-Cov ( Cui et al., 2025 )	26.47	41.60	53.58	40.55
+ Clip-Cov ( Cui et al., 2025 )	25.40	39.43	52.12	38.98
CalibRL	33.44	40.60	60.74	44.93

Table 6: Performance comparison of different entropy control.

Method	in-distribution	out-of-distribution		Avg.
Method	GeoEval	Geo3K	GeoQA	Avg.
GRPO	26.15	39.77	52.52	39.48
CalibRL
w/ ReLU	26.47	33.11	57.96	39.18
w/ Sigmoid	26.69	32.61	60.08	39.79
w/ Huber	24.22	27.62	55.70	35.85

Table 7: Ablations on activation functions.

Method	in-distribution	out-of-distribution		Avg.
Method	GeoEval	Geo3K	GeoQA	Avg.
CalibRL w/ expert baseline	33.44	40.60	60.74	44.93
CalibRL w/ ref policy baseline	27.87	40.77	55.57	42.74

Table 8: Ablations on reference baselines.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Table 9 — Table 9: Prompt template.

Method	MATH-500 (in-distribution)	AMC (out-of-distribution)
Qwen2.5-VL-7B	64.8	31.97
GRPO	68.5	31.59
CalibRL (ours)	70.2	32.08

Appendix

Table 10: Performance of the Qwen2.5-VL-7B model on math reasoning tasks.

Method	in-distribution	out-of-distribution		Avg.
Method	GeoEval	Geo3K	GeoQA	Avg.
Qwen2.5-VL-32B	37.94	48.24	60.87	48.60
GRPO	44.48	51.75	63.53	53.25
CalibRL(ours)	48.77	52.58	67.90	56.42

Appendix

Table 11: Performance of the Qwen2.5-VL-32B model.

Method	in-distribution	out-of-distribution		Avg.
Method	GeoEval	Geo3K	GeoQA	Avg.
CalibRL
w/o length norm	33.44	40.60	60.74	44.93
w/ length norm	29.26	39.10	60.34	42.90

Appendix

Table 12: Performance on length norm.

Method	in-distribution	out-of-distribution		Avg.
Method	GeoEval	Geo3K	GeoQA	Avg.
On Qwen data
GRPO	23.26	41.10	49.87	38.08
CalibRL	26.90	40.43	56.90	41.41 +3.33
On GPT data
GRPO	26.15	39.77	52.52	39.48

Appendix

Table 13: Trained on different expert data.

Component	GRPO	CalibRL	Additional Cost
Rollout sampling	G	G	0
Forward passes	G	G + 1	+1
Reward model queries	G	G	0
Backward passes	G	G	0

Appendix

Table 14: Per-prompt computational breakdown for a typical group size of G.

Figure 4: Relationship between $|\hat{A}_{i}|$ and reward frequency in a group of 10 samples.

Method	$\Delta\ell_{i}$	policy entropy	reward	response length
Method	$\Delta\ell_{i}$	policy entropy	reward	(expert length 382.19)
Qwen2.5-VL-7B	0.1459	0.4401	0.2029	437.76
GRPO	0.4256	0.2508	0.5381	431.79
LUFFY	0.0881	0.4688	0.4980	414.58
RL-PLUS	0.3452	0.2580	0.4975	425.08
CalibRL	-0.8025	1.4968	0.5667	271.71

Appendix

Table 15: Statistical data points from our trained checkpoints.

Figure 5: A case of CalibRL compared with baselines GRPO and SFT+GRPO.

Figure 6: A case of CalibRL compared with LUFFY and RL-PLUS.

RL Exploration

Reinforcement learning exploration studies how agents choose informative actions and states to learn effective policies under uncertainty.

See all 220 papers

Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs
Oct 5, 2025Zishang Jiang, Jinyi Han, Tingyun Li +7RL for Language Model Reasoning LLM-Guided RL
School of Data Science, Fudan University · Shanghai Institute of Artificial Intelligence for Education, East China Normal University · College of Computer Science and Artificial Intelligence, Fudan University +1
Boosting LLM Exploration via Weak-Model Guidance in RLVR
Aug 27, 2026Xingyu Shen, Huishuai Zhang, Peng Li +2RL for Language Model Reasoning RL for Language Models
Wangxuan Institute of Computer Technology, Peking University · National Engineering Research Center of New Electronic Publishing Technologies
Targeted Exploration via Unified Entropy Control for Reinforcement Learning
Apr 16, 2026Chen Wang, Lai Wei, Yanzhi Zhang +5Reinforcement Learning RL for Language Model Reasoning
College of Software, Nankai University · Zhongguancun Academy · Shanghai Jiao Tong University +3

Oct 5, 2025J/K move · Enter open · S save

Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs

Zishang Jiang, Jinyi Han, Tingyun Li, Xinyi Wang+6

School of Data Science, Fudan University · Shanghai Institute of Artificial Intelligence for Education, East China Normal University · College of Computer Science and Artificial Intelligence, Fudan University+1

RL for Language Model Reasoning LLM-Guided RL RL Exploration Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards (RLVR) has become a widely adopted technique for enhancing the reasoning ability of Large Language Models (LLMs). However, the effectiveness of RLVR strongly depends on the capability of base models. This issue arises because it requires the model to have sufficient capability to perform high-quality exploration, which involves both effectiveness and diversity. Unfortunately, existing methods address this issue by imitating expert trajectories, which improve effectiveness but neglect diversity. To address this, we argue that the expert only needs to provide guidance only at critical decision points rather than the entire reasoning path. Based on this insight, we propose MENTOR: Mixed-policy Expert Navigation for Token-level Optimization of Reasoning, a framework that provides expert guidance only at critical decision points to perform effective and diverse exploration in RLVR. Extensive experiments show that MENTOR enables models capture the essence of expert strategies rather than surface imitation, thereby performing high-quality exploration and achieving superior overall performance. Our code is available online.

Abstract

RL Exploration

Explore similar work

Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs

Boosting LLM Exploration via Weak-Model Guidance in RLVR

Targeted Exploration via Unified Entropy Control for Reinforcement Learning