Guided-RLVR: Guided Exploration in Reinforcement Learning with Verifiable Rewards

Bo Yang; Zishuai Zhang; Weihua Li; Qinnan Zhang; Sijia Wen; Hainan Zhang

cs.AIOct 27, 2026

Guided-RLVR: Guided Exploration in Reinforcement Learning with Verifiable Rewards

Authors: Bo Yang, Zishuai Zhang, Weihua Li, Qinnan Zhang, Sijia Wen, Hainan Zhang

Organizations: Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, Beihang University · School of Artificial Intelligence, Beihang University

Abstract

Reinforcement learning with verifiable rewards (RLVR) has shown strong empirical gains in improving the reasoning ability of large language models (LLMs) on tasks such as mathematics and programming. However, recent studies have shown that RLVR rollouts often collapse onto a narrow set of reasoning trajectories, limiting the explored reasoning space. Motivated by findings that thinking tokens at critical steps serve as critical structural elements in LLM reasoning pathways, we propose Guided-RLVR, a novel approach that augments RLVR with a lightweight controller. The controller strategically injects semantically meaningful steering tokens into the sampling distribution, encouraging the model to approach problems from multiple reasoning perspectives and thereby expanding the exploration space. Experiments on mathematical reasoning and code generation benchmarks show that Guided-RLVR consistently outperforms existing RLVR methods while producing more diverse reasoning trajectories.

Figures & tables

Figure 1: t-SNE ( van der Maaten and Hinton 2008 ) visualization of final-layer rollout representations from vanilla RLVR and guided steering, with 32 rollouts per problem. Total variance is computed in the standardized high-dimensional representation space before projection.

Figure 2: Overview of Guided-RLVR. (a) For each prompt

x

, policy

\pi_{\theta}

generates

G=10

rollouts: two unperturbed (greedy and stochastic) and eight guided. (b) At sentence boundaries where next-token entropy

H(\pi_{\theta})

exceeds the dynamic threshold

\tau_{t}

, the Logic Controller

\mu_{\phi}

maps aggregated hidden states

\sum h

to a semantic action

a_{t}

and steering token

v_{k}

. A bias

\delta

boosts its logit

z_{t}(v_{k})

toward probability

p_{\text{target}}=0.5

before sampling the next token

y_{t}

from the perturbed distribution

\pi^{\text{pert}}

. (c) Reward

r_{i}

combines task correctness

r_{\text{task}}

with a bonus

\alpha f(\Delta\ell)

, granted only to correct math rollouts where a promoted token was sampled. Here

\Delta\ell=\ell_{\text{base}}-\ell_{\text{cot}}

measures the same answer’s log-perplexity reduction when conditioned on the reasoning chain;

f

normalizes and clips this difference, and

\alpha

scales the bonus. Group mean

\mu_{r}

, standard deviation

\sigma_{r}

, and stabilizer

\epsilon

normalize rewards into advantages

\hat{A}_{i}

shared by policy and controller updates.

Qwen2.5-7B-Instruct
Method	MATH		AIME2024		AIME2025		AMC2023		Code-R1		AVG
Method	pass@1	pass@16	pass@1	pass@128	pass@1	pass@128	pass@1	pass@128	pass@1	pass@16	AVG
GRPO	62.1	79.4	11.20	46.67	5.99	50.00	43.23	90.36	82.0	94.8	56.58
GRPO + ENT	64.5	78.8	7.97	46.67	7.06	46.67	39.31	91.57	80.4	93.2	55.62
GRPO + REP	64.9	78.0	10.89	50.00	6.48	50.00	43.22	91.57	82.6	95.6	57.33
GRPO + Ours	65.6	80.6	12.11	53.33	5.03	53.33	42.77	92.77	82.6	96.0	58.41

Table 1: Main results across two backbones (Qwen2.5-7B-Instruct and Qwen3-4B) and two RLVR algorithms (GRPO and DAPO). All values are percentages. Ours denotes Guided-RLVR. AVG is the unweighted mean of the ten reported metrics. Within each backbone and RLVR algorithm block, the best value in each column is shown in bold; ties are also shown in bold.

Figure 3: Trajectory diversity across sampling budgets on MATH using Qwen2.5-7B-Instruct with GRPO. Solid lines denote TD and dashed lines denote ED; higher values indicate greater diversity.

Panel of Figure 4 — Figure 4: Pass@ $k$ scaling across all five benchmarks using Qwen3-4B with GRPO. The maximum sampling budget is $k{=}128$ for AIME2024, AIME2025, and AMC2023, and $k{=}16$ for MATH and Code-R1.

Method	MATH	AIME2024	AIME2025	AMC2023
GRPO	326.36	1224.94	935.95	818.50
GRPO + ENT	413.90	1224.84	910.15	724.80
GRPO + REP	407.15	1207.24	964.91	794.88
GRPO + Ours	338.04	1087.85	748.06	693.89

Table 2: Average generation length in tokens for the GRPO variants on four mathematical benchmarks using Qwen2.5-7B-Instruct, averaged over all completions. Lower values indicate shorter generations.

Variant	pass@1	pass@16
Vanilla GRPO	62.1	79.4
Guided-RLVR	65.6	80.6
(a) Random steering token selection	63.4	77.2
(b) Fixed steering token (“Wait”)	63.9	78.2
(c) No entropy threshold	62.1	79.2
(d) No log-perplexity reward	62.0	79.6

Table 3: Component ablation on MATH using Qwen2.5-7B-Instruct with GRPO. All values are percentages. Both pass@1 and pass@16 are estimated from 16 independent samples per problem at the selected checkpoint.

Figure 5: Effect of the log-perplexity reward coefficient $\alpha$ on MATH evaluation accuracy during training using Qwen2.5-7B-Instruct.

Method	GPU-h	s/step
GRPO	15.92	11.46
GRPO + ENT	14.22	10.24
GRPO + REP	16.55	11.92
GRPO + Ours	17.92	12.90

Table 4: RL-stage compute cost under the shared timing configuration. Ours denotes Guided-RLVR without supervised controller cold start.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Tokens	Semantic function
Reflection	Wait	Revisit or reconsider a step
Progression	Then	Advance to the next step
Causality	Therefore , Hence	Draw a consequence
Causal Reflection	Since , Because	Justify by a prior cause
Contrast	But , However	Introduce an opposing turn
Hypothesis	If	Pose a conditional assumption

Appendix

Table 5: Steering token vocabulary

\mathcal{T}

, grouped by the six semantic categories the controller predicts. Within a category, the injected token is drawn in proportion to its frequency in the cold-start DeepSeek-R1 traces.

Hyperparameter	Value
Sampling temperature (training)	1.0
Sampling temperature (evaluation)	0.7
KL penalty coefficient ( $\beta$ )	0
Decoupled clipping (high)	0.28
Decoupled clipping (low)	0.2
Rollouts per prompt ( $G$ )	10

Appendix

Table 6: Training hyperparameters shared across all methods and datasets.

Qwen2.5-7B-Instruct
Dataset	Token Length	Batch Size	Train Batch
MATH	1024	40	4
AIME2024/2025	8192	100	20
AMC2023	8192	100	20
Code-R1	8192	40	4
Qwen3-4B

Appendix

Table 7: Detailed training configuration for each backbone model and dataset.

Benchmark	# Problems	Difficulty	$K$
AIME2024 / 2025	30	Extremely hard	128
AMC2023	83	Hard	128
MATH	500	Moderate	16
Code-R1	500	Moderate	16

Appendix

Table 8: Sampling budget

K

used for pass@

K

evaluation on each benchmark.

Panel of Figure 6 — Figure 6: Controller behavior on MATH using Qwen2.5-7B-Instruct with GRPO. Panel (a) shows the cumulative proportion of each steering-token action over training steps. Panel (b) reports the number of interventions and the subsequent rollout accuracy for each token. The accuracy values are descriptive rather than causal because tokens are selected in different reasoning states and have unequal sample counts.

Figure 7: Training entropy on MATH using Qwen2.5-7B-Instruct with GRPO. Removing the entropy threshold (orange) leads to an overall increase after the initial decrease; the full Guided-RLVR configuration (blue) maintains lower entropy during later training.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,094 papers

Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs
Oct 5, 2025Zishang Jiang, Jinyi Han, Tingyun Li +7RL for Language Model Reasoning LLM-Guided RL
School of Data Science, Fudan University · Shanghai Institute of Artificial Intelligence for Education, East China Normal University · College of Computer Science and Artificial Intelligence, Fudan University +1
Boosting LLM Exploration via Weak-Model Guidance in RLVR
Aug 27, 2026Xingyu Shen, Huishuai Zhang, Peng Li +2RL for Language Model Reasoning RL for Language Models
Wangxuan Institute of Computer Technology, Peking University · National Engineering Research Center of New Electronic Publishing Technologies
LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Models
Jul 30, 2026Shuang Liang, Haoyang Zhou, Yifan Gong +2Numerical Reasoning in Language Models Prompt Selection for RL
Renmin University of China · AntGroup

Oct 5, 2025J/K move · Enter open · S save

Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs

Zishang Jiang, Jinyi Han, Tingyun Li, Xinyi Wang+6

School of Data Science, Fudan University · Shanghai Institute of Artificial Intelligence for Education, East China Normal University · College of Computer Science and Artificial Intelligence, Fudan University+1

RL for Language Model Reasoning LLM-Guided RL RL Exploration Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards (RLVR) has become a widely adopted technique for enhancing the reasoning ability of Large Language Models (LLMs). However, the effectiveness of RLVR strongly depends on the capability of base models. This issue arises because it requires the model to have sufficient capability to perform high-quality exploration, which involves both effectiveness and diversity. Unfortunately, existing methods address this issue by imitating expert trajectories, which improve effectiveness but neglect diversity. To address this, we argue that the expert only needs to provide guidance only at critical decision points rather than the entire reasoning path. Based on this insight, we propose MENTOR: Mixed-policy Expert Navigation for Token-level Optimization of Reasoning, a framework that provides expert guidance only at critical decision points to perform effective and diverse exploration in RLVR. Extensive experiments show that MENTOR enables models capture the essence of expert strategies rather than surface imitation, thereby performing high-quality exploration and achieving superior overall performance. Our code is available online.

Abstract

RL for Language Model Reasoning

Explore similar work

Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs

Boosting LLM Exploration via Weak-Model Guidance in RLVR

LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Models