Recycling Failures: Salvaging Exploration in RLVR via Fine-Grained Off-Policy Guidance
Authors: Yanwei Ren, Haotian Zhang, Likang Xiao, Xikai Zhang, Jiaxing Huang, Jiayan Qiu, Baosheng Yu, Quan Chen, Liu Liu
Organizations: School of Artificial Intelligence, Beihang University, Beijing, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, China · The Hong Kong Polytechnic University, Hong Kong SAR, China · University of Leicester, Leicester, United Kingdom · Nanyang Technological University, Singapore · Kuaishou Technology, Beijing, China
Reinforcement Learning from Verifiable Rewards (RLVR) has emerged as a powerful paradigm for enhancing the complex reasoning capabilities of Large Reasoning Models. However, standard outcome-based supervision suffers from a critical limitation that penalizes trajectories that are largely correct but fail due to several missteps as heavily as completely erroneous ones. This coarse feedback signal causes the model to discard valuable largely correct rollouts, leading to a degradation in rollout diversity that prematurely narrows the exploration space. Process Reward Models have demonstrated efficacy in providing reliable step-wise verification for test-time scaling, naively integrating these signals into RLVR as dense rewards proves ineffective.Prior methods attempt to introduce off-policy guided whole-trajectory replacement that often outside the policy model's distribution, but still fail to utilize the largely correct rollouts generated by the model itself and thus do not effectively mitigate the narrowing of the exploration space. To address these issues, we propose SCOPE (Step-wise Correction for On-Policy Exploration), a novel framework that utilizes Process Reward Models to pinpoint the first erroneous step in suboptimal rollouts and applies fine-grained, step-wise off-policy rectification. By applying precise refinement on partially correct rollout, our method effectively salvages partially correct trajectories and increases diversity score by 13.5%, thereby sustaining a broad exploration space. Extensive experiments demonstrate that our approach establishes new state-of-the-art results, achieving an average accuracy of 46.6% on math reasoning and exhibiting robust generalization with 53.4% accuracy on out-of-distribution reasoning tasks.
Figures & tables
Figure 1: SCOPE Overview. Top: On-policy RL rollouts terminate after the first error, causing partially correct trajectories to be discarded under sparse rewards. Bottom: SCOPE uses a PRM to identify the first incorrect step and applies step-wise off-policy rectification, while reusing correct on-policy steps to continue the rollout. Colored Lines: Black:Correct steps; Gray:Incorrect steps; Green:Recycled on-policy steps; Blue:Off-policy rectified steps.
Figure 2: Rollout recycling for sparse binary rewards. For each query, the policy samples rollouts {Oi}i=1n with rewards {Ri} . A PRM provides step scores {St} to pick the most promising failed trajectory, yielding O2′ by freezing its correct prefix and regenerating the erroneous suffix with a refiner. O2′ is merged with on-policy rollouts and rewards are recomputed to preserve useful prefixes.We provide case studies in Appendix C to illustrate how our method improves the model’s suffix completions.
Model
In-Distribution Performance
Out-of-Distribution Performance
AIME24
AIME25
AMC
MATH-500
Minerva
Olympiad
Avg.
ARC-c
GPQA*
MMLU-Pro
Avg.
Qwen-Base
11.5
4.9
31.3
43.6
7.4
15.6
19.1
18.2
11.1
16.9
15.4
Qwen-Instruct
12.5
10.2
48.5
80.4
32.7
41.0
37.6
70.3
24.7
34.1
43.0
Previous RLVR Methods
On-Policy RL
18.4
6.8
54.9
76.6
36.0
42.6
39.2
62.0
21.7
43.53
42.3
OpenReasoner-Zero
16.5
15.0
52.1
82.4
33.1
47.1
41.0
66.2
29.8
58.7
51.6
Table 1: Performance comparison (%) on the Qwen2.5-Math-7B backbone. We benchmark SCOPE against three categories of baselines: previous RLVR approaches, off-policy learning strategies, and off-policy guided methods. The best performance for each task is highlighted in bold.
Figure 3: Discarded valid reasoning trajectories among zero-reward samples. Compared to GRPO, our method significantly reduces the fraction of potentially valid trajectories that are discarded.
Figure 4: Training Dynamics Analysis. We present the training curves for (a) the average step keep ratio during rectification, (b) the reward comparison between our method and GRPO, (c) the policy entropy, and (d) the diversity and quality results from 10-sample evaluations on the AIME24 dataset at temperature T=1 .
Model
AIME24
AIME25
AMC
MATH-500
Minerva
Olympiad
Avg.
Qwen2.5-1.5B
Qwen2.5-1.5B-Base
0.0
1.0
1.5
4.0
1.8
1.8
1.7
Qwen2.5-1.5B-Instruct
2.3
0.9
21.2
49.4
15.8
16.4
17.7
On-Policy RL
4.7
0.0
17.2
32.6
6.3
13.3
12.3
SCOPE
4.1
1.7
24.7
57.0
19.9
21.5
21.5
Llama3.1-8B
Table 2: Scalability analysis on Qwen2.5-1.5B and Llama-3.1-8B. SCOPE consistently outperforms standard on-policy RL and stabilizes training across different model architectures where standard RL often fails.
Model
AIME24
AIME25
AMC
MATH-500
Minerva
Olympiad
Avg.
On-policy RL
18.4
6.8
54.9
76.6
36.0
42.6
39.2
Replace Entire Rollout
24.4
18.9
57.8
80.4
37.1
41.7
43.4
RL w/PRM Reward
12.9
7.2
52.5
78.6
34.5
40.4
37.7
Qwen2.5-32B-Instruct as Refiner
23.8
16.6
60.6
81.2
42.3
43.4
44.6
SCOPE(Ours)
30.7
17.6
62.4
82.4
39.3
47.0
46.6
Table 3: Ablation study on refinement strategies and reward integration. Results demonstrate that SCOPE’s step-wise rectification significantly outperforms whole-trajectory replacement and naive PRM reward densification.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Diversity ↑ (p50 [p10, p90])
Quality ↑
distinct-1
distinct-4
1-self-bleu
1-self-rouge
div-score
pass@1
pass@10
AIME24 ( K=10 samples per prompt)
GRPO (baseline)
0.11 [0.07, 0.16]
0.52 [0.34, 0.68]
0.27 [0.17, 0.41 ]
0.49 [ 0.37 , 0.63 ]
0.34 [0.27, 0.41]
16.67
53.33
Ours
0.14 [0.10, 0.17 ]
0.62 [ 0.52 , 0.76 ]
0.28 [ 0.19 , 0.39]
0.52 [0.34, 0.63 ]
0.39 [ 0.29 , 0.46 ]
26.67
60.00
Appendix
Table 4: Diversity and quality comparison on AIME-30 with K=10 samples per prompt. Diversity entries report median (p50) with p10/p90 in brackets; higher is better. Bold marks the better median between methods; bracketed ranges are included for robustness context.
Figure 5: Validation performance over training steps. We periodically evaluate the model on held-out validation sets during training. Each panel reports the performance trajectory on a different benchmark—(a) Olympiad, (b) Minerva, and (c) MATH500—comparing the GRPO baseline with our method.
Figure 6: Response length over training. Average generated response length (in tokens) as a function of training steps, comparing our method with the GRPO baseline.
Figure 7: Near-miss@k (r=0) over training steps. Near-miss@k denotes the fraction of zero-reward samples satisfying N−K≤k , where N is the judged total number of steps and K is the number of correct steps
Figure 12: SCOPE Rectification Process. Illustrated on a specific SCOPE rollout, for a selected suboptimal trajectory, the Process Reward Model (PRM) identifies and recycles valid steps based on confidence scores and a validity threshold. The subsequent erroneous steps are then targeted by the Refiner Model, which performs off-policy guided step-wise rectification to correct the reasoning logic, ultimately synthesizing a fully correct rollout.
Reinforcement learning with verifiable rewards (RLVR) optimizes LLMs using sparse verifiable final-answer rewards. This sparse anchor reliably verifies whether a trajectory succeeds but provides no direct feedback on the reasoning path that produced it. Before success, prerequisite progress on hard problems receives no reward signal; after success, outcome rewards cannot distinguish well-organized correct trajectories from redundant or locally flawed ones. We introduce SCOPE-RL (Scaffolded Chain Optimization with Process Efficiency), a two-stage framework that densifies this anchor while retaining the GRPO update: Adaptive Scaffolded RL adds prefix-decomposed verifiable rewards on answer-hidden sub-question chains before success, and Quality-Aware Process RL applies correctness-gated process-shape rewards to refine correct trajectories after success. An expert-validated Step-Quality Evaluation Protocol evaluates useful-step density, error localization, and token efficiency beyond final-answer accuracy. On Qwen3-8B-Instruct trained on DAPO-Math and Big-Math, SCOPE-RL improves average accuracy by up to 11.2 pp and reduces reasoning tokens by up to 27.1% over outcome-only GRPO; the gains hold under GSPO and on Qwen3-0.6B-Instruct, indicating that reward-signal densification is complementary to policy-update-level RLVR advances. Code and data are available at https://github.com/tokencraft-lab/SCOPE-RL.
Reinforcement learning with verifiable rewards (RLVR) has emerged as a scalable paradigm for improving the reasoning capabilities of large language models. However, its effectiveness is fundamentally limited by exploration: the policy can only improve on trajectories it has already sampled. While increasing the number of rollouts alleviates this issue, such brute-force scaling is computationally expensive, and existing approaches that modify the optimization objective provide limited control over what is explored. In this work, we propose NudgeRL, a framework for structured and diversity-driven exploration in RLVR. Our approach introduces Strategy Nudging, which conditions each rollout on lightweight, strategy-level contexts to induce diverse reasoning trajectories without relying on expensive oracle supervision. To effectively learn from such structured exploration, we further propose a unified objective, which decomposes the reward signal into inter- and intra-context components and incorporates a distillation objective to transfer discovered behaviors back to the base policy. Empirically, NudgeRL outperforms standard GRPO with up to 8 times larger rollout budgets, while outperforming oracle-guided RL baseline on average across five challenging math benchmarks. These results demonstrate that structured, context-driven exploration can serve as an efficient and scalable alternative to both brute-force rollout scaling and feasibility-oriented methods based on privileged information. Our code is available at https://github.com/tally0818/NudgeRL.
Reinforcement learning with verifiable rewards (RLVR) has shown strong empirical gains in improving the reasoning ability of large language models (LLMs) on tasks such as mathematics and programming. However, recent studies have shown that RLVR rollouts often collapse onto a narrow set of reasoning trajectories, limiting the explored reasoning space. Motivated by findings that thinking tokens at critical steps serve as critical structural elements in LLM reasoning pathways, we propose Guided-RLVR, a novel approach that augments RLVR with a lightweight controller. The controller strategically injects semantically meaningful steering tokens into the sampling distribution, encouraging the model to approach problems from multiple reasoning perspectives and thereby expanding the exploration space. Experiments on mathematical reasoning and code generation benchmarks show that Guided-RLVR consistently outperforms existing RLVR methods while producing more diverse reasoning trajectories.
Bo Yang, Zishuai Zhang, Weihua Li +3
Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, Beihang University · School of Artificial Intelligence, Beihang University