Recycling Failures: Salvaging Exploration in RLVR via Fine-Grained Off-Policy Guidance

Yanwei Ren; Haotian Zhang; Likang Xiao; Xikai Zhang; Jiaxing Huang; Jiayan Qiu; Baosheng Yu; Quan Chen

cs.AIFeb 27, 2026

Recycling Failures: Salvaging Exploration in RLVR via Fine-Grained Off-Policy Guidance

Authors: Yanwei Ren, Haotian Zhang, Likang Xiao, Xikai Zhang, Jiaxing Huang, Jiayan Qiu, Baosheng Yu, Quan Chen,

Organizations: School of Artificial Intelligence, Beihang University, Beijing, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, China · The Hong Kong Polytechnic University, Hong Kong SAR, China · University of Leicester, Leicester, United Kingdom ·

Abstract

Reinforcement Learning from Verifiable Rewards (RLVR) has emerged as a powerful paradigm for enhancing the complex reasoning capabilities of Large Reasoning Models. However, standard outcome-based supervision suffers from a critical limitation that penalizes trajectories that are largely correct but fail due to several missteps as heavily as completely erroneous ones. This coarse feedback signal causes the model to discard valuable largely correct rollouts, leading to a degradation in rollout diversity that prematurely narrows the exploration space. Process Reward Models have demonstrated efficacy in providing reliable step-wise verification for test-time scaling, naively integrating these signals into RLVR as dense rewards proves ineffective.Prior methods attempt to introduce off-policy guided whole-trajectory replacement that often outside the policy model's distribution, but still fail to utilize the largely correct rollouts generated by the model itself and thus do not effectively mitigate the narrowing of the exploration space. To address these issues, we propose SCOPE (Step-wise Correction for On-Policy Exploration), a novel framework that utilizes Process Reward Models to pinpoint the first erroneous step in suboptimal rollouts and applies fine-grained, step-wise off-policy rectification. By applying precise refinement on partially correct rollout, our method effectively salvages partially correct trajectories and increases diversity score by 13.5%, thereby sustaining a broad exploration space. Extensive experiments demonstrate that our approach establishes new state-of-the-art results, achieving an average accuracy of 46.6% on math reasoning and exhibiting robust generalization with 53.4% accuracy on out-of-distribution reasoning tasks.

Figures & tables

Figure 1: SCOPE Overview. Top: On-policy RL rollouts terminate after the first error, causing partially correct trajectories to be discarded under sparse rewards. Bottom: SCOPE uses a PRM to identify the first incorrect step and applies step-wise off-policy rectification, while reusing correct on-policy steps to continue the rollout. Colored Lines: Black:Correct steps; Gray:Incorrect steps; Green:Recycled on-policy steps; Blue:Off-policy rectified steps.

Figure 2: Rollout recycling for sparse binary rewards. For each query, the policy samples rollouts $\{O_{i}\}_{i=1}^{n}$ with rewards $\{R_{i}\}$ . A PRM provides step scores $\{S_{t}\}$ to pick the most promising failed trajectory, yielding $O_{2}^{\prime}$ by freezing its correct prefix and regenerating the erroneous suffix with a refiner. $O_{2}^{\prime}$ is merged with on-policy rollouts and rewards are recomputed to preserve useful prefixes.We provide case studies in Appendix C to illustrate how our method improves the model’s suffix completions.

Model	In-Distribution Performance							Out-of-Distribution Performance
Model	AIME24	AIME25	AMC	MATH-500	Minerva	Olympiad	Avg.	ARC-c	GPQA*	MMLU-Pro	Avg.
Qwen-Base	11.5	4.9	31.3	43.6	7.4	15.6	19.1	18.2	11.1	16.9	15.4
Qwen-Instruct	12.5	10.2	48.5	80.4	32.7	41.0	37.6	70.3	24.7	34.1	43.0
Previous RLVR Methods
On-Policy RL	18.4	6.8	54.9	76.6	36.0	42.6	39.2	62.0	21.7	43.53	42.3
OpenReasoner-Zero	16.5	15.0	52.1	82.4	33.1	47.1	41.0	66.2	29.8	58.7	51.6

Table 1: Performance comparison (%) on the Qwen2.5-Math-7B backbone. We benchmark SCOPE against three categories of baselines: previous RLVR approaches, off-policy learning strategies, and off-policy guided methods. The best performance for each task is highlighted in bold.

Figure 3: Discarded valid reasoning trajectories among zero-reward samples. Compared to GRPO, our method significantly reduces the fraction of potentially valid trajectories that are discarded.

Panel of Figure 4 — Figure 4: Training Dynamics Analysis. We present the training curves for (a) the average step keep ratio during rectification, (b) the reward comparison between our method and GRPO, (c) the policy entropy, and (d) the diversity and quality results from 10-sample evaluations on the AIME24 dataset at temperature $T{=}1$ .

Qwen2.5-1.5B
Model	AIME24	AIME25	AMC	MATH-500	Minerva	Olympiad	Avg.
Qwen2.5-1.5B-Base	0.0	1.0	1.5	4.0	1.8	1.8	1.7
Qwen2.5-1.5B-Instruct	2.3	0.9	21.2	49.4	15.8	16.4	17.7
On-Policy RL	4.7	0.0	17.2	32.6	6.3	13.3	12.3
SCOPE	4.1	1.7	24.7	57.0	19.9	21.5	21.5
Llama3.1-8B

Table 2: Scalability analysis on Qwen2.5-1.5B and Llama-3.1-8B. SCOPE consistently outperforms standard on-policy RL and stabilizes training across different model architectures where standard RL often fails.

Model	AIME24	AIME25	AMC	MATH-500	Minerva	Olympiad	Avg.
On-policy RL	18.4	6.8	54.9	76.6	36.0	42.6	39.2
Replace Entire Rollout	24.4	18.9	57.8	80.4	37.1	41.7	43.4
RL w/PRM Reward	12.9	7.2	52.5	78.6	34.5	40.4	37.7
Qwen2.5-32B-Instruct as Refiner	23.8	16.6	60.6	81.2	42.3	43.4	44.6
SCOPE(Ours)	30.7	17.6	62.4	82.4	39.3	47.0	46.6

Table 3: Ablation study on refinement strategies and reward integration. Results demonstrate that SCOPE’s step-wise rectification significantly outperforms whole-trajectory replacement and naive PRM reward densification.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

AIME24 ( $K{=}10$ samples per prompt)
Method	Diversity $\uparrow$ (p50 [p10, p90])					Quality $\uparrow$
Method	distinct-1	distinct-4	1-self-bleu	1-self-rouge	div-score	pass@1	pass@10
GRPO (baseline)	0.11 [0.07, 0.16]	0.52 [0.34, 0.68]	0.27 [0.17, 0.41 ]	0.49 [ 0.37 , 0.63 ]	0.34 [0.27, 0.41]	16.67	53.33
Ours	0.14 [0.10, 0.17 ]	0.62 [ 0.52 , 0.76 ]	0.28 [ 0.19 , 0.39]	0.52 [0.34, 0.63 ]	0.39 [ 0.29 , 0.46 ]	26.67	60.00

Appendix

Table 4: Diversity and quality comparison on AIME-30 with

K{=}10

samples per prompt. Diversity entries report median (p50) with p10/p90 in brackets; higher is better. Bold marks the better median between methods; bracketed ranges are included for robustness context.

Panel of Figure 5 — Figure 5: Validation performance over training steps. We periodically evaluate the model on held-out validation sets during training. Each panel reports the performance trajectory on a different benchmark—(a) Olympiad, (b) Minerva, and (c) MATH500—comparing the GRPO baseline with our method.

Figure 6: Response length over training. Average generated response length (in tokens) as a function of training steps, comparing our method with the GRPO baseline.

Panel of Figure 7 — Figure 7: Near-miss@k (r=0) over training steps. Near-miss@k denotes the fraction of zero-reward samples satisfying $N-K\leq k$ , where $N$ is the judged total number of steps and $K$ is the number of correct steps

Figure 12: SCOPE Rectification Process. Illustrated on a specific SCOPE rollout, for a selected suboptimal trajectory, the Process Reward Model (PRM) identifies and recycles valid steps based on confidence scores and a validity threshold. The subsequent erroneous steps are then targeted by the Refiner Model, which performs off-policy guided step-wise rectification to correct the reasoning logic, ultimately synthesizing a fully correct rollout.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,094 papers

SCOPE-RL: Optimizing Reasoning Paths Before and After Success
Jul 13, 2026Xiaojian Liu, Han Xu, Jianqiang Xia +6Reward Shaping RL for Language Model Reasoning
Baidu Inc. · Shandong University
Nudging Beyond the Comfort Zone: Efficient Strategy-Guided Exploration for RLVR
May 15, 2026Chanuk Lee, Sangwoo Park, Minki Kang +1RL for Language Model Reasoning RL Exploration
KAIST
Guided-RLVR: Guided Exploration in Reinforcement Learning with Verifiable Rewards
Oct 27, 2026Bo Yang, Zishuai Zhang, Weihua Li +3RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, Beihang University · School of Artificial Intelligence, Beihang University

Jul 13, 2026J/K move · Enter open · S save

SCOPE-RL: Optimizing Reasoning Paths Before and After Success

Xiaojian Liu, Han Xu, Jianqiang Xia, Zhixuan Li+5

Baidu Inc. · Shandong University

Reward Shaping RL for Language Model Reasoning Efficient Language Model Reasoning Reinforcement Learning with Verifiable Rewards Reward Design for RL

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) optimizes LLMs using sparse verifiable final-answer rewards. This sparse anchor reliably verifies whether a trajectory succeeds but provides no direct feedback on the reasoning path that produced it. Before success, prerequisite progress on hard problems receives no reward signal; after success, outcome rewards cannot distinguish well-organized correct trajectories from redundant or locally flawed ones. We introduce SCOPE-RL (Scaffolded Chain Optimization with Process Efficiency), a two-stage framework that densifies this anchor while retaining the GRPO update: Adaptive Scaffolded RL adds prefix-decomposed verifiable rewards on answer-hidden sub-question chains before success, and Quality-Aware Process RL applies correctness-gated process-shape rewards to refine correct trajectories after success. An expert-validated Step-Quality Evaluation Protocol evaluates useful-step density, error localization, and token efficiency beyond final-answer accuracy. On Qwen3-8B-Instruct trained on DAPO-Math and Big-Math, SCOPE-RL improves average accuracy by up to 11.2 pp and reduces reasoning tokens by up to 27.1% over outcome-only GRPO; the gains hold under GSPO and on Qwen3-0.6B-Instruct, indicating that reward-signal densification is complementary to policy-update-level RLVR advances. Code and data are available at https://github.com/tokencraft-lab/SCOPE-RL.

Abstract

RL for Language Model Reasoning

Explore similar work

SCOPE-RL: Optimizing Reasoning Paths Before and After Success

Nudging Beyond the Comfort Zone: Efficient Strategy-Guided Exploration for RLVR

Guided-RLVR: Guided Exploration in Reinforcement Learning with Verifiable Rewards