Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Space

Qiancheng Zhou; Ruizhe Li

cs.LGAug 29, 2026

Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Space

Authors: Qiancheng Zhou, Ruizhe Li

Organizations: School of Future Technology, Shanghai University · School of Computer Science, University of Birmingham

Abstract

Reinforcement learning with verifiable rewards (RLVR) substantially improves single-sample accuracy (pass@1) but causes the policy's solution space to contract, diminishing the returns of test-time scaling. In this work, we investigate where inside a reasoning trajectory this breadth is lost: does the policy fail to access a valid solution family, or does it fail to execute computation once initiated? To disentangle access from execution, we analyze the Countdown task, whose solution space can be exhaustively enumerated into discrete entrance families defined by the first operand and operator, across PPO on Qwen2.5-3B and GRPO on Qwen2.5-3B-Instruct. Across both training setups, solution coverage falls by up to 67%, halving even on problems solved across all checkpoints. We show that this contraction is heavily concentrated at the entrance: per-token likelihood shifts are 11x--16x larger prior to the first arithmetic operation than during downstream reasoning. Supplying only an unselected entrance prefix restores completion rates in low-access families by over an order of magnitude (0.018 -> 0.212 under PPO), demonstrating that alternative solutions remain executable but are no longer initiated. Guided by this localization, we find that while surface prompting fails to recover diversity, entrance-targeted interventions succeed: late-layer parameter interpolation with early checkpoints increases solution coverage by 37% at no loss in pass@1. Finally, we show that early-step entropy collapse recurs across six math benchmarks with 7B and 14B models, but is not an inevitable byproduct of reasoning optimization: an SFT baseline preserves more than double the coverage, and staged SFT--DPO--RLVR pipelines retain early-step entropy. In summary, reasoning breadth is lost at the door, not inside the room. Code: https://github.com/ershiyidian/early-branch-locking.

Figures & tables

Figure 1: Entrances partition the solution set. For

\{4,5,7,9\}\!\to\!24

, solver-defined entrances group valid solutions by their first operand and operator. The highlighted entrance

4\times

admits distinct continuations beginning with

4\times 7

and

4\times 9

PPO run
Checkpoint	pass@1	pass@64	pass@256	Solution cov.@320	Operator-class cov.
Base	0.001	0.076	0.228	0.097	0.178
Step 25	0.005	0.218	0.416	0.189	0.354
Step 50	0.051	0.500	0.596	0.337	0.538
Step 75	0.129	0.482	0.543	0.283	0.502
Step 150	0.323	0.441	0.457	0.152	0.362

Table 1: Accuracy and solution breadth over two independently trained RLVR runs on Countdown . PPO and GRPO use 150 and 135 held-out problems, respectively. Endpoints use 320 samples per problem and comparisons are within run. Intervals shown in Appendix B.9 .

Diagnostic	PPO	GRPO
NLL increase per token: entrance vs. execution	$+4.50$ vs. $+0.28$	$+8.08$ vs. $+0.71$
Designated completion: none $\to$ minimal entrance	$0.018\to 0.212$	$0.104\to 0.188$
Minimal-entrance completion: early $\to$ late	$0.113\to 0.212$	$0.131\to 0.188$

Table 2: Access and execution diagnostics on PPO and public GRPO Countdown. The first row gives early-to-late increase in NLL per token in entrance segment vs. subsequent execution; the second compares designated-family completion with and without a supplied minimal entrance at late checkpoint; the third tracks minimal-entrance completion between early and late checkpoints.

	Step 50	Step 275	Step 275 any-valid
Panel A: neutral scaffold, completion in the designated family
No entrance	0.009	0.018	0.060
Minimal entrance (operand and operator)	0.113	0.212	0.261
Completed first calculation	0.102	0.276	0.329
Panel B: grafts into the model’s own failed traces (step 275)
Feasible minimal entrance	0.176 (reference 0.044; excess $+0.121$ $[0.047,0.196]$ )		0.227

Table 3: Entrance interventions on the PPO Countdown run . Panel A reports designated-family completion over all 150 problems, with 8 to 16 continuations per prefix and a 0.070 path-enumeration reference conditional on the entrance. In Panel B we graft the same entrances into failed traces at step 275 (

S_{\mathrm{loss}}

: 33 problems, 32 with a usable retry point; reference 0.044). Panel C covers the 28 families observed at step 50 but unobserved in 320 free samples at step 275 with

n{=}64

continuations per cell. The GRPO counterpart is explained in Appendix D .

Figure 2: Access and execution move in opposite directions (PPO run). (a) From step 50 to step 275, normalized entrance entropy falls while completion from supplied entrances rises on matched problem and family keys; points are paired changes with 95% problem-bootstrap intervals. (b) At step 275, designated-family completion jumps after a minimal entrance and gains little more from the completed first calculation; the dashed line is the path-enumeration reference (Appendix C.1 ).

	Accuracy		Solution breadth
Intervention	pass@1	$\Delta$ pass@1	Coverage	$\Delta$ coverage
Step-275 policy (control)	0.278	—	0.111	—
Prompt diversification	$0.282^{\dagger}$	$+0.004$ $[-0.008,0.016]$	0.125	$+0.014$ $[0.000,0.038]$
Forced alternative operator	0.286	$+0.008$ $[-0.058,0.074]$	0.104	$-0.007$ $[-0.017,0.002]$
Answer-only logit mixing	$0.269^{\dagger}$	$-0.009$ $[-0.016,-0.003]$	0.105	$-0.006$ $[-0.015,0.000]$
High-temperature decoding ( $T{=}1.0$ )	0.264	$-0.014$ $[-0.026,-0.004]$	0.127	$+0.016$ $[0.007,0.027]$

Table 4: Paired intervention results on 100 held-out problems. Each intervention uses 64 samples per problem and is compared with the same step-275 control.

\Delta

columns report paired means with 95% problem-cluster bootstrap intervals from 10,000 draws.

\dagger

marks pass@1 equivalence under the pre-specified

\pm 0.02

TOST margin.

Figure 3: Early concentration on standard math benchmarks. (a) Same-trace scoring on Qwen2.5-7B: base minus RLVR per-token log-likelihood before the first complete calculation and during subsequent execution. (b) Origin-stratified difference-in-differences (DiD) between the two segments, with 95% problem-cluster bootstrap intervals (Eq. 5 ). Row values shown in Appendix F.1 .

Model	pass@1	pass@64	First-calc. $H$	Distinct-trace rate
Qwen Base 7B	0.379	0.754	1.063	0.780
Qwen SimpleRL 7B	0.510	0.760	0.744	0.444
Qwen Distill 7B	0.526	0.795	1.240	0.871
OLMo-3 7B (SFT)	0.447	0.781	1.189	0.770
OLMo-3 7B (DPO)	0.508	0.784	1.269	0.786
OLMo-3 7B (RLVR)	0.613	0.828	1.137	0.763

Table 5: Post-training comparison at 7B , using unweighted macro averages of problem means over six benchmarks; the 14B pair comparesion shown in Appendix G.1 . First-calculation entropy measures early-decision breadth and the distinct-trace rate is a length-sensitive complement.

Question	Henry wants to bake twice as many cookies as last year. When he finishes baking, he realizes he actually baked 15 more cookies than he meant to. He drops 5 of his cookies as he is putting them out to cool, and now has a total of 110 cookies. How many cookies did Henry bake last year? (Gold Answer: 50)
Qwen Base 7B Sample 6; correct Reverse Entrance	Henry now has 110 cookies. He dropped 5 cookies, so before dropping them he had 110 + 5 = 115 cookies. He baked 15 more than he meant to, so he meant to bake 115 - 15 = 100 cookies. This is twice last year, so he baked 100 / 2 = 50 cookies last year. The answer is 50.
Qwen SimpleRL 7B Direct RLVR; Sample 0 Forward Equation	Let $x$ be the number of cookies Henry baked last year. He intended to bake $2x$ , so he baked $2x+15$ cookies. After dropping 5, he has (2x + 15) - 5 = 110 cookies left. Simplifying gives $2x+10=110$ , so $2x=100$ and $x=\textbf{50}$ . The answer is 50.
OLMo-3 7B (RLVR) Staged RLVR; Sample 12 Reverse Entrance	Henry drops 5 cookies while cooling, and then has a total of 110 cookies left. So before dropping the 5 cookies, he had: 110 + 5 = 115 cookies. This 115 cookies is the actual number he baked: $2x+15=115\implies 2x=115-15=100\implies x=100/2=\textbf{50}$ . The answer is 50.

Table 6: Qualitative comparison of reasoning entrances on GSM8K Item 108. All models reach pass@64 =1. Direct RLVR ( Qwen SimpleRL ) collapses onto a dominant forward equation ( red ), completely extinguishing the backward-arithmetic entrance (

10\to 0

out of 64 rollouts). In contrast, the staged alignment pipeline ( OLMo-3 RLVR ) preserves access to the backward deduction ( blue ). Only initial arithmetic commitments are colored; full unedited traces shown in Appendix H .

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Panel A: access under two opening partitions among correct samples
	Step 50	Step 275
Operator classes per problem (feasible)	1.71	1.71
Coverage of feasible operator classes	0.538	0.253
First-evaluated-pair labels per problem (feasible)	3.99	3.99
Coverage of feasible first-evaluated pairs	0.368	0.117
Panel B: operator probability at matched model-generated pre-operator prefixes

Appendix

Table 7: Countdown opening-partition diagnostics (150 problems, 320 samples per problem). Panel B conditions on matched model-generated pre-operator prefixes and Panel C on solver-constructed canonical prefixes.

Checkpoint	Base	25	50	75	100	125	150	175	200	225	250	275
Coverage	0.178	0.354	0.538	0.502	0.448	0.412	0.362	0.328	0.300	0.262	0.256	0.253

Appendix

Table 8: Coverage of feasible operator classes among correct samples, base model and all eleven RLVR checkpoints (150 problems, 320 samples).

Checkpoint	Leaf retention	Zero-retention fraction	Still solved
Step 75	0.722 $[0.644,0.789]$	0.132	0.879
Step 100	0.644 $[0.570,0.717]$	0.176	0.824
Step 125	0.584 $[0.508,0.656]$	0.198	0.802
Step 150	0.476 $[0.399,0.547]$	0.264	0.758
Step 175	0.432 $[0.355,0.506]$	0.319	0.692
Step 200	0.380 $[0.306,0.452]$	0.319	0.681

Appendix

Table 9: Retention of step-50 correct leaves at later checkpoints, 91 step-50-solved problems, 320 samples, 2,000 problem-bootstrap draws. “Zero retention” is the fraction of problems retaining no step-50 leaf.

Set ( $n$ )	Ckpt.	pass@1	pass@64	Format-valid	Solution cov.	Operator-class cov.
$S_{\mathrm{both}}$ (58)	50	0.117	0.960	0.331	0.564	0.940
$S_{\mathrm{both}}$ (58)	275	0.737	0.971	0.749	0.286	0.654
$S_{\mathrm{loss}}$ (33)	50	0.027	0.586	0.228	0.542	0.795
$S_{\mathrm{loss}}$ (33)	275	0.000	0.000	0.000	0.000	0.000

Appendix

Table 10: Survivorship control, 320 samples per problem. Coverage columns are computed over correct samples.

Partition	Feasible labels/problem	Cov. step 50	Cov. step 275	Relative drop
Operator class (canonical)	1.71	0.538	0.253	0.53
Tree root operator	1.79	0.548	0.249	0.55
Full operator sequence	3.08	0.424	0.163	0.62
Tree signature	3.75	0.410	0.159	0.61
First evaluated pair	3.99	0.368	0.117	0.68
First merged numbers	2.49	0.483	0.167	0.66

Appendix

Table 11: Coverage under solver-enumerated partitions among correct samples, 150 problems, 320 samples, 2,000 bootstrap draws.

$T$	top- $p$	min- $p$	pass@1	pass@64	Solution coverage
0.7	0.90	0.00	0.284	0.377	0.108
0.7	0.90	0.05	0.283	0.376	0.108
0.7	1.00	0.00	0.277	0.406	0.138
0.7	1.00	0.05	0.283	0.385	0.109
1.0	0.90	0.00	0.274	0.402	0.130
1.0	0.90	0.05	0.274	0.402	0.138

Appendix

Table 12: Decoding sweep at step 275, 150 problems and 320 samples per cell.

Checkpoint	pass@1	pass@64	Solution coverage@320
Base	0.001 $[0.001,0.002]$	0.076 $[0.054,0.099]$	0.097 $[0.063,0.135]$
Step 25	0.005 $[0.004,0.006]$	0.218 $[0.174,0.263]$	0.189 $[0.145,0.235]$
Step 50	0.051 $[0.040,0.062]$	0.500 $[0.431,0.579]$	0.337 $[0.277,0.397]$
Step 75	0.129 $[0.100,0.159]$	0.482 $[0.407,0.561]$	0.283 $[0.230,0.337]$
Step 100	0.178 $[0.134,0.224]$	0.446 $[0.373,0.525]$	0.229 $[0.181,0.278]$
Step 125	0.276 $[0.214,0.340]$	0.446 $[0.373,0.526]$	0.205 $[0.156,0.254]$

Appendix

Table 13: Problem-bootstrap intervals for the PPO Countdown run, 150 problems and 320 samples per problem, 2,000 draws, seed 1729.

Checkpoint	Retry cue	First number	Local calc.	Generic plan	Misleading plan
Step 50	0.031	0.037	0.147	0.024	0.040
Step 275	0.015	0.015	0.211	0.023	0.004

Appendix

Table 14: Intermediate conditions on the retry scaffold, designated-family completion over 139 problems.

Relative position	Step 25	Step 275	$\Delta$	Tokens
$-4$	3.643 $[3.301,3.977]$	5.531 $[4.798,6.242]$	$+1.888$	264
$-3$	8.602 $[7.975,9.182]$	16.108 $[14.955,17.232]$	$+7.506$	1404
$-2$	2.098 $[1.779,2.448]$	3.315 $[2.715,3.979]$	$+1.217$	1530
$-1$	2.428 $[2.200,2.657]$	3.358 $[2.978,3.760]$	$+0.930$	1530
$0$	0.982 $[0.824,1.153]$	1.304 $[1.022,1.602]$	$+0.321$	1530
$+1$	0.762 $[0.668,0.863]$	1.230 $[1.050,1.420]$	$+0.468$	1530

Appendix

Table 15: Teacher-forced per-token negative log-likelihood by position relative to the first arithmetic operator. Position 0 is the first token at or after the boundary. Intervals use problem-cluster bootstrap with 10,000 draws.

Condition	Ckpt.	Problems $\times$ cont.	Prefix tokens	Rate
Panel A: state-only ablation
Numeric multiset only	200	$29\times 16$	12.9	0.002
Multiset and first operator (bare label)	200	$29\times 16$	14.9	0.017
Successful natural-language scaffold	200	$29\times 16$	171.1	0.998
Panel B: successful-trace boundary contrast
Before op 1	50	$184\times 64$	—	0.145

Appendix

Table 16: Successful-trace prefixes and gate-passing controls. Panel A is a state-only ablation at step 200 on 29 problems with 16 continuations per condition. Panel B uses 184 successful-trace boundary instances with 64 continuations. Panel C gives solver-constructed controls. The successful natural-language and successful-trace conditions fail the predictability gate; the others pass.

Checkpoint	pass@1	pass@64	Native-format	Solution cov.	Entrance cov.
Step 25	0.142	0.703	0.905	0.358	0.766
Step 50	0.181	0.710	0.947	0.384	0.735
Step 100	0.321	0.700	0.903	0.385	0.637
Step 150	0.450	0.657	0.864	0.304	0.540
Step 200	0.489	0.670	0.941	0.279	0.479
Step 300	0.507	0.673	0.947	0.267	0.479

Appendix

Table 17: Independent validation trajectory for the public GRPO run (300 problems, 64 samples per problem). The evaluation set used for endpoint comparisons is not used for checkpoint selection.

Checkpoint	pass@1	pass@64	pass@256	Native-format rate	Solution cov.	Entrance cov.	Entrance $H$
Step 25	0.121	0.633	0.750	0.900	0.467	0.932	1.585
Step 450	0.429	0.570	0.621	0.941	0.268	0.571	0.760

Appendix

Table 18: Endpoint evaluation for the public GRPO run (135 problems, 320 samples per problem).

Checkpoint or contrast	Entrance	Execution
Step 25	6.840 $[6.741,6.943]$	1.078 $[0.995,1.162]$
Step 450	14.917 $[14.731,15.119]$	1.788 $[1.593,1.984]$
Late $-$ early	$+8.077$ $[7.962,8.193]$	$+0.709$ $[0.578,0.846]$

Appendix

Table 19: Teacher-forced likelihood on the public GRPO run. Values are NLL per token with 95% problem-cluster bootstrap intervals.

Checkpoint	Prefix	Designated completion	Any-valid	Native-format rate
Step 25	none	0.056	0.116	0.911
Step 25	minimal	0.131	0.211	0.743
Step 25	first calc.	0.143	0.198	0.843
Step 450	none	0.104	0.434	0.941
Step 450	minimal	0.188	0.354	0.793
Step 450	first calc.	0.225	0.335	0.778

Appendix

Table 20: Supplied entrances on the public GRPO run. The primary quantity is valid completion in the designated entrance family.

Benchmark	DiD	95% CI	Problems	Traces
AIME24	0.205	$[0.132,0.304]$	9	129
AMC23	0.307	$[0.264,0.352]$	36	484
GSM8K	0.313	$[0.302,0.325]$	491	7736
MATH500	0.337	$[0.320,0.355]$	460	6018
Minerva	0.193	$[0.171,0.216]$	132	1499
OlympiadBench	0.273	$[0.254,0.292]$	322	4075

Appendix

Table 21: Origin-stratified difference in differences for Qwen2.5-7B, in nats per token, with 95% problem-cluster bootstrap intervals, 10,000 draws, seed 0.

Benchmark	Base-origin shift	RLVR-origin shift	Base traces	RLVR traces
AIME24	$-0.083$ $[-0.111,-0.056]$	$+0.055$ $[0.039,0.072]$	58	71
AMC23	$-0.142$ $[-0.178,-0.109]$	$+0.060$ $[0.050,0.069]$	230	242
GSM8K	$-0.226$ $[-0.240,-0.212]$	$+0.114$ $[0.110,0.117]$	5133	2500
MATH500	$-0.180$ $[-0.194,-0.166]$	$+0.064$ $[0.060,0.068]$	3404	2306
Minerva	$-0.079$ $[-0.096,-0.064]$	$+0.055$ $[0.050,0.061]$	689	690
OlympiadBench	$-0.113$ $[-0.123,-0.103]$	$+0.062$ $[0.058,0.066]$	2077	1760

Appendix

Table 22: Semantic-boundary early-minus-execution NLL shift for Qwen2.5-7B Base versus SimpleRL, stratified by the policy that generated the scored trace. Intervals use 10,000 problem-cluster bootstrap draws.

Model	Benchmark	pass@64	pass@256	Distinct first calcs.	First-calc. $H$	Distinct traces
Base 7B	GSM8K	0.993	1.000	8.98	1.054	148.5
SimpleRL 7B	GSM8K	0.973	0.975	2.20	0.247	22.0
Base 7B	MATH500	0.950	0.965	10.20	0.829	158.9
SimpleRL 7B	MATH500	0.937	0.950	3.32	0.288	25.2

Appendix

Table 23: Sampling depth at 256 samples per problem, 200 problems per benchmark.

Benchmark	Families	RLVR-unreached	Overall completion	Unreached completion
GSM8K	384	224	0.863 $[0.823,0.901]$	0.850 $[0.801,0.898]$
MATH500	337	155	0.715 $[0.661,0.768]$	0.713 $[0.645,0.780]$

Appendix

Table 24: Conditional execution from base-reachable first-calculation families. Intervals use problem-cluster bootstrap.

Model	pass@1	pass@64	First-calc. $H$	GSM8K correct-only $H$
Qwen Base 7B	0.379	0.754	1.063	0.894
Qwen SimpleRL 7B	0.510	0.760	0.744	0.285
Qwen Base 14B	0.377	0.774	1.207	0.970
Qwen SimpleRL 14B	0.546	0.774	0.711	0.300

Appendix

Table 25: Scale comparison for the Qwen base and SimpleRL pair. Macro columns average six benchmarks; the last column is matched GSM8K correct-only first-calculation entropy.

Model	Benchmark	pass@1	pass@64	First-calc. $H$	Effective first calcs.
Distilled base 1.5B	GSM8K	0.701	0.985	1.126	3.67
RL over distill 1.5B	GSM8K	0.766	0.980	1.206	4.02
Distilled base 1.5B	MATH500	0.633	0.955	1.168	4.26
RL over distill 1.5B	MATH500	0.732	0.955	1.130	3.93

Appendix

Table 26: RL after distillation at 1.5B; first-calculation statistics are measured at the first complete calculation, 1,000 bootstrap draws.

Supervision condition	pass@1	pass@64	pass@256	Solution cov.	Entrance cov.
GRPO step 450 (RLVR)	0.429	0.570	0.621	0.268	0.571
SFT ( $k{=}1$ )	0.120 $[0.095,0.145]$	0.737	0.856	0.471 $[0.412,0.532]$	0.604
SFT ( $k{=}2$ )	0.221 $[0.176,0.266]$	0.806	0.893	0.551 $[0.494,0.609]$	0.766
SFT ( $k{=}4$ , 8k)	0.310 $[0.254,0.366]$	0.853	0.923	0.725 $[0.669,0.782]$	0.825
SFT ( $k{=}4$ , 16k)	0.333 $[0.274,0.389]$	0.855	0.916	0.709 $[0.650,0.764]$	0.855
SFT ( $k{=}4$ , uniform)	0.269 $[0.219,0.315]$	0.836	0.928	0.779 $[0.725,0.835]$	0.821

Appendix

Table 27: Dose–response ablation over supervised solutions per problem

k

on 135 held-out Countdown tasks (320 samples per problem). All intervals are 2,000 problem-bootstrap draws (seed 1729).

Reasoning Diversity

Reasoning diversity studies how to represent, measure, and preserve distinct solution strategies in AI systems.

See all 20 papers

Sep 8, 2026

Difficulty-Adaptive Tree-Structured Policy Optimization for Expanding Reasoning Coverage in RLVR

Reinforcement Learning with Verifiable Rewards (RLVR) has been central to the recent success of Large Reasoning Models. However, while RLVR significantly improves single-sample accuracy, it often fails to expand the model's intrinsic reasoning coverage (pass@k) due to limited exploration during training. To address this, we optimize the structural design of train-time rollouts to enhance pass@k. Our analysis identifies three key design principles: (1) difficulty-adaptive rollout can play an important role in expanding pass@k, beyond serving as an efficiency heuristic; (2) tree-based rollout outperforms parallel sampling in discovering correct answers; and (3) sentence-entropy-guided forking overcomes the localization phenomenon of token-level branching to maximize semantic diversity. Building on these insights, we propose DATPO (Difficulty-Adaptive Sentence-entropy-guided Tree-structured Policy Optimization). DATPO integrates difficulty-adaptive tree search with a sibling-diversity advantage term, explicitly promoting semantic diversity to expand reasoning coverage during training. Experiments on mathematical reasoning benchmarks demonstrate that DATPO outperforms baselines especially in pass@k, which directly translates to superior test-time scaling performance.

Youngjun Yu, Sanghwan Jang, Hwanjo Yu

Pohang University of Science and Technology (POSTECH)

RL for Language Model Reasoning RL Exploration Reinforcement Learning with Verifiable Rewards LLM Reasoning

Ask PDF

Oct 17, 2026

What RL Brings to Reasoning: A Few Decisions the Base Model Would Not Make

Reinforcement learning with verifiable rewards (RLVR) substantially improves mathematical reasoning, but what it adds to the base model remains unclear. We investigate this through a sequence of generation-time interventions on Qwen3-4B-Base, measuring how much of its RL-trained counterpart's gain each intervention recovers on 120 AIME problems. Power sampling with the same 8,192-token output limit brings no gain beyond greedy decoding. Yet borrowing the RL model's token only where the two models' greedy choices differ, roughly 4% of generated positions, recovers 90.5% of the pass@1 gain, with similar recovery across three model pairs on MATH-500. We find that even a single opening token has a substantial effect: forcing the base model to begin with "Alright" recovers 35% of the RL gain. The opening alone is insufficient, however. Later replacements after the same opening add another 5.2 percentage points to pass@1, and the fixed rules we test for inserting reflective words such as "Wait" do not reproduce this benefit. Sparse replacement also retains much of the base model's diversity while staying close to the RL model at pass@1. At k=256, it solves 67.5% of the problems, significantly more than the RL model (61.4%) and a mixture of whole responses matched in pass@1 (61.45%). The comparison is at equal attempt counts rather than equal compute, since rl-replace consults both models at every decoding step. These results show that the opening can recover a meaningful part of the RL gain, but a larger contribution comes from later choices, which can be borrowed while retaining much of the base model's diversity. Code is available at https://github.com/CyrusZhang04/WhatRLBrings.

Siyu Zhang, Yuquan Lu, Yoshua Y. Li +7

University of California, San Diego · Meituan, LongCat Team · The Chinese University of Hong Kong, Shenzhen

RL for Language Model Reasoning LLM Reasoning Language Model Steering Language Model Reasoning Evaluation

Ask PDF

Jul 13, 2026

SCOPE-RL: Optimizing Reasoning Paths Before and After Success

Reinforcement learning with verifiable rewards (RLVR) optimizes LLMs using sparse verifiable final-answer rewards. This sparse anchor reliably verifies whether a trajectory succeeds but provides no direct feedback on the reasoning path that produced it. Before success, prerequisite progress on hard problems receives no reward signal; after success, outcome rewards cannot distinguish well-organized correct trajectories from redundant or locally flawed ones. We introduce SCOPE-RL (Scaffolded Chain Optimization with Process Efficiency), a two-stage framework that densifies this anchor while retaining the GRPO update: Adaptive Scaffolded RL adds prefix-decomposed verifiable rewards on answer-hidden sub-question chains before success, and Quality-Aware Process RL applies correctness-gated process-shape rewards to refine correct trajectories after success. An expert-validated Step-Quality Evaluation Protocol evaluates useful-step density, error localization, and token efficiency beyond final-answer accuracy. On Qwen3-8B-Instruct trained on DAPO-Math and Big-Math, SCOPE-RL improves average accuracy by up to 11.2 pp and reduces reasoning tokens by up to 27.1% over outcome-only GRPO; the gains hold under GSPO and on Qwen3-0.6B-Instruct, indicating that reward-signal densification is complementary to policy-update-level RLVR advances. Code and data are available at https://github.com/tokencraft-lab/SCOPE-RL.

Xiaojian Liu, Han Xu, Jianqiang Xia +6

Baidu Inc. · Shandong University

Reward Shaping RL for Language Model Reasoning Efficient Language Model Reasoning Reinforcement Learning with Verifiable Rewards Reward Design for RL

Ask PDF

Abstract

Reasoning Diversity

Explore similar work

Difficulty-Adaptive Tree-Structured Policy Optimization for Expanding Reasoning Coverage in RLVR

What RL Brings to Reasoning: A Few Decisions the Base Model Would Not Make

SCOPE-RL: Optimizing Reasoning Paths Before and After Success