What Outcome RL Buys: Finishing, Reliability, Rarely Reach

Shan Yang

cs.CLOct 26, 2026

What Outcome RL Buys: Finishing, Reliability, Rarely Reach

Authors: Shan Yang

Organizations: Individual Researcher.

Abstract

Outcome RL raises pass@1 but not pass@k at large k (Yue et al., 2025). But every pass@1 is scored under a token budget. Every recipe also trains under one. On physics and math problems at fixed token budgets, we compare the pass@1 of 20 released RL models (1.5B to 32B) with that of the thinking models they started from. We ask whether each pass@1 gain is changed behaviour or new knowledge. We divide each change in pass@1 into three parts. Finishing measures answering before the budget runs out. Reliability measures being right once finished. Reach measures problems RL solves that its base answered wrongly in all eight tries, minus problems the base solves that RL answered wrongly in all eight. We find four things. (1) Where the base often hits the token budget before answering, most of the change, gain or loss, is finishing (82% at 4k and 8k). Given more tokens, the base closes most of the finishing gap itself. (2) Where the base has enough tokens, the gain is reliability: the RL model is graded right more often, even on problems both finish. (3) By our definition, RL rarely solves what its base cannot. Reach adds at most 2.1 percentage points on any of our four physics and math sets. We checked 144 physics problems that only the RL model solved. Of these, 3.5% hold up when resampled or read. In a random sample, nine of ten new RL steps a human confirmed also appear in the base's samples. (4) In a small-scale RL run on an 8B model, the gain on new problems is finishing, with reliability flat. Supplying each problem's physical setup was not shown to improve on plain RL.

Figures & tables

Panel of Figure 1 — Figure 1: Outcome RL’s pass@1 gain on physics and competition math, split into finishing, reliability and net reach. Each bar is the mean over the recipes in its group. Recipes are grouped by how often their base finishes at the training cap on that set, with their number in parentheses. Reliability is measured on each model’s own finished derivations. A part left of zero lowers the score: the RL model hits the budget before answering more often (finishing), is right less often once finished (reliability), or solves fewer problems alone than the base does (reach).

Panel of Figure 2 — Figure 2: What outcome RL buys at each recipe’s own training cap and at the next budget above it, in pass@1 on physics. Each row is one recipe, split as in Figure 1 . Parts left of zero read as in Figure 1 . The number at the right of each row is the change, with the budget read. Red parts $<0.005$ marks a row whose parts are all below $0.005$ , too small to read. Rows are sorted by how often the base finishes at the cap on physics, least first. The lone-win panels count physics problems that only the RL model solves (black, right) or only the base (grey, left), while the other model finished all eight samples. † : the cap is not exact. DeepMath trains at a $24$ k cap but penalizes length from $20$ k. STILL-3 trains at $29$ k and is read at $28$ k.

Panel of Figure 3 — Figure 3: What outcome RL buys at each recipe’s own training cap on competition math, the recipes’ own domain. Rows are split as in Figure 2 . Parts left of zero read as in Figure 1 . Rows are sorted by how often the base finishes at the cap, least first.

base derivations that finish, physics / MATH-500
at $4$ k and $8$ k, range over bases	$12$ – $65\%$ / $56$ – $90\%$
at $16$ k	$82$ – $93\%$ / $92$ – $97\%$
finishing’s share of the change, $4$ k and $8$ k
both sets	$82\%$ [ $77$ , $86$ ]
physics / MATH-500	$71\%$ / $92\%$
RL’s change in length vs. in finishing	$\rho=-0.93$

Table 1: Finishing at tight budgets. Arrows read base to RL, except in the last block. Share

=\sum|F|/(\sum|F|+\sum|P|)

, with a

95\%

interval.

\rho

: Spearman’s correlation over recipes.

Panel of Figure 4 — Figure 4: Change in score against the finishing term $F$ , with Spearman $\rho=0.97$ over all $108$ points. Each point is one pair on one set at one budget. A point on the diagonal is a change carried by finishing alone. A point’s vertical distance from the diagonal is the rest of the change: reliability, reach and partial credit. The correlation is $0.99$ at $4$ k and $8$ k and $0.92$ at $16$ k.

competition math, matched split
physics at the cap, mean per recipe	$F$	$P$
four cap-bound recipes	$+0.078$	$+0.002$
matched split	$+0.058$	$+0.021$
Open-RS3, total change ∗	$+0.003$
sixteen recipes that are not cap-bound	$+0.002$	$+0.032$
all twenty recipes	$+0.017$	$+0.026$

Table 2: Finishing and reliability by recipe group, and the step check, on physics at the cap unless noted.

\rho

: Spearman’s correlation over recipes. At the next budget, a rise is a gain above

0.005

. ∗ With

95\%

interval [

-0.007

+0.014

physics	RL	base
solves what the other solves ( $16$ k)	$83\%$	$85\%$
lone wins, of $5{,}400$ pair–problems	$392$	$457$
lone wins, other wrong in all $8$	$128$	$153$
solves other’s all-wrong, at cap	$14\%$	$11\%$
with $24$ more samples	$23\%$	$27\%$
blind: a step the other lacks ( $143$ )	$80$	$71$

Table 3: Lone wins and new steps, for the RL model and its base. Rows one to three are at

16

k, rows four and five at the caps of the eighteen smaller recipes. Row one is the share of the other model’s solved problems this model also solves. The last three rows read

143

derivation pairs at

16

k (

23

on math), the RL one right and the base one wrong. They count steps one has and the other lacks, those an author upheld in a random sample of

50

pairs, and, on physics, those found in four other samples of the model lacking them.

RL lone wins at $16$ k on four bases
checked	$121$
solved by the base, $56$ more samples	$87$
no longer lone wins when regenerated	$22$
differ only in answer form	$9$
real physics	$3$
RL lone wins at the caps: real physics	$2$ of $23$

Table 4: Lone wins checked, and reach on other sets and at other sizes, physics at each recipe’s cap unless noted. § With MATH-500. The

7

B-and-up range covers four of five recipes.

\rho

: Spearman’s correlation over the

16

recipes with stated steps. ‡ The

16

regenerated pairs, three of them cap-bound (not e3). ¶ JustRL-DeepSeek’s would be

0.013

and Open-RS3’s

-0.001

channel	in training	held-out vs. plain RL
hint in the prompt	setup claims $55\to 92\%$	$+0.01$ [ $-0.01$ , $0.03$ ]
teacher derivation	touched problems $12\to 59\%$	$-0.05$ [ $-0.08$ , $-0.03$ ]
written setup	correct sections $0.29\to 0.45$	$-0.03$ [ $-0.05$ , $-0.01$ ]
question fine-tune	$99\%$ open with questions	at the base
setup reward	the policy gamed its scorer	not measurable
plain RL, the control		value

Table 5: The controlled probe on Qwen3-VL-8B-Thinking. Held-out differences from plain RL are on

87

hard problems, with

95\%

intervals. † Chosen after seeing the data.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

model	base	algorithm, scale	length shaping	RL data	training cap	rollout $T$	authors’ evaluation
DeepScaleR-1.5B-Preview	R1-Distill-1.5B	GRPO with a KL loss; about $1{,}750$ steps	none in the reward; cap raised $8$ k $\to 16$ k $\to 24$ k	$40$ k math	$8$ k to $24$ k	$0.6$	$32$ k, $T{=}0.6$
ProRL v2 (Nemotron-Research-Reasoning-Qwen-1.5B)	ProRL v1, from R1-Distill-1.5B	REINFORCE++ baseline, clip-higher, dynamic sampling, KL with reference resets; $3{,}000$ steps in all, counting ProRL v1’s $2{,}000$	cosine length penalty, switched on in scheduled cycles	$136$ k: math, code, STEM, puzzles, instruction following	$8$ k	n.d.	$32$ k, $T{=}0.6$
Open-RS3	R1-Distill-1.5B	GRPO with KL; $50$ steps	cosine length reward	$7$ k math	$3{,}584$	$0.7$	$32$ k, $T{=}0.6$
STILL-3-1.5B-preview	R1-Distill-1.5B	GRPO with KL and reference resets; about $210$ steps (inferred)	none documented	$30$ k math	$29$ k	$1.0$	$32$ k, $T{=}0.6$
Skywork-OR1-7B	R1-Distill-7B	GRPO without KL, adaptive entropy, rejection sampling; $1{,}320$ steps	none; cap raised $16$ k $\to 32$ k	$105$ k math, $14$ k code	$16$ k to $32$ k	$1.0$	$32$ k, $T{=}1.0$
Skywork-OR1-Math-7B	R1-Distill-7B	as above, two updates per step; $2{,}160$ steps	none; cap raised $8$ k $\to 32$ k	Skywork set (mix not stated)	$8$ k to $32$ k	$1.0$	$32$ k, $T{=}1.0$

Appendix

Table 6: What each released recipe did, from its model card, paper, blog or training scripts (n.d.: not documented). This part lists the eight recipes read first. The ten added are in the continuation. Rewards are outcome rewards unless a length term is listed. Outcome rewards are

0

1

, except

\pm 1

for STILL-3, JustRL and DeepMath and

\pm 5

for RLinf and AReaL. A truncated response usually scores as a wrong one. A truncated response scores

-2

under STILL-3, which docks a missing boxed answer, and under DeepMath’s overlong penalty. Some verifiers credit a truncated response if a correct boxed answer appears before the cut. “Authors’ evaluation” is the generation budget and temperature its authors report with. Figures and tables use short names. R1-1.5B to R1-32B are the DeepSeek-R1-Distill-Qwen bases, OMN-1.5B is OpenMath-Nemotron-1.5B, and Q3-1.7B and Q3-4B are the Qwen3 bases. JustRL-D and JustRL-N are JustRL-DeepSeek and JustRL-Nemotron. Skywork-7B, Skywork-Math and Skywork-32B are Skywork-OR1-7B, Skywork-OR1-Math-7B and Skywork-OR1-32B. AceReason and AceReason-14B are AceReason-Nemotron-7B and AceReason-Nemotron-14B. The other recipes go by their names without the size.

model	base	algorithm, scale	length shaping	RL data	training cap	rollout $T$	authors’ evaluation
JustRL-DeepSeek-1.5B	R1-Distill-1.5B	GRPO without KL or entropy bonus, clip-higher; $4{,}380$ steps	none; fixed $16$ k context	$17$ k math (DAPO-Math-17k)	$15$ k	$1.0$	$32$ k, $T{=}0.7$
DeepMath-1.5B	R1-Distill-1.5B	GRPO with a KL loss, clip-higher, dynamic sampling; $1{,}800$ steps	overlong penalty, $0$ to $-1$ over $20$ k to $24$ k; truncated $-2$	$103$ k math (DeepMath-103K)	$24$ k	$0.6$	$32$ k, $T{=}0.6$
RLinf-math-1.5B	R1-Distill-1.5B	GRPO without KL; steps n.d.	none	$106$ k math, with $10$ k logic puzzles	$27.6$ k	$1.0$	$32$ k, $T{=}0.6$
AReaL-1.5B-Preview-Stage-3	R1-Distill-1.5B, through two released earlier stages	critic-free PPO with a small KL; about $435$ steps over three stages	none; cap raised $8$ k $\to 16$ k $\to 24$ k	$40$ k math (DeepScaleR set)	$8$ k to $24$ k	$0.6$	$32$ k, $T{=}0.6$
JustRL-Nemotron-1.5B	OpenMath-Nemotron-1.5B	as JustRL-DeepSeek; $3{,}440$ steps	none; fixed $16$ k context	$17$ k math (DAPO-Math-17k)	$15$ k	$1.0$	$32$ k, $T{=}0.7$
QuestA-Nemotron-1.5B	OpenMath-Nemotron-1.5B	GRPO without KL, dynamic sampling; a partial solution in every prompt ( $50\%$ , then $25\%$ of it); $2{,}000$ steps	none	$9$ k hard math (OpenR1-Math)	$24$ k	$1.0$	$32$ k, $T{=}0.7$

Appendix

Table 7: (continued) The ten added recipes, registered before their outputs were generated. Below the rule are the

14

B and

32

B recipes, read in addition and not registered. QuestA and KnowRL put guidance in the training prompt. QuestA’s guidance is a prefix of a reference solution. KnowRL’s is knowledge points. The hints are absent from this paper’s prompts. QuestA’s authors also evaluate without the hints. KnowRL’s authors report both. Columns as above.

recipe	cap	fin@1 $(B)$ , base $\to$ RL	$F$	$P$	matched $P$	reach	change [interval]	$P$ , MATH-500
Open-RS3	4k	$0.21\to 0.21$	0.002	0.001	0.003	0	0.003 [ $-$ 0.007, 0.014]	$-$ 0.002
ProRL v2	8k	$0.45\to 0.99$	0.203	$-$ 0.026	0.023	0	0.178 [0.146, 0.211]	$-$ 0.025
JustRL-D	16k	$0.82\to 0.98$	0.063	0.032	0.048	$-$ 0.003	0.092 [0.068, 0.118]	0.008
e3	16k	$0.84\to 0.92$	0.043	$-$ 0.002	0.009	$-$ 0.007	0.034 [0.013, 0.056]	$-$ 0.018
JustRL-N	16k	$0.93\to 0.95$	0.013	0.024	0.032	0.003	0.039 [0.018, 0.060]	0.014
DeepScaleR	24k	$0.94\to 0.98$	0.020	0.027	0.039 ∗	0.004	0.051 [0.031, 0.071]	0.009

Appendix

Table 8: What RL buys at each recipe’s training cap, on physics: one row per recipe, the cells behind Figure 1 and the cap rows of Figure 2 .

F

P

and net reach split the change in full-credit pass@1, so the change can differ from a mean-credit score by up to

0.03

. Bold : the

95\%

paired bootstrap interval over problems excludes zero.

F

and

P

are computed outside the unsolved set

U

, so

F

’s sign can differ from the printed change in fin@1

(B)

. Matched

P

is reliability on problems both models finish seven times in eight. ∗ : computed on the second generation of Table 9 , which agrees within

0.007

on every term. Open-RS3’s matched

P

rests on ten problems. The Polaris pairs were not read on MATH-500 at their caps. The cells at the next budget above the cap, MATH-500 in full and every interval are in the released tables.

recipe	cap	$F$	$P$	net reach
ProRL v2	8k	$+0.219\to+0.240$	$-0.021\to-0.026$	$+0.001\to-0.001$
JustRL-D	16k	$+0.058\to+0.063$	$+0.032\to+0.028$	$-0.004\to-0.007$
DeepScaleR	24k	$+0.015\to+0.017$	$+0.040\to+0.039$	$-0.002\to+0.000$
DeepMath	24k	$+0.019\to+0.021$	$+0.070\to+0.069$	$+0.004\to+0.006$
RLinf	28k	$+0.012\to+0.013$	$+0.039\to+0.038$	$+0.000\to+0.002$
STILL-3	28k	$+0.006\to+0.006$	$+0.023\to+0.021$	$+0.000\to+0.000$

Appendix

Table 9: A second generation of

16

pairs on physics at each recipe’s cap, each cell matcher

\to

judge (judge protocol v2). Each model wrote one

64

k generation at temperature

0.6

, read at the cap by truncation, eight samples per problem on the

300

physics problems. Its columns use partial credit and split the whole change without taking reach out first, unlike Table 8 , which takes reach and its mirror out first and adds the mirror back to

P

. The

16

k rows and the reliabilities behind each term are in the released tables.

recipe	read at	by RL, never by base	by base, never by RL	by RL	by base
		solved in 4+ of 8 samples		solved at least once
Skywork-32B	24k	11 of 111	1 of 100	18	14
Skywork-Math	32k	9 of 122	2 of 114	21	16
RLinf	28k	5 of 141	1 of 180	25	28
Skywork-7B	32k	5 of 122	0 of 102	24	15
AceReason	32k	5 of 122	2 of 129	22	20

Appendix

Table 10: Robust wins by recipe, with the base’s. Each count is of problems one model solves while the other, finishing all eight samples, never does. “11 of 111” means the RL model solves

11

of the

111

problems its base leaves unsolved in eight. Physics and MATH-500 are pooled at each recipe’s cap, eight samples a side. ProRL v2 is read at its

8

k cap and Open-RS3 at

4

k, just above its cap. The JustRL pairs, trained at

15

k, and e3 are read at

16

k. The Polaris pairs are read on physics only.

recipe	cap	fin@1 $(B)$ , base $\to$ RL	$F$	$P$	net reach	robust net	RL solves 4+ of 8, never by base
ProRL v2	8k	$0.32\to 0.99$	0.137	$-$ 0.025	$-$ 0.004	$-$ 0.003	1 of 11
JustRL-D	16k	$0.75\to 0.95$	0.044	0.013	$-$ 0.006	$-$ 0.001	0 of 105
DeepScaleR	24k	$0.92\to 0.97$	0.016	0.015	$-$ 0.002	0	0 of 256
DeepMath	24k	$0.92\to 0.98$	0.018	0.028	0.001	0	0 of 256
RLinf	28k	$0.94\to 0.98$	0.015	0.026	0.005	0.001	2 of 284
STILL-3	28k	$0.94\to 0.95$	0.006	0.006	$-$ 0.001	0	0 of 284

Appendix

Table 11: The harder, unfiltered set at each recipe’s cap. The set has

471

harder problems and

200

drawn from the paper’s pools without the base filter. Each model writes eight samples per problem, scored by the fixed judge of Appendix C.2 with the deterministic checker and OlympiadBench’s per-problem tolerances. Robust reach counts wins in at least four of eight, net of the mirror. Bold : the paired bootstrap interval over problems excludes zero. The last column counts the problems the base finishes eight times and never solves that the RL model solves in at least four of eight. Every interval is in the released tables.

recipe	cap	fin@1 $(B)$ , base $\to$ RL	$F$	$P$	matched $P$	net reach	robust net	RL solves 4+ of 8, never by base
ProRL v2	8k	$0.20\to 0.89$	0.227	$-$ 0.055	0.006	$-$ 0.001	0	0 of 0
JustRL-D	16k	$0.49\to 0.87$	0.140	$-$ 0.004	0.046	0	0	0 of 8
DeepScaleR	24k	$0.74\to 0.97$	0.070	0.011	0.043	0	0	1 of 31
DeepMath	24k	$0.74\to 0.96$	0.070	0.011	0.050	0.005	0.004	1 of 31
RLinf	28k	$0.80\to 0.96$	0.057	0.045	0.069	0.007	0.014	4 of 44
STILL-3	28k	$0.80\to 0.89$	0.026	$-$ 0.006	$-$ 0.001	0.008	0.007	2 of 44

Appendix

Table 12: The competition-math read at each recipe’s cap. The set has

276

final-answer problems from eight 2025 and 2026 contests. Each model writes eight samples per problem, scored by the fixed judge of Appendix C.2 after the exact-match checker. Matched

P

is reliability on the problems both models finish at least seven times in eight, where selection into the finished set cannot act, weighted as

P

is. Matched reliability is the per-pair version of the matched split. Robust reach counts wins in at least four of eight, net of the mirror. Bold : the paired bootstrap interval over problems excludes zero. The last column counts the problems the base finishes eight times and never solves that the RL model solves in at least four of eight. Every interval is in the released tables.

Panel of Figure 5 — Figure 5: Left: the panel plots pass@ $k$ of the base model against sample count, on $160$ training problems never solved in the control’s first thirty steps. The curves are the registered decoding, a hotter sampler and a longer cap. The plotted points are read off the first $k$ samples, not the unbiased estimates quoted in the text. The dashed line is the gold setup in context at eight samples. Right: the panel plots fresh-rollout successes per $64$ on $60$ hard training problems solved at least once in $32$ samples at $16$ k. The count is taken visit by visit during training, over the first thirty updates of each run. The curves are with and without the policy’s own earlier successes replayed into the group (§ B.2 ). Replayed rows are excluded from the count.

arm	channel	corpus	what differs from its control	steps $\times$ groups
outcome RL (seeds 101, 202)	—	old	mined-milestone term, inert ( $4.7\%$ mass); the comparator	$60\times 96$
setup-reward arm (arm F)	reward	old	judged setup-waypoint reward on all-wrong groups	$60\times 8$
hint arm (arm H)	prompt	old	setup block, $\Pr(\text{hint})=1-s/59$	$60\times 96$
corpus v2 control (two seeds)	—	v2	none; seed 2 changes rollout seed and data order only	$60\times 96$
hint on v2	prompt	v2	the hint arm’s block, matched rows and draws	$60\times 96$
generic block	prompt	v2	content-free block, same draws	$60\times 96$

Appendix

Table 13: Trained arms. Groups are prompts per step. Every arm samples

16

rollouts per prompt except the setup-reward arm (

8

). One seed per arm except the control and the comparator. The letters are the names the appendices use. “Learnable” problems are the

60

of the

353

-problem text-only hard stratum with at least one full solve in

32

samples at

16

paired contrast (problem-level mean score, $87$ problems)	diff	$95\%$ CI	$\Pr(\leq 0)$
control s60 $-$ base ( $12$ k)	$+0.098$	$[+0.059,+0.142]$	$0.000$
hint on v2 s60 $-$ control s60 ( $12$ k)	$+0.009$	$[-0.014,+0.032]$	$0.232$
content-free block s60 $-$ control s60 ( $12$ k)	$-0.003$	$[-0.025,+0.018]$	$0.600$
teacher arm s60 $-$ control s60 ( $12$ k)	$-0.052$	$[-0.076,-0.030]$	$1.000$
setup-first (lenient judge) s30 $-$ control s30 ( $12$ k)	$-0.029$	$[-0.054,-0.007]$	$0.995$
setup-first (strict judge) s30 $-$ control s30 ( $12$ k)	$-0.036$	$[-0.060,-0.015]$	$1.000$

Appendix

Table 14: Clustered paired bootstrap (

20{,}000

draws) over held-out problems. Repeated reads were taken where held-out was flat. Deep sampling at

16

k has s.d.

0.003

over four reads. Replay at

16

k has s.d.

0.008

over seven. The matched rows read pass@1

(B\,|\,\text{fin})

on problems with a finished sample at both reads.

channel (scale)	how the setup is given	inside training	on held-out problems
reward the setup (pilot)	a judge rewards a stated setup on groups where every sample fails	no effect at the pilot’s scale; at full scale the policy gamed the scorer	not tested with a valid scorer
hint in the prompt (full)	the setup written into the prompt, faded out over training	the policy uses it: setup claims stated $55\to 92\%$	not kept once the hint is gone; a content-free block does as well
teacher derivations (full)	correct hinted derivations spliced into groups that failed	learned on the problems it touched ( $12\to 59\%$ solved)	below the control ( $-0.05$ ); no gain on similar problems
policy writes the setup (half)	the policy writes a setup section, which is judged	the sections become correct (strict content $0.29\to 0.45$ )	below the control ( $-0.03$ ); a section does not change which sample succeeds
fine-tune on questions (supervised)	$210$ derivations that open with a stronger model’s questions	the habit is learned ( $99\%$ open with questions)	no better than the same derivations without the questions; held-out at the base

Appendix

Table 15: Five ways of supplying the setup, and what each does. Full scale is the control’s sixty steps of

96

problems. Every comparison is against the control on the same problems or on the frozen held-out set.

fresh-rollout successes per $64$	visit 1	2	3	6	9	11
revisited, no replay ( $30$ updates)	$7.0$	$7.4$	$7.9$	$9.2$	$12.9$	$15.4$
revisited, with replay ( $30$ updates)	$7.3$	$8.3$	$7.8$	$10.1$	$14.0$	$17.7$

Appendix

Table 16: Revisiting

60

hard problems, with and without replay. Each cell counts successes among

64

fresh samples at one visit during training. Replayed samples are excluded.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,727 papers

What RL Brings to Reasoning: A Few Decisions the Base Model Would Not Make
Oct 17, 2026Siyu Zhang, Yuquan Lu, Yoshua Y. Li +7RL for Language Model Reasoning LLM Reasoning
University of California, San Diego · Meituan, LongCat Team · The Chinese University of Hong Kong, Shenzhen
Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Space
Aug 29, 2026Qiancheng Zhou, Ruizhe LiReasoning Diversity Reinforcement Learning with Verifiable Rewards
School of Future Technology, Shanghai University · School of Computer Science, University of Birmingham
What pass@k Cannot Measure: Evaluating Diversity and Capability Retention after Post-Training
Oct 5, 2026Subham Rath, Raj Dandekar, Rajat Dandekar +1LLM Evaluation Language Model Generation Evaluation
Accenture Strategy & Consulting · Vizuara AI Labs (vizuara.com) · Vizuara AI Labs (vizuara.ai)

Oct 17, 2026J/K move · Enter open · S save

What RL Brings to Reasoning: A Few Decisions the Base Model Would Not Make

Siyu Zhang, Yuquan Lu, Yoshua Y. Li, Yifan Ding+6

University of California, San Diego · Meituan, LongCat Team · The Chinese University of Hong Kong, Shenzhen

RL for Language Model Reasoning LLM Reasoning Language Model Steering Language Model Reasoning Evaluation

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) substantially improves mathematical reasoning, but what it adds to the base model remains unclear. We investigate this through a sequence of generation-time interventions on Qwen3-4B-Base, measuring how much of its RL-trained counterpart's gain each intervention recovers on 120 AIME problems. Power sampling with the same 8,192-token output limit brings no gain beyond greedy decoding. Yet borrowing the RL model's token only where the two models' greedy choices differ, roughly 4% of generated positions, recovers 90.5% of the pass@1 gain, with similar recovery across three model pairs on MATH-500. We find that even a single opening token has a substantial effect: forcing the base model to begin with "Alright" recovers 35% of the RL gain. The opening alone is insufficient, however. Later replacements after the same opening add another 5.2 percentage points to pass@1, and the fixed rules we test for inserting reflective words such as "Wait" do not reproduce this benefit. Sparse replacement also retains much of the base model's diversity while staying close to the RL model at pass@1. At k=256, it solves 67.5% of the problems, significantly more than the RL model (61.4%) and a mixture of whole responses matched in pass@1 (61.45%). The comparison is at equal attempt counts rather than equal compute, since rl-replace consults both models at every decoding step. These results show that the opening can recover a meaningful part of the RL gain, but a larger contribution comes from later choices, which can be borrowed while retaining much of the base model's diversity. Code is available at https://github.com/CyrusZhang04/WhatRLBrings.

Abstract

Reinforcement Learning

Explore similar work

What RL Brings to Reasoning: A Few Decisions the Base Model Would Not Make

Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Space

What pass@k Cannot Measure: Evaluating Diversity and Capability Retention after Post-Training