What RL Brings to Reasoning: A Few Decisions the Base Model Would Not Make

Siyu Zhang; Yuquan Lu; Yoshua Y. Li; Yifan Ding; Kun Zeng; Yu Huo; Yifu Guo; Rongxiang Weng

cs.CLOct 17, 2026

What RL Brings to Reasoning: A Few Decisions the Base Model Would Not Make

Authors: Siyu Zhang, Yuquan Lu, Yoshua Y. Li, Yifan Ding, Kun Zeng, Yu Huo, Yifu Guo, Rongxiang Weng,

Organizations: University of California, San Diego · Meituan, LongCat Team · The Chinese University of Hong Kong, Shenzhen

Abstract

Reinforcement learning with verifiable rewards (RLVR) substantially improves mathematical reasoning, but what it adds to the base model remains unclear. We investigate this through a sequence of generation-time interventions on Qwen3-4B-Base, measuring how much of its RL-trained counterpart's gain each intervention recovers on 120 AIME problems. Power sampling with the same 8,192-token output limit brings no gain beyond greedy decoding. Yet borrowing the RL model's token only where the two models' greedy choices differ, roughly 4% of generated positions, recovers 90.5% of the pass@1 gain, with similar recovery across three model pairs on MATH-500. We find that even a single opening token has a substantial effect: forcing the base model to begin with "Alright" recovers 35% of the RL gain. The opening alone is insufficient, however. Later replacements after the same opening add another 5.2 percentage points to pass@1, and the fixed rules we test for inserting reflective words such as "Wait" do not reproduce this benefit. Sparse replacement also retains much of the base model's diversity while staying close to the RL model at pass@1. At k=256, it solves 67.5% of the problems, significantly more than the RL model (61.4%) and a mixture of whole responses matched in pass@1 (61.45%). The comparison is at equal attempt counts rather than equal compute, since rl-replace consults both models at every decoding step. These results show that the opening can recover a meaningful part of the RL gain, but a larger contribution comes from later choices, which can be borrowed while retaining much of the base model's diversity. Code is available at https://github.com/CyrusZhang04/WhatRLBrings.

Figures & tables

Figure 1: Overview of the diagnostic tests of explanations for the RL gain. Each row pairs a question with its intervention and headline finding. All interventions act on the base model. Tests 0 and 1 are evaluated on MATH-500 and AIME; Tests 2 and 3 are evaluated on the 120 AIME problems.

Figure 2: Illustrative comparison of the base model, power sampling, direct RL decoding, and sparse RL-token replacement on AIME 2025 Problem 4. All methods receive the same prompt. The base model misses the overlap between two solution families, while the RL and RL-replace examples check for it and obtain the correct answer. Power sampling sharpens the base-model distribution but does not consistently recover the AIME gain. The bottom annotations report the corresponding opening and token-level statistics.

Figure 3: All interventions on the 120 AIME problems, with 16 samples per problem. Dashed lines mark the base model (grey) and the RL model (orange). Test 0 is represented by the gated MCMC variant of § 3.2 .

Figure 4: rl-replace decoding. At each step both models score the shared prefix; the RL model’s greedy token is emitted only where the two greedy tokens differ (about 4% of positions), and the base model samples everywhere else. The token sequence is illustrative.

		Base model only			Uses the RL model
Benchmark	Model	Greedy	Sampling	MCMC	rl-replace	RL
MATH-500	Qwen3-4B-Base	71.80	71.10	73.45	85.25	84.37
	Qwen3-8B-Base	74.60	75.98	78.53	87.74	89.26
	Llama-3.2-3B	46.80	46.15	49.65	57.50	59.27
AIME24	Qwen3-4B-Base	16.67	10.00	0 9.58	23.13	23.13
AIME24	Qwen3-8B-Base	20.00	20.00	16.67	27.08	26.67

Table 1: Power sampling (MCMC) versus standard sampling and greedy decoding, with methods that use the RL model as a reference. All values are pass@1 in %. Greedy decoding uses a single run; sampling and MCMC are averaged over 8 seeds. Standard sampling uses temperature 0.6. For Qwen3-4B, the rl-replace and RL results on AIME come from the main run summarized in Table 2 , where standard base sampling gives 12.50% on AIME24 and 9.79% on AIME25.

		pass@1 (%)				pass@16 (%)
Problems	#	Base	rl-replace	RL	Recovery	Base	rl-replace	RL
AIME24+25	60	11.15	22.08	23.02	92.1 [75.6, 121.2]	36.67	48.33	41.67
AIME22+23	60	0 8.85	14.79	15.63	87.7 [63.8, 120.9]	23.33	38.33	38.33
All	120	10.00	18.44	19.32	90.5 [77.0, 108.6]	30.00	43.33	40.00
MATH-500	500	71.10	85.25	84.37	106.6 [95.2, 120.4]	82.86	94.46	94.65

Table 2: rl-replace versus base and RL. AIME rows use

n=16

samples per problem. Recovery is the share of the RL pass@1 gain recovered by rl-replace , with 95% bootstrap intervals. On MATH-500, pass@1 of the base is standard sampling from Table 1 (temperature 0.6, averaged over 8 seeds), and pass@16 is the unbiased estimate from 64 samples per problem.

Table 7

Qwen3-4B
Method	pass@1	pass@4	pass@16	pass@64	pass@128	pass@256
Base	10.26	20.79	31.02	43.29	52.50	61.64
RL	19.77	29.61	39.41	51.85	57.10	61.40
rl-replace	18.98	29.74	41.69	54.73	60.96	67.50
Qwen3-8B
Base	19.77	27.31	38.64	50.18	58.09	68.33

Table 6: pass@

k

(%) on all 120 AIME problems, computed using the unbiased estimator with 512 samples per problem for the base and RL models and 256 for rl-replace . Bold indicates the highest value in each column within each model group. The Qwen3-4B run is independent of the main 16-sample run in Table 2 , so its pass@1 and pass@16 values differ slightly. Figure 5 shows the Qwen3-4B results by problem group.

Figure 5: Accuracy and coverage of Qwen3-4B as the sampling budget increases: (a) all 120 AIME problems, (b) AIME24+25, and (c) AIME22+23. Estimates use 512 samples per problem for the base and RL models and 256 for rl-replace , with the unbiased pass@ $k$ estimator. The response-level mixture selects the RL model with probability 0.92 and the base model otherwise for each complete response; its dashed line connects the reported values at $k=1,16,256$ in panel (a). All panels share the same vertical scale and use a logarithmic horizontal axis. Green endpoint labels report rl-replace pass@256. The comparisons use equal attempt counts; rl-replace consults both models at every decoding step.

Year	#	pass@1	pass@16	maj@16	Recovery	Replace rate
AIME 2024	30	12.50 / 23.13 / 23.13	36.67 / 43.33 / 40.00	16.67 / 30.00 / 26.67	100.0	4.5
AIME 2025	30	0 9.79 / 21.04 / 22.92	36.67 / 53.33 / 43.33	26.67 / 30.00 / 33.33	0 85.7	4.7
AIME 2022	30	0 7.08 / 14.38 / 15.21	20.00 / 36.67 / 36.67	13.33 / 16.67 / 23.33	0 89.8	4.3
AIME 2023	30	10.63 / 15.21 / 16.04	26.67 / 40.00 / 40.00	10.00 / 20.00 / 20.00	0 84.7	4.7

Table 7: Results by competition year (

n=16

, %). Base / rl-replace / RL. Recovery is computed on pass@1. maj@16 is the accuracy of the most frequent final answer among the 16 samples, grouping answers that Math-Verify judges equivalent. Replace rate is the share of responses’ tokens taken from the RL model, averaged per response.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Experiment	Where	Benchmark	Models	Samples	Decoding
Power sampling (Test 0)	Table 1	MATH-500, AIME	3 pairs / Qwen3-4B/8B	8 seeds; greedy once	MCMC, $\alpha=4$ , 16 blocks $\times$ 10 steps
Gate-MCMC	§ 3.2	AIME (120)	Qwen3-4B	16	MCMC where gate $\geq 0.8567$
rl-replace (Test 1)	Table 2	AIME (120), MATH-500	Qwen3-4B	16; 64 on MATH-500	Appendix A.1
Opening ablation (Test 2)	Table 4	AIME (120)	Qwen3-4B	16	Prefilled “Alright” / “Alright, I”
Greedy control	Table 4	AIME (120)	Qwen3-4B	16	“Alright”, base greedy at disagreements
Rules (Test 3)	Table 5	AIME (120)	Qwen3-4B	16	Hard insertion or logit bias $+2$ / $+4$

Appendix

Table 8: Settings of the experiments in the paper. “Samples” is per problem.

Setting	Value
Algorithm	GRPO [ Shao and others, 2024 ]
Framework	verl [ Sheng et al., 2024 ]
Initialization	Base model, no supervised fine-tuning
Training data	DAPO-Math-17k [ Yu and others, 2025 ]
Prompt format	DAPO-Math-17k format (Appendix B )
Learning rate	$10^{-6}$ , constant

Appendix

Table 9: RL training configuration, shared by all three model pairs.

Quantity	Metric	Estimate [95% CI]
Recovery rate of rl-replace (%)
AIME24+25	pass@1	92.1 [75.6, 121.2]
AIME22+23	pass@1	87.7 [63.8, 120.9]
All AIME	pass@1	90.5 [77.0, 108.6]
MATH-500	pass@1	106.6 [95.2, 120.4]
AIME24+25 minus AIME22+23	pass@1	4.4 [ $-$ 32, +41]

Appendix

Table 10: Point estimates and 95% bootstrap intervals (problem-level, paired across conditions; 10,000 resamples, and 5,000 for the difference to base + “Alright”). AIME results use the 120 problems unless a group is named.

Problems	Greedy	Standard sampling	MCMC
AIME 2025 (30)	13.33	0 9.58	13.75
AIME 2022–2024 (90)	11.11	10.00	10.00

Appendix

Table 11: Power sampling on AIME with Qwen3-4B-Base, pass@1 (%).

	Mismatch label	KL label (top 10%)
Chosen log-prob	0.993 / 0.993	0.944 / 0.960
Entropy	–	0.945 / 0.960
Top-1 probability	–	0.945 / 0.963
Margin	–	0.943 / 0.965
Relative position	–	0.547 / 0.572
Token rank	–	0.531 / 0.535

Appendix

Table 12: AUROC of base-only features for locating the positions where the two models differ (AIME24 / AIME25). The gate is a gradient-boosted classifier trained on all six features.

Figure 6: Results for the 19 problems on which the base model fails all 16 samples and rl-replace solves at least one, grouped by year. (a) Number of successful rl-replace samples out of 16. (b) Mean number of text-changing replacements per successful response. The dashed line marks the overall mean of approximately 240, weighted by the number of successful samples for each problem.

Position (%)	0–10	10–20	20–30	30–40	40–50	50–60	60–70	70–80	80–90	90–100
Share	14.0	12.1	11.6	10.0	10.7	8.2	7.8	8.0	8.5	9.1

Appendix

Table 13: Share of the 9,099 replacements by relative position in the response (%).

Category	Example	Count	Share (%)
Other content words		6,794	74.7
To L a T e X	the $\to$ (	607	6.7
From L a T e X	$\(\to$ :	467	5.1
To a first-person subject	the $\to$ we, let $\to$ I	292	3.2
Numbers	2 $\to$ 4	202	2.2
Opening and pause words	To $\to$ Alright, So $\to$ Wait	190	2.1

Appendix

Table 14: Replacement categories with direction. “To X” means the RL token is of type X where the base model’s token was not; “from X” is the reverse.

Function	Pair	Count
Opening (position 0)	To $\to$ Alright	21
	Assistant $\to$ Alright	9
	user $\to$ Alright	8
Pause before going on	### $\to$ Wait	30
	So $\to$ Wait	20
	But $\to$ Wait	8

Appendix

Table 15: Frequent replacement pairs (base token

\to

RL token) in the 38 responses, grouped by function.

Figure 7: Further interventions on the base model, evaluated on 120 AIME problems with 16 samples per problem. Bars show changes in (a) pass@1 and (b) pass@16 relative to the base model (10.00% and 30.00%, respectively), in percentage points. Red indicates a decrease and green an increase. The score columns give the absolute pass@ $k$ values (%). The two panels use different horizontal-axis ranges.

Figure 8: Temperature sensitivity of the Qwen3-4B RL baseline on all 120 AIME problems: (a) pass@1, (b) pass@16, (c) pass@64, and (d) pass@256 (%), with top- $p$ fixed at 0.95. Grey lines with square markers show the RL model at temperatures 0.4, 0.6, and 0.8; green triangles show rl-replace at temperature 0.6. For rl-replace , temperature applies to base-model sampling at agreement positions, while disagreements use the RL model’s greedy token. Each panel starts at zero and uses its own vertical-axis range.

Condition	Correct responses	Problems solved
Base	0 / 128	0 / 8
Base + “Alright”	2 / 128	2 / 8
Base + “Alright, I”	3 / 128	3 / 8
RL model	0 / 128	0 / 8
rl-replace	12 / 128	8 / 8

Appendix

Table 16: Correct responses on the eight problems that neither the base model nor the RL model solves in 16 samples (128 responses per condition).

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,094 papers

Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning
Jun 2, 2026Ziyue Wang, Aomufei Yuan, Yongfu Zhu +10LLM Alignment RL for Language Model Reasoning
1Peking University · 2JINGDONG · 3Shanghai Innovation Institute +2
From Base Rollouts to RL Reasoning: A Budgeted Search Perspective
Sep 1, 2026Wenhe Sun, Cunxiang Wang, Zijun Yao +1Inference-Time Search RL for Language Model Reasoning
1Fudan University · 2Zhipu AI · 3Tsinghua University +1
Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Space
Aug 29, 2026Qiancheng Zhou, Ruizhe LiReasoning Diversity Reinforcement Learning with Verifiable Rewards
School of Future Technology, Shanghai University · School of Computer Science, University of Birmingham

Jun 2, 2026J/K move · Enter open · S save

Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning

Ziyue Wang, Aomufei Yuan, Yongfu Zhu, Shuai Dong+9

1Peking University · 2JINGDONG · 3Shanghai Innovation Institute+2

LLM Alignment RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards LLM Mathematical Reasoning

Open paper AskPDF arXiv

Reinforcement Learning from Verifiable Rewards (RLVR) has become the dominant approach for improving mathematical reasoning in large language models, yet current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states. Investigating this structure, we find that at the anchor token (the position immediately before the answer marker), correct rollouts converge naturally because they must produce the same answer (cosine similarity ~0.84), yet each retains residual variance from its unique reasoning path. Encouraging full alignment at this point pushes the model to extract a unified "correct decision" representation, reducing sensitivity to which reasoning path was taken. Based on this observation, we propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, with zero overhead in both training and inference. On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales, supported by ablations on loss type, anchor position, layer depth, and loss weight.

Abstract

RL for Language Model Reasoning

Explore similar work

Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning

From Base Rollouts to RL Reasoning: A Budgeted Search Perspective

Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Space