How RL Reshapes LLM Reasoning: Transferability, Coverage, and Scaling Laws

Ziheng Cheng; Yixiao Huang; Hanlin Zhu; Somayeh Sojoudi

cs.LGOct 3, 2026

How RL Reshapes LLM Reasoning: Transferability, Coverage, and Scaling Laws

Authors: Ziheng Cheng, Yixiao Huang, Hanlin Zhu, Somayeh Sojoudi

Organizations: University of California, Berkeley

Abstract

Recent studies on reinforcement learning (RL) report seemingly conflicting evidence about large language model (LLM) reasoning. Training on mathematics can improve performance in other domains, yet gains in Pass@1 can coincide with lower Pass@ $N$ than the base model. This raises a fundamental question: does RL expand an LLM's reasoning boundary, or merely reweight its existing reasoning space? We revisit these phenomena across Qwen and Gemma model families, showing both cross-domain gains and forgetting, while coverage at large sampling budgets increases on some tasks and decreases on others. Detailed analysis of solution traces before and after RL indicates a shift in the reasoning strategies the model employs, motivating a two-stage autoregressive policy model that separates strategy selection from problem-specific execution. Within this framework, we prove how RL's implicit bias reshapes strategy preferences, allowing gains on some tasks while suppressing strategies required by others. This mechanism can also broaden or narrow coverage at a given sampling budget even without expanding strategy support. We further provide theoretical justifications for log-sigmoid and log-linear scaling laws in RL compute, and evaluate their predictive power. Together, these results connect changes in strategy selection to cross-domain transfer, reasoning coverage, and compute scaling.

Figures & tables

Figure 1: Transfer and forgetting depend on the benchmark, training data, and base model. Each column reports changes in performance from the base model after training. RL consistently improves in-domain performance, but transfer behavior beyond the mathematics and coding domains varies significantly. In contrast, SFT exhibits catastrophic forgetting across a broad range of domains.

Figure 2: RL can either expand or contract reasoning coverage at large sampling budgets. Higher Pass@1 can accompany either lower or higher Pass@1024 relative to the base model. The direction of the coverage change varies across training mixtures, benchmarks, and base models.

Figure 3: RL uses fewer routes, but also routes absent from base samples. (a) Mean numbers of distinct routes, using equal numbers of identifiable responses from both models for each question. (b) Percentage of RL responses whose route is absent from the same question’s 16 base responses.

Figure 4: Scaling behavior of RL compute. Both laws capture the fitting regime effectively with

\displaystyle\mathsf{R^{2}}>0.99

on average. Extrapolative predictions are highly reliable up to 2

\displaystyle\times

compute in the math domain and average scores, yet exhibit greater variation on coding and planning tasks due to noisy RL training dynamics.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: The theoretical approximation bounds are a few percentage points in a representative regime. We use

\displaystyle p=0.1

, initial target-strategy mass

\displaystyle q_{0}=0.5

, unit reward contrast, and feature overlap

\displaystyle c\in\{0.5,1\}

. We vary

\displaystyle a\in[0.30,0.65]

and

\displaystyle b-a\in[0,0.30]

, with the zero-width boundary shown by continuity. All panels use the same raw-reward color scale, and contours report percentage-point error. The asymptotic term

\displaystyle\varepsilon_{\rm fit}

is omitted.

Benchmark	Base	SFT	Math RL	Math-Code RL	Base	SFT	Math RL
	Qwen3-14B-Base				Gemma-3-12B-IT
Math
MATH500	75.5	87.7	87.4	88.3	80.9	80.3	82.9
AIME 2024	14.3	63.0	33.1	46.2	25.9	31.4	28.4
AIME 2025	14.0	48.7	25.2	31.4	17.4	23.9	21.8
OMEGA	11.4	21.3	17.7	25.1	14.0	12.0	13.8

Appendix

Table 1: Mathematics gains accompany transfer to some tasks and forgetting on others. Scores (%; higher is better) for the base models, SFT models evaluated with thinking enabled, and RL models at step 200, rounded to one decimal. Colors indicate changes from the corresponding base model before rounding.

Benchmark	Model	Pass@ $\displaystyle N$ (%)
		1	4	16	64	256	1024
Qwen3-14B-Base
MATH500	Base	73.00	86.24	91.36	94.17	95.92	97.80
	Math RL	87.13	91.23	93.42	94.84	95.71	96.60
	Math-Code RL	88.13	91.95	93.98	94.85	95.35	95.80
AIME 2024	Base	15.66	32.31	51.42	66.11	75.90	90.00

Appendix

Table 2: Detailed breakdown of Pass@

\displaystyle N

results for the RL and base models in Figure 2 .

Figure 6: Scaling behavior of RL compute. Both laws fit the observed training regime effectively.

RL data	Benchmark	$\displaystyle B$	$\displaystyle A$	$\displaystyle C_{\rm mid}$	Fit $\displaystyle\mathsf{R^{2}}$	$\displaystyle B_{\rm lin}(\times 100)$	$\displaystyle C_{\rm ref}$	Fit $\displaystyle\mathsf{R^{2}}$
		Log-sigmoid				Log-linear
Math RL	MATH500	$\displaystyle 0.54$	$\displaystyle 1.000$	$\displaystyle 41.7$	0.982	$\displaystyle 2.76$	$\displaystyle 0.6$	0.975
Math RL	AIME 2024	$\displaystyle 1.23$	$\displaystyle 0.361$	$\displaystyle 8.1$	0.981	$\displaystyle 5.66$	$\displaystyle 1.2$	0.982
Math RL	AIME 2025	$\displaystyle 1.71$	$\displaystyle 0.262$	$\displaystyle 9.1$	0.942	$\displaystyle 3.83$	$\displaystyle 1.8$	0.920
Math RL	LiveCodeBench	$\displaystyle 0.39$	$\displaystyle 1.000$	$\displaystyle{>}10^{4}$	0.881	$\displaystyle 1.28$	$\displaystyle 0.5$	0.845
Math RL	ACPBench	$\displaystyle 1.69$	$\displaystyle 0.693$	$\displaystyle 15.8$	0.861	$\displaystyle 4.56$	$\displaystyle 2.9$	0.851

Appendix

Table 3: Log-sigmoid and log-linear fits using GPU hours.

\displaystyle A

is the fitted log-sigmoid asymptote on the raw

\displaystyle[0,1]

score scale.

\displaystyle C_{\rm mid}

and

\displaystyle C_{\rm ref}

are in units of

\displaystyle 10^{2}

GPU-hours. Both laws fit the observed training regime effectively.

Figure 7: Training-step scaling on five benchmarks. Points show measured Qwen3-14B scores for Math RL and Math-Code RL. Solid and dashed curves show log-sigmoid and log-linear fits as functions of optimizer step, respectively. Each panel uses its own truncated score scale.

Figure 8: Scaling behavior using optimizer steps. Both laws are fit through step 200 and extrapolated through step 350, producing the same qualitative pattern as fits based on GPU-hours.

RL data	Benchmark	$\displaystyle B$	$\displaystyle A$	$\displaystyle C_{\rm mid}$	Fit $\displaystyle\mathsf{R^{2}}$	$\displaystyle B_{\rm lin}(\times 100)$	$\displaystyle C_{\rm ref}$	Fit $\displaystyle\mathsf{R^{2}}$
		Log-sigmoid				Log-linear
Math RL	MATH500	$\displaystyle 0.76$	$\displaystyle 1.000$	$\displaystyle 207.1$	0.985	$\displaystyle 3.77$	$\displaystyle 8.7$	0.976
Math RL	AIME 2024	$\displaystyle 1.61$	$\displaystyle 0.379$	$\displaystyle 77.3$	0.979	$\displaystyle 7.83$	$\displaystyle 16.5$	0.976
Math RL	AIME 2025	$\displaystyle 3.05$	$\displaystyle 0.259$	$\displaystyle 72.1$	0.953	$\displaystyle 5.22$	$\displaystyle 20.9$	0.902
Math RL	LiveCodeBench	$\displaystyle 0.53$	$\displaystyle 1.000$	$\displaystyle{>}10^{4}$	0.895	$\displaystyle 1.64$	$\displaystyle 8.1$	0.867
Math RL	ACPBench	$\displaystyle 2.60$	$\displaystyle 0.723$	$\displaystyle 137.2$	0.899	$\displaystyle 6.14$	$\displaystyle 31.3$	0.827

Appendix

Table 4: Log-sigmoid and log-linear fits using optimizer step.

\displaystyle A

is the fitted log-sigmoid asymptote on the raw

\displaystyle[0,1]

score scale.

\displaystyle C_{\rm mid}

and

\displaystyle C_{\rm ref}

are in steps.

Benchmark	Index	Question summary	Base	Math RL
AIME24	8	Count maximal placements of two colors of chips on a $\displaystyle 5\times 5$ grid.	3	0
AIME24	25	Count octagon colorings satisfying a rotation condition.	1	0
AIME25	11	Find the area of a bounded region defined by nonlinear inequalities in the plane.	0	24
AIME25	27	Compute a modular remainder from a long rational recurrence.	0	1

Appendix

Table 5: AIME questions accounting for the changes in Qwen’s observed Pass@1024 coverage after Math RL. Each row gives a question’s evaluation index and a brief problem summary. The Base and Math RL columns give the number of correct responses among 1,024 samples per question; zero indicates no correct response.

Explore similar work

Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning
May 7, 2026Ömer Faruk Akgül, Rajgopal Kannan, Willie Neiswanger +1RL for Language Model Reasoning Efficient Language Model Reasoning
University of Southern California · 2DEVCOM ARL
Can RL Teach Long-Horizon Reasoning to LLMs? Expressiveness Is Key
May 7, 2026Tianle Wang, Zhaoyang Wang, Guangchen Lan +4Logical Reasoning RL for Language Model Reasoning
Purdue University · UNC Chapel Hill · Georgia Tech +1
Understanding Reasoning from Pretraining to Post-Training
Jul 17, 2026Jingyan Shen, Ang Li, Salman Rahman +4Language Model Pretraining RL for Language Model Reasoning
New York University · Columbia University · University of California, Los Angeles +2

May 7, 2026J/K move · Enter open · S save

Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning

Ömer Faruk Akgül, Rajgopal Kannan, Willie Neiswanger, Viktor Prasanna

University of Southern California · 2DEVCOM ARL

RL for Language Model Reasoning Efficient Language Model Reasoning LLM Reasoning

Open paper AskPDF arXiv

Reinforcement learning has become the standard for improving reasoning in large language models, yet evidence increasingly suggests that RL does not teach new strategies; it redistributes probability mass over solutions the base model already contains. In this work, we ask: if RL merely steers the model toward paths it already knows, is the RL optimization loop itself necessary? Through token-level analysis across multiple model families and RL algorithms, we find that RL's beneficial footprint is a sparse, predictable correction concentrated at high-entropy decision points where the model is uncertain which branch to take. Only 1--3% of token positions are affected, the promoted token always lies within the base model's top-5 alternatives, and targeted corrections at those few positions causally recover a large fraction of RL's accuracy gain, while random corrections fail. The base model's own entropy identifies these positions without any RL-trained model, and the entire correction is low-dimensional, representable in a tiny fraction of model parameters. These findings reframe reasoning improvement as sparse policy selection, not capability acquisition. We translate this insight into ReasonMaxxer, a minimal RL-free method that applies contrastive loss only at entropy-gated decision points, using a few hundred base-model rollouts and no online generation. Across three model families, six scales, and six math reasoning benchmarks, ReasonMaxxer matches or exceeds full RL performance while requiring only tens of problems and minutes of single-GPU training, a reduction in training cost of roughly three orders of magnitude.

Abstract

Explore similar work

Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning

Can RL Teach Long-Horizon Reasoning to LLMs? Expressiveness Is Key

Understanding Reasoning from Pretraining to Post-Training