Where RL Breaks: Localizing and Repairing Failures via Targeted Continued Pretraining

Xinyi Wang; Danqi Chen

cs.CLOct 19, 2026

Where RL Breaks: Localizing and Repairing Failures via Targeted Continued Pretraining

Authors: Xinyi Wang, Danqi Chen

Organizations: Princeton Language and Intelligence

Abstract

Reinforcement learning with verifiable rewards (RLVR) is widely used to improve reasoning in language models, yet growing evidence suggests that it mostly sharpens what the base model can already do. This raises a practical question: when RL fails, what is the base model missing? We look inside the RL policy's own rollouts and find that failures are localized: whether a rollout fails is strongly predicted by a small set of characteristic n-grams, ranging from reflection phrases such as "let's recheck" to specific operations such as "the inverse of A". These failures persist throughout RL training, although they are not beyond the model's reach: other sampled answers to the same question are often correct. Building on this observation, we use the failure $n$ -grams to retrieve text from the pretraining corpus and continue pretraining the base model on it before RL. The resulting corpus has only 35M tokens, 3--4 orders of magnitude smaller than typical mid-training. We evaluate this approach on OLMo2 and Qwen3 models trained with RL on (1) GSM8K; (2) a mixture of math, commonsense, and knowledge questions; and (3) general math and science reasoning. Despite its size, the corpus yields consistent gains: our approach outperforms continued pretraining on the same amount of text sampled from the pretraining corpus across 10 out of 11 settings, including by $2.0$ points on GSM8K when inserted before mid-training. The retrieved text also transfers across model families: text retrieved for OLMo2-1B's failures improves Qwen3-4B on the mixture by $1.9$ points over the same baseline. Together, our results suggest that where RL fails is a cheap and effective signal for selecting pretraining data.

Figures & tables

Panel of Figure 1 — Figure 1: Overview. We mine failure $n$ -grams from RL rollouts, retrieve text containing them, continue pretraining on it, and re-run the same RL recipe. The resulting $\pi^{\prime}_{\mathrm{RL}}$ consistently outperforms training on an equal amount of text sampled from the same corpora.

	Failure $n$ -gram	$a(g)$	Count	$a(g)$ after CPT	Count after CPT
OLMo2-1B / GSM8K
	there’s a mistake	6.2%	745	8.1%	839
	Let’s recheck	5.6%	591	6.3%	2166
	failure five-grams	25.7%	8859	29.2%	16620
Qwen3-4B / multi-task
	correct answer. The correct	8.7%	126	16.7%	1360

Table 1: Failure

n

-grams mark low-accuracy states, and targeted CPT makes the model traverse them more often and more successfully. For each phrase we report its conditional accuracy

a(g)

and the number of rollouts containing it, late in RL, without and with our CPT. Summary rows aggregate over the retrieval queries, or over the failure five-grams mined from the no-CPT run, which were never used for retrieval. General Reasoner runs are compared at matched RL steps.

CPT inserted after stage 1 (before mid-training)
		Acc after RL
Setting	Model	None	Control	Ours
GSM8K	OLMo2-1B	37.4	36.2	38.2
Multi-task	OLMo2-1B	43.2	45.3	45.8
CPT on the released base model (after mid-training)
GSM8K	OLMo2-1B	59.3	60.0	60.6

Table 2: Main results: greedy test accuracy at the final RL step with the RL recipe fixed. The control uses the identical pipeline and budget with content-neutral queries; all CPT on

35

M tokens. Multi-task is the macro-average over GSM8K, MMLU, TriviaQA, and Social IQA. General Reasoner is the mean over eight math and science benchmarks. OLMo2-1B GSM8K reports the mean over three RL seeds; others single-seed. ‡ : OLMo2-1B-mined corpus. ¶ : compared at a matched RL step (Table 9 ).

Figure 2: Failure-state statistics over the course of RL for OLMo2-1B on GSM8K with CPT inserted after stage 1. The failure set (66 five-grams) is mined at steps 120–140 (shaded). RL leaves the entry rate flat, whereas our corpus doubles it. The accuracy panel starts at step 100, before which the three trajectories coincide; all curves are 3-point (30-step) moving averages.

Setting	Model	$\Pr[E_{\mathcal{G}}]$	$a(\mathcal{G})$	test acc.
GSM8K	OLMo2-1B (after stage 1)	$5.7\to 10.6$	$25.7\to 29.2$	$\mathbf{36.2\to 39.2}$
GSM8K	OLMo2-1B (after stage 1, 3 seeds)	$6.8\to 7.6$	$24.3\to 28.4$	$\mathbf{37.4\to 38.2}$
General Reasoner	Qwen3-1.7B (base)	$7.4\to 7.3$	$\mathbf{16.9\to 27.7}$	$34.1\to 35.0$

Table 3: Entry rate into the failure states, success rate from them, and test accuracy before

\to

after CPT on

35

M retrieved tokens, same RL recipe. After stage 1 and for OLMo2-7B all three rise; in the General Reasoner setting the success rate rises at an unchanged entry rate.

Figure 3: pass@

k

with 32 samples at

T{=}0.7

; hollow diamonds mark greedy accuracy. Left: OLMo2-1B on GSM8K, CPT after stage 1, mean over three RL seeds. Middle: Qwen3-1.7B on MATH-500 in the General Reasoner setting (step 226; ours is the self-mined corpus, which has no matched control). Right: Qwen3-4B on TriviaQA after multi-task RL. The OLMo2-1B base rarely produces the required #### answer, hence its near-zero curve. RL raises pass@1 far more than pass@32; the repaired model leads the control at every

k

and, relative to no CPT, leads on OLMo2-1B and Qwen3-4B and, at

k{=}32

, edges above it on Qwen3-1.7B.

	After stage 1 (before mid-training)		Released base (after mid-training)
CPT data	GSM8K	multi-task	GSM8K	multi-task
none	37.4	43.2	59.3	52.6
matched control	36.2	45.3	60.0	49.5
failure $n$ -grams (ours)	38.2	45.8	60.6	51.5
ours $-$ none	$+0.8$	$+2.6$	$+1.3$	$-1.1$
ours $-$ matched control	$+2.0$	$+0.5$	$+0.6$	$+2.0$

Table 4: Where to insert the repair: greedy accuracy of OLMo2-1B at the final RL step with the same

35

M-token pipeline at both insertion points; the bottom rows give the gain of our corpus over no CPT and over the control.

Task	Model	Insertion	0	35M	100M	1B
GSM8K	OLMo2-1B	After stage 1	37.4	38.2	37.7	33.4
GSM8K	OLMo2-1B	Released base	59.3	60.6	60.5	59.9
Multi-task	Qwen3-0.6B	Released base	52.4	52.8	52.6	51.7
Multi-task	Qwen3-1.7B	Released base	62.4	62.5	63.4	63.2
General Reasoner	Qwen3-4B	Released base	37.5	40.5	35.2	18.8

Table 5: CPT token budget: greedy accuracy at the final RL step against the number of retrieved tokens; the last row evaluates the CPT model before RL. OLMo2-1B GSM8K at

0

and

35

M: three-seed means.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

	GSM8K	multi-task	General Reasoner
rollouts per prompt $G$	5	5	8
prompts per step	512	512	1024
mini-batch	128	128	256
learning rate	$10^{-6}$	$10^{-6}$	$5\times 10^{-7}$
clip $\epsilon$	0.2	0.2	0.3 (clip-higher)
KL weight $\beta$	$10^{-3}$	$10^{-3}$	$10^{-4}$

Appendix

Table 6: RL and continued-pretraining hyperparameters. GRPO uses the low-variance KL estimator; “clip-higher” denotes the asymmetric clipping of DAPO with

\epsilon_{\text{high}}=0.3

. CPT with the “after stage 1” insertion uses the OLMo trainer and continues the stage-1 schedule; all other CPT uses a fresh AdamW optimizer.

Corpus	Index	Queries
GSM8K, OLMo2-1B, after stage 1 (ours, 35M; also the Qwen3 GSM8K and multi-task “OLMo2-1B-mined” corpora)	Olmo-mix (API index)	“let’s correct”, “still incorrect”, “solve it correctly”, “wait”
Matched control for the above (35M; also the Qwen3 controls)	Olmo-mix (API index)	“1”, “first”, “step 1”
GSM8K, OLMo2-1B, after stage 1, 175M (ours: 20 lowest-accuracy 6-grams)	Olmo-mix	“, which totals to ”, “seems there’s a mistake”, “’s a mistake in the”, “there’s a mistake in”, “, which is not possible.”, “.{}nWe are given that”, “the nearest whole number, which”, “recheck the steps.{}n”, “Let’s recheck the”, “. Let’s recheck”, “.{}nHe buys ”, “be a whole number, and”, “must be a whole number,”, “whole number, which is ”, “we can’t have a”, “can’t have a fraction”, “t have a fraction of a”, “ving for x, we get”, “there was a misunderstanding in”, “standing in the problem statement.”
GSM8K, OLMo2-1B, after stage 1, 175M control (20 highest-accuracy 6-grams)	Olmo-mix	“, he needs to buy ”, “as follows:{}nLet”’, “months in a year, so”, “find out how many packs”, “. So, to find out”, “lollipops.{}n”, “miles.{}nTo find out”, “”{}nThe original price of”, “they eat a total of ”, “ounces.{}n####”, “.{}nHe gets paid $”, “.{}nNow, let”’, “, we need to subtract the”, “uesday.{}n#### ”, “out how many packs he”, “pieces. So, the total”, “.{}nHe sells each”, “.{}nSo, the cost”, “cubic inches.{}n####”, “ so he ate ”
GSM8K, OLMo2-1B released base (self-mined, $n{=}5$ )	Dolmino-mix, OLMoE-mix	“which is incorrect”, “Let’s correct”, “there’s a mistake”, “doesn’t make sense”, “However, let’s”, “Let’s recheck”, “have a negative number of”, “we’ll round”, “can’t have half”, “we know it takes”, “or a mistake in”, “because we can’t”, “there’s a misunderstanding”, “However, based on the”, “need to recheck the”, “be a mistake in”
Multi-task, OLMo2-1B released base ( $n{=}4$ )	Dolmino-mix, OLMoE-mix	“to calculate the area”, “a negative number of”, “Let’s re”, “recheck”, “: we are given”, “in the problem statement”, “the nearest whole number”, “recheck the steps”, “since we can”, “a mistake in the”, “it seems there”, “a fraction of a”, “we set up”, “find out how long”, “set up the equation”, “’t make sense”

Appendix

Table 7: Retrieval queries. “API index” denotes the public infini-gram v4_olmo-mix-1124_llama index; the others are local indexes of the named corpora.

Setting	Model	Base $+$ RL	Base $+$ control CPT $+$ RL	Base $+$ our CPT $+$ RL
multi-task	OLMo2-1B	52.6 $\pm$ 0.4	49.5	51.5 $\pm$ 0.1
multi-task	Qwen3-1.7B	62.4	63.4	62.5

Appendix

Table 8: Multi-task settings in which CPT did not help, omitted from Table 2 (same protocol: greedy macro-average at the final RL step; OLMo2-1B no-CPT and ours: mean

\pm

s.d. over two RL seeds; rest single-seed).

Model	CPT corpus	RL step	mean acc.
Qwen3-4B	none	35	44.0
Qwen3-4B	reflection queries (OLMo2-1B-mined), 35M	35	44.7
Qwen3-4B	self-mined $n{=}3$ , 35M	60	43.4
Qwen3-4B	self-mined $n{=}5$ , 35M	60	43.1
Qwen3-4B	self-mined $n{=}3{+}5$ , 35M	40	44.5
Qwen3-4B	self-mined $n{=}3$ , 100M	40	42.2

Appendix

Table 9: General Reasoner setting: mean greedy accuracy over eight benchmarks (GSM8K, MATH-500, Minerva, OlympiadBench, AMC, AIME 2024, AIME 2025, GPQA-Diamond). RL runs with self-mined corpora were stopped by compute limits before 226 steps, so checkpoints are compared at the indicated step; the three Qwen3-1.7B runs were later completed to 226 steps. Answers are graded by normalized exact match with symbolic equivalence (math-verify). AIME and AMC contain 30–40 problems each, so differences below

{\sim}2

points are noise: at step 226 the three Qwen3-1.7B runs are within

1

point on GSM8K (

80.1

–

81.2

) and MATH-500 (

66.0

–

66.6

), and their means differ through AMC and GPQA.

CPT	GSM8K	MATH-500	GPQA	mean (8 tasks)
none	71.7	71.0	27.8	37.5
OLMo2-1B-mined (transfer), 35M	55.8	59.6	20.7	29.1
targeted $n{=}3$ , 35M	85.3	75.6	29.8	40.5
targeted $n{=}3$ , 100M	82.5	72.2	27.3	35.2
targeted $n{=}3$ , 1B	29.5	43.8	17.7	18.8
targeted $n{=}5$ , 35M	83.6	71.4	27.8	38.0

Appendix

Table 10: Qwen3-4B and Qwen3-1.7B before RL (the source of the before-RL rows of Tables 2 and 5 ): greedy accuracy of the base model and of the CPT models, same eight-benchmark suite and grader as Table 9 . The transfer row is the OLMo2-1B-mined corpus of Table 2 ; targeted corpora are self-mined from Dolmino-mix (

n{=}3

: the corpora of Table 9 ;

n{=}5

: a set mined from validation rollouts); random = uniformly sampled Olmo-mix documents; mix = targeted and random documents shuffled together. The lower block repeats the sweep for Qwen3-1.7B-Base.

setting	none	ours 35M	random only	35M ours $+$ random	large ours $+$ random	large ours alone
GSM8K / OLMo2-1B, after stage 1	37.4 $\pm$ 1.1	38.2 $\pm$ 1.5	35.5 $\pm$ 1.0	35.4 c	35.5 $\pm$ 0.6	33.4 b
GSM8K / OLMo2-1B, released base	59.3 $\pm$ 0.7	60.6 $\pm$ 0.9	58.5 a	59.9 a	59.7 a	59.9 b
Qwen3-4B, before RL (8 tasks)	37.5	40.5	28.5	27.4	18.8	18.8
multi-task / OLMo2-1B, released base	52.6 $\pm$ 0.4	51.5 $\pm$ 0.1	51.4 $\pm$ 1.0	51.4 $\pm$ 0.3	51.9 $\pm$ 1.3	51.7 $\pm$ 0.4

Appendix

Table 11: Replay sweep: uniformly sampled Olmo-mix documents mixed into the CPT corpus. OLMo2-1B rows: final RL accuracy on GSM8K, mean

\pm

s.d. over RL seeds (

n{=}3

unless marked; a : two seeds,

59.7

–

59.9

; b : single seed; c :

34.3

and

36.5

on two seeds, the third seed’s RL never left

8\%

). Qwen3-4B row: the pre-RL eight-benchmark mean of Table 10 . Sizes: after stage 1, random

=400

M, large

=435

M targeted; released base and Qwen3-4B, random

=1

B, large

=500

M targeted

+

500

M random, and “ours large”

=1

B targeted. Multi-task row: macro average over the four tasks at RL step 595, mean

\pm

s.d. over two RL seeds, self-mined

n{=}4

corpus; the full targeted-fraction sweep is in Table 12 .

targeted tokens	0	35M	100M	200M	500M	700M	1B
targeted fraction	0%	3.5%	10%	20%	50%	70%	100%
accuracy	51.4 $\pm$ 1.0	51.4 $\pm$ 0.3	52.1 $\pm$ 0.6	51.8 $\pm$ 0.2	51.9 $\pm$ 1.3	51.9 $\pm$ 0.6	51.7 $\pm$ 0.4

Appendix

Table 12: Targeted fraction at a fixed

1

B-token CPT budget on the multi-task mixture (OLMo2-1B released base, self-mined

n{=}4

corpus, remainder uniformly sampled Olmo-mix documents): final macro accuracy over the four tasks, mean

\pm

s.d. over two RL seeds. Reference points: no CPT

52.6\pm 0.4

, the

35

M targeted slice alone

51.5\pm 0.1

, matched control

49.5

(one seed).

Insertion	CPT data	tokens	accuracy
after stage 1	high-accuracy $n$ -grams	175M	36.2
after stage 1	failure $n$ -grams (ours)	175M	37.7
released base	none	0	59.3 $\pm$ 0.7
released base	high-accuracy $n$ -grams	35M	59.4 $\pm$ 1.2
released base	MegaMath-Web-Pro (curated math)	35M	60.7
released base	failure $n$ -grams (ours)	35M	60.6 $\pm$ 0.9

Appendix

Table 13: Alternative CPT data for the runs of Table 4 (OLMo2-1B, GSM8K, final RL accuracy; mean

\pm

s.d. over three RL seeds where given). High-accuracy

n

-grams: the retrieval pipeline run with the highest-

a(g)

n

-grams instead of the failure set, i.e. targeting the states RL already handles; MegaMath-Web-Pro:

35

M tokens of a curated math web corpus with no retrieval.

Setting / model	CPT corpus	step	$\Pr[E_{\mathcal{G}}]$	$a(\mathcal{G})$	mean $a(g)$ ( $k$ )	rollout acc.
GSM8K / OLMo2-1B, after stage 1	failure $n$ -grams, 35M	150–210	$5.7\to 10.6$	$25.7\to 29.2$	–	$42.4\to 43.7$
GSM8K / OLMo2-1B, after stage 1	matched control, 35M	150–210	$5.7\to 6.7$	$25.7\to 28.7$	–	$42.4\to 42.4$
GSM8K / OLMo2-1B, after stage 1, RL seed 2	failure $n$ -grams, 35M	150–210	$4.7\to 5.4$	$23.0\to 28.5$	–	$40.8\to 44.5$
GSM8K / OLMo2-1B, after stage 1, RL seed 3	failure $n$ -grams, 35M	150–210	$10.1\to 6.9$	$24.1\to 27.4$	–	$43.6\to 42.2$
GSM8K / OLMo2-1B, released	self-mined $n{=}5$ , 35M	210	$1.1\to 1.8$	$18.5\to 10.9$	– (1)	$78.3\to 77.0$
GSM8K / Qwen3-0.6B	OLMo2-1B-mined, 35M	210	$0.7\to 1.1$	$26.3\to 22.2$	– (0)	$90.4\to 91.7$

Appendix

Table 14: Entry rate into the targeted failure states (

\Pr[E_{\mathcal{G}}]

, fraction of rollouts containing any

g\in\mathcal{G}

), success rate from them (

a(\mathcal{G})

), and mean

a(g)

over

n

-grams with

\geq 10

occurrences in both runs (

k

= number of such

n

-grams), before

\to

after CPT.

\dagger

: the Qwen3-1.7B multi-task set is dominated by a formatting fragment ( process is that , 664

\to

3 occurrences).

\lx@sectionsign

\mathcal{G}

is the set of 23 failure five-grams of Table 1 , mined from the no-CPT run at steps 480–520, rather than the retrieval queries; the matched control enters them in

0.4\%

of rollouts with

6.2\%

success.

	no CPT		matched control		ours
query	$n$	$a(g)$	$n$	$a(g)$	$n$	$a(g)$
doesn’t make sense	1605	7.5%	93	8.6%	1486	5.4%
have a negative number of	921	7.8%	800	8.4%	481	12.3%
there’s a mistake	745	6.2%	20	15.0%	839	8.1%
be a mistake in	731	8.3%	46	6.5%	684	6.1%
Let’s correct	706	5.2%	123	1.6%	289	4.5%

Appendix

Table 15: Every retrieval query of the OLMo2-1B GSM8K corpus on all training rollouts of steps 150–210 (stage-1 insertion, RL seed 1): number of rollouts containing the phrase and their success rate, without CPT, with the matched control, and with our corpus. The last two rows aggregate over the 16 queries and over the 66 five-grams of Figure 2 .

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,094 papers

Jul 17, 2026

Understanding Reasoning from Pretraining to Post-Training

Reinforcement learning (RL) has become central to improving large language models (LLMs) on complex reasoning tasks, yet RL post-training is largely studied in isolation from the pretraining that precedes it. As a result, two basic questions remain open: (1) how do pretraining choices (model size, data) shape the returns to RL compute, and (2) what does RL actually do to the model? These questions are difficult to study in the standard LLM setting: pretraining corpora are vast and uncontrolled, making it hard to attribute behaviors to pretraining versus RL, and systematic compute sweeps across both stages are prohibitively expensive. To address these challenges, we use chess as a controlled testbed for studying reasoning across the full pretraining-to-post-training pipeline. We follow the standard LLM training pipeline by pretraining language models from 5M to 1B parameters on human chess games, supervised fine-tuning on synthetic reasoning traces, and running RL on chess puzzles with verifiable rewards. Using this framework, we find that the post-RL performance at given RL compute level is well-predicted from the pretraining loss, and slope of the RL reward curves improves approximately linearly with the pretraining tokens. Beyond scaling, we find that RL does not simply sharpen the SFT policy: on easy puzzles it amplifies correct moves the SFT policy already preferred, while on hard puzzles it surfaces correct moves that were nearly absent under SFT. We further test whether our findings transfer beyond chess by training a 1B language model on math-domain text, where the same predictive pattern emerges: longer-pretrained checkpoints reach higher post-RL performance and improve faster under RL. In sum, we provide a quantitative account of the pretraining-to-RL interface and a controlled testbed for studying the science of reasoning across the full pretraining-to-post-training pipeline.

Jingyan Shen, Ang Li, Salman Rahman +4

New York University · Columbia University · University of California, Los Angeles +2

Language Model Pretraining RL for Language Model Reasoning RL Post-Training

Ask PDF

Jun 2, 2026

RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training

The standard LLM training pipeline applies reinforcement learning (RL) only after pre-training and supervised fine-tuning (SFT). We question this status quo by training a LLM from scratch and applying RL, SFT, and SFT followed by RL directly to intermediate pre-training checkpoints. We find that RL is effective very early, and often matches the full SFT

\to

RL pipeline early as well. Through experiments on harder problems, we find that targeted pre-training data composition is a strong lever for RL effectiveness, even more so than model scale. Beyond reasoning accuracy, applying RL directly to base checkpoints expands the model's distribution; the sharpening effect reported in recent work arises only when RL follows SFT. The general capabilities of the model remain essentially unchanged by RL, while they degrade following SFT. Finally, we merge RL and SFT objectives by parallel averaging, which outperforms across all other training methods discussed, across metrics, while preserving general capabilities. Together, these results suggest that LLM training might benefit from an expanded use of RL.

Rachit Bansal, Clara Mohri, Tian Qin +2

Harvard University

Language Model Pretraining Supervised Fine-Tuning RL for Language Models Reinforcement Learning RL Post-Training

Ask PDF

Aug 24, 2026

Demystifying Reinforcement Learning Post-Training of Language Models

Reinforcement learning (RL) post-training has emerged as a powerful framework for enhancing the capabilities of large language models (LLMs), enabling impressive reasoning, math, and coding capabilities. Yet for many researchers and practitioners, the principles behind classical RL remain a "black box". In this work, we deconstruct the RL post-training algorithm, investigating each step to clarify what is actually happening beneath the surface. By isolating the mechanics of RL with Verifiable Rewards in a controlled and simplified environment, we examine how RL outcomes are shaped by the base model's prior distribution, the granularity of the reward signal, the diversity of the prompt distribution, and model scale. We use the entropy of the policy's output distribution as a lens to compare the distributions learned through pretraining, SFT, and RL post-training, revealing how each stage shapes model certainty. Our investigation sheds light on how these choices interact to affect post-training success. For example, we show that the effect of so-called 'spurious rewards' depends on the prompt distribution used for post-training. We also provide insight into why the success of RL post-training depends on whether the base model already places sufficient probability mass on the desired behavior, linking it to the classical concept of exploration in RL. Ultimately, we provide this primer as a resource to those in the NLP community wishing to incorporate RL as a tool in their toolbox.

Donovan Clay, Saket Gollapudi, Sankar Harilal +4

University of Washington · Allen Institute for AI

RL for Language Models RL Post-Training Reinforcement Learning with Verifiable Rewards RL Exploration

Ask PDF

Abstract

RL for Language Model Reasoning

Explore similar work

Understanding Reasoning from Pretraining to Post-Training

RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training

Demystifying Reinforcement Learning Post-Training of Language Models