Demystifying Reinforcement Learning Post-Training of Language Models

Donovan Clay; Saket Gollapudi; Sankar Harilal; Min Jang; Jacob Morrison; Sewoong Oh; Natasha Jaques

cs.LGAug 24, 2026

Demystifying Reinforcement Learning Post-Training of Language Models

Authors: Donovan Clay, Saket Gollapudi, Sankar Harilal, Min Jang, Jacob Morrison, Sewoong Oh, Natasha Jaques

Organizations: University of Washington · Allen Institute for AI

Abstract

Reinforcement learning (RL) post-training has emerged as a powerful framework for enhancing the capabilities of large language models (LLMs), enabling impressive reasoning, math, and coding capabilities. Yet for many researchers and practitioners, the principles behind classical RL remain a "black box". In this work, we deconstruct the RL post-training algorithm, investigating each step to clarify what is actually happening beneath the surface. By isolating the mechanics of RL with Verifiable Rewards in a controlled and simplified environment, we examine how RL outcomes are shaped by the base model's prior distribution, the granularity of the reward signal, the diversity of the prompt distribution, and model scale. We use the entropy of the policy's output distribution as a lens to compare the distributions learned through pretraining, SFT, and RL post-training, revealing how each stage shapes model certainty. Our investigation sheds light on how these choices interact to affect post-training success. For example, we show that the effect of so-called 'spurious rewards' depends on the prompt distribution used for post-training. We also provide insight into why the success of RL post-training depends on whether the base model already places sufficient probability mass on the desired behavior, linking it to the classical concept of exploration in RL. Ultimately, we provide this primer as a resource to those in the NLP community wishing to incorporate RL as a tool in their toolbox.

Figures & tables

Figure 1: Overview of the RL Post-Training Process. Our experiments isolate the specific impact of the Base Distribution, the Prompt Distribution ( $\mathcal{D}$ ), and the Reward Function ( $r$ ). We exclude algorithmic variation from our scope, deferring to the extensive existing research on RL algorithmic choices and engineering techniques that enhance performance (see e.g. ( Khatri et al., 2025 ) )

Table 1: Probability of producing the target quote before and after RL post-training, (eval = 10,000 samples).

Configuration	SFT+	Base	SFT-
No Reward	26.6%	3.92%	0.00%
Sparse Reward	85.9%	10.2%	0.00%
Dense Reward	86.7%	92.2%	0.00%

Table 2: Probability of correctly solving AIME Problem 4 before and after RL post-training on Qwen2.5-7B-Instruct (eval = 128 samples).

Panel of Figure 2 — Figure 2: Overlay of sparse vs. dense learning dynamics. RL learning curves for (a) Qwen3-1.7B on the movie quote task; (b) Qwen2.5-7B-Instruct on the AIME task. Both figures show an example of how dense rewards can successfully guide the model to learn a behavior with very low probability in the base model, which it is not possible to learn with sparse rewards.

Figure 3: Acc@1 on the MATH and AMC datasets after training on narrow and broad $\mathcal{D}$ ; training on narrow $\mathcal{D}$ yields higher accuracy.

Panel of Figure 4 — Figure 4: Effect of training OLMo models on spurious (random) rewards). (a) Average entropy on broad $\mathcal{D}$ (10k prompts). Each training stage (Base $\to$ SFT $\to$ DPO) delays entropy increase onset, showing increasing resilience. (b) Broad $\mathcal{D}$ SFT model—entropy spike at step $\sim$ 400 coincides with collapse of all eval metrics (GSM8K, MMLU, IFEval), confirming global unlearning. (c) Narrow $\mathcal{D}$ SFT model (100 math prompts)—entropy decreases , MMLU and IFEval are preserved, but GSM8K collapses, showing targeted corruption of the training domain.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: RL post-training dynamics and evaluation metrics for Qwen2.5-7B-Instruct - Math Reasoning (Top Left) Prior to RL, the SFT+ model shows a strong initial prior (26.6%), and the Base model exhibits a small but non-zero starting mass (3.9%). (Bottom Left) Final RL evaluation reveals that Dense rewards on both the SFT+ and Base models successfully and acquire the target behavior ( $>85\%$ match rates) however under sparse rewards the Base model was unable to acquire the target behavior while the SFT+ model acquires the target behavior. SFT- unable to gain momentum with either reward structure. (Right) The raw reward curves highlight a difference in the RL Training between Base, SFT+ and SFT- models. Experiments show that even through complex reasoning tasks, the priors of the models impact the role of RL post training.

Panel of Figure 6 — Figure 6: RL post-training dynamics and evaluation metrics for Qwen2-7B. (Top Left) Prior to RL, the SFT+ model shows a strong initial prior (28.8%), and the Base model exhibits a small but non-zero starting mass (3.5%). (Bottom Left) Final RL evaluation reveals that both the SFT+ and Base models successfully and perfectly acquire the target behavior ( $>98\%$ match rates) under both dense and sparse rewards, while SFT- completely fails. (Right) The raw reward curves highlight a delayed but successful optimization for the Base model. While SFT+ climbs immediately, the Base model requires roughly 40 steps to escape its initial plateau before rapidly converging to a 1.0 reward. The SFT- model remains strictly suppressed.

Panel of Figure 7 — Figure 7: RL post-training dynamics and evaluation metrics for Qwen2-1.5B. (Top Left) Initial substring match performance prior to RL shows the SFT+ model possesses only marginal probability mass for the target sequence (8.0%), while Base (0.3%) and SFT- (0.0%) exhibit near-zero coverage. (Bottom Left) Final RL evaluation demonstrates that only the SFT+ model successfully acquires the target behavior (achieving $>90\%$ match rates under both reward regimes). The Base model gains a slight benefit from dense rewards (7.0%) but largely fails, alongside SFT-. (Right) Raw reward curves confirm these dynamics: the SFT+ model successfully optimizes both signals, while the Base and SFT- variants flatline, failing to escape their initial exploration plateaus.

Panel of Figure 8 — Figure 8: RL post-training dynamics and evaluation metrics for Qwen3-1.7B. (Top Left) Initial coverage is limited, with SFT+ at 14.0% and Base/SFT- near zero. (Bottom Left) Final evaluation demonstrates the specific power of dense rewards. While the Base model only achieves a 10.0% match rate under sparse rewards, the granular gradient of the dense reward allows it to reach 48.8%. The SFT+ model succeeds broadly across both signals. (Right) The reward curves explicitly show this dynamic: under dense rewards, the Base model successfully begins climbing (reaching $\sim$ 0.5), whereas under sparse rewards, it remains completely flat at 0. SFT- fails entirely across all regimes.

Panel of Figure 9 — Figure 9: RL post-training dynamics and evaluation metrics for Qwen3-8B. (Top Left) The SFT+ model starts with a highly accessible prior (42.8%), while the Base and SFT- models start with exactly 0.0% probability mass. (Bottom Left) Final evaluation shows a stark binary outcome: the SFT+ model perfectly converges to a 100.0% match rate under both sparse and dense rewards, while the Base and SFT- models achieve absolute 0%. (Right) The raw reward curves demonstrate rapid, smooth convergence for SFT+ under both regimes. In contrast, the Base and SFT- models remain completely flat, unable to generate the target sequence a single time during exploration, confirming the strict bottleneck imposed by a 0% base prior.

Explore similar work

Understanding Reasoning from Pretraining to Post-Training
Jul 17, 2026Jingyan Shen, Ang Li, Salman Rahman +4Language Model Pretraining RL for Language Model Reasoning
New York University · Columbia University · University of California, Los Angeles +2
Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Models
Apr 27, 2026Dan Shi, Zhuowen Han, Simon Ostermann +3Reinforcement Learning RL for Language Model Reasoning
TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China · German Research Center for Artificial Intelligence (DFKI), Saarbrücken, Germany · Saarland University, Saarbrücken, Germany
Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning
Apr 9, 2026Shiwan Zhao, Zhihu Wang, Xuyang Zhao +10LLM Post-Training RL for Language Models
Nankai University, China · Huawei Technologies Ltd., China

Jul 17, 2026J/K move · Enter open · S save

Understanding Reasoning from Pretraining to Post-Training

Jingyan Shen, Ang Li, Salman Rahman, Yifan Sun+3

New York University · Columbia University · University of California, Los Angeles+2

Language Model Pretraining RL for Language Model Reasoning RL Post-Training

Open paper AskPDF arXiv

Reinforcement learning (RL) has become central to improving large language models (LLMs) on complex reasoning tasks, yet RL post-training is largely studied in isolation from the pretraining that precedes it. As a result, two basic questions remain open: (1) how do pretraining choices (model size, data) shape the returns to RL compute, and (2) what does RL actually do to the model? These questions are difficult to study in the standard LLM setting: pretraining corpora are vast and uncontrolled, making it hard to attribute behaviors to pretraining versus RL, and systematic compute sweeps across both stages are prohibitively expensive. To address these challenges, we use chess as a controlled testbed for studying reasoning across the full pretraining-to-post-training pipeline. We follow the standard LLM training pipeline by pretraining language models from 5M to 1B parameters on human chess games, supervised fine-tuning on synthetic reasoning traces, and running RL on chess puzzles with verifiable rewards. Using this framework, we find that the post-RL performance at given RL compute level is well-predicted from the pretraining loss, and slope of the RL reward curves improves approximately linearly with the pretraining tokens. Beyond scaling, we find that RL does not simply sharpen the SFT policy: on easy puzzles it amplifies correct moves the SFT policy already preferred, while on hard puzzles it surfaces correct moves that were nearly absent under SFT. We further test whether our findings transfer beyond chess by training a 1B language model on math-domain text, where the same predictive pattern emerges: longer-pretrained checkpoints reach higher post-RL performance and improve faster under RL. In sum, we provide a quantitative account of the pretraining-to-RL interface and a controlled testbed for studying the science of reasoning across the full pretraining-to-post-training pipeline.

Abstract

Explore similar work

Understanding Reasoning from Pretraining to Post-Training

Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Models

Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning