Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning

Shiwan Zhao; Zhihu Wang; Xuyang Zhao; Jiaming Zhou; Caiyue Xu; Chenfei Liu; Liting Zhang; Yuhang Jia

cs.CLApr 9, 2026

Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning

Authors: Shiwan Zhao, Zhihu Wang, Xuyang Zhao, Jiaming Zhou, Caiyue Xu, Chenfei Liu, Liting Zhang, Yuhang Jia, +5 more

Organizations: Nankai University, China · Huawei Technologies Ltd., China

Abstract

Post-training has become central to turning pretrained large language models (LLMs) into aligned, capable, and deployable systems. Recent progress spans supervised fine-tuning (SFT), preference optimization, reinforcement learning (RL), process supervision, verifier-guided methods, distillation, and multi-stage pipelines. Yet these methods are often discussed in fragmented ways, organized by labels or objectives rather than by the behavioral bottlenecks they address. This survey argues that LLM post-training is best understood as structured intervention on model behavior. We organize the field first by trajectory provenance, which defines two primary regimes: off-policy learning on externally supplied trajectories and on-policy learning on learner-generated rollouts. We then interpret methods through two recurring roles -- effective support expansion, which makes useful behaviors more reachable, and policy reshaping, which improves behavior within already reachable regions -- together with a complementary systems-level role, behavioral consolidation, which preserves, transfers, and amortizes useful behavior across stages and model transitions. Under this view, SFT may serve either support expansion or policy reshaping; preference optimization is usually off-policy reshaping, though online variants move closer to learner-generated states. On-policy RL often improves behavior on learner-generated states, but stronger guidance can also make hard-to-reach reasoning paths reachable. Distillation is often better understood as consolidation rather than only compression, and hybrid pipelines emerge as coordinated multi-stage compositions. Overall, the framework helps diagnose post-training bottlenecks and reason about stage composition, suggesting that progress increasingly depends on coordinated systems design rather than any single dominant objective.

Figures & tables

Figure 1. Unified overview of the survey’s organizational and analytical framework. Trajectory provenance serves as the top-level organizational axis, which motivates the review of off-policy and on-policy post-training in Sections 4 and 5 . Within each regime, the method sections are organized by major algorithm families. Supervision interfaces are retained as a secondary descriptive layer for clarifying how training signals are attached within and across method families, whereas functional roles serve as the main explanatory lens for interpreting behavioral change. Hybrid pipelines are synthesized later in Section 6 as coordinated multi-stage compositions rather than as a third regime parallel to off-policy and on-policy learning. The figure is intended as both a reading map and a compact preview of the framework formalized in Section 3 ; it does not imply fixed one-to-one mappings between regime, interface, and role.

Survey cluster	Off/On axis	Functional roles	Distill. transfer	Multi-stage synthesis	Main emphasis
Instruction tuning ( Zhang et al., 2026a )	$-$	$-$	$-$	$-$	Instruction tuning / supervised fine-tuning, instruction following, dataset construction, and applications across domains
Preference optimization ( Liu et al., 2025c )	$-$	$-$	$-$	$-$	Direct preference optimization, preference optimization variants, benchmarking, applications, and open challenges
RLHF / alignment ( Casper et al., 2023 ; Wang et al., 2023b )	$-$	$-$	$-$	$\circ$	LLM alignment methods, RLHF, evaluation, and RLHF limitations / open problems
Process / verifier ( Zheng et al., 2025b ; Guan et al., 2024 )	$-$	$\circ$	$-$	$\circ$	Process reward modeling, process supervision, verifier engineering, and verifier-guided post-training
Distillation ( Xu et al., 2024b )	$-$	$\circ$	$\bullet$	$-$	Teacher–student transfer, compression, self-improvement, and knowledge distillation methods
General post-training ( Tie et al., 2025 )	$-$	$-$	$-$	$-$	Broad post-training taxonomy spanning fine-tuning, alignment, reasoning, efficiency, and integration / adaptation

Table 1. Qualitative positioning of representative prior surveys under the organizational lens adopted in this paper. ‘Off/On axis’ asks whether off-policy versus on-policy learning is used as a top-level organizing principle; ‘Functional roles’ asks whether methods are analyzed by their dominant behavioral role rather than mainly by method family; ‘Distill. transfer’ asks whether distillation is treated as cross-stage transfer rather than only as compression; and ‘Multi-stage synthesis’ asks whether hybrid and related multi-stage compositions are synthesized explicitly. The comparison reflects the specific organizational lens of this work rather than a general assessment of survey quality, coverage, or depth. The symbols indicate recurring organizational emphasis:

\bullet

denotes a recurring primary organizing emphasis,

\circ

denotes secondary or implicit treatment, and

-

denotes that the dimension is not foregrounded as a recurring organizing principle.

Dominant role	Typical positive evidence	Main question answered	Typical evidence against dominance	Attrition risk to monitor
Expansion	Newly occupied effective regions; improved reachability of useful prefixes or locally important actions; emergence of solution patterns that were previously absent or difficult to realize reliably; supervision that exposes the learner to behavior it rarely realized before	Does training make previously negligible or inaccessible useful behavior practically reachable?	Gains are concentrated mainly on already reachable responses, styles, branches, or state-action choices, with little evidence that new effective regions became available	Previously effective regions may shrink while new ones expand; apparent gains may mask losses of earlier capabilities elsewhere
Reshaping	Better ranking among already reachable candidates; improved local branch choice on learner-visited states; stronger calibration, alignment, or response quality without clear widening of effective support; probability redistribution within shared effective support	Does training improve behavior mainly by reweighting state-action choices that were already effectively available?	Improvements depend primarily on importing new trajectories, new prefixes, or previously unseen state-action regions, rather than on improving choice within retained effective support	Harmful reweighting inside retained regions may degrade previously reliable behavior even when support is preserved
Consolidation	Preservation of gains across stage transitions; successful transfer to a smaller or cheaper student; retention after handoff; amortization of costly search, reflection, or teacher guidance into a reusable policy; robust deployment without the original scaffold	Can useful behavior be preserved, transferred, compressed, or amortized across stages, models, or deployment settings?	The main gains exist only during the inducing stage and disappear once teacher support, search, or runtime scaffolding is removed	Stage transitions may cause support attrition if useful induced behavior is not faithfully retained in the next model or regime

Table 2. Operational proxies for assigning a dominant functional role. The table is not intended as a strict checklist or deterministic decision rule; rather, it provides practical cues for deciding whether a post-training stage is primarily expansion-dominant, reshaping-dominant, or consolidation-dominant in a given setting. Support attrition is listed separately as a diagnostic risk rather than as a fourth dominant role.

Method family	Prov.	Typical train states	Interface	Dominant role	Boundary note	Main strength	Typical attrition / handoff risk
SFT	Off-pol.	Curated prompts and prefixes	Token targets	Expansion or reshaping	Weakens under staged synthetic rewriting or strong teacher pipelines	Efficient behavior import; strong format alignment	Exposure mismatch and forgetting of previously effective behavior
Offline preference optimization	Usually off-pol.	Curated chosen/rejected responses	Ranked preferences	Usually reshaping	Weakens under online preference refresh	Strong offline ranking and alignment calibration	Weak learner-state correction; may over-optimize style while rollout failures remain
Off-policy teacher-guided transfer	Off-pol.	Teacher traces or logits	Teacher-guided targets	Usually consolidation	Weakens when teacher guidance is attached to learner rollouts	Efficient teacher transfer and amortization	Teacher–deployment mismatch; subtle behavior may be lost in compression
On-policy RLHF / RLAIF / RLVR	On-pol.	Learner rollouts	Rewards, evaluators, tests	Learner-state reshaping	Weakens when replay or scaffolds dominate training states	Direct correction of rollout-time failures	Expensive, evaluator-sensitive, and hard to preserve after handoff
Online preference optimization	Mixed / on-pol.	Current-policy preference candidates	Ranked preferences	Usually reshaping	Weakens when data stay mostly offline	Better learner-relevant preference alignment than offline preference optimization	Limited by generated candidates, especially under sparse refresh or rare desirable behaviors
On-policy verifier / process guidance	On-pol.	Learner intermediate states	Process feedback, verifiers	Fine-grained learner-state reshaping	Weakens when the same verifier mainly curates offline data	Denser local correction than terminal rewards alone	Depends on verifier faithfulness and reachable-state coverage

Table 6. Comparison of major post-training paradigms under the unified lens. Entries summarize dominant tendencies rather than exhaustive or mutually exclusive assignments. Functional role refers to typical dominant usage; the boundary-note column indicates when the family-level label is most likely to weaken.

Explore similar work

Apr 20, 2026

Post-training is (Massive) Supervised Learning

The prevailing paradigm for training LLMs has evolved to rely on a massive post-training phase consisting of SFT and RL. In this position paper, we argue that this methodology effectively marks a reversion to the

pre-train then fine-tune'' approach of the BERT era, explicitly tailoring models to the desired behaviors and specific benchmarks on which they are evaluated. We begin with a historical overview of LLMs, describing the different phases of the LLM evolution. We argue that the current landscape is remarkably similar to the early days of LLMs, where task performance heavily relied on fitting the models to in-distribution datasets. To empirically demonstrate this, we compare pre-trained models to randomly initialized ones, by fine-tuning both variants on modern reasoning datasets and evaluating them on competitive math and code benchmarks. We show that models post-trained from scratch yield highly non-trivial performance. Our findings suggest that current post-training methodologies function primarily as a distribution-fitting mechanism. We finish by positing that developing generally capable models and systems requires moving beyond extensive post-training for predefined behaviors, shifting instead toward training procedures where models

learn how to learn''.

Michael Hassid, Yossi Adi, Roy Schwartz

1FAIR, Meta AI · 2The Hebrew University of Jerusalem

Supervised Fine-Tuning Language Model Post-Training

Ask PDF

Sep 1, 2026

From Rollouts to Recipes: Self-Contained Post-Training for LLMs

Post-training large language models usually applies a single training recipe to all samples, even though the model's own rollouts reveal different sample-level learning states. We propose Self-Routing, a behavior-conditioned post-training framework that uses rollout correctness and confidence to decide how each sample should be optimized. Depending on its behavior state, a sample is routed to GRPO, on-policy self-distillation, regularization, or skipping, allowing training to adapt without external teachers, extra annotations, or additional sampling. Experiments on mathematical reasoning across Qwen3 and Qwen3.5 backbones show that Self-Routing consistently improves over uniform GRPO, uniform OPSD, fixed mixtures, and simpler routing baselines. Further analyses show that the routing distribution changes over training and reduces unnecessary updates on low-signal or already stable samples.

Yifei Li, Lingling Zhang, Muye Huang +3

School of Computer Science and Technology, Xi’an Jiaotong University, China · Shaanxi Province Key Laboratory of Big Data Knowledge Engineering · Zhongguancun Academy, Beijing, China +1

Self-Training for Language Models RL for Language Model Reasoning LLM Post-Training

Ask PDF

Aug 24, 2026

Demystifying Reinforcement Learning Post-Training of Language Models

Reinforcement learning (RL) post-training has emerged as a powerful framework for enhancing the capabilities of large language models (LLMs), enabling impressive reasoning, math, and coding capabilities. Yet for many researchers and practitioners, the principles behind classical RL remain a "black box". In this work, we deconstruct the RL post-training algorithm, investigating each step to clarify what is actually happening beneath the surface. By isolating the mechanics of RL with Verifiable Rewards in a controlled and simplified environment, we examine how RL outcomes are shaped by the base model's prior distribution, the granularity of the reward signal, the diversity of the prompt distribution, and model scale. We use the entropy of the policy's output distribution as a lens to compare the distributions learned through pretraining, SFT, and RL post-training, revealing how each stage shapes model certainty. Our investigation sheds light on how these choices interact to affect post-training success. For example, we show that the effect of so-called 'spurious rewards' depends on the prompt distribution used for post-training. We also provide insight into why the success of RL post-training depends on whether the base model already places sufficient probability mass on the desired behavior, linking it to the classical concept of exploration in RL. Ultimately, we provide this primer as a resource to those in the NLP community wishing to incorporate RL as a tool in their toolbox.

Donovan Clay, Saket Gollapudi, Sankar Harilal +4

University of Washington · Allen Institute for AI

RL for Language Models RL Post-Training Reinforcement Learning with Verifiable Rewards RL Exploration

Ask PDF