Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models

Haitao Jiang; Wenbo Zhang; Jiarui Yao; Hengrui Cai; Sheng Wang; Rui Song

cs.AIMar 14, 2026

Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models

Authors: Haitao Jiang, Wenbo Zhang, Jiarui Yao, Hengrui Cai, Sheng Wang, Rui Song

Organizations: North Carolina State University · UC Irvine · University of Illinois Urbana-Champaign · University of Washington · Amazon

Abstract

Pre-trained Large Language Model (LLM) exhibits broad capabilities, yet, for specific tasks or domains their attainment of higher accuracy and more reliable reasoning generally depends on post-training through Supervised Fine-Tuning (SFT) or Reinforcement Learning (RL). Although often treated as distinct methodologies, recent theoretical and empirical developments demonstrate that SFT and RL are closely connected. This study presents a comprehensive and unified perspective on LLM post-training with SFT and RL. We first provide an in-depth overview of both techniques, examining their objectives, algorithmic structures, and data requirements. We then systematically analyze their interplay, highlighting frameworks that integrate SFT and RL, hybrid training pipelines, and methods that leverage their complementary strengths. Drawing on a representative set of recent application studies from 2023 to 2025, we identify emerging trends, characterize the rapid shift toward hybrid post-training paradigms, and distill key takeaways that clarify when and why each method is most effective. By synthesizing theoretical insights, practical methodologies, and empirical evidence, this study establishes a coherent understanding of SFT and RL within a unified framework and outlines promising directions for future research in scalable, efficient, and generalizable LLM post-training.

Figures & tables

Panel of Figure 1 — Figure 1: A taxonomy of large language model (LLM) post-training alignment methods via supervised fine-tuning (SFT) and reinforcement learning (RL). We organize prior work along three categories: (1) algorithm-centric versus data-centric approaches within SFT and RL, (2) comparative, unifying, and hybrid frameworks that integrate SFT and RL objectives, and (3) representative downstream application domains, including reasoning, mathematics, agentic behavior, and code-related tasks.

Figure 2: Trends in task focus, training methodologies, and ground-truth data sources from 2023 to 2025. Results show rapid growth across all surveyed domains, with substantial increases in research volume and diversification of application areas; increasing convergence toward hybrid SFT–RL pipelines, supported by more mature training infrastructures, libraries, and preference datasets; and a continued shift from API-based labeling to data generated with increasingly capable open-weight models. Projections for 2025 and all reported proportions are derived from surveyed publications; see Appendix B.4 for further discussion.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description
Models and Policies
$\pi_{\theta},\pi_{\phi}$	Current policy/model parameterized by $\theta$ or $\phi$
$\pi_{\text{ref}},\pi_{\text{base}}$	Reference or base policy (typically frozen)
$\pi_{\text{old}}$	Policy from a previous optimization step
$\pi_{\text{expert}}$	Expert policy used for imitation or IRL
Data and Sequences

Appendix

Table 2: Summary of Notation and Symbols

Domain	Input	CoT / Reasoning	Output
General QA	Moderate	Moderate	Moderate
Mathematical	Moderate	Long	Moderate
Agentic	Long	Long(Multi-Step)	Moderate
Code Gen.	Long	Moderate	Long

Appendix

Table 3: Summary of LLM application domains and their characteristics.

Agentic

webshop, webarena, wind2web, miniwob++, scienceworld, alfworld,

tdw-mat, c-wah, alfred, rlcard

hotpotqa, strategyqa, triviaqa, pubmedqa, musique,

2wikimultihopqa, qasper

Math

Appendix

Table 4: Datasets used as search keys for benchmark-oriented paper classification.

	QA			Math			Agentic			Code
Keyword count	23	24	25	23	24	25	23	24	25	23	24	25
>0	771	1644	1308	12673	20405	13177	1310	1826	1373	334	1091	1302
>1	580	1321	1088	3870	7143	4889	330	541	437	209	767	921
>2	453	1022	858	1673	3295	2684	162	310	260	178	621	717
>3	384	861	730	932	1957	1953	125	234	207	148	549	604
>4	338	746	621	655	1391	1587	108	196	167	130	481	513

Appendix

Table 5: Paper counts under different keyword-mention thresholds across four domains.

Explore similar work

Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning
Apr 9, 2026Shiwan Zhao, Zhihu Wang, Xuyang Zhao +10LLM Post-Training RL for Language Models
Nankai University, China · Huawei Technologies Ltd., China
Demystifying Reinforcement Learning Post-Training of Language Models
Aug 24, 2026Donovan Clay, Saket Gollapudi, Sankar Harilal +4RL for Language Models RL Post-Training
University of Washington · Allen Institute for AI
Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Models
Apr 27, 2026Dan Shi, Zhuowen Han, Simon Ostermann +3Reinforcement Learning RL for Language Model Reasoning
TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China · German Research Center for Artificial Intelligence (DFKI), Saarbrücken, Germany · Saarland University, Saarbrücken, Germany

Apr 9, 2026J/K move · Enter open · S save

Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning

Shiwan Zhao, Zhihu Wang, Xuyang Zhao, Jiaming Zhou+9

Nankai University, China · Huawei Technologies Ltd., China

LLM Post-Training RL for Language Models RL Post-Training

Open paper AskPDF arXiv

Post-training has become central to turning pretrained large language models (LLMs) into aligned, capable, and deployable systems. Recent progress spans supervised fine-tuning (SFT), preference optimization, reinforcement learning (RL), process supervision, verifier-guided methods, distillation, and multi-stage pipelines. Yet these methods are often discussed in fragmented ways, organized by labels or objectives rather than by the behavioral bottlenecks they address. This survey argues that LLM post-training is best understood as structured intervention on model behavior. We organize the field first by trajectory provenance, which defines two primary regimes: off-policy learning on externally supplied trajectories and on-policy learning on learner-generated rollouts. We then interpret methods through two recurring roles -- effective support expansion, which makes useful behaviors more reachable, and policy reshaping, which improves behavior within already reachable regions -- together with a complementary systems-level role, behavioral consolidation, which preserves, transfers, and amortizes useful behavior across stages and model transitions. Under this view, SFT may serve either support expansion or policy reshaping; preference optimization is usually off-policy reshaping, though online variants move closer to learner-generated states. On-policy RL often improves behavior on learner-generated states, but stronger guidance can also make hard-to-reach reasoning paths reachable. Distillation is often better understood as consolidation rather than only compression, and hybrid pipelines emerge as coordinated multi-stage compositions. Overall, the framework helps diagnose post-training bottlenecks and reason about stage composition, suggesting that progress increasingly depends on coordinated systems design rather than any single dominant objective.

Abstract

Explore similar work

Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning

Demystifying Reinforcement Learning Post-Training of Language Models

Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Models