Supervised Fine-Tuning

Also known as SFT

Supervised fine-tuning trains a pretrained model to follow examples of desired behavior, from answering questions to reasoning, using tools, and interpreting images, audio, or other domain-specific inputs. Common approaches curate or generate instruction and agent-trajectory data, distill teacher outputs, and use methods such as LoRA or selective token and layer updates to control training cost and influence. Current work also adjusts what the model learns through token weighting, self-distillation, and constraints that preserve its original output distribution. The central tension is gaining reliable task performance without losing general capabilities or limiting later learning; how SFT should complement reinforcement learning depends on the data, task, and model.

Latest papers

214 papers29 papers in the last four weeks, up 12% on the four weeks before. 0.2% of all new papers.

Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Models
Feb 2, 2026Hao Wang, Hao Gu, Hongming Piao +6Supervised Fine-Tuning Language Model Distillation
1City University of Hong Kong · 2The Hong Kong University of Science and Technology · 3The Chinese University of Hong Kong
Bilevel Data Curation for LLM Fine-tuning: Offline Selection and Online Self-Refining Generation
Nov 26, 2025Quan Xiao, Yutong Xuan, Gaowen Liu +2Supervised Fine-Tuning LLM Self-Refinement
Cornell University · Cisco Research
Reclaiming Low-Quality Instruction Data with Rule-Guided LLM Reconstruction
Oct 27, 2025Zile Yang, Ling Li, Na Di +5Instruction-Tuning Data Selection Supervised Fine-Tuning
The Hong Kong University of Science and Technology (Guangzhou) · University of California, Santa Cruz · Hong Kong Baptist University
Learning Task Mixtures from Task Affinities: A Probabilistic Graphical Model for Supervised Fine-Tuning
Jul 16, 2025Prateek Chanda, Saral Sureka, Parth Pratim Chatterjee +3Supervised Fine-Tuning Multi-Task Learning
IIT Bombay · IBM Research · Red Hat AI Innovation +1

Feb 2, 2026J/K move · Enter open · S save

Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Models

Hao Wang, Hao Gu, Hongming Piao, Kaixiong Gong+5

1City University of Hong Kong · 2The Hong Kong University of Science and Technology · 3The Chinese University of Hong Kong

Supervised Fine-Tuning Language Model Distillation Self-Distillation Large Reasoning Models

Open paper AskPDF arXiv

The standard post-training recipe for large reasoning models, supervised fine-tuning followed by reinforcement learning (SFT-then-RL), may limit the benefits of the RL stage: while SFT imitates expert demonstrations, it often causes overconfidence and reduces generation diversity, leaving RL with a narrowed solution space to explore. Adding entropy regularization during SFT is not a cure-all; it tends to flatten token distributions toward uniformity, increasing entropy without improving meaningful exploration capability. In this paper, we propose CurioSFT, an entropy-preserving SFT method designed to enhance exploration capabilities through intrinsic curiosity. It consists of (a) Self-Exploratory Distillation, which distills the model toward a self-generated, temperature-scaled teacher to encourage exploration within its capability; and (b) Entropy-Guided Temperature Selection, which adaptively adjusts distillation strength to mitigate knowledge forgetting by amplifying exploration at reasoning tokens while stabilizing factual tokens. Extensive experiments on mathematical reasoning tasks demonstrate that, in SFT stage, CurioSFT outperforms the vanilla SFT by 2.5 points on in-distribution tasks and 2.9 points on out-of-distribution tasks. We also verify that exploration capabilities preserved during SFT successfully translate into concrete gains in RL stage, yielding an average improvement of 5.0 points.

Latest papers

Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Models

Bilevel Data Curation for LLM Fine-tuning: Offline Selection and Online Self-Refining Generation

Reclaiming Low-Quality Instruction Data with Rule-Guided LLM Reconstruction

Learning Task Mixtures from Task Affinities: A Probabilistic Graphical Model for Supervised Fine-Tuning