Why does supervised fine-tuning (SFT) lead to more forgetting than reinforcement fine-tuning (RFT), even when all teacher demonstrations are semantically correct? We study this question on classification tasks where tokens within each semantic class express the same semantic answer in different styles. The tasks share an underlying semantic rule but differ in their prompt distributions and teachers' stylistic preferences. Using a tractable linear-softmax policy, we derive an exact decomposition of the updates into semantic and style components. We show that, at a common policy and prompt, SFT and RFT have parallel semantic updates but differ in their style dynamics. Starting from a policy with no within-class style preference, RFT with exact policy gradients preserves this symmetry, whereas SFT with a nonuniform teacher develops off-axis style drift along a nonzero task mean under population updates. We use this drift to establish a separation under explicit conditions: for population updates from a common perfectly fitted checkpoint, SFT forgetting admits a strictly positive lower bound over a finite training interval, while RFT retains zero semantic error. Simulations over task sequences support these theoretical predictions.
Figures & tables
Figure 1: Semantic classes and expression styles. (a) The same token space is partitioned into semantic classes A0 and A1 . The prompt determines which class is rewarded; all tokens within a class receive the same reward. Here, x denotes the prompt’s vector representation. (b) Teachers favor different expression styles across tasks.
Figure 2: SFT and RFT under population and finite-sample updates. (a) Generalization–forgetting plane, for finite-sample (dashed, faint) and population (solid) updates; selected learning rates are annotated. (b) Final off-axis parameter energy ∥WT⊥∥F2 , averaged across seeds, versus learning rate, with a logarithmic energy axis.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure D.1: Two-task population dynamics at a fixed step size. (a) Old-task error, equal to forgetting from the common perfectly fitted checkpoint. (b) New-task error. (c) Off-axis style norm ∥β⊥∥F . The horizontal axis is logarithmic outside a small linear neighborhood of zero. Shading marks 0<τ≤τ0 ; the inset compares the first three SFT steps with the old-task error lower bound in Theorem 3 (dotted). Later times illustrate behavior beyond that guaranteed interval.
Figure E.1: Zero task means. We set μt=0 for every task. As in Figure 2 , panel (a) shows generalization error versus forgetting, and panel (b) shows final off-axis energy versus learning rate. Solid and dashed curves denote population and finite-sample updates, respectively.
Figure E.2: Uniform teachers. We retain the nonzero Gaussian task means of Figure 2 and set ρt,b(a)=1/M for every task and class. The layout and line styles match Figure E.1 .
Figure E.3: Non-Gaussian prompts. Prompt noise coordinates are drawn independently from Laplace(0,1/2) around the task means of Figure 2 , giving identity covariance. The layout and line styles match Figure E.1 .
Figure E.4: Forgetting under two assumption ablations. Top: MC population updates; bottom: finite-sample updates. Columns vary reward corruption and the number of REINFORCE actions per prompt, respectively. All panels use η≈0.121 and the same vertical scale. Training prompts are paired across methods. Shading gives pointwise 95% Student- t intervals over 10 MC seeds and mean ±1.96 standard errors over 50 finite-sample seeds, conditional on the fixed world and test sets. Negative forgetting indicates backward improvement.
Domain-specific supervised fine-tuning (SFT) often improves in-domain performance at the cost of degrading a model's general capabilities. We view this degradation through two practical gaps in domain SFT: a supervision-compatibility gap, where domain targets differ in style and reasoning format from the original model's natural responses, and a trajectory-preservation gap, where teacher-forced SFT optimizes fixed target tokens without constraining the model's behavior on its own generated prefixes. This process fails to preserve the model's original behavior. We propose RAFT (Data Refinement and Adaptive Distillation for Domain Fine-Tuning with Alleviated Forgetting), a two-stage framework that addresses both factors. First, RAFT constructs model-compatible supervision through self-conditioned rewriting, semantic filtering, and answer fusion. Second, RAFT performs Answer-Conditioned On-Policy Distillation, where the original instruction-tuned model provides soft targets on student-generated trajectories while being conditioned on the fused answer as helpful context. We further introduce top-K temperature distillation and EMA-based adaptive loss balancing to stabilize the domain-general trade-off. Across three instruction-tuned backbones and five domains, RAFT improves average domain accuracy by 23.2% over standard SFT, while recovering part of the SFT-induced degradation on MS-Bench and IFEval, with relative improvements of 18.2% and 10.2%, respectively. These results show that coupling data refinement with trajectory-level preservation provides an effective recipe for domain fine-tuning with alleviated forgetting.
Yuduo Li, Xiaofeng Shi, Qian Kou +2
Beijing Academy of Artificial Intelligence (BAAI) · Beijing Jiaotong University (BJTU)
Fine-tuning large language models (LLMs) frequently induces catastrophic forgetting of prior capabilities. Recent work has shown that reinforcement learning (RL) retains prior capabilities more effectively than supervised fine-tuning (SFT), attributing this to policy-gradient updates remaining closer to the base policy \cite{shenfeld2025rl}. We extend this behavioral account to the mechanistic level and ask whether RL's advantage is mirrored by stronger preservation of internal computational circuits. We introduce differential circuit vulnerability, a head-level measure of how much a circuit degrades under fine-tuning, and use it to compare RL and SFT on Qwen2.5-3B-Instruct adapted to scientific question-answering. We find a clear mechanistic trade-off: SFT adapts more rapidly to the target task but produces substantially greater circuit disruption and forgetting of prior capabilities, whereas RL preserves a larger fraction of the base circuit at the cost of slower task adaptation. These findings suggest that circuit preservation may help explain why RL is more robust to catastrophic forgetting. We released our code here: https://github.com/rl-sft-circuit-research/differential-circuit-vulnerability.
Jeanmely Rojas Nunez, Viraj Sawant, Nathan Allen +4
Adapting language models (LMs) to new tasks via post-training carries the risk of degrading existing capabilities -- a phenomenon classically known as catastrophic forgetting. In this paper, toward identifying guidelines for mitigating this phenomenon, we systematically compare the forgetting patterns of two widely adopted post-training methods: supervised fine-tuning (SFT) and reinforcement learning (RL). Our experiments reveal a consistent trend across LM families (Llama, Qwen) and tasks (instruction following, general knowledge, and arithmetic reasoning): RL leads to less forgetting than SFT while achieving comparable or higher target task performance. To investigate the cause for this difference, we consider a simplified setting in which the LM is modeled as a mixture of two distributions, one corresponding to prior knowledge and the other to the target task. We identify that the mode-seeking nature of RL, which stems from its use of on-policy data, enables keeping prior knowledge intact when learning the target task. We then verify this insight by demonstrating that the use on-policy data underlies the robustness of RL to forgetting in practical settings, as opposed to other algorithmic choices such as the KL regularization or advantage estimation. Lastly, as a practical implication, our results highlight the potential of mitigating forgetting using approximately on-policy data, which can be substantially more efficient to obtain than fully on-policy data.
Howard Chen, Noam Razin, Karthik Narasimhan +1
Princeton Language and Intelligence, Princeton University.