Why does supervised fine-tuning (SFT) lead to more forgetting than reinforcement fine-tuning (RFT), even when all teacher demonstrations are semantically correct? We study this question on classification tasks where tokens within each semantic class express the same semantic answer in different styles. The tasks share an underlying semantic rule but differ in their prompt distributions and teachers' stylistic preferences. Using a tractable linear-softmax policy, we derive an exact decomposition of the updates into semantic and style components. We show that, at a common policy and prompt, SFT and RFT have parallel semantic updates but differ in their style dynamics. Starting from a policy with no within-class style preference, RFT with exact policy gradients preserves this symmetry, whereas SFT with a nonuniform teacher develops off-axis style drift along a nonzero task mean under population updates. We use this drift to establish a separation under explicit conditions: for population updates from a common perfectly fitted checkpoint, SFT forgetting admits a strictly positive lower bound over a finite training interval, while RFT retains zero semantic error. Simulations over task sequences support these theoretical predictions.
Figures & tables
Figure 1: Semantic classes and expression styles. (a) The same token space is partitioned into semantic classes A0 and A1 . The prompt determines which class is rewarded; all tokens within a class receive the same reward. Here, x denotes the prompt’s vector representation. (b) Teachers favor different expression styles across tasks.
Figure 2: SFT and RFT under population and finite-sample updates. (a) Generalization–forgetting plane, for finite-sample (dashed, faint) and population (solid) updates; selected learning rates are annotated. (b) Final off-axis parameter energy ∥WT⊥∥F2 , averaged across seeds, versus learning rate, with a logarithmic energy axis.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure D.1: Two-task population dynamics at a fixed step size. (a) Old-task error, equal to forgetting from the common perfectly fitted checkpoint. (b) New-task error. (c) Off-axis style norm ∥β⊥∥F . The horizontal axis is logarithmic outside a small linear neighborhood of zero. Shading marks 0<τ≤τ0 ; the inset compares the first three SFT steps with the old-task error lower bound in Theorem 3 (dotted). Later times illustrate behavior beyond that guaranteed interval.
Figure E.1: Zero task means. We set μt=0 for every task. As in Figure 2 , panel (a) shows generalization error versus forgetting, and panel (b) shows final off-axis energy versus learning rate. Solid and dashed curves denote population and finite-sample updates, respectively.
Figure E.2: Uniform teachers. We retain the nonzero Gaussian task means of Figure 2 and set ρt,b(a)=1/M for every task and class. The layout and line styles match Figure E.1 .
Figure E.3: Non-Gaussian prompts. Prompt noise coordinates are drawn independently from Laplace(0,1/2) around the task means of Figure 2 , giving identity covariance. The layout and line styles match Figure E.1 .
Figure E.4: Forgetting under two assumption ablations. Top: MC population updates; bottom: finite-sample updates. Columns vary reward corruption and the number of REINFORCE actions per prompt, respectively. All panels use η≈0.121 and the same vertical scale. Training prompts are paired across methods. Shading gives pointwise 95% Student- t intervals over 10 MC seeds and mean ±1.96 standard errors over 50 finite-sample seeds, conditional on the fixed world and test sets. Negative forgetting indicates backward improvement.