cs.LGApr 15, 2026
Save(How) Learning Rates Regulate Catastrophic Overtraining
Organizations: EPFL
Abstract
Supervised fine-tuning (SFT) is a common first stage of LLM post-training, teaching the model to follow instructions and shaping its behavior as a helpful assistant. At the same time, SFT may harm the fundamental capabilities of an LLM, particularly after long pretraining: a phenomenon known as catastrophic overtraining (Springer et al., 2025). To understand overtraining, we first investigate catastrophic forgetting in finetuning through the lens of implicit regularization of the learning rate. For models trained to the same SFT loss, we identify how the learning rate mediates optimization: finetuning with large and small steps converges to qualitatively different models. Next, we link forgetting to overtraining: learning rate decay increases the sharpness of the pretrained model, which in turn exacerbates catastrophic forgetting during SFT, leading to overtraining. Our findings paint a picture of the overtraining mechanism in LLMs and broadly contribute to the understanding of the interplay between optimization dynamics during pretraining and finetuning.
Figures & tables
Figure 1: SFT training loss vs OOD performance for different models and LRs. Checkpoints taken throughout finetuning are shown by dots and connected within each run. The horizontal red line represents the performance of a base model pre-finetuning (without the chat template). All models lose a part of capabilities due to catastrophic forgetting. The curves for higher LRs lie below the ones for low LRs, indicating stronger forgetting.
Figure 2: SFT training loss vs MPA between the representations of the base and finetuned models throughout SFT (defined in Section 3.2 ). The angles increase as SFT progresses and the model moves from its initialization; but for any fixed level of training loss, higher LRs cause more rotation.
Figure 3: Loss landscape analysis for OLMo 1 1B. Panels 3(a) , 3(b) , 3(e) , 3(f) : the dynamics of SFT loss and MPA (layer 10) during the start of finetuning. Panels 3(c) , 3(d) , 3(g) , 3(h) : and along each gradient step. Top row: fixed checkpoint, each line is a different LR. Bottom row: fixed LR, each line is a different checkpoint.
Figure 4: Finetuning with Diagonal Networks. We train a diagonal network with using mini-batch SGD (10 runs) and plot the final distribution of absolute value of co-ordinates of weights. Here . In (a), when initialized without pretraining features, the model recovers the finetuning features, while larger step sizes lead to greater forgetting of pretraining features, as seen in the region . In (b,c), two runs with different initial sharpness and with same LR. Sharpness is measured as . We observe that the more time the model spends in sharper regions, the greater the forgetting.
Figure 5: Evolution of model sharpness throughout pretraining (x-axis reflects pretraining progress). Sharpness grows as LR is decayed; for two models trained with WSD schedule, a rapid increase in sharpness coincides with the LR cooldown phase.
Figure 6: The dynamic of accuracy drop (the difference between the OOD score between the pretrained and finetuned models) throughout pretraining (x-axis shows the pretraining progress). Forgetting increases in later checkpoints.
Figure 7: The principal angles between representations of base and finetuned SmolLM3-3B as a function of pretraining steps. The representational dissimilarity stays constant until the pretraining LR is decreased, when it surges.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: L2 on SAE representations vs MPA for Gemma 3 1B.
Figure 9: L2 on SAE representations vs SFT train loss for Gemma 3 1B.
Figure 10: Left: dynamics of estimated sharpness and true sharpness closely follow each other through pretraining. Right: same data, true vs estimated sharpness.
Figure 11: SFT train loss vs OOD score during supervised finetuning (same as Figure 1 ) for finetuning on the Tülu 3 dataset. The plots appear mostly horizontal, indicating that models go through a rapid forgetting phase in the beginning of finetuning and stay stable afterwards.
Figure 12: MPA vs SFT train loss during supervised finetuning (same as Figure 2 ) for Gemma 3 and Hubble.
Figure 13: MPA vs SFT train loss during supervised finetuning (same as Figures 2 and 12 ) but for finetuning on the Tülu 3 dataset.
Figure 14: SFT training loss vs OOD performance for each OOD task separately. Same setup as Figure 1 in the paper, Anthropic HH dataset.
Figure 15: SFT training loss vs OOD performance for each OOD task separately. Same setup as Figure 1 in the paper, Tülu 3 dataset.
Figure 16: Validation loss on SFT datasets when finetuning with different learning rates (for the same setup as Figure 1 in the paper). Top row: finetuning on Anthropic HH, bottom row: finetuning on Tülu 3.
Figure 17: Loss landscape analysis (same as Figure 3 ) for OLMo 2 1B.
Figure 18: MPA as a function of stepsize with variable learning rates (analogous to Figures 3(d) , 17(d) ) for different layers of OLMo 1 (top row) and OLMo 2 (bottom row).
Figure 19: MPA as a function of stepsize with variable checkpoints (analogous to Figure 3(h) ) for different layers of OLMo 1.
Figure 20: MPA as a function of stepsize with variable checkpoints (analogous to Figure 17(h) ) for different layers of OLMo 2.
Figure 21: Forgetting (the drop in OOD score between the base and finetuned models) increases throughout pretraining. Same as Figure 6 , but for the models finetuned on the Tülu 3 dataset.
Figure 22: The principal angles between representations of base and finetuned models as a function of pretraining step, Same as Figure 7 , but for all models.
Figure 23: Same as Figures 7 and 22 , but for the models finetuned on Tülu 3.
Figure 24: The dynamic of sharpness during a partial pretraining run of SmolLM 3 3B with and without LR decay at step 1000.