(How) Learning Rates Regulate Catastrophic Overtraining

Mark Rofin; Aditya Varre; Nicolas Flammarion

cs.LGApr 15, 2026

(How) Learning Rates Regulate Catastrophic Overtraining

Authors: Mark Rofin, Aditya Varre, Nicolas Flammarion

Organizations: EPFL

Abstract

Supervised fine-tuning (SFT) is a common first stage of LLM post-training, teaching the model to follow instructions and shaping its behavior as a helpful assistant. At the same time, SFT may harm the fundamental capabilities of an LLM, particularly after long pretraining: a phenomenon known as catastrophic overtraining (Springer et al., 2025). To understand overtraining, we first investigate catastrophic forgetting in finetuning through the lens of implicit regularization of the learning rate. For models trained to the same SFT loss, we identify how the learning rate mediates optimization: finetuning with large and small steps converges to qualitatively different models. Next, we link forgetting to overtraining: learning rate decay increases the sharpness of the pretrained model, which in turn exacerbates catastrophic forgetting during SFT, leading to overtraining. Our findings paint a picture of the overtraining mechanism in LLMs and broadly contribute to the understanding of the interplay between optimization dynamics during pretraining and finetuning.

Figures & tables

Figure 1: SFT training loss vs OOD performance for different models and LRs. Checkpoints taken throughout finetuning are shown by dots and connected within each run. The horizontal red line represents the performance of a base model pre-finetuning (without the chat template). All models lose a part of capabilities due to catastrophic forgetting. The curves for higher LRs lie below the ones for low LRs, indicating stronger forgetting.

Figure 2: SFT training loss vs MPA between the representations of the base and finetuned models throughout SFT (defined in Section 3.2 ). The angles increase as SFT progresses and the model moves from its initialization; but for any fixed level of training loss, higher LRs cause more rotation.

Panel of Figure 3 — Figure 3: Loss landscape analysis for OLMo 1 1B. Panels 3(a) , 3(b) , 3(e) , 3(f) : the dynamics of SFT loss and MPA (layer 10) during the start of finetuning. Panels 3(c) , 3(d) , 3(g) , 3(h) : $\Delta\mathrm{Loss}$ and $\Delta\mathrm{MPA}$ along each gradient step. Top row: fixed checkpoint, each line is a different LR. Bottom row: fixed LR, each line is a different checkpoint.

Panel of Figure 4 — Figure 4: Finetuning with Diagonal Networks. We train a diagonal network with $d=100$ using mini-batch SGD (10 runs) and plot the final distribution of absolute value of co-ordinates of weights. Here $(a,b,c,d)=(0,23,45,66)$ . In (a), when initialized without pretraining features, the model recovers the finetuning features, while larger step sizes lead to greater forgetting of pretraining features, as seen in the region $\mathcal{S}{\mathrm{FT}}\,\triangle\,\mathcal{S}{\mathrm{PreT}}$ . In (b,c), two runs with different initial sharpness and with same LR. Sharpness is measured as $\mathrm{Tr}(\nabla^{2}L)$ . We observe that the more time the model spends in sharper regions, the greater the forgetting.

Figure 5: Evolution of model sharpness throughout pretraining (x-axis reflects pretraining progress). Sharpness grows as LR is decayed; for two models trained with WSD schedule, a rapid increase in sharpness coincides with the LR cooldown phase.

Figure 6: The dynamic of accuracy drop (the difference between the OOD score between the pretrained and finetuned models) throughout pretraining (x-axis shows the pretraining progress). Forgetting increases in later checkpoints.

Figure 7: The principal angles between representations of base and finetuned SmolLM3-3B as a function of pretraining steps. The representational dissimilarity stays constant until the pretraining LR is decreased, when it surges.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: L2 on SAE representations vs MPA for Gemma 3 1B.

Figure 9: L2 on SAE representations vs SFT train loss for Gemma 3 1B.

Figure 10: Left: dynamics of estimated sharpness and true sharpness closely follow each other through pretraining. Right: same data, true vs estimated sharpness.

Figure 11: SFT train loss vs OOD score during supervised finetuning (same as Figure 1 ) for finetuning on the Tülu 3 dataset. The plots appear mostly horizontal, indicating that models go through a rapid forgetting phase in the beginning of finetuning and stay stable afterwards.

Panel of Figure 12 — Figure 12: MPA vs SFT train loss during supervised finetuning (same as Figure 2 ) for Gemma 3 and Hubble.

Panel of Figure 13 — Figure 13: MPA vs SFT train loss during supervised finetuning (same as Figures 2 and 12 ) but for finetuning on the Tülu 3 dataset.

Figure 14: SFT training loss vs OOD performance for each OOD task separately. Same setup as Figure 1 in the paper, Anthropic HH dataset.

Figure 15: SFT training loss vs OOD performance for each OOD task separately. Same setup as Figure 1 in the paper, Tülu 3 dataset.

Figure 16: Validation loss on SFT datasets when finetuning with different learning rates (for the same setup as Figure 1 in the paper). Top row: finetuning on Anthropic HH, bottom row: finetuning on Tülu 3.

Panel of Figure 17 — Figure 17: Loss landscape analysis (same as Figure 3 ) for OLMo 2 1B.

Panel of Figure 18 — Figure 18: MPA as a function of stepsize with variable learning rates (analogous to Figures 3(d) , 17(d) ) for different layers of OLMo 1 (top row) and OLMo 2 (bottom row).

Panel of Figure 19 — Figure 19: MPA as a function of stepsize with variable checkpoints (analogous to Figure 3(h) ) for different layers of OLMo 1.

Panel of Figure 20 — Figure 20: MPA as a function of stepsize with variable checkpoints (analogous to Figure 17(h) ) for different layers of OLMo 2.

Figure 21: Forgetting (the drop in OOD score between the base and finetuned models) increases throughout pretraining. Same as Figure 6 , but for the models finetuned on the Tülu 3 dataset.

Panel of Figure 22 — Figure 22: The principal angles between representations of base and finetuned models as a function of pretraining step, Same as Figure 7 , but for all models.

Panel of Figure 23 — Figure 23: Same as Figures 7 and 22 , but for the models finetuned on Tülu 3.

Figure 24: The dynamic of sharpness during a partial pretraining run of SmolLM 3 3B with and without LR decay at step 1000.

Explore similar work

May 7, 2026

Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Less

Optimizers play an important role in both pretraining and finetuning stages when training large language models (LLMs). In this paper, we present an observation that full finetuning with the same optimizer as in pretraining achieves a better learning-forgetting tradeoff, i.e., forgetting less while achieving the same or better performance on the new task, than other optimizers and, possibly surprisingly, LoRA, during the supervised finetuning (SFT) stage. We term this phenomenon optimizer-model consistency. To better understand it, through controlled experiments and theoretical analysis, we show that: 1) optimizers can shape the models by having regularization effects on the activations, leading to different landscapes around the pretrained checkpoints; 2) in response to this regularization effect, the weight update in SFT should follow some specific structures to lower forgetting of the knowledge learned in pretraining, which can be obtained by using the same optimizer. Moreover, we specifically compare Muon and AdamW when they are employed throughout the pretraining and SFT stages and find that Muon performs worse when finetuned for reasoning tasks. With a synthetic language modeling experiment, we demonstrate that this can come from Muon's strong tendency towards rote memorization, which may hurt pattern acquisition with a small amount of data, as for SFT.

Yuxing Liu, Jianyu Wang, Tong Zhang

1UIUC∗ · 2Apple

Supervised Fine-Tuning LLM Fine-Tuning Catastrophic Forgetting

Ask PDF

May 19, 2026

Fine-Tuning Without Forgetting via Loss-Adaptive Learning Rates

Fine-tuning large language models on new data improves task performance but degrades capabilities learned during pretraining, a phenomenon known as catastrophic forgetting. Existing methods mitigate this by modifying the fine-tuning objective to suppress high-loss tokens or sequences, but these tokens are essential for learning new tasks, especially those with poor pretraining coverage. In such settings, hard tokens should still contribute to learning, so forgetting must be controlled without suppressing them. We identify a simple mechanism for doing so: per-step forgetting is bounded by the product of the learning rate and the square root of the current training loss. This suggests that high-loss batches are especially prone to inducing forgetting. Motivated by this observation, we introduce FINCH, a loss-adaptive learning-rate schedule that reduces the learning rate on high-loss batches and increases it as the model converges, while leaving the fine-tuning objective unchanged. Across knowledge acquisition, science, and low-resource language adaptation benchmarks, FINCH reduces forgetting by 93% on average while matching the task performance of standard fine-tuning. On Qwen3-4B knowledge acquisition, FINCH cuts TruthfulQA degradation by 5x and reverses HaluEval degradation, while better preserving confidence calibration. Overall, our results show that learning-rate schedules are an effective tool to shape model behavior during fine-tuning, beyond just target-task optimization.

Parjanya Prajakta Prashant, Jiongli Zhu, Aldan Creo +1

University of California San Diego

Learning Rate Scheduling LLM Fine-Tuning Catastrophic Forgetting

Ask PDF

Apr 11, 2026

Why Supervised Fine-Tuning Fails to Learn: A Systematic Study of Incomplete Learning in Large Language Models

Supervised Fine-Tuning (SFT) is the standard approach for adapting large language models (LLMs) to downstream tasks. However, we observe a persistent failure mode: even after convergence, models often fail to correctly reproduce a subset of their own supervised training data. We refer to this behavior as the Incomplete Learning Phenomenon(ILP). This paper presents the first systematic study of ILP in LLM fine-tuning. We formalize ILP as post-training failure to internalize supervised instances and demonstrate its prevalence across multiple model families, domains, and datasets. Through controlled analyses, we identify five recurrent sources of incomplete learning: (1) missing prerequisite knowledge in the pre-trained model, (2) conflicts between SFT supervision and pre-training knowledge, (3) internal inconsistencies within SFT data, (4) left-side forgetting during sequential fine-tuning, and (5) insufficient optimization for rare or complex patterns. We introduce a diagnostic-first framework that maps unlearned samples to these causes using observable training and inference signals, and study several targeted mitigation strategies as causal interventions. Experiments on Qwen, LLaMA, and OLMo2 show that incomplete learning is widespread and heterogeneous, and that improvements in aggregate metrics can mask persistent unlearned subsets. The findings highlight the need for fine-grained diagnosis of what supervised fine-tuning fails to learn, and why.

Chao Xue, Yao Wang, Mengqiao Liu +11

University of New South Wales, Australia · Tencent Hunyuan, China · Tencent Yuanbao, China +2

LLM Fine-Tuning Supervised Fine-Tuning LLM Training

Ask PDF