A Quantitative Characterization of Forgetting in Post-Training

Krishnakumar Balasubramanian; Shiva Prasad Kasiviswanathan

cs.LGMar 12, 2026

A Quantitative Characterization of Forgetting in Post-Training

Authors: Krishnakumar Balasubramanian, Shiva Prasad Kasiviswanathan

Organizations: Department of Statistics, University of California, Davis · Amazon

Abstract

Continual post-training of generative models is widely used, yet a principled understanding of when and why forgetting occurs remains limited. We develop theoretical results under a two-mode mixture abstraction (representing old and new tasks), proposed by Chen et al. (2025) (arXiv:2510.18874), and formalize forgetting in two forms: (i) mass forgetting, where the old mixture weight collapses to zero, and (ii) old-component drift, where an already-correct old component shifts during training. For equal-covariance Gaussian modes, we prove that forward-KL objectives trained on data from the new distribution drive the old weight to zero, while reverse-KL objectives converge to the true target (thereby avoiding mass forgetting) and perturb the old mean only through overlap-gated misassignment probabilities controlled by the Bhattacharyya coefficient, yielding drift that decays exponentially with mode separation and a locally well-conditioned geometry with exponential convergence. We further quantify how replay interacts with these objectives. For forward-KL, replay must modify the training distribution to change the population optimum; for reverse-KL, replay leaves the population objective unchanged but prevents finite-batch old-mode starvation through bounded importance weighting. Finally, we analyze three recently proposed near-on-policy post-training methods, SDFT (arxiv:2601.19897), TTT-Discover (arxiv:2601.16175), and OAPL (arxiv:2602.19362), via the same lens and derive explicit conditions under which each retains old mass and exhibits overlap-controlled drift. Overall, our results show that forgetting can by precisely quantified based on the interaction between divergence direction, geometric behavioral overlap, sampling regime, and the visibility of past behavior during training.

Figures & tables

Method	Prevents Mass Forgetting?	Controls Old-Component Drift?
Forward-KL (SFT)	✗	✓
	(Theorem 2.1 )	(but unimportant
		as mass collapses)
Reverse-KL (RL)	✓	✓ Exponentially small in $\delta$
	(Theorem 2.2 )	(Theorem 2.3 )
SDFT	✓ If demonstrator strength is $>0$	✓ Finite total drift ⋆

Table 1: Summary of forgetting behavior across training objectives. “Prevents Mass Forgetting” means the population optimum satisfies

\beta^{\star}>0

. “Controls Old-Component Drift” means the gradient

\|\nabla_{m_{o}}L\|

is provably small when

m_{o}=\mu_{o}

. Mode separation

\delta=\|\mu_{n}-\mu_{o}\|_{\Sigma^{-1}}

\star

: We show it is exponentially small under additional assumptions; see Remark 3.2 .

LLM Post-Training

Large language model post-training adapts pretrained models into more useful, reliable, and specialized assistants through supervised and preference-based learning.

See all 246 papers

May 25, 2026

Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replay

Models trained on a new task typically degrade on prior tasks, a phenomenon known as forgetting. Traditionally, mitigating forgetting has required replaying stored exemplars from prior tasks, which is often impractical. By contrast, language models can sample from their own training distribution, and we show that these self-generated samples serve as effective replay data, nearly eliminating forgetting. We find that forgetting nonetheless persists when the model has little remaining capacity: models pretrained close to saturation cannot absorb new information without overwriting prior knowledge. When capacity is not the limiting factor, low learning rates reduce forgetting but require substantially more training steps. Replay breaks this tradeoff, enabling fast, high-learning-rate finetuning without forgetting.

Martin Marek, Dongkyu Cho, Shikai Qiu +3

New York University

Continual Learning for LLMs LLM Fine-Tuning Replay-Based Continual Learning Catastrophic Forgetting Fine-Tuning Language Models

Ask PDF

Oct 21, 2025

Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting

Adapting language models (LMs) to new tasks via post-training carries the risk of degrading existing capabilities -- a phenomenon classically known as catastrophic forgetting. In this paper, toward identifying guidelines for mitigating this phenomenon, we systematically compare the forgetting patterns of two widely adopted post-training methods: supervised fine-tuning (SFT) and reinforcement learning (RL). Our experiments reveal a consistent trend across LM families (Llama, Qwen) and tasks (instruction following, general knowledge, and arithmetic reasoning): RL leads to less forgetting than SFT while achieving comparable or higher target task performance. To investigate the cause for this difference, we consider a simplified setting in which the LM is modeled as a mixture of two distributions, one corresponding to prior knowledge and the other to the target task. We identify that the mode-seeking nature of RL, which stems from its use of on-policy data, enables keeping prior knowledge intact when learning the target task. We then verify this insight by demonstrating that the use on-policy data underlies the robustness of RL to forgetting in practical settings, as opposed to other algorithmic choices such as the KL regularization or advantage estimation. Lastly, as a practical implication, our results highlight the potential of mitigating forgetting using approximately on-policy data, which can be substantially more efficient to obtain than fully on-policy data.

Howard Chen, Noam Razin, Karthik Narasimhan +1

Princeton Language and Intelligence, Princeton University.

Continual Learning for LLMs RL Post-Training Catastrophic Forgetting

Ask PDF

Feb 19, 2026

CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training

Large language model (LLM) post-training enhances latent skills, unlocks value alignment, improves performance, and enables domain adaptation. Unfortunately, post-training is known to induce forgetting, especially in the ubiquitous use-case of leveraging third-party pre-trained models, which is typically understood as a loss of parametric or factual knowledge. We argue that this accuracy-centric view is insufficient for modern foundation models and instead define forgetting as systematic model drift that degrades behavior and user experience. In this context, we introduce CapTrack, a capability-centric framework for analyzing forgetting in LLMs that combines a behavioral taxonomy with an evaluation suite centered on capability-specific metrics. Using CapTrack, we conduct a large-scale empirical study across post-training algorithms, domains, and model families, including models up to 80B parameters. We find that forgetting extends beyond parametric knowledge, with pronounced drift in robustness and default behaviors. Instruction fine-tuning induces the strongest relative drift, while preference optimization is more conservative and can partially recover lost capabilities. Differences across model families persist, and no universal mitigation emerges.

Lukas Thede, Stefan Winzeck, Zeynep Akata +1

Thomson Reuters Foundational Research, 5 Canada Square, London · Tübingen AI Center, University of Tübingen · Helmholtz Munich +2

LLM Post-Training Catastrophic Forgetting LLM Evaluation Language Model Robustness

Ask PDF

Abstract

LLM Post-Training

Explore similar work

Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replay

Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting

CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training