When Forgetting is not Catastrophic: On the Mechanics of Spurious Forgetting
Authors: Vedant Palit, Florent Draye, Nicolas Zucchet, Zhijing Jin, Bernhard Schölkopf
Organizations: MPI for Intelligent Systems, Tübingen · Jinesis Lab, University of Toronto & Vector Institute · EuroSafeAI · Hector Foundation · Stanford University · ELLIS Institute Tübingen
Knowledge that a language model appears to forget during finetuning often remains stored and can be recovered, a phenomenon called spurious forgetting. Finetuning on new facts can even produce forgetting that undoes itself: recall of the old facts collapses, recovers as training continues on new facts alone, and only then erodes for good. We seek to understand when such forgetting is not catastrophic. A minimal associative memory reproduces these dynamics with three ingredients: keys with shared structure, concentrated new values, and normalization in the network. Finetuning moves all old representations along a common direction, hiding the old facts while preserving their relative geometry; normalization withdraws this shift once the new facts are learned, whereas fact-specific changes accumulate and cause the erosion. Moreover, subtracting the common shift eliminates the collapse in a Transformer trained on synthetic data, and removing a single direction from each weight update restores old facts in a pretrained language model. Forgetting thus combines a shared, reversible loss of access with a slow erosion of individual facts, and only the second is catastrophic. Which one dominates depends on whether the new data move old memories together or apart.
Figures & tables
Figure 1: Changing only the finetuning data turns catastrophic forgetting into spurious forgetting. A transformer is pretrained on two non-overlapping sets of facts, A and B , and then finetuned on new facts only, without any replay. When the new facts overlap with both pretraining sets (left, finetuning on C ), performance on both declines slowly and steadily. When they overlap only with B (right, fine-tuning on B′ ), performance on B is mostly intact, but performance on A collapses, recovers although A is never seen again, and later erodes. How the different sets are built and what it means for the new facts to overlap with the old ones is detailed in Section 4 .
Figure 2: Shared keys, concentrated new values and normalization are all needed for the collapse, recovery and erosion. The model maps keys, which share a component of strength α , to values split into two regions, through one hidden layer with RMS normalization. As in Figure 1 , it is pretrained on A and B , whose values lie in different regions, then finetuned only on new facts B′ , whose values lie in the region of B ; curves show recall of A . Reducing α weakens the collapse and removes the recovery (1), drawing the new values from both regions leaves only gradual forgetting (2), and replacing the normalization removes the recovery (3). The model is described in Section 3 .
Figure 3: Normalization reverses the shift responsible for forgetting. The old representations first move together along a shared direction. Once the new facts are learned and the individual margin becomes positive, the common shift retracts and the old facts are recalled again. Without a normalization layer, this reversal does not occur, which is why the collapse is permanent (Figure 2 , panel 3). (left) Old hidden states before normalization, projected on the shared direction and an orthogonal one, after pretraining, at the end of the collapse and during erosion. (middle) Common shift magnitude s of the old hidden states, with and without normalization. (right) Shared and individual margins of the new facts, which determine the dynamics of s through Equation 5 . Same setup as in Figure 2 ; the dotted line marks the end of the collapse.
Figure 4: The Transformer initially forgets through the same common shift as the minimal model. (left) The Transformer is trained on synthetic biographies of fictitious individuals, each stating several facts about them, such as where they come from. The possible answers of each attribute are split into two subsets: the individuals of A draw their answers from the first, and those of B from the second. The model is pretrained on A and B , then finetuned on new individuals B′ whose answers come from the same subset as B ; this is the setup of Figure 1 (right). (middle) The common shift Δhˉ of the hidden states of A grows fast and retracts at the end of the collapse, as in the minimal model (Figure 3 ). The fact-specific displacements εa grow more slowly and eventually become larger. (right) Accuracy on A , and when only the common part of the change of its logits is kept or removed. The dotted line indicates the end of the collapse.
Figure 5: The collapse persists in a pretrained language model, but training reverses it only for arbitrary new facts. OLMo 2 1B is finetuned without replay, in separate runs, on synthetic individuals or on real entities. Old facts (CounterFact facts the model knows) are solid, new facts dotted; three seeds. (left) Recall of the old facts drops for both kinds of new facts, but recovers only for synthetic individuals. (middle) Removing the top singular direction of each weight update (light) restores recall of the old facts for both kinds, whereas removing a random direction of the same norm leaves it where training did (dark). (right) With a larger set of synthetic individuals, the collapse goes deeper and lasts longer.
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
Minimal
Transformer
OLMo 2
Analysis
model
attention-only
with MLP blocks
1B
Collapse, recovery and erosion
Fig. 2
Fig. 1
App. C.5
Fig. 5
New answers in both regions
Fig. 2
Fig. 1
App. C.5
–
Common shift and fact-specific drift
App. B.10
Fig. 4
App. C.5
–
Removing the common shift
App. B.9
Fig. 4
App. C.5
–
Ordering within the region
App. B.8
App. C.1
App. C.5
–
Appendix
Table 1: Reference location for each analysis, for each model.
Figure 6: Examples from the three datasets. Each fact is learned from training phrasings and evaluated on held-out phrasings; answers in bold. The old facts of the language model are facts it already knows, and the new facts concern synthetic individuals, whose answers are unrelated to their names, or little-known real entities.
Figure 7: Equation 38 reproduces the dynamics of the common shift. (left) Length ∥hˉA∥ of the old facts’ mean hidden state, measured and obtained by integrating Equation 38 (with normalization) or s˙=αγˉsh (without normalization) from the point at which the mean state has aligned with the shared direction, after which its length follows the same dynamics as its displacement s . (right) Average shared margin γˉsh and individual margin γˉind of the new facts, with normalization. The dotted line marks the peak; ten seeds.
Figure 8: How the shift is written. (left) Cosine between the update of each old hidden state and their mean update, and relative error of Equation 4 . (middle) Speed of the update at the start of finetuning against α . (right) Displacement of the old and new hidden states along the shared direction.
Figure 9: The shift favors the half of the new answers. (left) Common shift with the new answers in one half of the vocabulary or anywhere. (middle) How far it pushes the logits of A toward the half of the new answers. (right) Old facts whose correct answer is still ranked first within their own half, against their accuracy.
Figure 10: Normalization rotates the old states toward the shift. (left) Mean cosine between the old hidden states. (middle) Norm of their individual parts after normalization, relative to the start of finetuning. (right) Share of the predictions for A that fall in the half of the new answers, and their correlation with how often each answer occurs among the new facts.
Figure 11: The readout of the answers of A is barely touched. (left) Relative change of the readout weights of the answers of A and of the half of the new answers. (right) Gradient on the readout weights of the answers of A , and the probability the model puts on these answers for the new facts, both relative to the start of finetuning.
Figure 12: The shift hides A ; the drift erodes A and B . (left) Accuracy of A and B as trained and with the common part of the change of their logits removed. (right) Among the facts of A recalled wrongly, the share that names the answer of the closest new key, against that of a random new key.
Figure 13: Removing the shared part of the update restores the old facts. Accuracy on the old (left) and new (right) facts as trained and with ΔW1μμ⊤ removed from the store, the readout kept as trained. The dotted line marks the trough; ten seeds.
Figure 14: The spread of the old facts is set by the Frobenius norm of the update. (left) Measured spread εrms of the old hidden states during finetuning, compared with (1−α)/d∥ΔW1∥F and with (1−α)/d∥ΔW⊥∥F (Equation 47 ). (right) Measured against predicted; the prediction runs slightly low only while the shared column of the update is large. Ten seeds.
Figure 15: In the minimal model, removing the common shift removes the collapse. (left) Common shift and fact-specific drift of the old hidden states after normalization. (right) Old-fact accuracy as trained, with only the common part of the change of their logits, and with it removed; the dotted line marks the bottom of the collapse.
Figure 16: Removing the top direction of the update in the minimal model. Accuracy on the old (left) and new (right) facts as trained, with the top singular component of the update of W1 removed, and, for the old facts, with a random rank-one matrix of the same norm removed.
Figure 17: Part of the bias towards the new answer region is stored in the readout. Accuracy on the old and new facts during finetuning, with the readout trained and frozen; the learning rate of the frozen run is matched so that the new facts are learned at the same pace. Ten seeds.
Figure 18: Replaying a few old facts protects the others from the collapse, but not from erosion. (left) Accuracy of the old facts that are never replayed, when 0, 4, 16 or 64 of the 128 old facts are replayed in every batch (16 replayed examples per step; five seeds). (right) Gain over no replay at the trough and at the end of finetuning (3,000 steps), with 16 or 4 replayed examples per step.
Figure 19: The collapse leaves the old facts’ ranking within their region intact. Accuracy of the old facts of A over all answers, and fraction of them whose correct answer still ranks first among the answers of its own half, for the transformer of Figure 1 (right; three seeds). The dotted line marks the trough.
Figure 20: The end of the collapse follows the learning of the new facts. Old-fact (left) and new-fact (right) accuracy under five finetuning conditions; circles and dotted lines mark each condition’s trough. The troughs span steps 34 to 100, yet the new facts are at 36–44% accuracy at every one of them. Three seeds, evaluated every 2 steps.
Figure 21: With fewer new facts, the collapse ends later in their learning. Old-fact (left) and new-fact (right) accuracy of the Transformer finetuned on 125, 250 or 500 new individuals; circles and dotted lines mark the bottom of each collapse. Three seeds.
Figure 22: In the minimal model, the shift peaks when the individual margin turns positive. (left) Common shift over finetuning with 32, 128 and 256 new facts, scaled to its peak; circles mark where γˉind turns positive. (middle) The step of the peak against that step, for different numbers of new facts (filled) and for different learning rates and batch sizes (open). (right) Accuracy on the new facts at that step: it depends on their number, but not on the learning rate or the batch size. Ten seeds.
Weights restored
Share of the common shift
Attention value and output
84%
Attention query and key
11%
Input embedding
2%
LayerNorms
1%
Unembedding (control)
0%
Appendix
Table 2: Share of the common shift removed by restoring each group of weights to its value before finetuning, at the trough (step 50; three seeds, standard deviation below 0.3 percentage points).
Figure 23: The Transformer with MLP blocks reproduces Figure 1 . Accuracy of the old and new facts when finetuning on C (left) and on B′ (right).
Figure 24: With MLP blocks, removing the common shift still removes the collapse. (left) Common shift and fact-specific drift of the old hidden states. (middle) Old-fact accuracy as trained, with only the common part of the change of their logits, and with it removed. (right) Old facts whose correct answer still ranks first within their own half, against their accuracy. Three seeds; the dotted line marks the bottom of the collapse.
Figure 25: What turns an old fact wrong. For the old facts recalled wrongly, the margin of the correct answer over its strongest competitor, split into its pretrained value and the contributions of the common and fact-specific parts of the change of the logits.
Figure 26: Timing in the Transformer with MLP blocks. Old-fact (left) and new-fact (right) accuracy under five learning rates and batch sizes; circles and dotted lines mark the bottom of each collapse. Three seeds evaluated at every 2 steps
Old facts
New facts
New facts
Step
trained
top removed
random removed
trained
top removed
Synthetic individuals
80
0.28
0.77
0.28
0.03
0.01
400
0.38
0.70
0.38
0.90
0.40
Real entities
80
0.73
0.88
0.73
0.46
0.25
400
0.55
0.79
0.55
1.00
0.66
Appendix
Table 3: Accuracy of OLMo 2 1B at the trough (step 80) and after 400 steps of finetuning, with the top or a random direction of each weight update removed. Before finetuning, the old facts are recalled with accuracy 0.87 . Small finetuning set for the synthetic individuals; three seeds.
Figure 27: Removing the top direction of the update in the controlled Transformers. (left, middle) Accuracy on the old (solid) and new (dotted) facts as trained, with the top singular component of every block-weight update removed, and with a random rank-one matrix of the same norm removed. (right) Cosine between the change the edit makes to the readout states of the old facts and their common shift. Three seeds.
Models trained on a new task typically degrade on prior tasks, a phenomenon known as forgetting. Traditionally, mitigating forgetting has required replaying stored exemplars from prior tasks, which is often impractical. By contrast, language models can sample from their own training distribution, and we show that these self-generated samples serve as effective replay data, nearly eliminating forgetting. We find that forgetting nonetheless persists when the model has little remaining capacity: models pretrained close to saturation cannot absorb new information without overwriting prior knowledge. When capacity is not the limiting factor, low learning rates reduce forgetting but require substantially more training steps. Replay breaks this tradeoff, enabling fast, high-learning-rate finetuning without forgetting.
As new knowledge rapidly accumulates, language models (LMs) with pretrained knowledge quickly become obsolete. A common approach to updating LMs is fine-tuning them directly on new knowledge. However, recent studies have shown that fine-tuning for memorization may be ineffective in storing knowledge or may exacerbate hallucinations. In this work, we introduce a setting we call continual memorization, where a model must memorize and retain a set of factoids through multiple stages of fine-tuning on subsequent datasets. We characterized the forgetting patterns through extensive experiments and show that LMs widely suffer from forgetting, especially when needing to memorize factoids in the second stage. We posit that forgetting can be alleviated by modifying training dynamics: (1) protecting the memorization process when learning factoids or (2) reducing interference from subsequent training stages. Intriguingly, we find that mixing randomly generated word sequences or generic data sampled from pretraining corpora at different training stages effectively mitigates forgetting REMIX: Random and Generic Data Mixing). REMIX can recover performance from severe forgetting, outperforming replay methods and other continual learning baselines. We analyze how REMIX influences the learning process and find that robust memorization follows a distinct pattern: the model stores factoids in earlier layers than usual and diversifies the layers that retain them, which results in easier recall and manipulate of the learned factoids.
Howard Chen, Jiayi Geng, Adithya Bhaskar +2
Princeton Language and Intelligence (PLI), Princeton University
Updating a language model's knowledge through fine-tuning is essential for keeping its outputs current, yet can also induce factual forgetting and new hallucinations. Prior work shows that long-tail knowledge is harder to acquire and newly memorized long-tail facts are difficult to retain during later fine-tuning. We study a complementary question: among facts that a model has encoded correctly, which are most vulnerable to collateral corruption during other updates? To investigate this question under a realistic factual distribution, we construct a large-scale graph FACTPROP of verified Wikipedia facts by linking triples that share head or tail entities, thereby preserving connections among factual knowledge. We fine-tune models on factual statements and measure correct-to-incorrect facts after each update. Our results reveal a pattern distinct from prior findings on long-tail vulnerability during acquisition and retention: among facts that models already answer correctly, those associated with highly connected entities are more likely to be corrupted by neighboring updates, and updates to such facts propagate errors more broadly. Structural popularity therefore predicts both vulnerability and downstream damage. Inspired by this finding, we propose Popularity-based Anchoring (PopAnchor), a lightweight rehearsal strategy that preserves a small set of popular facts and reduces forgetting.
Yuji Zhang, Weibing Wang, Cheng Qian +5
University of Illinois Urbana-Champaign · City University of New York · Massachusetts Institute of Technology +1