cs.CLOct 6, 2026
SaveWhen Forgetting is not Catastrophic: On the Mechanics of Spurious Forgetting
Organizations: MPI for Intelligent Systems, Tübingen · Jinesis Lab, University of Toronto & Vector Institute · EuroSafeAI · Hector Foundation · Stanford University · ELLIS Institute Tübingen
Abstract
Knowledge that a language model appears to forget during finetuning often remains stored and can be recovered, a phenomenon called spurious forgetting. Finetuning on new facts can even produce forgetting that undoes itself: recall of the old facts collapses, recovers as training continues on new facts alone, and only then erodes for good. We seek to understand when such forgetting is not catastrophic. A minimal associative memory reproduces these dynamics with three ingredients: keys with shared structure, concentrated new values, and normalization in the network. Finetuning moves all old representations along a common direction, hiding the old facts while preserving their relative geometry; normalization withdraws this shift once the new facts are learned, whereas fact-specific changes accumulate and cause the erosion. Moreover, subtracting the common shift eliminates the collapse in a Transformer trained on synthetic data, and removing a single direction from each weight update restores old facts in a pretrained language model. Forgetting thus combines a shared, reversible loss of access with a slow erosion of individual facts, and only the second is catastrophic. Which one dominates depends on whether the new data move old memories together or apart.
Figures & tables
Figure 1: Changing only the finetuning data turns catastrophic forgetting into spurious forgetting. A transformer is pretrained on two non-overlapping sets of facts, and , and then finetuned on new facts only, without any replay. When the new facts overlap with both pretraining sets (left, finetuning on ), performance on both declines slowly and steadily. When they overlap only with (right, fine-tuning on ), performance on is mostly intact, but performance on collapses, recovers although is never seen again, and later erodes. How the different sets are built and what it means for the new facts to overlap with the old ones is detailed in Section 4 .
Figure 2: Shared keys, concentrated new values and normalization are all needed for the collapse, recovery and erosion. The model maps keys, which share a component of strength , to values split into two regions, through one hidden layer with RMS normalization. As in Figure 1 , it is pretrained on and , whose values lie in different regions, then finetuned only on new facts , whose values lie in the region of ; curves show recall of . Reducing weakens the collapse and removes the recovery (1), drawing the new values from both regions leaves only gradual forgetting (2), and replacing the normalization removes the recovery (3). The model is described in Section 3 .
Figure 3: Normalization reverses the shift responsible for forgetting. The old representations first move together along a shared direction. Once the new facts are learned and the individual margin becomes positive, the common shift retracts and the old facts are recalled again. Without a normalization layer, this reversal does not occur, which is why the collapse is permanent (Figure 2 , panel 3). (left) Old hidden states before normalization, projected on the shared direction and an orthogonal one, after pretraining, at the end of the collapse and during erosion. (middle) Common shift magnitude of the old hidden states, with and without normalization. (right) Shared and individual margins of the new facts, which determine the dynamics of through Equation 5 . Same setup as in Figure 2 ; the dotted line marks the end of the collapse.
Figure 4: The Transformer initially forgets through the same common shift as the minimal model. (left) The Transformer is trained on synthetic biographies of fictitious individuals, each stating several facts about them, such as where they come from. The possible answers of each attribute are split into two subsets: the individuals of draw their answers from the first, and those of from the second. The model is pretrained on and , then finetuned on new individuals whose answers come from the same subset as ; this is the setup of Figure 1 (right). (middle) The common shift of the hidden states of grows fast and retracts at the end of the collapse, as in the minimal model (Figure 3 ). The fact-specific displacements grow more slowly and eventually become larger. (right) Accuracy on , and when only the common part of the change of its logits is kept or removed. The dotted line indicates the end of the collapse.
Figure 5: The collapse persists in a pretrained language model, but training reverses it only for arbitrary new facts. OLMo 2 1B is finetuned without replay, in separate runs, on synthetic individuals or on real entities. Old facts (CounterFact facts the model knows) are solid, new facts dotted; three seeds. (left) Recall of the old facts drops for both kinds of new facts, but recovers only for synthetic individuals. (middle) Removing the top singular direction of each weight update (light) restores recall of the old facts for both kinds, whereas removing a random direction of the same norm leaves it where training did (dark). (right) With a larger set of synthetic individuals, the collapse goes deeper and lasts longer.
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
| Minimal | Transformer | OLMo 2 | ||
|---|---|---|---|---|
| Analysis | model | attention-only | with MLP blocks | 1B |
| Collapse, recovery and erosion | Fig. 2 | Fig. 1 | App. C.5 | Fig. 5 |
| New answers in both regions | Fig. 2 | Fig. 1 | App. C.5 | – |
| Common shift and fact-specific drift | App. B.10 | Fig. 4 | App. C.5 | – |
| Removing the common shift | App. B.9 | Fig. 4 | App. C.5 | – |
| Ordering within the region | App. B.8 | App. C.1 | App. C.5 | – |
Table 1: Reference location for each analysis, for each model.
Figure 6: Examples from the three datasets. Each fact is learned from training phrasings and evaluated on held-out phrasings; answers in bold. The old facts of the language model are facts it already knows, and the new facts concern synthetic individuals, whose answers are unrelated to their names, or little-known real entities.
Figure 7: Equation 38 reproduces the dynamics of the common shift. (left) Length of the old facts’ mean hidden state, measured and obtained by integrating Equation 38 (with normalization) or (without normalization) from the point at which the mean state has aligned with the shared direction, after which its length follows the same dynamics as its displacement . (right) Average shared margin and individual margin of the new facts, with normalization. The dotted line marks the peak; ten seeds.
Figure 8: How the shift is written. (left) Cosine between the update of each old hidden state and their mean update, and relative error of Equation 4 . (middle) Speed of the update at the start of finetuning against . (right) Displacement of the old and new hidden states along the shared direction.
Figure 9: The shift favors the half of the new answers. (left) Common shift with the new answers in one half of the vocabulary or anywhere. (middle) How far it pushes the logits of toward the half of the new answers. (right) Old facts whose correct answer is still ranked first within their own half, against their accuracy.
Figure 10: Normalization rotates the old states toward the shift. (left) Mean cosine between the old hidden states. (middle) Norm of their individual parts after normalization, relative to the start of finetuning. (right) Share of the predictions for that fall in the half of the new answers, and their correlation with how often each answer occurs among the new facts.
Figure 11: The readout of the answers of is barely touched. (left) Relative change of the readout weights of the answers of and of the half of the new answers. (right) Gradient on the readout weights of the answers of , and the probability the model puts on these answers for the new facts, both relative to the start of finetuning.
Figure 12: The shift hides ; the drift erodes and . (left) Accuracy of and as trained and with the common part of the change of their logits removed. (right) Among the facts of recalled wrongly, the share that names the answer of the closest new key, against that of a random new key.
Figure 13: Removing the shared part of the update restores the old facts. Accuracy on the old (left) and new (right) facts as trained and with removed from the store, the readout kept as trained. The dotted line marks the trough; ten seeds.
Figure 14: The spread of the old facts is set by the Frobenius norm of the update. (left) Measured spread of the old hidden states during finetuning, compared with and with (Equation 47 ). (right) Measured against predicted; the prediction runs slightly low only while the shared column of the update is large. Ten seeds.
Figure 15: In the minimal model, removing the common shift removes the collapse. (left) Common shift and fact-specific drift of the old hidden states after normalization. (right) Old-fact accuracy as trained, with only the common part of the change of their logits, and with it removed; the dotted line marks the bottom of the collapse.
Figure 16: Removing the top direction of the update in the minimal model. Accuracy on the old (left) and new (right) facts as trained, with the top singular component of the update of removed, and, for the old facts, with a random rank-one matrix of the same norm removed.
Figure 17: Part of the bias towards the new answer region is stored in the readout. Accuracy on the old and new facts during finetuning, with the readout trained and frozen; the learning rate of the frozen run is matched so that the new facts are learned at the same pace. Ten seeds.
Figure 18: Replaying a few old facts protects the others from the collapse, but not from erosion. (left) Accuracy of the old facts that are never replayed, when 0, 4, 16 or 64 of the 128 old facts are replayed in every batch (16 replayed examples per step; five seeds). (right) Gain over no replay at the trough and at the end of finetuning (3,000 steps), with 16 or 4 replayed examples per step.
Figure 19: The collapse leaves the old facts’ ranking within their region intact. Accuracy of the old facts of over all answers, and fraction of them whose correct answer still ranks first among the answers of its own half, for the transformer of Figure 1 (right; three seeds). The dotted line marks the trough.
Figure 20: The end of the collapse follows the learning of the new facts. Old-fact (left) and new-fact (right) accuracy under five finetuning conditions; circles and dotted lines mark each condition’s trough. The troughs span steps 34 to 100, yet the new facts are at 36–44% accuracy at every one of them. Three seeds, evaluated every 2 steps.
Figure 21: With fewer new facts, the collapse ends later in their learning. Old-fact (left) and new-fact (right) accuracy of the Transformer finetuned on 125, 250 or 500 new individuals; circles and dotted lines mark the bottom of each collapse. Three seeds.
Figure 22: In the minimal model, the shift peaks when the individual margin turns positive. (left) Common shift over finetuning with 32, 128 and 256 new facts, scaled to its peak; circles mark where turns positive. (middle) The step of the peak against that step, for different numbers of new facts (filled) and for different learning rates and batch sizes (open). (right) Accuracy on the new facts at that step: it depends on their number, but not on the learning rate or the batch size. Ten seeds.
| Weights restored | Share of the common shift |
|---|---|
| Attention value and output | |
| Attention query and key | |
| Input embedding | |
| LayerNorms | |
| Unembedding (control) |
Table 2: Share of the common shift removed by restoring each group of weights to its value before finetuning, at the trough (step 50; three seeds, standard deviation below 0.3 percentage points).
Figure 23: The Transformer with MLP blocks reproduces Figure 1 . Accuracy of the old and new facts when finetuning on (left) and on (right).
Figure 24: With MLP blocks, removing the common shift still removes the collapse. (left) Common shift and fact-specific drift of the old hidden states. (middle) Old-fact accuracy as trained, with only the common part of the change of their logits, and with it removed. (right) Old facts whose correct answer still ranks first within their own half, against their accuracy. Three seeds; the dotted line marks the bottom of the collapse.
Figure 25: What turns an old fact wrong. For the old facts recalled wrongly, the margin of the correct answer over its strongest competitor, split into its pretrained value and the contributions of the common and fact-specific parts of the change of the logits.
Figure 26: Timing in the Transformer with MLP blocks. Old-fact (left) and new-fact (right) accuracy under five learning rates and batch sizes; circles and dotted lines mark the bottom of each collapse. Three seeds evaluated at every 2 steps
| Old facts | New facts | |||||
|---|---|---|---|---|---|---|
| New facts | Step | trained | top removed | random removed | trained | top removed |
| Synthetic individuals | 80 | 0.28 | 0.77 | 0.28 | 0.03 | 0.01 |
| 400 | 0.38 | 0.70 | 0.38 | 0.90 | 0.40 | |
| Real entities | 80 | 0.73 | 0.88 | 0.73 | 0.46 | 0.25 |
| 400 | 0.55 | 0.79 | 0.55 | 1.00 | 0.66 | |
Table 3: Accuracy of OLMo 2 1B at the trough (step 80) and after 400 steps of finetuning, with the top or a random direction of each weight update removed. Before finetuning, the old facts are recalled with accuracy . Small finetuning set for the synthetic individuals; three seeds.
Figure 27: Removing the top direction of the update in the controlled Transformers. (left, middle) Accuracy on the old (solid) and new (dotted) facts as trained, with the top singular component of every block-weight update removed, and with a random rank-one matrix of the same norm removed. (right) Cosine between the change the edit makes to the readout states of the old facts and their common shift. Three seeds.