cs.LGSep 28, 2026

Don't Forget! Decomposing the Training Dynamics of Memorization in Language Models

Authors: Florian Eichin, Philipp Mondorf, Andrei Mircea, Yupei Du, Barbara Plank, Michael A. Hedderich

Organizations: MaiNLP, Center for Information and Language Processing, LMU Munich & Munich Center for Machine Learning (MCML), Germany · Mila – Quebec AI Institute & University of Montreal, Canada · Saarland University, Germany

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Memorization has been proposed as a mechanism to explain how language models fit the tail of their training distributions, but its training dynamics are not understood well. In this work, we take a fine-grained look at memorization by decomposing the loss trajectory of memorized sequences over training and model parameters. Across the Pythia family, we study memorization of duplicated training sequences (recitation) and rare ones (recollection). We find that memorization in both cases is characterized by sequence-level gradient alignment, though recitation suffers from misalignment with other training influences which causes forgetting, explaining the necessity for higher duplication of these examples. We further show that the lower model layers are the most involved in memorization and forgetting. Predicting memorization, our decomposition improves over a cross-entropy baseline, especially in larger models and early in training. Intervening on a small set of highly influential parameters we are able to ablate memorization in the final model. Together, these findings advance our understanding of how memorization develops during training and offer insights for predicting and intervening on it.

Figures & tables

Figure 1: Pythia-1B training dynamics. We plot the loss and extractability of our sampled sequences over time and find that a lot of memorization happens early in training. Other models in App. Fig. 9 .
Figure 2: Pythia-1B update magnitude, test sensitivity, and alignment. We plot the mean and standard deviation over the samples. See Eq. 5 for definitions. Other models in App. Fig. 14 and 12
Figure 3: Memorization classification. We plot the mean accuracy on held-out set and mean weights of the decomposition features of the decomposition and CE+decomposition. Others in App. Fig. 19 .
Figure 4: Pythia-1B update magnitude and test sensitivity by layer depth. Others in App. Fig. 14 .
Figure 5: Pythia-1B alignment by layer depth and type. Other models, control in App. Fig. 15 , 16 .
Figure 6: Pythia-1B train/weight decay alignment by model region. Other models in App. Fig. 17
Figure 7: Pythia-1B memorization ablation. Left: Number of 32-extractable sequences in the memorized sample, right: CE loss on the control sample. Other models in App. Fig. 20 .
Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Alignment with other injected instances. Other models in App. Fig. 18
Figure 9: Pythia training dynamics across scales. We plot the loss over our sampled sequences and 32-extractability over time.
Figure 10: Pythia BLiMP accuracy across scales.
Figure 11: Pythia update magnitudes and test sensitivities across scales. We plot each at early and later pretraining.
Figure 12: Pythia self alignment across scales. We plot early and later training stages.
Figure 13: Pythia weight decay alignment across scales. We plot the alignment of the instances we decompose with the parameter update introduced by weight decay, early and late in training.
Figure 14: Pythia update magnitudes and test sensitivity by model region across scales.
Figure 15: Pythia self alignment by model region across scales.
Figure 16: Pythia self alignment by layer type across scales.
Figure 17: Pythia train and regularization alignment by model region across scales.
Figure 18: Alignment with other injected instances across scales.
Figure 19: Memorization vs. not memorization classification across model scales. We also plot the weights of the decomposition features for the decomposition only and CE+decomposition model.
Figure 20: Pythia memorization ablations across scales. Left: Number of 32-extractable sequences in the memorized sample, right: CE loss on the control sample.
Figure 21: Pythia memorization ablations affected parameters, embeddings included. Ablation at step 10,00010,000 with 0.000040.00004 of all model parameters ablated.
Figure 22: Pythia memorization ablations affected parameters, embeddings excluded. Ablation at step 10,00010,000 with 0.000040.00004 of all model parameters ablated.
Figure 23: Attention head ablation in Pythia (1/2). We ablate each attention head, one by one, and plot number of 32-extractable memorized sequences and CE on control examples in intervened model.
Figure 24: Attention head ablation in Pythia (2/2). We ablate each attention head, one by one, and plot number of 32-extractable memorized sequences and CE on control examples in intervened model.

Explore similar work

CardsList

  1. Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replay

    May 25, 2026Martin Marek, Dongkyu Cho, Shikai Qiu +3Catastrophic ForgettingForgetting

    New York University

  2. Mitigating Memorization In Language Models

    Oct 3, 2024Mansi Sakarvadia, Aswathy Ajith, Arham Khan +6MemorizationLarge Language Model Unlearning

    University of Chicago · Argonne National Laboratory · Lawrence Berkeley National Laboratory +3

  3. Continual Memorization of Factoids in Language Models

    Nov 11, 2024Howard Chen, Jiayi Geng, Adithya Bhaskar +2MemorizationModel Fine-Tuning

    Princeton Language and Intelligence (PLI), Princeton University