Understanding the Interplay between Memorization and Learning in Large Language Models

Bishwamittra Ghosh; Soumi Das; Qinyuan Wu; Mohammad Aflah Khan; Deepak Garg; Krishna P. Gummadi; Evimaria Terzi

cs.LGJul 20, 2025

Understanding the Interplay between Memorization and Learning in Large Language Models

Authors: Bishwamittra Ghosh, Soumi Das, Qinyuan Wu, Mohammad Aflah Khan, Deepak Garg, Krishna P. Gummadi, Evimaria Terzi

Organizations: Max Planck Institute for Software Systems, Germany · Boston University, USA

Abstract

We investigate foundational questions about the interplay between memorization and learning in large language models (LLMs). When an LLM generates a string, can we disentangle the roles of rote memorization and contextual learning? Rote memorization results in an LLM regurgitating specific details of a training string, while contextual learning leads to the generation of new strings that follow generalizable patterns of the underlying language. Related questions of interest include: can an LLM avoid memorization when optimally learning a language?, and if not, can we design training schemes to reduce memorization and improve learning? To address these questions, we propose contextual memorization, a new measure that precisely disentangles memorization from learning. Using this measure, we establish that different training strings are memorized at different epochs in training, and that memorization of some training strings is unavoidable when optimally learning a language. Finally, we propose memorization-aware training, a scheme that dynamically down-samples training strings based on their memorization, and in doing so, equalizes learning across strings, reduces memorization, and improves learning. Our conclusions are supported by extensive experiments on multiple LLMs using formal languages as a controlled testbed, enabling precise language specification, exact string sampling, and elimination of data contamination; and further validated on natural languages.

Figures & tables

Figure 1: Contextual Memorization. The solid curve is the training loss and the dotted curve is the contextual loss of a string

s

, when it is inside and outside the training dataset

D\sim L

, respectively (details in Section 4.2 ). Contextual memorization begins when training loss drops below the lowest contextual loss, i.e., the LLM generates

s

better than it would from learning the underlying language.

Figure 2: A probabilistic and hierarchical context-free grammar (top), and a sampled string generated by the grammar (bottom), adapted from Ghosh et al. [30] . The grammar produces a skewed language, where some rules (patterns) are applied more frequently than others.

Panel of Figure 3 — Figure 3: Disagreement Among Memorization Measures. The beginning of memorization (vertical dotted lines) for strings $s_{0},s_{1},s_{2}$ of decreasing frequency. Recollection-based : training loss (solid) drops below threshold $\tau=0.2$ (Figure 3a ). Counterfactual : training loss deviates from counterfactual test loss (dotted curves) (Figure 3b ). Contextual : training loss drops below the string-specific optimal contextual loss (Figure 3c ). The optimal contextual loss of the mid-frequent string $s_{1}$ is usually close to the lowest average test loss over $L$ .

Panel of Figure 4 — Figure 4: Memorization of training strings across languages of different entropy. The vertical dotted line marks $e^{*}$ , the epoch of optimal learning (see Figure 33 ). At $e^{*}$ , memorization is nonzero.

Panel of Figure 5 — Figure 5: Contradiction among measures on memorization of the top $10\%$ most frequent and bottom $10\%$ least frequent strings in low-entropy language $L_{2}$ (extended results in Figure 34 ).

Panel of Figure 7 — Figure 7: Base training dynamics on language $L_{2}$ using Mistral-7B, by frequency group. Optimal epoch and mean test loss are marked using vertical and horizontal lines, respectively (Figure 7b ).

Panel of Figure 8 — Figure 8: Training dynamics under MAT in the same setting as in Figure 7 . We additionally show the fraction of training strings removed per frequency group in MAT (leftmost).

Panel of Figure 9 — Figure 9: MAT improves learning by reducing the variance of test loss (often with lower mean loss as well), while also reducing contextual memorization and training cost.

Appendix figures & tables62 assets

Supplementary material from the paper’s appendix.

Appendix

Memorization Measure	Motivation	Memorization Threshold	Ease of Operationalization	Strictness of Measure
Recollection	Disclosing private information	Manual	Easy	Variable
Counterfactual	Differential privacy	Adaptive	Hard	Medium
Contextual (ours)	Local over-fitting	Adaptive	Hard	High

Appendix

Table 1: Characteristics of memorization measures.

Panel of Figure 10 — Figure 10: Contextual memorization is stricter than counterfactual memorization. Top row : the contextual threshold ( red dash-dot line) lies below the counterfactual threshold ( blue dash-dot line), since contextual learning keeps improving the recollection of $s$ even after the epoch at which counterfactual memorization already permits a membership-inference attack. Contextual memorization of $s$ therefore begins at the same epoch or later, with its score lower-bounded by the counterfactual score (length of the horizontal gray arrow). Bottom row : the same relationship holds when measured by accuracy instead of loss; since higher accuracy indicates better recollection, the contextual threshold ( red ) now lies above the counterfactual threshold ( blue ), and contextual memorization again begins at the same epoch or later.

		$\displaystyle{\color[rgb]{1,0,0}S}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A16}\;{\color[rgb]{0,0,1}[1]}$
		$\displaystyle{\color[rgb]{1,0,0}A16}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A15}\;{\color[rgb]{1,0,0}A14}\;{\color[rgb]{1,0,0}A13}\;{\color[rgb]{0,0,1}[0.50]}$
		$\displaystyle{\color[rgb]{1,0,0}A16}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A13}\;{\color[rgb]{1,0,0}A15}\;{\color[rgb]{1,0,0}A14}\;{\color[rgb]{0,0,1}[0.50]}$
		$\displaystyle{\color[rgb]{1,0,0}A13}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A11}\;{\color[rgb]{1,0,0}A12}\;{\color[rgb]{0,0,1}[0.50]}$
		$\displaystyle{\color[rgb]{1,0,0}A13}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A12}\;{\color[rgb]{1,0,0}A11}\;{\color[rgb]{0,0,1}[0.50]}$
		$\displaystyle{\color[rgb]{1,0,0}A14}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A11}\;{\color[rgb]{1,0,0}A10}\;{\color[rgb]{1,0,0}A12}\;{\color[rgb]{0,0,1}[0.50]}$
		$\displaystyle{\color[rgb]{1,0,0}A14}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A10}\;{\color[rgb]{1,0,0}A11}\;{\color[rgb]{1,0,0}A12}\;{\color[rgb]{0,0,1}[0.50]}$

Appendix

Figure 11: Production rules of

{G}_{1}

(left) and

{G}_{2}

(right). Compared to

G_{1}

, the grammar

G_{2}

generates more skewed distribution (or lower entropy) strings, since one out of two production rules for each non-terminal is selected with higher probability.

		$\displaystyle{\color[rgb]{1,0,0}S}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}S5}\;{\color[rgb]{0,0,1}[1]}$
		$\displaystyle{\color[rgb]{1,0,0}S5}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}B4}\;{\color[rgb]{1,0,0}C1_{1}}\;{\color[rgb]{1,0,0}E4}\;{\color[rgb]{1,0,0}T1_{1}}\;{\color[rgb]{0,0,1}[0.25]}$
		$\displaystyle{\color[rgb]{1,0,0}S5}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}B4}\;{\color[rgb]{1,0,0}C1_{2}}\;{\color[rgb]{1,0,0}E4}\;{\color[rgb]{1,0,0}T1_{2}}\;{\color[rgb]{0,0,1}[0.25]}$
		$\displaystyle{\color[rgb]{1,0,0}S5}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}B4}\;{\color[rgb]{1,0,0}C1_{3}}\;{\color[rgb]{1,0,0}E4}\;{\color[rgb]{1,0,0}T1_{3}}\;{\color[rgb]{0,0,1}[0.25]}$
		$\displaystyle{\color[rgb]{1,0,0}S5}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}B4}\;{\color[rgb]{1,0,0}C1_{4}}\;{\color[rgb]{1,0,0}E4}\;{\color[rgb]{1,0,0}T1_{4}}\;{\color[rgb]{0,0,1}[0.25]}$
		$\displaystyle{\color[rgb]{1,0,0}B4}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}B3}\;{\color[rgb]{0,0,1}[0.3333]}$
		$\displaystyle{\color[rgb]{1,0,0}B4}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}B3}\;{\color[rgb]{1,0,0}B3}\;{\color[rgb]{1,0,0}B3}\;{\color[rgb]{0,0,1}[0.3333]}$

Appendix

Figure 12: Production rules of

{G}_{3}

(left) and

{G}_{4}

(right). Compared to

G_{3}

, the grammar

G_{4}

generates more skewed distribution (or lower entropy) of strings, since one out of three production rules of non-terminal

{\color[rgb]{1,0,0}B1}

and

{\color[rgb]{1,0,0}E1}

is selected with higher probability.

		$\displaystyle{\color[rgb]{1,0,0}S}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A16}\;{\color[rgb]{0,0,1}[1]}$
		$\displaystyle{\color[rgb]{1,0,0}A16}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A15}\;{\color[rgb]{1,0,0}A13}\;{\color[rgb]{0,0,1}[0.50]}$
		$\displaystyle{\color[rgb]{1,0,0}A16}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A13}\;{\color[rgb]{1,0,0}A15}\;{\color[rgb]{1,0,0}A14}\;{\color[rgb]{0,0,1}[0.50]}$
		$\displaystyle{\color[rgb]{1,0,0}A13}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A11}\;{\color[rgb]{1,0,0}A12}\;{\color[rgb]{0,0,1}[0.50]}$
		$\displaystyle{\color[rgb]{1,0,0}A13}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A12}\;{\color[rgb]{1,0,0}A11}\;{\color[rgb]{0,0,1}[0.50]}$
		$\displaystyle{\color[rgb]{1,0,0}A14}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A11}\;{\color[rgb]{1,0,0}A10}\;{\color[rgb]{1,0,0}A12}\;{\color[rgb]{0,0,1}[0.50]}$
		$\displaystyle{\color[rgb]{1,0,0}A14}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A10}\;{\color[rgb]{1,0,0}A11}\;{\color[rgb]{1,0,0}A12}\;{\color[rgb]{0,0,1}[0.50]}$

Appendix

Figure 13: Production rules of

{G}_{5}

(left) and

{G}_{6}

(right). These grammars are adapted from

{G}_{1}

and

{G}_{3}

respectively, by allowing non-uniform lengths of tokens in the lowest level production rules.

		$\displaystyle{\color[rgb]{1,0,0}S}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A16}\;{\color[rgb]{0,0,1}[1]}$
		$\displaystyle{\color[rgb]{1,0,0}A16}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A15}\;{\color[rgb]{1,0,0}A13}\;{\color[rgb]{0,0,1}[0.50]}$
		$\displaystyle{\color[rgb]{1,0,0}A16}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A13}\;{\color[rgb]{1,0,0}A15}\;{\color[rgb]{1,0,0}A14}\;{\color[rgb]{0,0,1}[0.50]}$
		$\displaystyle{\color[rgb]{1,0,0}A13}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A11}\;{\color[rgb]{1,0,0}A12}\;{\color[rgb]{0,0,1}[0.50]}$
		$\displaystyle{\color[rgb]{1,0,0}A13}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A12}\;{\color[rgb]{1,0,0}A11}\;{\color[rgb]{0,0,1}[0.50]}$
		$\displaystyle{\color[rgb]{1,0,0}A14}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A11}\;{\color[rgb]{1,0,0}A10}\;{\color[rgb]{1,0,0}A12}\;{\color[rgb]{0,0,1}[0.50]}$
		$\displaystyle{\color[rgb]{1,0,0}A14}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A10}\;{\color[rgb]{1,0,0}A11}\;{\color[rgb]{1,0,0}A12}\;{\color[rgb]{0,0,1}[0.50]}$

Appendix

Figure 14: Production rules of

{G}_{7}

(left) and

{G}_{8}

(right). These grammars are adapted from

{G}_{5}

and

{G}_{6}

respectively, by replacing numerical tokens with Latin character tokens.

		$\displaystyle{\color[rgb]{1,0,0}S}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A13}\;{\color[rgb]{0,0,1}[1]}$
		$\displaystyle{\color[rgb]{1,0,0}A13}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A10}\;{\color[rgb]{1,0,0}A12}\;{\color[rgb]{0,0,1}[0.8413447461]}$
		$\displaystyle{\color[rgb]{1,0,0}A13}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A9}\;{\color[rgb]{1,0,0}A10}\;{\color[rgb]{0,0,1}[0.1586552539]}$
		$\displaystyle{\color[rgb]{1,0,0}A12}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A5}\;{\color[rgb]{1,0,0}A7}\;{\color[rgb]{1,0,0}A5}\;{\color[rgb]{1,0,0}A8}\;{\color[rgb]{0,0,1}[0.8413447461]}$
		$\displaystyle{\color[rgb]{1,0,0}A12}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A6}\;{\color[rgb]{1,0,0}A6}\;{\color[rgb]{0,0,1}[0.1586552539]}$
		$\displaystyle{\color[rgb]{1,0,0}A11}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A5}\;{\color[rgb]{1,0,0}A7}\;{\color[rgb]{1,0,0}A8}\;{\color[rgb]{0,0,1}[0.8413447461]}$
		$\displaystyle{\color[rgb]{1,0,0}A11}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A6}\;{\color[rgb]{0,0,1}[0.1586552539]}$

Appendix

Figure 15: Production rules of

{G}_{9}

. The hierarchical grammar has a maximum depth

4

, maximum breadth

4

, rules per non-terminals

2

, and

5

numerical tokens.

		$\displaystyle{\color[rgb]{1,0,0}S}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A19}\;{\color[rgb]{0,0,1}[1]}$
		$\displaystyle{\color[rgb]{1,0,0}A19}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A12}\;{\color[rgb]{1,0,0}A18}\;{\color[rgb]{1,0,0}A14}\;{\color[rgb]{1,0,0}A17}\;{\color[rgb]{1,0,0}A11}\;{\color[rgb]{1,0,0}A17}\;{\color[rgb]{1,0,0}A15}\;{\color[rgb]{1,0,0}A10}\;{\color[rgb]{0,0,1}[0.16335]}$
		$\displaystyle{\color[rgb]{1,0,0}A19}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A15}\;{\color[rgb]{1,0,0}A13}\;{\color[rgb]{0,0,1}[0.2032909216]}$
		$\displaystyle{\color[rgb]{1,0,0}A19}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A10}\;{\color[rgb]{1,0,0}A12}\;{\color[rgb]{0,0,1}[0.2376710982]}$
		$\displaystyle{\color[rgb]{1,0,0}A19}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A18}\;{\color[rgb]{0,0,1}[0.2594097969]}$
		$\displaystyle{\color[rgb]{1,0,0}A19}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A16}\;{\color[rgb]{1,0,0}A15}\;{\color[rgb]{1,0,0}A15}\;{\color[rgb]{0,0,1}[0.1346583572]}$
		$\displaystyle{\color[rgb]{1,0,0}A19}\;{\color[rgb]{0,0,0}\rightarrow}\;{\color[rgb]{1,0,0}A11}\;{\color[rgb]{1,0,0}A18}\;{\color[rgb]{1,0,0}A13}\;{\color[rgb]{1,0,0}A10}\;{\color[rgb]{0,0,1}[0.0016209134]}$

Appendix

Figure 16: For simulating pre-training, we consider a large skewed grammar

G_{10}

with maximum depth

3

, maximum breadth

10

, rules per non-terminals

9

, and

20

numerical tokens.

Panel of Figure 17 — Figure 17: Length distribution of strings sampled from multiple formal languages. The length distribution is based on $10000$ sampled strings per language.

Panel of Figure 18 — Figure 18: Representative strings from different languages, annotated with non-terminals applied in different positions by the respective hierarchical grammar.

Panel of Figure 19 — Figure 19: Representative strings from different languages, annotated with non-terminals applied in different positions by the respective hierarchical grammar.

Panel of Figure 20 — Figure 20: Validation of the Approximation of the Lowest Contextual Loss. (a) Ground truth vs. approximated lowest contextual loss (Figure 20a ): points cluster tightly around the diagonal, indicating strong agreement between the two. (b) Relative error (ratio of absolute error to ground truth) vs. ground truth (Figure 20b ): relative error is higher when the ground-truth loss is small, a numerical artifact of dividing a similar absolute error by a smaller denominator. (c) Relative error by training size, grouped by model (Figure 20c ): the approximation error decreases as training size increases, and this trend is consistent across models.

Panel of Figure 21 — Figure 21: Start of memorization of selected strings in Language $L_{2}$ .

Panel of Figure 23 — Figure 23: Start of memorization of selected strings in language $L_{4}$ (specifically, a modified version of $L_{4}$ as explained below). The observation is consistent with language $L_{2}$ , as shown in Figure 3 , where frequency of strings correlates with the beginning of recollection-based memorization. Similarly, frequency often inversely correlates with counterfactual and contextual memorization, with an exception that both $s_{1}$ and $s_{2}$ are memorized at the same epoch in the counterfactual memorization. Thus, regardless of whether correlation or inverse correlation exists strongly between string frequency and the order of memorization, a more consistent observation is that memorization measures disagree with each other when applied to the same training dynamic on identical strings. In this experiment, to better differentiate the strings $s_{0},s_{1},s_{2}$ based on frequency, we modify $L_{4}$ to be even more skewed. We apply high probability to one random production rule in each non-terminal in all levels, beyond the lowest level non-terminals in $L_{4}$ , as shown in Figure 12 .

Panel of Figure 25 — Figure 25: Disagreement Among Memorization Measures on Natural Language. We select three training strings $s_{0}$ , $s_{1}$ , $s_{2}$ from the Tiny Shakespeare dataset [ 42 ] with low, medium, and high loss under the base model (before any fine-tuning), respectively. Similar to our results in formal language, different memorization measures disagree on when a string in natural language begins to be memorized, shown using vertical lines. More importantly in Figure 25c , strings have different values of optimal contextual loss, making the case for contextual memorization.

Panel of Figure 28 — Figure 28: Chinese , drawn from FineWeb2 [ 58 ] . Same results of Figure 27 as observed in the Chinese language.

Panel of Figure 31 — Figure 31: Memorization of training strings in languages of different entropy across different memorization measures. Results are for language $L_{1}$ , which is a high entropy language.

Panel of Figure 33 — Figure 33: Continuing Figure 4 , we demonstrate associated loss with weighted memorization.

Panel of Figure 34 — Figure 34: Continuing Figure 6 , contradiction between recollection-based and contextual (or counterfactual) memorization on determining memorization of top $10\%$ frequent strings and bottom $10\%$ infrequent strings in a low entropy language. The results are for Mistral- $7$ B on language $L_{2}$ , which is a low entropy language.

Panel of Figure 35 — Figure 35: Continuing Figure 34 , contradiction between recollection-based and contextual (or counterfactual) memorization on determining memorization of top $10\%$ frequent strings and bottom $10\%$ infrequent strings in a low entropy language. The results are for Mistral- $7$ B on language $L_{4}$ , which is a low entropy language.

Panel of Figure 36 — Figure 36: Tradeoffs between optimal learning and memorization among comparable $\approx 7$ B parameter size models on language $L_{1}$ , which is a high entropy language.

Panel of Figure 37 — Figure 37: Tradeoffs between optimal learning and memorization among comparable $\approx 7$ B parameter size models on language $L_{2}$ , which is a low entropy language.

Panel of Figure 38 — Figure 38: Tradeoffs between optimal learning and memorization among comparable $\approx 7$ B parameter size models on language $L_{3}$ , which is a high entropy language.

Panel of Figure 39 — Figure 39: Tradeoffs between optimal learning and memorization among comparable $\approx 7$ B parameter size models on language $L_{4}$ , which is a low entropy language.

Panel of Figure 40 — Figure 40: Tradeoffs between optimal learning and memorization among comparable $\approx 7$ B parameter size models on language $L_{5}$ , which is a high entropy language.

Panel of Figure 41 — Figure 41: Tradeoffs between optimal learning and memorization among comparable $\approx 7$ B parameter size models on language $L_{6}$ , which is a high entropy language.

Panel of Figure 42 — Figure 42: Tradeoffs between optimal learning and memorization among comparable $\approx 7$ B parameter size models on language $L_{7}$ , which is a high entropy language.

Panel of Figure 43 — Figure 43: Tradeoffs between optimal learning and memorization among comparable $\approx 7$ B parameter size models on language $L_{8}$ , which is a high entropy language.

Panel of Figure 44 — Figure 44: Contextual memorization vs. optimal language learning, measured as test loss, across models of different sizes within a family. Results are on language $L_{1}$ , which is a high entropy language.

Panel of Figure 45 — Figure 45: Contextual memorization vs. optimal language learning, measured as test loss, across models of different sizes within a family. Results are on language $L_{2}$ , which is a low entropy language.

Panel of Figure 46 — Figure 46: Contextual memorization vs. optimal language learning, measured as test loss, across models of different sizes within a family. Results are on language $L_{3}$ , which is a high entropy language.

Panel of Figure 47 — Figure 47: Contextual memorization vs. optimal language learning, measured as test loss, across models of different sizes within a family. Results are on language $L_{4}$ , which is a low entropy language.

Panel of Figure 48 — Figure 48: Contextual memorization vs. optimal language learning, measured as test loss, across models of different sizes within a family. Results are on language $L_{5}$ , which is a high entropy language.

Panel of Figure 49 — Figure 49: Contextual memorization vs. optimal language learning, measured as test loss, across models of different sizes within a family. Results are on language $L_{6}$ , which is a high entropy language.

Panel of Figure 50 — Figure 50: Contextual memorization vs. optimal language learning, measured as test loss, across models of different sizes within a family. Results are on language $L_{7}$ , which is a high entropy language and contains Latin characters as tokens.

Panel of Figure 51 — Figure 51: Contextual memorization vs. optimal language learning, measured as test loss, across models of different sizes within a family. Results are on language $L_{8}$ , which is a high entropy language and contains Latin characters as tokens.

Panel of Figure 52 — Figure 52: Training dynamics under Base Training, Dedup, MAT, and MAT* on language $L_{2}$ using Mistral- $7$ B (seed $0$ ). Each row corresponds to a training scheme; columns show train loss, test loss, and contextual memorization by frequency group across epochs. For MAT and MAT*, the fraction of training strings removed per group is shown leftmost. Vertical lines mark the optimal learning epoch. In the case of deduplication, all the training strings have the same frequency, hereby grouped as high-frequency.

Panel of Figure 53 — Figure 53: Training dynamics under Base Training, Dedup, MAT, and MAT* on language $L_{2}$ using Mistral- $7$ B (seed $1$ ). Each row corresponds to a training scheme; columns show train loss, test loss, and contextual memorization by frequency group across epochs. For MAT and MAT*, the fraction of training strings removed per group is shown leftmost. Vertical lines mark the optimal learning epoch. In the case of deduplication, all the training strings have the same frequency, hereby grouped as high-frequency.

Panel of Figure 54 — Figure 54: Training dynamics under Base Training, Dedup, MAT, and MAT* on language $L_{2}$ using Mistral- $7$ B (seed $2$ ). Each row corresponds to a training scheme; columns show train loss, test loss, and contextual memorization by frequency group across epochs. For MAT and MAT*, the fraction of training strings removed per group is shown leftmost. Vertical lines mark the optimal learning epoch. In the case of deduplication, all the training strings have the same frequency, hereby grouped as high-frequency.

Panel of Figure 55 — Figure 55: Training dynamics under Base Training, Dedup, MAT, and MAT* on language $L_{4}$ using Mistral- $7$ B (seed $0$ ). Each row corresponds to a training scheme; columns show train loss, test loss, and contextual memorization by frequency group across epochs. For MAT and MAT*, the fraction of training strings removed per group is shown leftmost. Vertical lines mark the optimal learning epoch. In the case of deduplication, all the training strings have the same frequency, hereby grouped as high-frequency.

Panel of Figure 56 — Figure 56: Training dynamics under Base Training, Dedup, MAT, and MAT* on language $L_{4}$ using Mistral- $7$ B (seed $1$ ). Each row corresponds to a training scheme; columns show train loss, test loss, and contextual memorization by frequency group across epochs. For MAT and MAT*, the fraction of training strings removed per group is shown leftmost. Vertical lines mark the optimal learning epoch. In the case of deduplication, all the training strings have the same frequency, hereby grouped as high-frequency.

Panel of Figure 57 — Figure 57: Training dynamics under Base Training, Dedup, MAT, and MAT* on language $L_{4}$ using Mistral- $7$ B (seed $2$ ). Each row corresponds to a training scheme; columns show train loss, test loss, and contextual memorization by frequency group across epochs. For MAT and MAT*, the fraction of training strings removed per group is shown leftmost. Vertical lines mark the optimal learning epoch. In the case of deduplication, all the training strings have the same frequency, hereby grouped as high-frequency.

Panel of Figure 58 — Figure 58: Training dynamics under Base Training, Dedup, MAT, and MAT* on language $L_{9}$ using Pythia- $1$ B (seed $0$ ). Each row corresponds to a training scheme; columns show train loss, test loss, and contextual memorization by frequency group across epochs. For MAT and MAT*, the fraction of training strings removed per group is shown leftmost. Vertical lines mark the optimal learning epoch. In the case of deduplication, all the training strings have the same frequency, hereby grouped as high-frequency.

Panel of Figure 59 — Figure 59: Training dynamics under Base Training, Dedup, MAT, and MAT* on language $L_{9}$ using Pythia- $1$ B (seed $1$ ). Each row corresponds to a training scheme; columns show train loss, test loss, and contextual memorization by frequency group across epochs. For MAT and MAT*, the fraction of training strings removed per group is shown leftmost. Vertical lines mark the optimal learning epoch. In the case of deduplication, all the training strings have the same frequency, hereby grouped as high-frequency.

Panel of Figure 60 — Figure 60: Training dynamics under Base Training, Dedup, MAT, and MAT* on language $L_{9}$ using Pythia- $1$ B (seed $2$ ). Each row corresponds to a training scheme; columns show train loss, test loss, and contextual memorization by frequency group across epochs. For MAT and MAT*, the fraction of training strings removed per group is shown leftmost. Vertical lines mark the optimal learning epoch. In the case of deduplication, all the training strings have the same frequency, hereby grouped as high-frequency.

Panel of Figure 61 — Figure 61: Summary results on learning, memorization, and training cost efficiency on different formal languages, averaged across multiple runs. Lower values are better in all dimensions.

Panel of Figure 62 — Figure 62: Detailed results of the standard deviation and mean of test loss across test strings from different frequency groups. The benefit of MAT and MAT* is also pronounced in mid and low-frequency strings.

Panel of Figure 63 — Figure 63: Training dynamics under Base training and MAT with down-sampling rate $0.5x$ , $x$ , and $2x$ on language $L_{2}$ using Mistral- $7$ B (seed $0$ ), where $x$ is the fraction of test sequences whose loss increases relative to the previous epoch. Each row corresponds to a training scheme; columns show train loss, test loss, and contextual memorization across epochs.

Panel of Figure 64 — Figure 64: Same as Figure 63 , but on language $L_{4}$ .

Panel of Figure 65 — Figure 65: Same as Figure 63 , but on language $L_{9}$ .

Panel of Figure 66 — Figure 66: Summary of MAT’s threshold sensitivity (Base, $0.5x$ , $x$ , $2x$ ) across languages $L_{2}$ , $L_{4}$ , $L_{9}$ (top to bottom) using Mistral- $7$ B, averaged over three seeds. All statistics are measured at the epoch of optimal learning (lowest test loss) for each variant, not at the end of training. $x$ wins on memorization in every language tested. $0.5x$ occasionally improves stability or mean test loss, most notably in $L_{4}$ , where it beats $x$ on both std and mean, but always at a cost in memorization ( $L_{4}$ : 0.081 vs. $x$ ’s 0.054). $L_{4}$ is the one setting where $x$ trades away some stability for its memorization gains; in $L_{2}$ and $L_{9}$ , $x$ is best or near-best on both fronts. From an explainability perspective, $x$ remains the simplest to justify – it down-samples exactly the fraction of training strings matching the fraction of test strings whose loss worsened.

Panel of Figure 67 — Figure 67: Training dynamics under Base, MAT (test-derived), and MAT (w/ val) on $L_{2}$ , $L_{4}$ , $L_{9}$ using Mistral- $7$ B (seed $0$ ).

Panel of Figure 68 — Figure 68: When comparing test-driven split MAT and validation-driven split MAT (w/ val), memorization is comparable or better under the validation split. It improves further in $L_{2}$ and $L_{9}$ , and stays close in $L_{4}$ . Learning stability (std) is slightly better under the test-derived split in all three languages, though the gap is small in $L_{2}$ and $L_{9}$ and only visible in $L_{4}$ . Thus, the memorization benefit transfers cleanly to a realistic held-out-validation setup, at only a small cost in stability.

Panel of Figure 69 — Figure 69: (Column 1 and 3): Test loss of strings from different frequency groups in a language. (Column 2 and 4): Contextual memorization of training strings. Vertical line indicates the lowest test loss of all test strings. Results are obtained using Mistral- $7$ B on $L_{2}$ .

Panel of Figure 70 — Figure 70: Impact of different training schemes on learning, memorization, and training cost. A scheme on the lower-left side of the first two plots is better. In the third plot, lower training size is better. For methods (keep) bottom, middle, and top, we keep $33\%$ training strings, ranked by loss with respect to the ‘base’ (default) training scheme at respective optimal epoch. Compared to down-sampling heuristics of Marion et al. [51] , both MAT and MAT* result in lower mean and standard deviation of test loss and lower contextual memorization. Result is the average of $3$ runs of Mistral- $7$ B on $L_{2}$ .

Panel of Figure 71 — Figure 71: Learning, memorization, and training cost efficiency in a large training setup, where new training strings are added after each epoch, shown for $L_{10}$ . Therefore, the experiment mimics a pre-training setup, where the LLM passes over the training dataset only once and there is no repetition. Yet, MAT is effective in learning, memorization, and training cost efficiency.

Panel of Figure 72 — Figure 72: MAT on natural-language pre-training (1/2). We pre-train a Qwen- $2.5$ - $1.5$ B model from scratch without any data repetition (i.e., a single epoch), independently on Tiny Shakespeare, FineWeb-Edu [ 49 ] , and the German split of FineWeb2 [ 58 ] . Columns show, left to right: test loss vs. epoch, test loss vs. training size, and the mean/variance test-loss trade-off at the optimal learning point (when mean test loss is minimized). In all three datasets, Base and MAT track closely by epoch and training size, with MAT terminating earlier, and MAT always reaches a lower test-loss variance than Base. On Tiny Shakespeare this comes with a lower mean test loss as well; on FineWeb-Edu and German it comes at the cost of a marginally higher mean – the same pattern continues for Chinese, Swahili, and Bengali in Figure 73 .

Panel of Figure 73 — Figure 73: MAT on natural-language pre-training (2/2), continued from Figure 72 . Same setup, on the Chinese, Swahili, and Bengali splits of FineWeb2 [ 58 ] . As with German, Base and MAT track closely by epoch and training size, with MAT terminating earlier, and MAT reaches a lower test-loss variance at the cost of a marginally higher mean in every language here. Therefore, MAT is effective in reducing test loss variance and training size in natural language as well.

Panel of Figure 74 — Figure 74: Counterfactual influence of training string $s_{0}$ on its own loss (a) and on test strings at increasing tree distances (b)–(j). Solid lines ( $s\in D_{\text{train}}$ ) versus dotted lines ( $s\notin D_{\text{train}}$ ) reveal a distance-dependent pattern: nearby test strings exhibit substantially lower loss under inclusion, while the benefit diminishes, and occasionally reverses, for structurally distant test strings.

Panel of Figure 75 — Figure 75: Counterfactual influence of training string $s_{1}$ on its own loss (a) and on test strings at increasing tree distances (b)–(j). The same distance-dependent pattern holds: inclusion lowers test loss for close test strings but has negligible or negative impact on distant ones.

Panel of Figure 76 — Figure 76: Counterfactual influence of training string $s_{2}$ on its own loss (a) and on test strings at increasing tree distances (b)–(j). Consistent with Figures 74 and 75 , the influence of a memorized training string is concentrated among structurally close test strings and attenuates, or reverses, at larger tree distances.

Prompt + Generation	Accuracy of Generation		Remark
	Training	$\text{Contextual}^{\text{UB}}$
( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( (	$1.00$	$1.00$	Repetitions
orem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum l orem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum l	$1.00$	$1.00$	Repeated L a T e X code
29, int t30, int t31, int t32, int t33, int t34, int t35, int t36, int t 37, int t38, int t39, int t40, int t41, int t42, int t43, int t44, int t	$1.00$	$1.00$	Predictable
ICO CITY PLEASE COME TO MEXICO CITY PLEASE COME TO MEXICO CITY PLEASE COME TO MEXICO CITY PLEASE COM E TO MEXICO CITY PLEASE COME TO MEXICO CITY PLEASE COME TO MEXICO CITY PLEASE COME TO MEXICO	$1.00$	$1.00$	Repetition
, ’2014-07-22’ , ’2014-07-23’ , ’2014-07-24’ , ’2014-07-25’ , ’2014-07-26’ , ’2014-07-27’ , ’2014-07-28’ , ’2014-07-29’	$1.00$	$1.00$	Predictable

Appendix

Table 2: List of recollection-based memorized strings by Pythia-

1

B-deduped [ 5 ] , where many strings can be contextually recollected, i.e., repeated words, predictable generation, etc. We report the upper bound (UB) of contextual accuracy using a reference model OLMo-

1

B, which is trained on a different dataset than used in Pythia-

1

B-deduped. Considering the high accuracy of the OLMo-

1

B on memorized strings by Pythia-

1

B-deduped, we suspect that the highlighted generations are not contextually memorized .

Memorization in Language Models

Memorization in language models studies how training examples and facts are stored, recalled, generalized, and exposed by model parameters.

See all 111 papers

Feb 21, 2026

A Prior-Aware Metric for Efficiently Distinguishing Memorization from Generalization in Large Language Models

Training data leakage from Large Language Models (LLMs) raises serious concerns related to privacy, security, and copyright compliance. A central challenge in assessing this risk is distinguishing prefix-specific memorization of training data from the generation of statistically common sequences. Existing approaches to measuring memorization often conflate these phenomena, labeling outputs as memorized even when they arise from generalization over common patterns. Counterfactual memorization and other related metrics \citep{zhang2023counterfactual, wang2025generalization, lesci2024causal} provide principled solutions, however, their reliance on retraining multiple baseline models or parsing through the training data makes them computationally impractical at scale. This work introduces Prior-Aware memorization, a theoretically grounded, lightweight and training-free criterion for identifying prefix-specific memorization in LLMs. The key idea is to evaluate whether a candidate suffix is strongly associated with its specific training prefix or whether it appears with high probability across many IID sampled sequences from the training data distribution due to statistical commonality. We correlate our metric with counterfactual memorization, and also evaluate it on the training corpora of two pre-trained models, LLaMA and OPT. Our results show that between 55% and 90% of sequences previously labeled as memorized fail our criterion and are consistent with statistical commonality

Trishita Tiwari, Ari Trachtenberg, G. Edward Suh

Cornell University · Boston University · NVIDIA, Cornell University

Memorization in Language Models LLM Auditing Privacy Leakage in Language Models

Ask PDF

Oct 3, 2024

Mitigating Memorization In Language Models

Language models (LMs) can "memorize" information, i.e., encode training data in their weights in such a way that inference-time queries can lead to verbatim regurgitation of that data. This ability to extract training data can be problematic, for example, when data are private or sensitive. In this work, we investigate methods to mitigate memorization: three regularizer-based, three finetuning-based, and eleven machine unlearning-based methods, with five of the latter being new methods that we introduce. We also introduce TinyMem, a suite of small, computationally-efficient LMs for the rapid development and evaluation of memorization-mitigation methods. We demonstrate that the mitigation methods that we develop using TinyMem can successfully be applied to production-grade LMs, and we determine via experiment that: regularizer-based mitigation methods are slow and ineffective at curbing memorization; fine-tuning-based methods are effective at curbing memorization, but overly expensive, especially for retaining higher accuracies; and unlearning-based methods are faster and more effective, allowing for the precise localization and removal of memorized information from LM weights prior to inference. We show, in particular, that our proposed unlearning method BalancedSubnet outperforms other mitigation methods at removing memorized information while preserving performance on target tasks.

Mansi Sakarvadia, Aswathy Ajith, Arham Khan +6

University of Chicago · Argonne National Laboratory · Lawrence Berkeley National Laboratory +3

Memorization in Language Models LLM Fine-Tuning Neural Network Memorization LLM Memory Privacy Leakage in Language Models Memorization in Generative Models+1

Ask PDF

Sep 28, 2026

Don't Forget! Decomposing the Training Dynamics of Memorization in Language Models

Memorization has been proposed as a mechanism to explain how language models fit the tail of their training distributions, but its training dynamics are not understood well. In this work, we take a fine-grained look at memorization by decomposing the loss trajectory of memorized sequences over training and model parameters. Across the Pythia family, we study memorization of duplicated training sequences (recitation) and rare ones (recollection). We find that memorization in both cases is characterized by sequence-level gradient alignment, though recitation suffers from misalignment with other training influences which causes forgetting, explaining the necessity for higher duplication of these examples. We further show that the lower model layers are the most involved in memorization and forgetting. Predicting memorization, our decomposition improves over a cross-entropy baseline, especially in larger models and early in training. Intervening on a small set of highly influential parameters we are able to ablate memorization in the final model. Together, these findings advance our understanding of how memorization develops during training and offer insights for predicting and intervening on it.

Florian Eichin, Philipp Mondorf, Andrei Mircea +3

MaiNLP, Center for Information and Language Processing, LMU Munich & Munich Center for Machine Learning (MCML), Germany · Mila – Quebec AI Institute & University of Montreal, Canada · Saarland University, Germany

Memorization in Language Models Data Repetition in Language Model Training Gradient Alignment

Ask PDF

Abstract

Memorization in Language Models

Explore similar work

A Prior-Aware Metric for Efficiently Distinguishing Memorization from Generalization in Large Language Models

Mitigating Memorization In Language Models

Don't Forget! Decomposing the Training Dynamics of Memorization in Language Models