How Learning Governs Unlearning across the Memorization-Generalization Spectrum

Hwiyeong Lee; Hyelim Lim; Ingyu Bang; Hoki Kim; Taeuk Kim

cs.LGOct 6, 2026

How Learning Governs Unlearning across the Memorization-Generalization Spectrum

Authors: Hwiyeong Lee, Hyelim Lim, Ingyu Bang, Hoki Kim, Taeuk Kim

Organizations: Seoul National University · Hanyang University · Chung-Ang University

Abstract

While unlearning seeks to negate undesired capabilities acquired through learning, little research has examined how the way models learn shapes their subsequent unlearning. In this paper, we investigate this connection from the perspectives of memorization and generalization, the two most representative yet competing strategies that models employ during training. We first classify memorization- and generalization-heavy models using grokking in modular addition and compare their responses to unlearning, showing that the latter suffer greater retain damage, i.e., a larger performance drop on the retain set. Furthermore, we conduct a finer-grained analysis by introducing bucketed modular addition, in which the respective contributions of the two strategies can be explicitly controlled across the memorization-generalization spectrum. In this setup, we reaffirm that the same trend persists and is nearly monotonic. We further demonstrate that this relationship also holds in LLM unlearning across verbatim and factual recall settings. Finally, we provide two practical insights for developing better unlearning methods, highlighting the importance of accounting for learning dynamics in unlearning.

Figures & tables

Figure 1: Analysis of unlearning across the memorization–generalization spectrum. We argue that retain damage in unlearning grows with a model’s reliance on generalization over memorization during training. We study this in three steps: contrasting memorizing and generalizing models separated by grokking (§ 3 ), extending this contrast to a fine-grained analysis across the spectrum, quantified by the memorization floor (§ 4 ), and confirming the relationship in LLM unlearning (§ 5 ).

Phase	Epoch	RL	GD
Memorization	1,000	59.5 ± 3.3	23.0 ± 6.5
Circuit formation	3,800	52.4 ± 7.3	19.3 ± 5.3
Cleanup	10,000	14.2 ± 1.5	8.7 ± 3.2
Grokked	25,300	8.1 ± 0.7	3.3 ± 1.0

Table 1: Retain accuracy (%). Mean

\pm

95% CI across five

\mathcal{D}_{f}

selection seeds when forget accuracy first falls to a threshold

\tau

or below.

Figure 2: Learning strategy changes the retain damage of unlearning. Unlearning trajectories under random labeling (RL) at four checkpoints from memorization to grokking (

\mathcal{D}_{f}

selection seed 1). Top: forget accuracy falls much faster than retain accuracy in the memorizing model, whereas both decline together in the generalized model, incurring greater retain damage. Bottom: the Fourier circuit contributes little in the memorizing model; in the generalized model, falling FVE and rising restricted loss reveal its degradation. Restricted loss is normalized per trajectory.

Figure 3: Grokking and the forget–retain trade-off in BMA. Top: For

M\leq 64

, test bucket accuracy (dashed) rises long after training exact accuracy (solid) saturates, revealing grokking in bucket selection. Dotted lines mark chance levels. Bottom: Forget–retain trade-offs in

P_{f}

and

P_{r}

during unlearning (5% forget set, seed 17). With larger memorization floor, the trade-off curves generally lie closer to the starred ideal target:

P_{f}=1/256

and

P_{r}=1

Memorization Floor	Passage	TF-IDF	Bits
High	… those of Humelbergius , the editor -physician of Z ü rich , will be enjoyed and read with profit by every antiquary . The labors of Bernhold and Schuch are meritorious …	7.29	1268
Low	… just as obvious as the principles taken for granted. For no very good reason, three of these principles have been singled out by tradition under the name of ‘Laws of Thought’ …	6.68	614

Table 2: Rarer vocabulary, higher memorization floor. One example passage from each forget set; words occurring five or fewer times in inserted Gutenberg Popular passages are highlighted.

Figure 4: A higher memorization floor leaves less retain damage. (Left) Damage on ID

\pm

95% CI after unlearning the High and Low forget sets. (Right) In-relation damage

\pm

95% CI for each target relation for each method, ordered by the relation’s memorization floor. Heavy line represents the means over the four methods. Nat. and Birth. abbreviate nationality and birthplace respectively.

Relation	Subject $\,\rightarrow\,$ Attribute	Mem. Floor
Email	Maurice Clauss $\,\rightarrow\,$ clauss@gmail.com	0.0002
Nat.	Ricardo da Sousa $\,\rightarrow\,$ Brazil	0.2013
UUID	Tony Nieminen $\,\rightarrow\,$ 766e00 … 8b12ce	5.6064

Table 3: Predictable attributes have a lower memorization floor. Hubble-8B-500B examples; memorization floor in bits.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Transformer layers	1
Attention heads	4
Model dimension	128
MLP width	512
Activation	ReLU
Layer normalization	None

Appendix

Table 4: Model architecture for modular addition.

Setting	Value
Task	$y=(a+b)\bmod 113$
Train/test pairs	3,830 / 8,939 (30% / 70%)
Optimizer	Full-batch AdamW
Learning rate	$10^{-3}$
Weight decay	1
Adam coefficients	$(0.9,0.98)$

Appendix

Table 5: Training configuration for modular addition.

Setting	Value
Optimizer	Full-batch AdamW
Learning rate	$10^{-4}$ (RL); $10^{-4}$ or $3\times 10^{-4}$ (GD)
Weight decay	0 (RL); 1 (GD)
Retain weight $\lambda$	0 (RL); 10 (GD)
Evaluation interval (updates)	1
Forget/retain examples	383 / 3,447

Appendix

Table 6: Unlearning configuration for modular addition. Learning rates are those used for the reported chance-threshold results.

Memorization
Intervention	Retain accuracy (%)	FVE	Restricted loss
RL	59.5 ± 3.3	0.0104 ± 0.0005	5.382 ± 0.448
RL + filtering (non-key)	59.7 ± 3.1	0.0104 ± 0.0005	5.384 ± 0.465
RL + filtering (key)	60.2 ± 2.6	0.0124 ± 0.0004	5.296 ± 0.473
Circuit formation
RL	52.4 ± 7.3	0.0239 ± 0.0029	4.653 ± 0.142

Appendix

Table 7: Fourier gradient filtering across training phases. Results at forget accuracy

\leq\tau

, reported as mean

\pm

95% CI across five

\mathcal{D}_{f}

selection seeds.

Intervention	Retain acc. (%)	FVE	Restricted loss
RL	8.1 ± 0.7	0.186 ± 0.026	2.324 ± 0.183
RL + filtering (non-key)	8.2 ± 0.7	0.187 ± 0.027	2.316 ± 0.187
RL + filtering (key)	30.6 ± 4.1	0.509 ± 0.020	0.167 ± 0.167

Appendix

Table 8: Protecting the Fourier circuit reduces retain damage. Unlearning (RL) results for the grokked model at forget accuracy

\leq\tau

, using the same optimizer settings across conditions. Filtering removes logit-gradient components along the specified Fourier directions. Entries are mean

\pm

95% CI across five

\mathcal{D}_{f}

selection seeds.

Setting	Value
Task	$y=M((a+b)\bmod p)+u_{a,b}$ , $pM=256$
Bucket size $M$ / Memorization Floor $\log_{2}M$	$1,4,16,64,256$ / $0,2,4,6,8$ bits
Output classes	256
Train/test pairs	19,660 / 45,876 (30% / 70%)
Optimizer	Full-batch AdamW
Learning rate	$10^{-3}$

Appendix

Table 9: Training configuration for bucketed modular addition. Settings are fixed across bucket sizes.

Setting	Value
Optimizer / weight decay	Full-batch AdamW / 0
Forget/retain examples	197 / 19,463 (1%); 983 / 18,677 (5%); 1,966 / 17,694 (10%)
Forget-set selection seeds	17, 26, 42
Stopping rule	$P_{f}\leq 0.0039$ , or update budget
Comparison criterion	$P_{f}=0.1$

Appendix

Table 10: Unlearning configuration for bucketed modular addition. Settings are fixed across bucket sizes and selection seeds; only GD depends on the forget fraction.

Figure 5: A lower memorization floor leaves greater retain damage. Retain damage

1-P_{r}

P_{f}=0.1

. Points and error bars show means and 95% CIs across three

\mathcal{D}_{f}

selection seeds. Dashed lines mark damage

0.9

, the nonselective baseline where

P_{f}=P_{r}=0.1

Forget	Method	$\rho_{s}$	$r$
1%	GD	$-$ 0.900	$-$ 0.950
	RL	$-$ 1.000	$-$ 0.995
	DPO	$-$ 1.000	$-$ 0.994
	ME	$-$ 0.900	$-$ 0.924
5%	GD	$-$ 1.000	$-$ 0.975
	RL	$-$ 1.000	$-$ 0.972

Appendix

Table 11: Memorization floor and retain damage. Spearman’s

\rho_{s}

and Pearson’s

r

between memorization floor (

\log_{2}M

) and mean retain damage (

1-P_{r}

) at

P_{f}=0.1

, across five bucket sizes (

n=5

), calculated separately for each forget fraction and method.

Size	Pretraining budget	Repository
8B	100B tokens	allegrolab/hubble-8b-100b_toks-{ standard , perturbed }-hf
8B	500B tokens	allegrolab/hubble-8b-500b_toks-{ standard , perturbed }-hf
1B	100B tokens	allegrolab/hubble-1b-100b_toks-{ standard , perturbed }-hf
1B	500B tokens	allegrolab/hubble-1b-500b_toks-{ standard , perturbed }-hf

Appendix

Table 12: Hubble model pairs. The two members of a pair share a pretraining procedure and differ only in whether the controlled insertions are present.

Corpus	Dataset
Gutenberg Popular	allegrolab/passages_gutenberg_popular
Wikipedia	allegrolab/passages_wikipedia
YAGO biographies	allegrolab/biographies_yago

Appendix

Table 13: Inserted corpora. The Hubble insertions these experiments use. Each is added to the base corpus when the perturbed model is pretrained and is absent from the standard model; Hubble inserts further corpora that we do not use.

Role	Group	Hubble 8B-100B	Hubble 8B-500B	Hubble 1B-100B	Hubble 1B-500B	Source and selection
Forget	High	24	24	24	18	Gutenberg Popular, top 30%
Forget	Low	24	24	24	18	Gutenberg Popular, bottom 30%
Retain	ID-seen	32	32	32	21	Gutenberg Popular, middle 40%
Retain	OOD-seen	76	76	76	16	Wikipedia, $256\times$ inserted
Retain	Unseen	400	400	400	400	200 held-out passages of each corpus

Appendix

Table 14: Forget and retain groups for verbatim recall. Entries are passage counts. High and Low are the top and bottom 30% of the

256\times

Gutenberg Popular tier by memorization floor. For Hubble-1B-500B the tier is first restricted to

\bar{P}_{\mathrm{pert}}\geq 0.6

, which shrinks every seen group.

Group	Hubble 8B-100B	Hubble 8B-500B	Hubble 1B-100B	Hubble 1B-500B
High	1136 ± 89	1048 ± 88	1233 ± 96	1062 ± 97
Low	881 ± 82	780 ± 88	985 ± 83	838 ± 80

Appendix

Table 15: Memorization floor of the verbatim forget sets. Per-passage floor in bits, as mean

\pm

95% CI over the passages of the group.

Method	Learning rate	Retain objective
GA	$10^{-5}$	None
GD	$10^{-5}$	Cross-entropy on ID-seen ( $\lambda=1$ )
NPO	$10^{-5}$	None
CEU	$10^{-5}$	None

Appendix

Table 16: Unlearning hyperparameters for verbatim recall. Each run uses AdamW, gradient clipping at 1.0, and one pass over its forget set per epoch. NPO uses

\beta=0.1

Figure 6: Document-level TF-IDF and memorization floor. Each point is a passage of the two forget sets; panel titles give the Pearson correlation and its

p

-value.

		Retain damage (%) $\downarrow$
Model	Retain	GA		GD		NPO		CEU
Model	Retain	High	Low	High	Low	High	Low	High	Low
Hubble 8B-100B	ID-seen	45.5 ± 3.8	54.0 ± 3.2	17.3 ± 2.3	17.9 ± 1.8	45.0 ± 3.8	52.8 ± 3.3	54.2 ± 6.5	74.0 ± 5.1
	OOD-seen	3.4 ± 0.5	5.4 ± 0.9	2.9 ± 0.5	4.0 ± 0.7	3.4 ± 0.5	5.2 ± 0.8	9.8 ± 3.0	18.7 ± 4.3
	Unseen	0.5 ± 0.1	0.7 ± 0.1	0.6 ± 0.1	0.8 ± 0.1	0.5 ± 0.1	0.7 ± 0.1	$-$ 0.6 ± 0.1	$-$ 0.4 ± 0.1
Hubble 8B-500B	ID-seen	14.8 ± 2.1	26.4 ± 2.9	0.2 ± 0.6	3.6 ± 1.1	13.4 ± 2.1	25.2 ± 2.9	44.1 ± 6.1	63.1 ± 4.2

Appendix

Table 17: Retain damage in verbatim recall. Hubble-8B and Hubble-1B,

256\times

Gutenberg Popular. Damage is

100(\bar{P}_{r,\mathrm{before}}-\bar{P}_{r,\mathrm{after}})

\bar{P}_{f}=0.1

, as mean

\pm

95% CI over the retain passages of a group. Bold marks the smaller damage of a High–Low pair.

		Memorization floor (bits)
Relation	Prompt	Hubble-8B-100B	Hubble-8B-500B
Email	{subject} receives email at	0.93 ± 0.64	0.43 ± 0.42
Nationality	{subject} is from	1.61 ± 0.44	1.09 ± 0.31
Birthplace	{subject} was born in	4.30 ± 2.05	2.30 ± 1.31
UUID	{subject} has the unique identifier	5.00 ± 0.62	5.70 ± 0.70

Appendix

Table 18: Relations, prompts, and memorization floors. Each prompt opens the biography sentence that states the attribute, with the subject’s name in place of {subject} , and is completed by the attribute value. Floors are mean

\pm

95% CI over the relation’s selected facts.

Setting	Hubble-1B	Hubble-8B
Learning rate	$3\times 10^{-7}$	$10^{-5}$
Initial update budget	240	480
Continuation for unmatched runs	None	Up to 20 updates

Appendix

Table 19: Unlearning settings for factual recall. Only these three settings depend on model size; both sizes use the 100B and 500B pretraining budgets, 15 forget facts out of 50 with selection seed 42, GD retain weight 5, and

\beta=0.1

for NPO and DPO. Settings are shared by all methods and relations.

			Retain damage (%p) $\downarrow$
Model	Relation	Mem. Floor	GA		GD		NPO		DPO
Model	Relation	Mem. Floor	In-relation	Out-relation	In-relation	Out-relation	In-relation	Out-relation	In-relation	Out-relation
Hubble 8B-100B	Email	0.93 ± 0.64	77.1 ± 14.6	3.6 ± 3.1	14.3 ± 12.2	2.1 ± 2.4	45.7 ± 17.4	3.6 ± 3.0	45.7 ± 17.4	2.9 ± 2.7
	Nationality	1.61 ± 0.44	48.6 ± 17.4	2.9 ± 2.7	42.9 ± 17.2	3.6 ± 3.0	45.7 ± 17.4	3.6 ± 3.0	25.7 ± 15.2	2.9 ± 2.8
	Birthplace	4.30 ± 2.05	17.1 ± 13.1	1.4 ± 2.0	5.7 ± 8.1	2.1 ± 2.4	25.7 ± 15.2	1.4 ± 2.0	17.1 ± 13.1	1.4 ± 2.0
	UUID	5.00 ± 0.62	11.4 ± 11.1	1.4 ± 2.0	14.3 ± 12.2	1.4 ± 2.0	14.3 ± 12.2	0.0 ± 0.0	14.3 ± 12.2	0.7 ± 1.4

Appendix

Table 20: Retain damage in factual recall. Hubble-8B and Hubble-1B,

64\times

YAGO. The memorization floor is in bits and damage is the decrease in accuracy in percentage points, both as mean ± 95% CI; in-relation intervals are paired across facts. GA, GD, NPO, and DPO all reach the forget target. Within a model, bold marks the relation on which a method leaves the least in-relation damage.

Model	Relation	In-relation	Out-relation	$\Delta$
Hubble-8B-100B	Email	45.7 ± 40.8	3.0 ± 1.1	42.7 ± 39.9
	Nationality	40.7 ± 16.3	3.2 ± 0.7	37.5 ± 16.1
	Birthplace	16.4 ± 13.1	1.6 ± 0.6	14.8 ± 13.6
	UUID	13.6 ± 2.3	0.9 ± 1.1	12.7 ± 3.0
Hubble-8B-500B	Email	36.4 ± 15.9	5.4 ± 2.7	31.1 ± 16.6
	Nationality	39.3 ± 15.0	8.4 ± 2.3	30.9 ± 14.8

Appendix

Table 21: Mean retain damage across methods. Accuracy decrease (percentage points), averaged over GA, GD, NPO, and DPO, with 95%

t

intervals across methods.

\Delta

is the difference between in-relation and out-relation damage.

Model	Method	$r$	$\rho_{s}$
Hubble-8B-100B	GA	$-$ 0.966	$-$ 1.000
	GD	$-$ 0.517	$-$ 0.316
	NPO	$-$ 0.978	$-$ 0.949
	DPO	$-$ 0.889	$-$ 1.000
Hubble-8B-500B	GA	$-$ 0.943	$-$ 1.000
	GD	$-$ 0.829	$-$ 1.000

Appendix

Table 22: Memorization floor–damage correlations in factual recall. Pearson's

r

and Spearman's

\rho_{s}

over the relations of one model and method: four for Hubble-8B and three for Hubble-1B. With three relations

\rho_{s}

can only take the values

0

\pm 0.5

and

\pm 1

, so the Hubble-1B rank coefficients are coarse.

$P$	$M$	Relabeling target	Retain	Forget bucket
131	2	Unrestricted	15.4	12.5
131	2	Within bucket	75.7	99.9
277	4	Unrestricted	28.0	14.7
277	4	Within bucket	71.3	99.7

Appendix

Table 23: Within-bucket redistribution preserves retain performance. RL with unrestricted or within-bucket incorrect targets at 10% forget exact accuracy. Entries report retain exact and forget bucket accuracies (%) from one trained model and split per setting.

Method	Weight $\omega_{t}$	Tokens weighted most
GA	$1$	all equally
WGA ( Wang et al., 2025 )	$p_{t}^{\beta}$	tokens the model still predicts confidently
TNPO ( Wang et al., 2025 )	$2p_{t}^{\beta}/\bigl(p_{t}^{\beta}+(p_{t}^{\mathrm{ref}})^{\beta}\bigr)$	tokens whose probability has dropped least
Imp ( Yang et al., 2025 )	$(1-p_{t})^{\beta}$	tokens the model predicts poorly
SatImp ( Yang et al., 2025 )	$p_{t}^{\beta_{1}}(1-p_{t})^{\beta_{2}}$	confident but not yet saturated tokens
ETW ( Koh et al., 2026 )	$n_{x}H_{t}/\sum_{s}H_{s}$	tokens with uncertain predictions

Appendix

Table 24: Token-wise regularizers compared in Appendix E.2 .

p_{t}

: the current model’s probability of the gold token;

p_{t}^{\mathrm{ref}}

: the same before unlearning;

H_{t}

: predictive entropy at temperature

T=1.5

;

n_{x}

: document length. Exponents are hyperparameters:

\beta=5

for WGA,

4

for TNPO and

1

for Imp;

(\beta_{1},\beta_{2})=(5,1)

for SatImp.

Method	AUC	Retain
SatImp	0.65 ± 0.05	0.236 ± 0.007
WGA	0.61 ± 0.07	0.246 ± 0.019
TNPO	0.61 ± 0.06	0.243 ± 0.014
ETW	0.55 ± 0.04	0.199 ± 0.013
GA	0.50 ± 0.00	0.217 ± 0.015
SCE	0.45 ± 0.00	0.191 ± 0.006

Appendix

Table 25: Regularizers that better separate tokens above from those below the one-bit memorization floor (higher AUC) retain more. Both measured when the forget probability reaches 0.1; mean

\pm

95% CI over three seeds.

Explore similar work

Oct 7, 2026cs.LG

Why Forget-Only Unlearning Needs Memorization

Machine unlearning asks for a deletion algorithm whose output is close to retraining from scratch without the selected forget examples. In this work, we study forget-only unlearning, where the deletion algorithm receives only the trained model and the examples to forget, with no retained data or extra training information. We ask whether forget-only unlearning is always possible. We first show that this depends on the learning method: different datasets can produce the same trained model but require very different outputs after the same examples are removed. Using this observation, we derive lower bounds on how accurately unlearning can match retraining and instantiate them for several standard learning algorithms. We then ask what must be true when forget-only unlearning succeeds. To this end, we derive lower bounds on what an algorithm must memorize about the training data to handle arbitrary deletion requests. For simple threshold learners, the required information can be as large as the entire dataset, even though ordinary training keeps only one boundary point. Overall, our results show that information discarded during ordinary learning may be needed later for deletion, so models designed for forget-only unlearning may need to retain more information than standard training does.

Luka Radić, Vikrant Singhal, Amartya Sanyal

Department of Computer Science, University of Copenhagen

Topic pending

Ask PDF

Sep 28, 2026cs.LG

Causal Routing for Unlearning

LLMs cannot forget the way we delete a file. Strangely, we are asked to remove something that was never put anywhere in particular. What the model took from a piece of text is now smeared across billions of weights. Existing methods rewrite all of them to change one thing, and none of them say which part produced that change. To address this, we introduce Causal Routing for Unlearning (CRU) by asking where the concept is expressed in the model and suppressing only that part. One untrained forward pass over the forget set ranks neurons by how their activations vary. Then, small routing modules on those neurons gate and suppress only the concepts that need to be forgotten. In CRU, the base model is frozen, and any change in behavior is caused only by the gated neurons; hence, why the routing is causal. Due to our parameter efficiency (only ~0.01% as many parameters as the base model), unlearning a concept costs 14 GiB, whereas the baselines require 71 GiB. On TOFU, CRU is indistinguishable from the retained model (p > 0.05, KS test) and is never Pareto-dominated, whereas every compared baseline matches its forgetting on the larger-forget batches only by collapsing utility. On RWKU, it achieves an adversarial-probe recall of 0.052, compared to 0.250 for the strongest baseline, meaning the knowledge is gone, not merely harder to reach. Thus, deciding on the intervention at query time, rather than fixing it beforehand, is the axis along which we argue that unlearning should proceed.

Bardh Prenkaj, Andrea D'Angelo, Davide Mottin +4

Sapienza University of Rome · Aarhus University

Large Language Model Memory Causal Batch

Ask PDF

Sep 28, 2026cs.LG

Making LLMs Truly Forget: Deep Unlearning by Searching, Selecting, and Severing Knowledge Paths

While an unlearned language model may no longer recall a fact directly, the fact often remains recoverable through multi-hop reasoning over related knowledge. Most existing unlearning techniques overlook this vulnerability, targeting facts in isolation while leaving their supporting knowledge intact. To achieve true forgetting, we propose a general deep unlearning framework compatible with existing unlearning algorithms. Our approach adaptively explores both explicit responses and latent internal representations to discover valid reasoning paths, compiles them into a confidence-aware supporting subgraph, and we apply a graph minimum cut to sever all recovery paths while preserving unrelated knowledge. To rigorously evaluate deep unlearning, we introduce a model-specific pipeline that extracts and completes knowledge graphs from raw text, filtering them by calibrated model confidence to reflect what the model genuinely retains. Comprehensive experiments demonstrate that selectively unlearning supporting knowledge yields substantially deeper forgetting than superficial methods while preserving model utility, highlighting that genuine unlearning requires breaking the relational structures that enable factual reconstruction.

Jialu Wang, Peizhi Niu, Haoteng Yin +3

Tongji University · University of Illinois Urbana-Champaign · Purdue University +1

Large Language Model Unlearning Forgetting Large Language Model Memory Knowledge Graphs

Ask PDF