Mitigating Memorization Where It Happens

Xuanqi Zhang; Haoyang Shang; Xiaoxiao Li

cs.LGFeb 9, 2026

Mitigating Memorization Where It Happens

Authors: Xuanqi Zhang, Haoyang Shang, Xiaoxiao Li

Organizations: University of British Columbia · Vector Institute · Independent Researcher

Abstract

Large language models (LLMs) can memorize and reproduce training sequences verbatim, raising privacy and copyright concerns. Existing mitigation methods apply interventions uniformly, often degrading performance on the majority of tokens that generalize normally. We study a token-level log-likelihood ratio between a target model and a domain-competent reference with lower exposure to the evaluated sequences. Exposure-controlled experiments support its use for localizing positions associated with sequence-level verbatim recall. Across evaluated settings, a minority of positions carries a disproportionate share of the positive signal, motivating selective activation correction. From the ideal memorization reduction objective, we derive a probe--steer framework which decomposes intervention into a probe that detects memorization-relevant activations and a steer that applies targeted correction only when the probe exceeds a threshold. We propose Gated Subspace Steering (GSS) as its practical instantiation: the optimal probe--steer pair admits a closed-form solution as the leading singular vectors of a gradient-weighted activation matrix. Across four benchmarks, GSS matches or exceeds state-of-the-art memorization reduction while requiring 10--100 $\times$ less compute than optimization-based alternatives.

Figures & tables

Panel of Figure 1 — Figure 1: Token-level excess likelihood on Qwen3-GSM8K. (a) The distribution of $\omega_{t}$ is right-skewed (skewness $2.89$ ), with some positions exhibiting much larger target–reference likelihood differences than others. (b) Consecutive runs of positions with $\omega_{t}>0$ have a mean length of $1.64$ ; $60.8\%$ of runs contain one position. These statistics describe the likelihood signal, rather than the lengths of memorized spans or ground-truth transitions between recall and generalization. Cross-setting concentration measurements are reported in Appendix C.1 .

Figure 2: Gated Subspace Steering (GSS) overview. (a) Calibration. From the token-level signal $\omega_{t}$ , we partition activations into memorization- and generalization-aligned subsets, then derive decoupled probe ( $u_{k}$ ) and steer ( $v_{k}$ ) directions from the memorization matrix $\mathbf{M}$ and generalization manifold $\Sigma_{\text{gen}}$ via a generalized SVD. (b) Selective gating. At inference, the gate $\mathcal{G}$ fires when the probe response $\langle h,u_{k}\rangle$ exceeds threshold $\varepsilon$ , applying the steer only on memorization-dominant tokens and leaving generalization-dominant tokens untouched.

Figure 3: Comparison of intervention in the activation space. (a) Naive Steering subtracts a constant vector from every token, an “unconstrained shift” that distorts generalized representations. (b) GSS establishes a “safe subspace”: the intervention activates only on memorization-dominant tokens that exceed the gate threshold, leaving the generalization manifold untouched.

Method	Year	Math Model						Language Model
		Noise			Backdoor			Noise			Backdoor
		%Mem $\downarrow$	Acc $\uparrow$	Time $\downarrow$	%Mem $\downarrow$	Acc $\uparrow$	Time $\downarrow$	%Mem $\downarrow$	Perp $\downarrow$	Time $\downarrow$	%Mem $\downarrow$	Perp $\downarrow$	Time $\downarrow$
Baseline	–	34.55	97.00	–	99.44	96.81	–	17.60	57.73	–	100.00	58.45	–
Spectral reg	2024	0.17	96.84	12254	99.89	96.68	1131.5	0.00	57.92	10850.2	100.00	90.42	238.6
Loss trunc	2024	30.58	96.97	2391	99.67	99.96	347.7	0.00	57.92	16080	100.00	58.45	6005
Both FT	2024	0.00	97.17	28.45	0.00	91.36	20.35	0.00	51.63	1680	0.00	59.19	1629

Table 1: TinyMem Models [ 48 ] . Comparison of memorization mitigation strategies on multiplicative math and language models under noise and backdoor settings. The results are averaged with 3 seeds. Metrics include

(n,k)

-memorization, test accuracy / perplexity, and wall-clock time of the mitigation pipeline (seconds).

	Pythia 2.8B			Pythia 6.9B
Method	%Mem $\downarrow$	PPL $\downarrow$	Time $\downarrow$	%Mem $\downarrow$	PPL $\downarrow$	Time $\downarrow$
Baseline	52.87	21.75	–	89.31	19.46	–
HC	44.55	21.35	2.25	87.55	17.56	36.01
Slim	14.46	23.41	1.96	33.46	18.90	280.35
Act	14.46	24.43	13.01	30.30	18.34	36.02
Dura	6.93	35.49	17.44	14.65	23.34	44.34

Table 2: Comparison on Pythia models across

(n,k)

-memorization, test perplexity, and time.

Figure 4: Pareto Frontier Analysis. The plots visualize the trade-off between memorization reduction measured (x-axis, normalized by baseline) and downstream utility measured in log-likelihood (y-axis, normalized by baseline). The ideal method occupies the top-right corner. (Left: GSM8K) On reasoning tasks, our method dominates the frontier. We observe utility recovery where mild steering improves performance ( $>1.0$ ). (Right: UltraChat) Our method consistently outperforms Editing baselines and maintains a Pareto frontier in the high-utility regime

Figure 5: Gated vs. Ungated Intervention. Gating exploits heavy-tailed sparsity.

Panel of Figure 6 — Figure 6: Reference model ablation. GSS is robust to the choice of reference model. Across reference checkpoints, the memorization rate remains around $6.5$ – $7.1\%$ .

Panel of Figure 7 — Figure 7: Visualize the Pareto frontier . Utility score vs. memorization reduction.

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Target weight update	Calibration / optimization access
GSS	No	Target, reference, calibration corpus
NPO implementation	Yes	Forget loss, frozen reference scores, retain loss
Task Vector	Yes	Fine-tuning corpus and parameter copies
LUNAR implementation	Yes	Unlearning/reference activations
ROME / MEMIT	Yes	Edit requests and implementation-specific statistics

Appendix

Table 3: Weight updates and data access required by each method as evaluated.

Statistic	Range
Top-10% positive-mass share	30–89%
Relative to uniform allocation	$3.0$ – $8.9\times$
Gini coefficient	0.50–0.91
Normalized effective support	0.088–0.532
Positions with $\omega>0$	47–94%

Appendix

Table 4: Concentration of positive likelihood mass: ranges across twelve settings (11,833 sequences, 1.72 million positions).

Panel of Figure 8 — Figure 8: Distribution of the likelihood signal $\omega_{t}$ . Low signal is not a generalization label, distributional shape is setting-dependent, and run frequency does not measure intervention magnitude or causal importance.

Score	Positions	Sequences	AUROC	95% CI
Target confidence	120780	2745	0.909	[0.904, 0.914]
$\omega_{t}$	120780	2745	0.583	[0.567, 0.599]

Appendix

Table 5: Token-level ranking of 20-token verbatim continuations on Pythia-2.8B, with 95% sequence-bootstrap intervals. No intervention is applied.

Partition	Math Noise			Math Backdoor			LM Noise			LM Backdoor
Partition	Mem. $\downarrow$	Acc. $\uparrow$	Time $\downarrow$	Mem. $\downarrow$	Acc. $\uparrow$	Time $\downarrow$	Mem. $\downarrow$	PPL $\downarrow$	Time $\downarrow$	Mem. $\downarrow$	PPL $\downarrow$	Time $\downarrow$
Random partition	18.42	96.71	0.001	21.36	96.44	0.001	27.85	64.02	0.002	31.17	64.36	0.003
Inverse partition $(\omega_{t}<0)$	34.79	96.55	0.001	39.28	96.21	0.001	46.63	64.71	0.002	49.85	65.08	0.003
High- $\omega_{t}$ partition (GSS)	0.00	96.98	0.001	0.00	96.82	0.001	0.00	63.13	0.002	0.00	63.17	0.003

Appendix

Table 6: TinyMem partition ablation. Calibrating on high-signal positions gives the lowest reproduction.

Reference	Sequence AUROC	95% CI
Matched, unexposed	1.000	[1.000, 1.000]
Matched, exposed	0.556	[0.456, 0.657]
Independent seed, unexposed	1.000	[1.000, 1.000]
Independent seed, exposed	0.676	[0.585, 0.763]
Early checkpoint	0.998	[0.994, 1.000]
Late checkpoint	0.000	[0.000, 0.000]

Appendix

Table 7: Sequence AUROC of

\omega

under different reference models in the canary setting. No intervention is applied.

Reference step	Enrichment	Sequence AUROC
512	36.90	0.916
1000	9.42	0.711
2000	7.62	0.660
16000	2.55	0.498
32000	1.66	0.469
72000	1.39	0.488

Appendix

Table 8: Reference checkpoint sweep: verbatim-20 enrichment of positive versus nonpositive

\omega

positions. Sequence AUROC uses mean signal to distinguish the memorization cohort from the random cohort (505 versus 1,024 sequences).

Reference Model Epoch	4-layer Noisy (%)	8-layer Noisy (%)	4-layer Clean (%)
50	0.5	5.1	0.0
100	4.5	5.2	0.0
150	11.0	10.0	0.0
200	16.4	9.2	0.0
250	6.8	20.0	0.0
300	20.1	11.7	0.0

Appendix

Table 9: TinyMem reproduction rates (percent) across reference choices.

Reference Model Step	Mem. Rate	PPL
2000	6.45%	31.49
4000	7.11%	29.49
6000	6.93%	28.26
8000	7.11%	28.64
10000	7.11%	28.74

Appendix

Table 10: Pythia-2.8B reproduction and PPL across reference checkpoints.

$\beta$	$r$	Exact	PPL	Trigger	Number preservation
3	1	0.1667	23.45	0.435	0.434
3	2	0.1732	23.15	0.325	0.439
10	1	0.0965	25.04	0.379	0.333
10	2	0.1118	24.15	0.271	0.379

Appendix

Table 11: Pythia-2.8B, rank-one C,

\eta=0.1

. Trigger is the fraction of evaluated positions at which the gate fires; number preservation measures overlap with numbers in the reference continuation and is not task accuracy.

$r$	Exact-30	Accuracy	Trigger	Number preservation
1	0.515	0.533	0.078	0.854
2	0.770	0.600	0.021	0.937

Appendix

Table 12: Qwen2.5-7B on GSM8K, C,

\beta=3

\eta=0.1

, single seed. Accuracy is measured on 300 test items.

Seed	$r$	$\beta$	Exact-30	Number preservation	Trigger
1	1	10	0.243	0.697	0.076
1	2	10	0.622	0.863	0.021
1	2	20	0.566	0.836	0.020
1	2	30	0.536	0.814	0.018
2	1	10	0.137	0.533	0.173
2	2	10	0.387	0.669	0.100

Appendix

Table 13: Qwen2.5-7B gate strength sweep with C,

\eta=0.1

, and 30-token exact reproduction, reported per seed.

Mask	Exact	PPL	Number	Entity
Full gate	0.085	24.69	0.324	0.323
No whitespace/punctuation	0.093	24.31	0.377	0.349
No function words	0.089	24.26	0.340	0.361
No numbers	0.089	24.58	0.340	0.321
No entities	0.091	24.48	0.330	0.357
No numbers/entities/function words	0.099	23.95	0.358	0.392

Appendix

Table 14: Mask ablations on Pythia-2.8B (rank-one C,

\beta=10

\eta=0.1

). “No

X

” masks exclude positions of class

X

from the gate.

Class	Fraction	95% CI
Redundant	0.757	[0.682, 0.836]
Generic/interchangeable	0.126	[0.068, 0.180]
Harmful	0.081	[0.038, 0.127]
Verbatim-critical	0.036	[0.000, 0.062]

Appendix

Table 15: Counterfactual single-token classifications (rank-one C,

\beta=10

\eta=0.1

) with 95% bootstrap intervals.

Evaluation text	Positions	Exceedance
Pile (held-out)	31600	0.1006
UltraChat instructions	73388	0.1265
Long context (600 tokens)	86580	0.1132
Entity-heavy text	61668	0.1068
GSM8K	31096	0.0572
Number-heavy answers	29381	0.0380

Appendix

Table 16: Probe exceedance rates under distribution shift with a Pile-calibrated threshold (nominal rate 0.1).

Figure 9: Threshold ablation on Pythia-2.8B.

Noisy samples	Gated PPL $\downarrow$	Ungated PPL $\downarrow$	$\Delta$
500	58.97	62.17	$-3.2$
800	60.40	62.90	$-2.5$
1000	59.37	63.17	$-3.8$
1300	62.20	65.10	$-2.9$
1600	61.60	65.70	$-4.1$
1900	63.30	66.30	$-3.0$

Appendix

Table 17: Gated versus ungated intervention on the TinyMem language model under increasing numbers of noisy training samples. Memorization is 0.0% for both variants.

Seed	$\beta$	Reproduction	Accuracy
1	–	0.884	0.593
1	1.0	0.833	0.597
1	3.0	0.513	0.543
1	10.0	0.246	0.460
2	–	0.854	0.597
2	1.0	0.793	0.543

Appendix

Table 18: Qwen2.5-7B on GSM8K: reproduction and accuracy per seed (

r=1

). Rows with – are the unmodified model.

Setting	$\beta$	Score	Change	95% CI	$p_{\mathrm{TOST}}$
GSM8K (seed 1)	1.0	7.94	-0.23	[-0.52, 0.05]	0.0361
GSM8K (seed 1)	3.0	8.02	-0.15	[-0.49, 0.17]	0.0203
GSM8K (seed 1)	10.0	7.07	-1.11	[-1.56, -0.65]	0.9958
GSM8K (seed 2)	1.0	7.71	-0.25	[-0.58, 0.09]	0.0758
GSM8K (seed 2)	3.0	7.23	-0.73	[-1.13, -0.35]	0.8703
GSM8K (seed 2)	10.0	6.47	-1.49	[-1.91, -1.05]	1.0000

Appendix

Table 19: Qwen2.5-7B-Instruct judge scores (C,

r=1

). Change is relative to the unmodified model; intervals and TOST

p

-values come from paired tests.

Domain	$\eta$	$\beta$	Exact-50-char	Control PPL	Trigger
healthcare	–	–	0.913	1.089	–
healthcare	0.05	1	0.233	2.019	0.049
healthcare	0.05	3	0.153	3.213	0.049
healthcare	0.05	10	0.133	4.332	0.050
healthcare	0.1	1	0.053	4.356	0.097
healthcare	0.1	3	0.040	12.173	0.097

Appendix

Table 20: Real-document domains, C-M with

r=1

. Reproduction is measured on the first 50 characters and PPL on control documents. Rows with – are the unmodified model.

Copies	$\beta$	Full record	Email	Accuracy
1	–	0.00	0.02	0.597
1	3.0	0.00	0.03	0.337
1	10.0	0.00	0.02	0.340
3	–	0.77	0.86	0.610
3	3.0	0.59	0.73	0.550
3	10.0	0.01	0.04	0.220

Appendix

Table 21: Synthetic PII extraction by number of copies (single-position gate). Full-record and email columns are extraction rates; accuracy measures task utility. Rows with – are the unmodified model.

Model	$\eta$	$\beta$	Reproduction	Accuracy
Qwen3-14B	–	–	0.772	0.703
Qwen3-14B	0.05	0.3	0.529	0.680
Qwen3-14B	0.05	1.0	0.054	0.503
Qwen3-14B	0.05	3.0	0.008	0.280
Qwen3-14B	0.1	0.3	0.491	0.673
Qwen3-14B	0.1	1.0	0.045	0.483

Appendix

Table 22: GSM8K sweep across models (single-position gate). Rows with – are the unmodified models; L16 and L24 denote the intervention layer. Llama-3-8B at L16 collapses at all eight settings tested (

\eta\in\{0.05,0.1\}

\beta\in\{0.1,0.3,0.5\}

, and additionally

\beta\in\{1,10\}

\eta=0.1

Method	Reproduction	Clean PPL
Baseline	0.06513	321.49
ROME (clamp 1)	0.00000	831.76
ROME (clamp 4)	0.00000	1590547.73
MEMIT	0.00000	324.88
AlphaEdit	0.00000	381.30
EMMET	0.02204	322.33

Appendix

Table 23: Knowledge editors and GSS on the same TinyMem checkpoint. Zero is the floor of the reproduction metric.

$\beta$	Exact	4-gram precision	Cosine	Idea-only
Baseline	0.535	0.811	0.913	0.307
1.0	0.248	0.545	0.785	0.356
3.0	0.141	0.425	0.690	0.299
5.0	0.107	0.365	0.641	0.259
15.0	0.081	0.307	0.592	0.218
20.0	0.077	0.286	0.575	0.198

Appendix

Table 24: Exact and semantic reproduction on Pythia-2.8B across steering strengths. Weak steering increases idea-only reproduction, so exact suppression does not imply semantic suppression.

Metric	Before	After	Delta
Exact memorized %	23.97	0.00	$-23.97$
Approx-only memorized %	76.03	3.64	$-72.39$
ROUGE-L mean	0.9664	0.3749	$-0.5916$

Appendix

Table 25: TinyMem exact and approximate reproduction.

Metric	Before	After	Delta
Exact memorized %	46.14	8.91	$-37.23$
Approx-only memorized %	42.18	34.06	$-8.12$
ROUGE-L mean	0.8529	0.4740	$-0.3790$

Appendix

Table 26: Pythia-2.8B exact and approximate reproduction.

Attack template	Prefix fraction	Base ASR (%)	Steered ASR (%)	Reduction (%)
Instruction-0	0.2	9.7	5.1	47
Instruction-0	0.5	18.5	9.2	50
Instruction-2	0.3	16.4	5.1	69
Paraphrase-0	0.3	16.4	5.1	69
Paraphrase-0	0.5	18.5	7.2	61

Appendix

Table 27: Attack settings in which steering reduces attack success rate (a subset of the evaluated grid).

Template	Prefix Fraction	# Examples	Metric	Base	Steered
Raw prefix	0.5	145	Exact Match ASR	0.00	0.00
Raw prefix	0.5	145	ASR@50% Token Overlap	54.48	0.00
Raw prefix	0.5	145	ASR@80% Token Overlap	2.07	0.00
Raw prefix	0.5	145	Mean Token Overlap	0.4469	0.1390
Raw prefix	0.5	145	Mean LCP Ratio	0.2855	0.0172
Raw prefix	0.5	145	Mean ROUGE-L	0.6404	0.3792

Appendix

Table 28: Best-of-8 extraction on 145 sequences. Exact-match ASR is already zero at baseline, so overlap-based ASR is the informative outcome.

Model	Metric	Rank 4	Rank 8
Qwen3-0.6B	AUC	0.783	0.811
Qwen3-0.6B	Spearman	0.530	0.603
Llama-3.2-1B	AUC	0.899	0.929
Llama-3.2-1B	Spearman	0.771	0.839

Appendix

Table 29: Sequence-level discrimination performance of the whitened probe subspace on memorized vs. non-memorized sequences.

Direction	$\cos$
$u$ (probe)	$0.999\pm 0.001$
$v$ (steer)	$0.943\pm 0.067$

Appendix

Table 30: Stability of the learned probe and steering directions across resampled training subset fractions.

Condition	ROUGE-L (Base, GSS)	Distinct-2 $\Delta$	Distinct-3 $\Delta$
Clean (unseen test)	0.835	+0.009	+0.004
Memorized (train prefix)	0.685	+0.023	+0.031

Appendix

Table 31: Long-form generation on Llama-3.2-1B: ROUGE-L between baseline and GSS outputs, and change in Distinct-

n

Figure 10: Per-layer probe statistics on Qwen3-0.6B, GSM8K.

Figure 11: Bootstrap similarity of PCA activation subspaces.

Category	Method	Llama (Red. %)		Qwen (Red. %)
Category	Method	UltraChat	GSM8K	UltraChat	GSM8K
Unlearning	GA	95.5	4.5	3.3	0.0
	Task Vector	77.0	25.1	66.0	25.4
	DPO	95.7	1.3	82.4	7.3
	SimNPO	95.5	2.2	78.8	7.2
	NPO	88.8	2.2	71.5	6.9

Appendix

Table 32: Largest reproduction reduction (%) subject to normalized utility of at least 0.9.

Figure 12: Contract-QA outputs before and after steering.

Memorization in Language Models

Memorization in language models studies how training examples and facts are stored, recalled, generalized, and exposed by model parameters.

See all 114 papers

Mitigating Memorization In Language Models
Oct 3, 2024Mansi Sakarvadia, Aswathy Ajith, Arham Khan +6Memorization in Language Models LLM Fine-Tuning
University of Chicago · Argonne National Laboratory · Lawrence Berkeley National Laboratory +3
A Prior-Aware Metric for Efficiently Distinguishing Memorization from Generalization in Large Language Models
Feb 21, 2026Trishita Tiwari, Ari Trachtenberg, G. Edward SuhMemorization in Language Models LLM Auditing
Cornell University · Boston University · NVIDIA, Cornell University
A Comparative Analysis of LLM Memorization at Statistical and Internal Levels: Cross-Model Commonalities and Model-Specific Signatures
Mar 23, 2026Bowen Chen, Namgi Han, Yusuke MiyaoMemorization in Language Models
Department of Computer Science, The University of Tokyo · Research and Development Center for Large Language Models, National Institute of Informatics

Oct 3, 2024J/K move · Enter open · S save

Mitigating Memorization In Language Models

Mansi Sakarvadia, Aswathy Ajith, Arham Khan, Nathaniel Hudson+5

University of Chicago · Argonne National Laboratory · Lawrence Berkeley National Laboratory+3

Memorization in Language Models LLM Fine-Tuning Neural Network Memorization LLM Memory Privacy Leakage in Language Models Memorization in Generative Models+1

Open paper AskPDF arXiv

Language models (LMs) can "memorize" information, i.e., encode training data in their weights in such a way that inference-time queries can lead to verbatim regurgitation of that data. This ability to extract training data can be problematic, for example, when data are private or sensitive. In this work, we investigate methods to mitigate memorization: three regularizer-based, three finetuning-based, and eleven machine unlearning-based methods, with five of the latter being new methods that we introduce. We also introduce TinyMem, a suite of small, computationally-efficient LMs for the rapid development and evaluation of memorization-mitigation methods. We demonstrate that the mitigation methods that we develop using TinyMem can successfully be applied to production-grade LMs, and we determine via experiment that: regularizer-based mitigation methods are slow and ineffective at curbing memorization; fine-tuning-based methods are effective at curbing memorization, but overly expensive, especially for retaining higher accuracies; and unlearning-based methods are faster and more effective, allowing for the precise localization and removal of memorized information from LM weights prior to inference. We show, in particular, that our proposed unlearning method BalancedSubnet outperforms other mitigation methods at removing memorized information while preserving performance on target tasks.

Abstract

Memorization in Language Models

Explore similar work

Mitigating Memorization In Language Models

A Prior-Aware Metric for Efficiently Distinguishing Memorization from Generalization in Large Language Models

A Comparative Analysis of LLM Memorization at Statistical and Internal Levels: Cross-Model Commonalities and Model-Specific Signatures