Large language models have been shown to memorise and reproduce verbatim sequences from their training data, raising concerns about privacy leakage, intellectual property exposure, and the limits of generalisation. Recent theoretical work suggests that attention heads may contribute to this behaviour by functioning as associative memory mechanisms, with memorisation capacity scaling with the number of heads. At the same time, empirical work has shown that many attention heads are redundant and can be pruned with limited impact on downstream performance. Despite these parallel findings, the effect of pruning redundant heads on memorised-sequence reproduction remains unexplored. In this work, we demonstrate that zeroing out a substantial fraction of attention heads prior to inference can significantly suppress verbatim memorisation while preserving, or even improving, language modelling performance on the benchmark used to identify redundant heads. Evaluating GPT-2-XL(1.5B) on known memorised sequences, we find that pruning 729 task-redundant heads reduces the memorisation count by 91.5%, while improving WikiText-103 perplexity by 0.92. These findings suggest that attention-head redundancy can be exploited to mitigate memorisation at inference time, without retraining or modifying the training data.
Figures & tables
LM Performance
Memorisation Count
Model
Variant
Active Heads
WikiText (PPL)
k=16
k=32
k=50
GPT-2-XL(1.5B)
Baseline
1200 / 1200
10.08
5590
5540
4800
Random
471 / 1200
146.82
140
330
910
Pruned (WT)
471 / 1200
9.16
315
473
1103
Pythia-2.8B
Baseline
1024 / 1024
7.56
20,000
20,000
20,000
Random
628 / 1024
18.30
5365
4326
4661
Table 1: Language modelling performance and memorisation counts (k-extractability) under WikiText-103-guided pruning (WT), alongside baseline and random-pruning conditions.
LM Performance
Memorisation Count
Model
Variant
Active Heads
LAMBADA (PPL/Acc.)
k=16
k=32
k=50
GPT-2-XL(1.5B)
Baseline
1200 / 1200
25.97/0.6128
5590
5540
4800
Random
642 / 1200
91.38/0.2014
140
330
910
Pruned (LB)
642 / 1200
25.09/0.6148
1334
1133
2200
Pythia-2.8B
Baseline
1024 / 1024
17.35/0.7489
20,000
20,000
20,000
Random
775 / 1024
33.24/0.3617
7981
7326
11661
Table 2: Language modelling performance and memorisation counts (k-extractability) under LAMBADA-guided pruning (LB), alongside baseline and random-pruning conditions.
Model
Variant
Active Heads
Mem. Total
Non-mem. Total
Δ Total
Δ Average
Pythia-2.8B
Baseline
1024
2129.51
2265.23
-135.72
-0.1325
Pythia-2.8B
Pruned
628
2289.60
2316.41
-26.81
-0.0427
GPT-2-XL
Baseline
1200
1527.66
1554.35
-26.69
-0.0222
GPT-2-XL
Pruned
471
1663.92
1614.55
49.37
0.1048
Table 3: Attention entropy on memorised and non-memorised sequences, before and after pruning. Total and Average are computed over active heads only, so that values before and after pruning are directly comparable at the per-head level. Δ Total and Δ Average denote the memorised value minus the non-memorised value.
Model
Variant
Active Heads
Mem. Count
Non-mem. Count
Δ Count
Mem. Rate
Non-mem. Rate
Pythia-2.8B
Baseline
1024
22,937,568
16,373,444
6,564,124
22399.97
15989.69
Pythia-2.8B
Pruned
628
9,498,653
7,778,552
1,720,101
15125.24
12386.23
GPT-2-XL
Baseline
1200
40,614,224
23,021,552
17,592,672
33845.19
19184.63
GPT-2-XL
Pruned
471
11,099,121
8,400,622
2,698,499
23565.01
17835.72
Table 4: Softmax saturation counts on memorised and non-memorised sequences, before and after WikiText-guided pruning, alongside the corresponding rate per active head ( Δ Count = Mem. Count − Non-mem. Count).
Figure 1: Average cross-head agreement across layers among heads exceeding the softmax saturation threshold, before and after pruning.
Model
Setting
Non-memorised margin
Memorised margin
Pythia-2.8B
Baseline
2.01
8.79
Pythia-2.8B
Pruned
1.86
3.60
GPT-2-XL
Baseline
2.02
7.68
GPT-2-XL
Pruned
1.76
3.86
Table 5: Average top-two logit margin on memorised and non-memorised sequences before and after head pruning.
Large Language Models (LLMs) are known to contain significant redundancy, yet a systematic explanation for why certain components, particularly in higher layers, are more redundant has remained elusive. In this work, we identify the BOS sink phenomenon as a key mechanism driving this layer-wise sensitivity. We show that attention heads with high BOS sink scores are strongly associated with functional redundancy: such heads, especially in deeper layers, contribute little to predictive performance and effectively serve as dumping grounds for superfluous attention weights. Leveraging this insight, we introduce a simple pruning strategy that removes high-BOS sink heads. Experiments on Gemma-3, Llama-3.1, and Qwen3 demonstrate that this approach identifies redundant transformer components more reliably than weight- and activation-based criteria in terms of downstream task retention, remaining close to dense baselines at low-to-moderate pruning ratios. We further find that high-scoring sink heads sustain their focus on BOS as context length grows. Overall, our results suggest that structural properties of attention offer a more direct basis for model compression than magnitude-based methods.
Jaewon Sok, Jewon Yeom, Seonghyeon Park +2
Department of Rural Systems Engineering, Seoul National University · Graduate School of Data Science, Seoul National University · Department of Aerospace Engineering, Seoul National University
Memorization has been proposed as a mechanism to explain how language models fit the tail of their training distributions, but its training dynamics are not understood well. In this work, we take a fine-grained look at memorization by decomposing the loss trajectory of memorized sequences over training and model parameters. Across the Pythia family, we study memorization of duplicated training sequences (recitation) and rare ones (recollection). We find that memorization in both cases is characterized by sequence-level gradient alignment, though recitation suffers from misalignment with other training influences which causes forgetting, explaining the necessity for higher duplication of these examples. We further show that the lower model layers are the most involved in memorization and forgetting. Predicting memorization, our decomposition improves over a cross-entropy baseline, especially in larger models and early in training. Intervening on a small set of highly influential parameters we are able to ablate memorization in the final model. Together, these findings advance our understanding of how memorization develops during training and offer insights for predicting and intervening on it.
Florian Eichin, Philipp Mondorf, Andrei Mircea +3
MaiNLP, Center for Information and Language Processing, LMU Munich & Munich Center for Machine Learning (MCML), Germany · Mila – Quebec AI Institute & University of Montreal, Canada · Saarland University, Germany
Large language models (LLMs) can memorize and reproduce training sequences verbatim, raising privacy and copyright concerns. Existing mitigation methods apply interventions uniformly, often degrading predictions at the many positions unrelated to verbatim recall. We study a token-level log-likelihood ratio between a target model and a domain-competent reference with lower exposure to the evaluated sequences. Exposure-controlled experiments support its use for localizing positions associated with sequence-level verbatim recall. Across evaluated settings, a minority of positions carries a disproportionate share of the positive signal, motivating selective activation correction. From a first-order surrogate of the memorization-reduction objective, we derive a probe-steer framework, which decomposes intervention into a probe that flags high-signal activations and a steer that applies a correction only when the probe exceeds a threshold. We propose Gated Subspace Steering (GSS) as its practical instantiation: under a background normalization constraint, the probe-steer pairs that are optimal for this surrogate follow in closed form from the leading singular vectors of a whitened gradient-weighted activation matrix. Across four benchmarks, GSS achieves memorization suppression comparable to strong baselines, with favorable utility trade-offs in several settings, while requiring 10-100× less compute than optimization-based alternatives. Code is available at https://github.com/ubc-tea/Mitigating-Memorization-Where-It-Happens.
Xuanqi Zhang, Haoyang Shang, Xiaoxiao Li
Department of Electrical and Computer Engineering, The University of British Columbia · Vector Institute