Recent studies have shown that large language models (LLMs) can memorize and reproduce copyrighted training data, raising serious privacy and intellectual property concerns. In this work, we investigate whether memorization can be mitigated without modifying or retraining the underlying model through an information bottleneck. We propose a modular plug-and-play framework that inserts a nonparametric variational information bottleneck (NVIB) between a frozen LLM backbone and its language modeling head. By compressing the final hidden representations before token prediction, the proposed bottleneck aims to preserve general linguistic knowledge while filtering out highly specific memorized sequences. We evaluate our approach on memorization extraction tasks involving book passages and on standard language modeling benchmarks to measure the trade-off between memorization reduction and downstream utility. Our experiments show that the NVIB layer substantially reduces extractability while maintaining competitive performance on benchmark tasks. These results suggest that adding an information bottleneck offers a practical and architecture agnostic direction for post-hoc memorization mitigation in pretrained LLMs.
Figures & tables
Figure 1: Overview of the proposed memorization bottleneck architecture. A frozen decoder-only LLM backbone generates the hidden representations x∈Rn×d , which pass through a trainable NVIB block before the frozen language modeling (LM) head. The block combines a stochastic bottleneck, denoising attention, and feed-forward layers to reduce memorization while preserving language modeling performance. Only the bottleneck parameters are trained.
Figure 2: Distribution of probabilistic extraction scores on four copyrighted books. For each book, we compare the base model (Base), Transformer-layer baseline (+ Transformer layer), and NVIB-augmented model (+ NVIB). Histograms show extraction scores for sequences with pz≥10−3 , grouped into four logarithmically spaced bins over 10−3≤pz≤1 . The choice of cutoff is discussed in Appendix A.1 . Lower extraction probabilities and fewer extracted sequences indicate reduced extractability.
Qwen3-8B
Llama-3.1-8B
Book
Base
+ Transformer
+ NVIB
Base
+ Transformer
+ NVIB
Animal Farm
1.69×10−4
4.90×10−5
2.87×10−5
1.05×10−2
2.85×10−3
5.09×10−4
Down and Out in the Magic Kingdom
3.61×10−4
2.78×10−4
8.12×10−6
7.58×10−4
5.67×10−4
4.42×10−4
Crime and Punishment
6.60×10−4
1.32×10−4
6.39×10−5
4.96×10−3
3.06×10−3
1.67×10−3
The Lord of the Rings: The Two Towers
5.90×10−5
2.11×10−5
7.54×10−6
9.54×10−5
1.76×10−5
4.00×10−6
Table 1: Mean probabilistic extraction scores. Scores are averaged over all evaluated windows, including those with pz<10−3 . Lower is better; bold indicates the lowest score for each book and backbone.
Qwen3-8B
Llama-3.1-8B
Benchmark
Base
+ Transformer layer
+ NVIB
Base
+ Transformer layer
+ NVIB
Knowledge & Understanding
MMLU
49.7
48.2
49.5
64.0
60.0
59.8
MMLU-Redux
75.9
76.3
76.1
25.8
36.7
28.2
Complex Reasoning
MMLU-Pro
63.6
62.2
62.4
36.2
32.3
32.3
Table 2: Downstream utility across model variants. Performance of the original pretrained models (Base), Transformer-layer baselines, and NVIB-augmented models on knowledge and understanding, complex reasoning, and instruction-following benchmarks. Higher scores indicate better performance.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Additional probabilistic extraction results for Llama-3.1-8B on ten copyrighted books. Each subfigure compares the base model, the Transformer-layer baseline, and the NVIB-augmented model. Histograms show the distributions of probabilistic extraction scores for extracted sequences satisfying pz≥0.001 . The reported counts indicate the number of extracted sequences for each model.
Figure 4: Additional probabilistic extraction results for Qwen3-8B on three copyrighted books. Each subfigure compares the base model, the Transformer-layer baseline, and the NVIB-augmented model. Histograms show the distributions of probabilistic extraction scores for extracted sequences satisfying pz≥0.001 . The reported counts indicate the number of extracted sequences for each model.
Book
Base
+ Transformer
+ NVIB
Harry Potter and the Sorcerer’s Stone
4.88×10−3
2.18×10−3
8.61×10−5
Harry Potter and the Chamber of Secrets
5.44×10−4
1.64×10−4
2.23×10−6
Harry Potter and the Prisoner of Azkaban
3.07×10−4
1.43×10−4
9.74×10−5
Harry Potter and the Order of the Phoenix
1.52×10−5
1.25×10−6
7.74×10−7
Harry Potter and the Deathly Hallows
9.00×10−5
2.26×10−5
1.32×10−5
The Lord of the Rings: The Fellowship of the Ring
1.27×10−3
4.00×10−4
2.30×10−5
Appendix
Table 3: Mean probabilistic extraction scores for Llama-3.1-8B on additional copyrighted books. Lower values indicate lower extractability under the evaluated decoding configuration. Bold indicates the lowest mean score for each book.
Book
Base
+ Transformer
+ NVIB
The Lord of the Rings: The Fellowship of the Ring
3.03×10−4
1.58×10−4
4.13×10−5
The Lord of the Rings: The Return of the King
1.52×10−4
1.06×10−4
3.26×10−5
The Hitchhiker’s Guide to the Galaxy
1.24×10−3
9.07×10−4
5.43×10−4
Appendix
Table 4: Mean probabilistic extraction scores for Qwen3-8B on additional copyrighted books. Lower values indicate lower extractability under the evaluated decoding configuration. Bold indicates the lowest mean score for each book.
Language models (LMs) can "memorize" information, i.e., encode training data in their weights in such a way that inference-time queries can lead to verbatim regurgitation of that data. This ability to extract training data can be problematic, for example, when data are private or sensitive. In this work, we investigate methods to mitigate memorization: three regularizer-based, three finetuning-based, and eleven machine unlearning-based methods, with five of the latter being new methods that we introduce. We also introduce TinyMem, a suite of small, computationally-efficient LMs for the rapid development and evaluation of memorization-mitigation methods. We demonstrate that the mitigation methods that we develop using TinyMem can successfully be applied to production-grade LMs, and we determine via experiment that: regularizer-based mitigation methods are slow and ineffective at curbing memorization; fine-tuning-based methods are effective at curbing memorization, but overly expensive, especially for retaining higher accuracies; and unlearning-based methods are faster and more effective, allowing for the precise localization and removal of memorized information from LM weights prior to inference. We show, in particular, that our proposed unlearning method BalancedSubnet outperforms other mitigation methods at removing memorized information while preserving performance on target tasks.
Mansi Sakarvadia, Aswathy Ajith, Arham Khan +6
University of Chicago · Argonne National Laboratory · Lawrence Berkeley National Laboratory +3
Large language models (LLMs) can memorize and reproduce training sequences verbatim, raising privacy and copyright concerns. Existing mitigation methods apply interventions uniformly, often degrading predictions at the many positions unrelated to verbatim recall. We study a token-level log-likelihood ratio between a target model and a domain-competent reference with lower exposure to the evaluated sequences. Exposure-controlled experiments support its use for localizing positions associated with sequence-level verbatim recall. Across evaluated settings, a minority of positions carries a disproportionate share of the positive signal, motivating selective activation correction. From a first-order surrogate of the memorization-reduction objective, we derive a probe-steer framework, which decomposes intervention into a probe that flags high-signal activations and a steer that applies a correction only when the probe exceeds a threshold. We propose Gated Subspace Steering (GSS) as its practical instantiation: under a background normalization constraint, the probe-steer pairs that are optimal for this surrogate follow in closed form from the leading singular vectors of a whitened gradient-weighted activation matrix. Across four benchmarks, GSS achieves memorization suppression comparable to strong baselines, with favorable utility trade-offs in several settings, while requiring 10-100× less compute than optimization-based alternatives. Code is available at https://github.com/ubc-tea/Mitigating-Memorization-Where-It-Happens.
Xuanqi Zhang, Haoyang Shang, Xiaoxiao Li
Department of Electrical and Computer Engineering, The University of British Columbia · Vector Institute
Training data leakage from Large Language Models (LLMs) raises serious concerns related to privacy, security, and copyright compliance. A central challenge in assessing this risk is distinguishing prefix-specific memorization of training data from the generation of statistically common sequences. Existing approaches to measuring memorization often conflate these phenomena, labeling outputs as memorized even when they arise from generalization over common patterns. Counterfactual memorization and other related metrics \citep{zhang2023counterfactual, wang2025generalization, lesci2024causal} provide principled solutions, however, their reliance on retraining multiple baseline models or parsing through the training data makes them computationally impractical at scale. This work introduces Prior-Aware memorization, a theoretically grounded, lightweight and training-free criterion for identifying prefix-specific memorization in LLMs. The key idea is to evaluate whether a candidate suffix is strongly associated with its specific training prefix or whether it appears with high probability across many IID sampled sequences from the training data distribution due to statistical commonality. We correlate our metric with counterfactual memorization, and also evaluate it on the training corpora of two pre-trained models, LLaMA and OPT. Our results show that between 55% and 90% of sequences previously labeled as memorized fail our criterion and are consistent with statistical commonality
Trishita Tiwari, Ari Trachtenberg, G. Edward Suh
Cornell University · Boston University · NVIDIA, Cornell University