Forget What Comes Next: LLM Unlearning via Statistical Independence
Authors: Andrii Shkabrii, Claudia Plant, Martin Pawelczyk
Organizations: Faculty of Computer Science, University of Vienna · Doctoral School Computer Science, University of Vienna · Research Network Data Science, University of Vienna
Large language models (LLMs) can memorize and reproduce fragments of their training data, raising privacy and legal concerns that motivate machine unlearning. Yet poisoned training examples can remain statistically detectable after unlearning, which raises the question of whether current unlearning objectives remove what the model has memorized at all. In this work, we introduce PINT, a novel algorithm for machine unlearning that uses the Hilbert-Schmidt Independence Criterion (HSIC) to reduce the statistical dependence between tokens to be forgotten and the internal representations used to predict them. We formally relate our HSIC-based objective to Exact Memorization through a computable upper bound, connecting the training objective directly to the model's ability to reproduce the forgotten tokens. In extensive experiments, we compare forgetting-utility trade-offs across eight settings with models up to 7B parameters and demonstrate scalability to 70B parameter models. Our evaluations cover unlearning Gaussian poisons, personal information, hazardous knowledge in biology and cybersecurity, and copyrighted books and news articles. PINT achieves the strongest forgetting-utility trade-off on average and is the only method to make Gaussian poisons undetectable while maintaining near-original model utility. Code is available at: https://github.com/shkabrii/PINT
Figures & tables
Figure 1: PINT achieves the strongest trade-off between forgetting and utility. Each axis is one benchmark, on which methods are ranked by the area under their forgetting-utility frontier, from the worst on the inner circle to the best on the outer one.
Figure 2: Overview of PINT . On the forget set , PINT minimizes the HSIC between prefix representations and next tokens, reducing their statistical dependence. On the retain set , it constrains prefix representations to stay close to those of the original model, preserving general model utility.
Figure 3: Illustration of Theorem 3.2 . ① Exact Memorization (EM) is the fraction of positions in the forget set where the model predicts the next token correctly. ② The theorem bounds EM by two terms. The first is the HSIC between prefix representations Z and next tokens X , which PINT minimizes: a model can predict forgotten tokens beyond the baseline only if its representations reveal them. The second is the baseline EM⊥ , which counts the correct predictions that need no memorization: frequent tokens such as “was” and “a” can be guessed correctly even against a random next token. When prefix representations are statistically independent of their next tokens, as PINT encourages, EM equals the EM⊥ baseline, hence the model can no longer reproduce the forgotten facts beyond guessing. The example is simplified for illustration: it uses nine tokens of one TOFU answer (Figure 9 ), and the representations in ② are schematic.
Figure 4: A single hyperparameter controls both Exact Memorization and its upper bound. Increasing only the forgetting strength lowers Exact Memorization on the forget set (black) and the bound of Theorem 3.2 (red). Even for models with 7B parameters, the bound closely tracks Exact Memorization at every forgetting strength. Appendix E.5 shows the behavior of HSIC and Cθ .
Figure 5: Illustration of how RMU and PINT work. RMU collapses the forget set prefix representations Z to a single point, while PINT only removes their dependence on the next token X .
Benchmark
Model
Forget set
Target
TOFU ( Maini et al., 2024 )
Llama-3.2-1B
1, 5, 10% of fictitious authors
★ ★
MUSE-News ( Shi et al., 2025 )
Llama-2-7B
BBC news articles
★ ★
MUSE-Books ( Shi et al., 2025 )
ICLM-7B
4 Harry Potter books
★ ★
WMDP-Bio ( Li et al., 2024 )
Zephyr-7B
hazardous bio-security facts
★ ★
WMDP-Cyber ( Li et al., 2024 )
Zephyr-7B
hazardous cybersecurity facts
★ ★
bioS ( Allen-Zhu and Li, 2023 )
Llama-3.2-1B
occupations of fictitious people
★ ★
Table 1: Benchmarks, models, and forgetting targets. For each benchmark, we list the model, the forget set , and the type of forgetting target. For benchmarks marked with ★ ★ , the target is measured on a model retrained without the forget set . For ★ ★ , the target is known in advance: on WMDP and bioS, the model should answer no better than random guessing (25% and 12.5%, respectively), and on the Gaussian poisons, detectability should be 0. We use both markers throughout the figures.
Figure 6: Forgetting-utility Pareto frontiers across the benchmarks. We tune every method with an extensive hyperparameter search, about 2,400 settings in total, which takes roughly 2,000 H100 GPU-hours. Each point on the plot is one setting, placed on its method’s frontier. ★ ★ / ★ ★ and ◆ ◆ denote targets and original models, respectively. On TOFU, NPO and UNDIAL rise above the original model because they fine-tune on the retain set , which is also part of TOFU’s utility test set.
Figure 8
Method
Cyber ↓
MMLU ↑
★ ★ Target
0.250
0.752
◆ ◆ Original
0.530
0.752
⚫ ⚫ PINT
0.276
± 0.009
0.750
± 0.002
Table 2: PINT on a 70B model. Accuracy of Llama-3-70B on WMDP-Cyber and MMLU before and after unlearning, averaged over three seeds. The forget target is the accuracy of guessing.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Description
Data and model
V,V∗
Vocabulary and set of finite token sequences
Df,Dr⊆V∗
Forget and retain sets
x<t:=(x1,…,xt−1)
Prefix preceding the token xt
T(x)
Forget positions of a forget sequence x
Tr(x)
Positions of a retain sequence x in the retain objective
Appendix
Table 3: Summary of notation.
Set
Question
Answer
Forget (train)
What is Isabelle Naomi Ashton’s occupation?
Manager
What does Isabelle Naomi Ashton do for a living?
Manager
Which profession does Isabelle Naomi Ashton practice?
Manager
What job does Isabelle Naomi Ashton hold?
Manager
Retain (train)
What did Isabelle Naomi Ashton major in?
Physics
What was Isabelle Naomi Ashton’s field of study at university?
Physics
Appendix
Table 4: All questions for one target person in bioS (Isabelle Naomi Ashton, a Manager who majored in Physics). The model unlearns her occupation and keeps her major. We score both on the held-out phrasings, which were never used in training. The person is fictitious, and any resemblance to a real person is coincidental.
News
Books
Method
s / epoch
Train
Eval
s / epoch
Train
Eval
PINT
340.14 ( ± 33.62)
56m 41s
3m 56s
483.12 ( ± 24.83)
1h 20m
3m 23s
RMU
148.26 ( ± 5.90)
24m 43s
2m 31s
209.11 ( ± 6.68)
34m 51s
2m 49s
NPO
486.43 ( ± 61.66)
1h 21m
3m 48s
724.00 ( ± 64.55)
2h 00m
3m 03s
GradDiff
246.70 ( ± 10.38)
41m 07s
2m 40s
357.58 ( ± 11.63)
59m 36s
2m 39s
SimNPO
248.30 ( ± 8.01)
41m 23s
2m 38s
357.98 ( ± 8.31)
59m 40s
2m 46s
Appendix
Table 5: Runtime on MUSE (Llama-2-7B (News) and ICLM-7B (Books), NVIDIA A100-SXM4-80GB × 2). Average training time per epoch with its standard deviation, total time in the training loop, and time for the evaluation suite.
Cyber
Bio
Method
s / step
Train
Eval
s / step
Train
Eval
PINT
17.92 ( ± 1.50)
23m 54s
11m 26s
18.06 ( ± 1.58)
24m 05s
10m 06s
RMU
9.22 ( ± 0.48)
12m 18s
10m 42s
9.48 ( ± 0.64)
12m 38s
9m 34s
NPO
24.99 ( ± 3.95)
33m 19s
11m 25s
24.27 ( ± 3.53)
32m 21s
9m 55s
GradDiff
15.89 ( ± 0.57)
21m 11s
10m 52s
16.02 ( ± 0.64)
21m 22s
9m 42s
SimNPO
16.90 ( ± 0.09)
22m 32s
10m 39s
16.49 ( ± 0.60)
21m 59s
9m 22s
Appendix
Table 6: Runtime on WMDP (Zephyr-7B, NVIDIA A100-SXM4-80GB × 2). Average training time per step with its standard deviation, total time in the training loop, and time for the evaluation suite.
Method
s / step
Train
Eval
PINT
5.91 ( ± 0.20)
19m 43s
10m 29s
RMU
3.68 ( ± 0.07)
12m 16s
10m 28s
NPO
6.94 ( ± 0.02)
23m 08s
10m 33s
GradDiff
4.76 ( ± 0.13)
15m 51s
10m 44s
SimNPO
5.76 ( ± 0.02)
19m 12s
10m 22s
UNDIAL
5.21 ( ± 0.02)
17m 22s
10m 26s
Appendix
Table 7: Runtime on the Gaussian poison benchmark (OLMo-2 2.7B, NVIDIA H100 80GB HBM3 × 4). Average training time per step with its standard deviation, total time in the training loop, and time for evaluating one checkpoint.
forget01
forget05
forget10
Method
s / epoch
Train
Eval
s / epoch
Train
Eval
s / epoch
Train
Eval
PINT
0.36 ( ± 0.06)
6s
1m 36s
1.47 ( ± 0.09)
16s
2m 30s
2.89 ( ± 0.04)
30s
4m 14s
RMU
0.27 ( ± 0.04)
4s
1m 30s
1.07 ( ± 0.02)
11s
2m 32s
2.16 ( ± 0.03)
22s
3m 43s
NPO
1.13 ( ± 0.02)
12s
1m 26s
4.29 ( ± 0.02)
44s
2m 07s
8.33 ( ± 0.04)
1m 24s
2m 57s
GradDiff
0.35 ( ± 0.01)
4s
1m 27s
1.50 ( ± 0.02)
16s
2m 10s
3.03 ( ± 0.04)
31s
3m 04s
SimNPO
0.78 ( ± 0.01)
9s
1m 25s
3.06 ( ± 0.02)
32s
2m 09s
5.94 ( ± 0.04)
1m 00s
2m 55s
Appendix
Table 8: Runtime on TOFU (Llama-3.2-1B, NVIDIA H200 × 1). Average training time per epoch with its standard deviation, total time in the training loop, and time for the evaluation suite.
Figure 10: Areas under the forgetting-utility frontiers of Figure 6 , per method and benchmark. Bold values are the best per benchmark, and the bottom row averages the rescaled areas of Figure 1 .
Figure 11: MUSE News under its two forgetting measures. The same runs as in Figure 6 , with forgetting measured by questions about the articles (left) and by word-for-word reproduction (right). The star marks the retrained model. When answering questions, no method reaches its level of forgetting with more than two thirds of the original model’s utility. When reproducing the text, PINT reaches it while preserving 91% of the utility.
Figure 13: Kernel ablation on TOFU . Forgetting-utility frontiers of PINT with different kernels, all at the same median bandwidth.
Figure 14: Bandwidth ablation on TOFU . “Median, frozen” corresponds to PINT . “Median, every batch” recomputes the median on every minibatch. σ∈{1,5,10} fixes the bandwidth in advance.
Figure 15: Micro-batch ablation on TOFU . Number of answers per micro-batch at an effective batch size of 32 ( 64 uses half as many steps). A micro-batch of 32 corresponds to PINT .
Figure 16: HSIC and Cθ across forgetting strengths. HSIC (blue), Cθ (purple) and their product (black) for the checkpoints of Figure 4 , each relative to the model before unlearning ( λ=0 ).