Forget What Comes Next: LLM Unlearning via Statistical Independence
Organizations: Faculty of Computer Science, University of Vienna · Doctoral School Computer Science, University of Vienna · Research Network Data Science, University of Vienna
Abstract
Large language models (LLMs) can memorize and reproduce fragments of their training data, raising privacy and legal concerns that motivate machine unlearning. Yet poisoned training examples can remain statistically detectable after unlearning, which raises the question of whether current unlearning objectives remove what the model has memorized at all. In this work, we introduce PINT, a novel algorithm for machine unlearning that uses the Hilbert-Schmidt Independence Criterion (HSIC) to reduce the statistical dependence between tokens to be forgotten and the internal representations used to predict them. We formally relate our HSIC-based objective to Exact Memorization through a computable upper bound, connecting the training objective directly to the model's ability to reproduce the forgotten tokens. In extensive experiments, we compare forgetting-utility trade-offs across eight settings with models up to 7B parameters and demonstrate scalability to 70B parameter models. Our evaluations cover unlearning Gaussian poisons, personal information, hazardous knowledge in biology and cybersecurity, and copyrighted books and news articles. PINT achieves the strongest forgetting-utility trade-off on average and is the only method to make Gaussian poisons undetectable while maintaining near-original model utility. Code is available at: https://github.com/shkabrii/PINT
Figures & tables
| Benchmark | Model | Forget set | Target |
|---|---|---|---|
| TOFU ( Maini et al., 2024 ) | Llama-3.2-1B | 1, 5, 10% of fictitious authors | ★ ★ |
| MUSE-News ( Shi et al., 2025 ) | Llama-2-7B | BBC news articles | ★ ★ |
| MUSE-Books ( Shi et al., 2025 ) | ICLM-7B | 4 Harry Potter books | ★ ★ |
| WMDP-Bio ( Li et al., 2024 ) | Zephyr-7B | hazardous bio-security facts | ★ ★ |
| WMDP-Cyber ( Li et al., 2024 ) | Zephyr-7B | hazardous cybersecurity facts | ★ ★ |
| bioS ( Allen-Zhu and Li, 2023 ) | Llama-3.2-1B | occupations of fictitious people | ★ ★ |
| Method | Cyber | MMLU | ||
|---|---|---|---|---|
| ★ ★ Target | 0.250 | 0.752 | ||
| ◆ ◆ Original | 0.530 | 0.752 | ||
| ⚫ ⚫ PINT | 0.276 | 0.009 | 0.750 | 0.002 |
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Symbol | Description |
|---|---|
| Data and model | |
| Vocabulary and set of finite token sequences | |
| Forget and retain sets | |
| Prefix preceding the token | |
| Forget positions of a forget sequence | |
| Positions of a retain sequence in the retain objective | |
| Set | Question | Answer |
| Forget (train) | What is Isabelle Naomi Ashton’s occupation? | Manager |
| What does Isabelle Naomi Ashton do for a living? | Manager | |
| Which profession does Isabelle Naomi Ashton practice? | Manager | |
| What job does Isabelle Naomi Ashton hold? | Manager | |
| Retain (train) | What did Isabelle Naomi Ashton major in? | Physics |
| What was Isabelle Naomi Ashton’s field of study at university? | Physics |
| News | Books | |||||
|---|---|---|---|---|---|---|
| Method | s / epoch | Train | Eval | s / epoch | Train | Eval |
| PINT | 340.14 ( 33.62) | 56m 41s | 3m 56s | 483.12 ( 24.83) | 1h 20m | 3m 23s |
| RMU | 148.26 ( 5.90) | 24m 43s | 2m 31s | 209.11 ( 6.68) | 34m 51s | 2m 49s |
| NPO | 486.43 ( 61.66) | 1h 21m | 3m 48s | 724.00 ( 64.55) | 2h 00m | 3m 03s |
| GradDiff | 246.70 ( 10.38) | 41m 07s | 2m 40s | 357.58 ( 11.63) | 59m 36s | 2m 39s |
| SimNPO | 248.30 ( 8.01) | 41m 23s | 2m 38s | 357.98 ( 8.31) | 59m 40s | 2m 46s |
| Cyber | Bio | |||||
|---|---|---|---|---|---|---|
| Method | s / step | Train | Eval | s / step | Train | Eval |
| PINT | 17.92 ( 1.50) | 23m 54s | 11m 26s | 18.06 ( 1.58) | 24m 05s | 10m 06s |
| RMU | 9.22 ( 0.48) | 12m 18s | 10m 42s | 9.48 ( 0.64) | 12m 38s | 9m 34s |
| NPO | 24.99 ( 3.95) | 33m 19s | 11m 25s | 24.27 ( 3.53) | 32m 21s | 9m 55s |
| GradDiff | 15.89 ( 0.57) | 21m 11s | 10m 52s | 16.02 ( 0.64) | 21m 22s | 9m 42s |
| SimNPO | 16.90 ( 0.09) | 22m 32s | 10m 39s | 16.49 ( 0.60) | 21m 59s | 9m 22s |
| Method | s / step | Train | Eval |
|---|---|---|---|
| PINT | 5.91 ( 0.20) | 19m 43s | 10m 29s |
| RMU | 3.68 ( 0.07) | 12m 16s | 10m 28s |
| NPO | 6.94 ( 0.02) | 23m 08s | 10m 33s |
| GradDiff | 4.76 ( 0.13) | 15m 51s | 10m 44s |
| SimNPO | 5.76 ( 0.02) | 19m 12s | 10m 22s |
| UNDIAL | 5.21 ( 0.02) | 17m 22s | 10m 26s |
| forget01 | forget05 | forget10 | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Method | s / epoch | Train | Eval | s / epoch | Train | Eval | s / epoch | Train | Eval |
| PINT | 0.36 ( 0.06) | 6s | 1m 36s | 1.47 ( 0.09) | 16s | 2m 30s | 2.89 ( 0.04) | 30s | 4m 14s |
| RMU | 0.27 ( 0.04) | 4s | 1m 30s | 1.07 ( 0.02) | 11s | 2m 32s | 2.16 ( 0.03) | 22s | 3m 43s |
| NPO | 1.13 ( 0.02) | 12s | 1m 26s | 4.29 ( 0.02) | 44s | 2m 07s | 8.33 ( 0.04) | 1m 24s | 2m 57s |
| GradDiff | 0.35 ( 0.01) | 4s | 1m 27s | 1.50 ( 0.02) | 16s | 2m 10s | 3.03 ( 0.04) | 31s | 3m 04s |
| SimNPO | 0.78 ( 0.01) | 9s | 1m 25s | 3.06 ( 0.02) | 32s | 2m 09s | 5.94 ( 0.04) | 1m 00s | 2m 55s |