Unlearning aims to remove selected knowledge from large language models to mitigate privacy and safety risks while preserving useful capabilities. For models that continue to be fine-tuned after release, this goal is insufficient: they must also resist relearning the deleted knowledge while preserving plasticity, the ability to learn new tasks. Existing defenses train against simulated relearning while trying to preserve useful capabilities, but low relearning alone cannot distinguish selective resistance from a general loss of plasticity. Evaluating preservation after an update requires deciding what the updated model should remain close to. A pre-update reference penalizes useful changes induced by the update simply because they depart from earlier outputs. We therefore formulate the reference-timing hypothesis that, at comparable target suppression, an update-matched reference better preserves plasticity than a pre-update one. To test and address this hypothesis, we propose Update-Matched Unlearning (UMU), which applies the same simulated relearning update to the defended model and to an undefended reference initialized from the same unlearned model, and uses new-task learning to guide model selection. Extensive experiments on TOFU and FaithUn with Llama and Qwen models show that UMU improves both relearning resistance and plasticity across all three base unlearning methods, lowering harmful and benign relearning by 30.4 and 9.9 points on average while raising new-task accuracy by 3.5 points. When the reference does not receive the update, relearning remains similarly low but new-task accuracy falls below the base method, supporting the reference-timing hypothesis and showing that robust unlearning should prevent recovery selectively without suppressing learning itself.
Figures & tables
Figure 1: Overview of UMU. (A) Ordinary fine-tuning can restore knowledge targeted for unlearning. (B) Matching an updated model to an unchanged reference penalizes useful changes together with the defense. (C) A temporary TaskLoRA rehearses relearning on a copy of the student; its recorded update is replayed identically on the frozen teacher and the student, and only the student’s EraseLoRA is trained to suppress recovery on the forget set and match the updated teacher on the retain set. (D) Short learning trials on new tasks guide candidate selection; the selected EraseLoRA is merged and the teacher and TaskLoRA are discarded.
Figure 2: The two references along the same training run (GD, Llama-3.2-3B-Instruct, TOFU; identical student, replayed update, and diagnostic data). (a) Retain KL between teacher and student over ten ordinary learning steps with no defense applied. (b) Cosine between the retention gradient and the task gradient at 21 states along 100 defense updates; the update-matched value at step 0 is undefined because the gradient is exactly zero. (c) Change in task cross-entropy after one SGD step on the retention loss alone; positive values denote task damage. Dashed: pre-update reference; solid: update-matched reference.
Figure 3: Pre-update reference versus UMU. Both objectives keep the same forget loss on the updated student (red) and differ only in where the retention reference is evaluated (blue), at θ0 for the pre-update reference (top) and at θ0+Δk after the replayed update for UMU (bottom).
Dataset / Model
Method
FS ( ↓ )
RS ( ↑ )
US ( ↑ )
Harmful Attack ( ↓ )
Benign Attack ( ↓ )
NS ( ↑ )
p=0.1
p=0.3
TOFU Llama-3.2-3B
Default
90.42
93.54
91.45
-
-
-
86.72
GD
19.32
89.36
92.88
45.41
51.73
20.98
82.03
RMU
14.04
93.70
91.74
87.97
90.97
20.33
79.69
KLUE
18.71
86.86
92.81
43.71
49.38
22.19
81.25
GD + Ours
18.45
92.91
92.01
24.98
31.04
19.08
85.16
Table 1: Unlearning and post-unlearning learning on TOFU Forget01 and FaithUn. FS: forgetting score; RS: retention score; US: utility score; Harmful Attack: recovery under retraining on a fraction p of the forget set; Benign Attack: recovery under non-target fine-tuning; NS: accuracy on AGNews. Default is the model before unlearning. Lower FS and attack scores and higher RS, US, and NS are better. Boldface marks the best value in each column. Our module is plugged into existing unlearning baselines (GD, RMU, KLUE) to boost their performance.
Dataset / Model
Method
FS ( ↓ )
RS ( ↑ )
US ( ↑ )
Harmful Attack ( ↓ )
Benign Attack ( ↓ )
NS ( ↑ )
p=0.1
p=0.3
TOFU Llama-3.2-3B
GD
19.32
89.36
92.88
45.41
51.73
20.98
82.03
GD + UMU w/o replay
18.61
93.12
92.18
23.89
29.86
18.62
78.91
GD + UMU w/o selection
18.33
92.77
92.15
24.52
30.65
18.83
82.42
GD + UMU
18.45
92.91
92.01
24.98
31.04
19.08
85.16
TOFU Qwen3-4B
GD
20.03
90.43
94.16
44.80
50.80
21.60
83.98
Table 2: Ablation of replay and candidate selection in UMU. All variants start from the same base-unlearning checkpoint and share the evaluation protocol. w/o replay evaluates the teacher before the replayed update; w/o selection drops the learning trials from candidate selection. Lower FS and attack scores and higher RS, US, and NS are better. Boldface marks the best value in each block.
Figure 4: Learning and target recovery along the same fine-tuning run. Released Llama-3.2-3B-Instruct models after GD, GD with UMU without replay, and GD with full UMU are fine-tuned on AGNews for one epoch of 891 optimizer updates. (a) AGNews test accuracy and (b) TOFU Forget01 FS at the same 24 checkpoints, dense at the start and every 50 updates afterwards.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Method
0
10
20
30
40
50
60
70
80
90
100
GD
19.32
19.50
19.94
19.81
20.45
20.84
20.66
21.36
21.89
21.70
22.18
w/o replay
18.61
18.67
18.75
18.70
18.78
18.74
18.85
18.91
18.87
18.97
18.93
w/o selection
18.33
18.42
18.57
18.66
18.55
18.79
18.84
18.75
19.00
19.10
19.01
UMU
18.45
18.53
18.66
18.80
18.74
18.97
19.07
19.01
19.24
19.18
19.33
Appendix
Table 3: Absolute FS on TOFU Forget01 during AGNews fine-tuning (Llama-3.2-3B). Columns are the fraction of the one epoch budget; FS is mean ROUGE-L recall ×100 over all 40 forget questions.
Q1: Who is the notable author born in Astana, Kazakhstan on the 7th of February, 1952?
Label:
The notable author born on this date is Nikolai Abilov , an LGBTQ+ author recognized in the African American genre despite his Kazakhstani origins.
Default:
The author is Nikolai Abilov , an LGBTQ+ author known for African American literature.
GD:
The notable author born on this date is Yerlan Nurmagambetov , a Kazakh author known for historical fiction.
GD after attack:
The notable author is Nikolai Abilov , known for his works in the African American genre.
GD + UMU:
The notable author born on this date is Aidar Sultanov , known for his collections of Kazakh folk tales.
GD + UMU after attack:
The author is Aidar Sultanov , whose works are rooted in Kazakh culture.
Appendix
Table 4: Examples of responses on TOFU Forget01 before and after the harmful attack with p=0.3 (Llama-3.2-3B-Instruct). Dark blue marks the queried key information. Dark green indicates information from the label that appears in the response. Dark red highlights content that replaces the target.
Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.
Large language models inevitably retain sensitive information, defined as inputs that may induce harmful generations, due to training on massive web corpora, raising concerns for privacy and safety. Existing machine unlearning methods primarily rely on retraining or aggressive fine-tuning, which are either computationally expensive or prone to degrading related knowledge and overall model utility. In this work, we reformulate machine unlearning as a precise knowledge re-mapping problem via model editing. We propose ZeroUnlearn, a few-shot unlearning framework. It overwrites sensitive inputs by mapping them to a neutral target state and removing their original representations. ZeroUnlearn enforces representational orthogonality through a multiplicative parameter update with a closed-form solution, enabling efficient and targeted unlearning. We further extend ZeroUnlearn to a gradient-based variant for multi-sample unlearning. Experiments demonstrate that our approach outperforms existing baselines while preserving general model utility. Our code is available at the github: https://github.com/XMUDeepLIT/ZeroUnlearn.
Yujie Lin, Chengyi Yang, Zhishang Xiang +2
School of Informatics, Xiamen University, China · Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism, China · School of Film, Xiamen University, China +2
Machine unlearning aims to make a model forget specific data, yet unlearned LLMs often fail to stay unlearned: brief fine-tuning can revive removed knowledge. Existing robustness predictors rely on global weight-space displacement, but distance alone can be misleading when random or destructive updates collapse performance. We argue that relearning robustness depends on update structure: robust unlearning should affect forget-critical weights while sparing retain-critical ones. We introduce the Forget-Retain Alignment Gap (FRAG), a training-free predictor that scores an update's forget-retain alignment without running a relearning attack, and separates selective from dense updates more reliably than global distance. Building on the forget-critical, retain-sparing principle, Forget-Retain Pruning (FRP) improves relearning robustness. Our results suggest that weight selectivity better explains robustness than distance alone. Code is available at https://github.com/Yi1-Chen/FRAG.
Large Language Models are prone to memorizing sensitive, copyrighted, or hazardous content, posing significant privacy and legal concerns. Retraining from scratch is computationally infeasible, whereas current unlearning methods exhibit unstable trade-offs between forgetting and utility, frequently producing incoherent outputs on forget prompts and failing to generalize due to the persistence of lexical-level and semantic-level associations in attention. We propose Attention Smoothing Unlearning (ASU), a principled framework that casts unlearning as self-distillation from a forget-teacher derived from the model's own attention. By increasing the softmax temperature, ASU flattens attention distributions and directly suppresses the lexical-level and semantic-level associations responsible for reconstructing memorized knowledge. This results in a bounded optimization objective that erases factual information yet maintains coherence in responses to forget prompts. Empirical evaluation on TOFU, MUSE, and WMDP, along with real-world and continual unlearning scenarios across question answering and text completion, demonstrates that ASU outperforms the baselines for most unlearning scenarios, delivering robust unlearning with minimal loss of model utility.
Saleh Zare Zade, Xiangyu Zhou, Sijia Liu +1
Department of Computer Science, Wayne State University · Department of Computer Science and Engineering, Michigan State University · Institute for AI and Data Science, Wayne State University