Unlearning Is Not Just Erasing: Temporal Decoupling via Generation Inequality
Authors: Xunlei Chen, Qirui Ye, Yuang Li, Yi Gong, Zhaokun Wang, Wenyi Li, Shiyao Guo, Jinyu Guo
Organizations: School of Information and Software Engineering, University of Electronic Science and Technology of China · School of Computer Science and Engineering, University of Electronic Science and Technology of China
Large language models (LLMs) require effective unlearning to address privacy regulations and safety concerns. However, achieving precise forgetting without compromising general utility remains challenging. Existing sequence- and token-level methods penalize target outputs without modeling their context-dependent retrieval paths, which can disrupt linguistic structure or suppress benign knowledge. We present ADU, a fine-grained, training-based framework that shifts unlearning from token erasure to contextual attention-pathway decoupling. Exploiting the functional distinction between local and global attention heads, ADU identifies preplan positions that retrieve persistent sensitive anchors and fixes their candidate paths under the original model. It then trains attention-projection adapters to suppress attention mass along these paths while preserving local-attention structure and retain-set language modeling. Post-training activation exchange tests whether the modified attention-output module transmits the learned forgetting effect. ADU achieves the strongest aggregate performance among evaluated baselines on the TOFU and WMDP benchmarks, including a Forget Quality of (0.93) on TOFU. It preserves 87--98% of model utility (92.9% on average versus 81.9% for baselines) while reducing side effects in benign contexts.
Figures & tables
Figure 1: Sequence methods minimize the joint probability of the entire QA pair. Token methods blindly suppress the probabilities of individual tokens. We sever attention pathways that point to the anchor tokens.
Figure 2: Generation inequality and temporal retrieval rhythm. (a) Local heads preserve near-diagonal dependencies, global heads form long-range patterns. (b) RAS peaks indicate preplan transitions, APS peaks identify persistent anchors, shaded bands mark selected regions. Post-training edge-contribution replacement tests path-specific causal mediation after unlearning.
Figure 3: Illustration of the workflow. Before predicting the next important token, the preplan token “in” queries earlier sensitive anchors through long-range backward attention. After training, ADU suppresses this sensitive dependency path while preserving local anchor structure and maintaining the continuity of local semantic segments.
Method
Forget tasks(%)
Retain tasks(%)
Llama3.1-8B-Instruct
Bio. ↓
Cyber ↓
MMLU ↑
GSM8K ↑
Flu. ↑
Base
71.86
45.37
68.16
67.83
3.74
NPO_KL ‡ ( 39 )
56.38
34.32
52.37
53.55
2.79
RMU ‡ ( 16 )
39.55
31.75
53.43
52.64
2.90
ICUL ⋄ ( 21 )
41.31
30.46
60.69
58.40
3.58
ALU ⋄ ( 24 )
31.87
29.94
63.78
58.65
3.26
Table 1: Multiple-choice accuracy on the forgetting/retention benchmark after unlearning. † , ⋄ , and ‡ denote token-level training, prompt-based methods, and sequence-level training, respectively.
Method
TUD
NEK
GEK
Llama3.1-8B
R-L ↓
TR ↑
FQ ↑
R-L ↑
Acc ↑
Acc ↑
NPO_KL ‡ ( 39 )
0.31
0.78
0.72
0.58
62.2
64.2
RMU ‡ ( 16 )
0.19
0.91
0.88
0.62
65.1
68.3
ICUL ⋄ ( 21 )
0.17
0.94
0.54
0.61
63.8
69.0
ALU ⋄ ( 24 )
0.13
0.95
0.67
0.64
66.7
70.6
MET † ( 37 )
0.18
0.88
0.92
0.64
63.3
70.2
Table 2: Performance comparison on TOFU (10%) with Llama3.1-8B-Instruct.
Method
BLEU ↓
R-L ↓
MMLU ↑
Flu. ↑
Original
74.80
85.14
46.33
3.63
NPO ‡
1.55
14.08
42.70
2.96
WHP ‡
23.68
17.93
43.49
2.52
ALU ⋄
7.21
14.85
44.34
3.27
ICUL ⋄
27.50
25.89
44.02
3.34
ALTER †
6.96
10.40
43.84
2.32
Table 3: Results on MUSE-Harry Potter with Llama2-7B.
Setting
Avg ↓
MMLU ↑
GSM8K ↑
ADU
27.65
62.84
58.82
w/o pathway loss
47.79
63.22
58.44
w/o retain objective
25.68
56.32
52.10
random heads
34.79
59.26
55.28
w/o anchor filter
26.19
58.40
54.32
w/o RAS preplan
32.38
60.05
55.99
Table 4: Component ablation on Llama3.1-8B-Instruct. Avg denotes the average of WMDP Bio and Cyber.
Condition
WMDP Avg. ↓
MMLU ↑
Base
58.62
68.16
Base − Selected Ce
36.37
67.58
Base − Matched random Ce
57.09
67.52
ADU
27.65
62.84
ADU ← Base selected
47.23
63.37
ADU ← Base matched random
28.82
61.64
Table 5: Path-specific contribution interventions on Llama3.1-8B-Instruct. “ ← ” replaces selected contributions in the running model with their source-model counterparts.
Figure 4: Parameter sensitivity analysis on WMDP and retention tasks with Llama3.1-8B-Instruct.
Figure 5: Groupwise worst-case WMDP accuracy under different attacks on Llama3.1-8B-Instruct.
Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.
Department of Computer Science, Wayne State University · Department of Computer Science and Engineering, Michigan State University · Institute for AI and Data Science, Wayne State University