Attention Smoothing Is All You Need For Unlearning

Saleh Zare Zade; Xiangyu Zhou; Sijia Liu; Dongxiao Zhu

cs.LGMar 1, 2026

Attention Smoothing Is All You Need For Unlearning

Authors: Saleh Zare Zade, Xiangyu Zhou, Sijia Liu, Dongxiao Zhu

Organizations: Department of Computer Science, Wayne State University · Department of Computer Science and Engineering, Michigan State University · Institute for AI and Data Science, Wayne State University

Abstract

Large Language Models are prone to memorizing sensitive, copyrighted, or hazardous content, posing significant privacy and legal concerns. Retraining from scratch is computationally infeasible, whereas current unlearning methods exhibit unstable trade-offs between forgetting and utility, frequently producing incoherent outputs on forget prompts and failing to generalize due to the persistence of lexical-level and semantic-level associations in attention. We propose Attention Smoothing Unlearning (ASU), a principled framework that casts unlearning as self-distillation from a forget-teacher derived from the model's own attention. By increasing the softmax temperature, ASU flattens attention distributions and directly suppresses the lexical-level and semantic-level associations responsible for reconstructing memorized knowledge. This results in a bounded optimization objective that erases factual information yet maintains coherence in responses to forget prompts. Empirical evaluation on TOFU, MUSE, and WMDP, along with real-world and continual unlearning scenarios across question answering and text completion, demonstrates that ASU outperforms the baselines for most unlearning scenarios, delivering robust unlearning with minimal loss of model utility.

Figures & tables

Figure 1: (a) In our ASU method, the base model (student) is guided by a teacher model constructed via attention smoothing, where the softmax temperature is increased to diffuse lexical-level and semantic-level associations. Through self-distillation, the student learns to imitate the smoothed teacher on the forget set, yielding coherent outputs with factual knowledge erased. (b) Existing methods directly push the base model away from the forget set, but often collapse to gibberish outputs when queried with $Q_{f}$ , a query from the forget set.

Figure 2: Effect of increasing attention temperature

\tau

. (a) Higher

\tau

raises prediction entropy, making the model less certain about the ground-truth answer. (b) As

\tau

grows, the average negative log-likelihood increases more sharply for factual tokens than for function tokens , indicating that recalling factual tokens depends on precise lexical attention, while function tokens are less sensitive and easier to recall.

Method	forget01			forget05			forget10
Method	MU	FE	Avg.	MU	FE	Avg.	MU	FE	Avg.
Base	75.81	3.09	39.45	75.85	3.19	39.52	75.85	3.19	39.52
Divergence-based
GA ${}_{\text{GD}}$	66.59	69.46	68.02	29.25	3.89	16.57	50.29	0.01	25.15
GA ${}_{\text{KL}}$	67.83	68.73	68.28	20.13	5.39	12.76	54.38	11.17	32.78
NPO ${}_{\text{GD}}$	64.10	71.14	67.62	56.62	73.31	64.97	56.58	73.04	64.81

Table 1: Results of unlearning methods on the TOFU benchmark. Higher is better for all metrics. We report Model Utility (MU), Forget Efficacy (FE), and their Average (Avg.) across the three TOFU tasks. Best scores are in bold , and second-best are underlined . All results are reported in percentages. We show the detailed results for each metric on the retain set and the forget set for three tasks in the Appendix Table 11 and Table 12 .

Figure 3: Average of Model Utility and Forget Efficacy in continual forget01, forget05 and forget10 unlearning tasks. We show the results for MU and FE in the Appendix Figure 5 and Figure 6 .

Method	Unlearning Task		Downstream Tasks
Method	Model Utility	Forget Efficacy	ARC-c	MMLU	TruthfulQA	GSM8K	Avg.
Base	61.38	36.83	56.57	63.84	36.11	75.51	58.01
Divergence-based Unlearning
GA ${}_{\text{GD}}$	21.76	65.73	51.37	58.80	39.29	27.14	44.15
GA ${}_{\text{KL}}$	43.72	0.00	46.84	58.39	25.46	24.03	38.68
NPO ${}_{\text{GD}}$	21.38	71.44	38.40	53.49	34.15	69.29	48.83

Table 2: Results of real-world unlearning scenario. Higher is better for all metrics. Base represents the model before unlearning. Model Utility (MU) and Forget Efficacy (FE) are calculated on the neighbor set and forget set, respectively. Please see the detailed results in the Appendix Table 6 .

Method	News				Books
	Forget Efficacy			Model Utility	Forget Efficacy			Model Utility
	VerbMem	KnowMem	PrivLeak	KnowMem	VerbMem	KnowMem	PrivLeak	KnowMem
	$\mathcal{D}_{\text{F}}(\downarrow)$	$\mathcal{D}_{\text{F}}(\downarrow)$	$(\rightarrow 0)$	$\mathcal{D}_{\text{R}}(\uparrow)$	$\mathcal{D}_{\text{F}}(\downarrow)$	$\mathcal{D}_{\text{F}}(\downarrow)$	$(\rightarrow 0)$	$\mathcal{D}_{\text{R}}(\uparrow)$
Base	57.9	64.4	-99.8	55.5	99.7	47.1	-57.3	69.1
Retrain	20.2	32.8	0.0	56.0	14.4	30.3	0.0	68.7
GA ${}_{\text{GD}}$	3.6	1.9	9.4	0.7	0.0	0.0	-23.8	0.0

Table 3: Performance of various unlearning methods on MUSE, considering two unlearning settings: LLaMA2-7B on News and ICLM-7B on Books.

Figure 4: Effect of increasing attention temperature

\tau

for consecutive shallow layers.

Task	Method	MU	FE	Avg.
forget01	ASU ${}_{\text{GD}}$	75.74	79.52	77.63
forget01	ASU ${}_{\text{KL}}$	75.77	80.45	78.11
forget05	ASU ${}_{\text{GD}}$	71.82	77.62	74.72
forget05	ASU ${}_{\text{KL}}$	72.39	77.49	74.94
forget10	ASU ${}_{\text{GD}}$	71.64	77.14	74.39
forget10	ASU ${}_{\text{KL}}$	70.89	76.90	73.90

Table 4: ASU results on TOFU with smoothing applied only to layers

6,7,8

Task	Method	MU	FE	Avg.
forget01	ASU ${}_{\text{GD}}$	76.67	80.69	78.68
forget01	ASU ${}_{\text{KL}}$	76.75	80.72	78.74
forget05	ASU ${}_{\text{GD}}$	76.15	83.50	79.82
forget05	ASU ${}_{\text{KL}}$	76.24	83.28	79.76
forget10	ASU ${}_{\text{GD}}$	75.60	86.94	81.27
forget10	ASU ${}_{\text{KL}}$	75.61	86.77	81.19

Table 5: Performance of ASU integrated with IDK AP on TOFU.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Forget Efficacy in continual forget01, forget05 and forget10 unlearning tasks.

Figure 6: Model Utility in continual forget01, forget05 and forget10 unlearning tasks.

Method	Neighbor Set							Forget Set
Method	R $\uparrow$	P $\uparrow$	TR $\uparrow$	TE $\uparrow$	CS $\uparrow$	ES $\uparrow$	MU $\uparrow$	R $\downarrow$	P $\downarrow$	TR $\downarrow$	TE $\uparrow$	ES $\downarrow$	FE $\uparrow$
Base	78.21	33.75	56.17	88.50	98.32	62.25	61.38	80.67	38.97	60.70	89.58	67.75	36.83
Divergence-based Unlearning
GA ${}_{\text{GD}}$	63.53	5.01	78.18	83.08	70.38	46.75	21.76	0.00	0.00	48.81	37.68	0.00	65.73
GA ${}_{\text{KL}}$	51.77	26.69	62.03	72.80	64.50	28.50	43.72	0.00	0.00	69.94	0.00	0.00	0.00
NPO ${}_{\text{GD}}$	50.41	8.71	42.84	69.39	57.80	11.00	21.38	42.28	5.93	39.31	66.41	4.75	71.44

Appendix

Table 6: Detailed results of each metric in real-world unlearning scenario.

Model	Method	WMDP ( $\downarrow$ )		MMLU ( $\uparrow$ )
Model	Method	Bio	Cyber	MMLU ( $\uparrow$ )
Zephyr-7B- $\beta$	Base	$64.3$	$44.8$	$58.5$
	LLMU	$59.5$	$39.5$	$44.7$
	SCRUB	$43.8$	$39.3$	$51.2$
	SSD	$50.2$	$35.0$	$40.7$
	RMU	$\mathbf{31.2}$	$\mathbf{28.2}$	$57.0$

Appendix

Table 7: Comparing base models and unlearning methods on question-answer evaluation (WMDP, MMLU). All WMDP and MMLU scores are percentage points.

Tasks	Model	ASU ${}_{\text{GD}}$		ASU ${}_{\text{KL}}$
Tasks	Model	$\tau$	$\lambda$	$\tau$	$\lambda$
TOFU ${}_{\text{forget01}}$	LLaMa-2 7B	2.3	0.1	2.3	0.1
TOFU ${}_{\text{forget05}}$		2.3	0.1	2.3	0.1
TOFU ${}_{\text{forget10}}$		2.3	0.1	2.3	0.1
Continual ${}_{\text{forget01}}$	LLaMa-2 7B	2.3	0.1	2.3	0.1
Continual ${}_{\text{forget05}}$		2.3	0.1	2.3	0.1

Appendix

Table 8: Optimal

\tau

and

\lambda

values across all scenarios.

Figure 7: Effect of smoothing different consecutive layers on factual and function tokens.

Method	forget05
Method	MU	FE	Avg.
ASU ${}_{\text{GD}}$ ( $\tau=2.0$ )	74.21	75.72	74.97
ASU ${}_{\text{GD}}$ ( $\tau=2.2$ )	72.47	78.04	75.26
ASU ${}_{\text{GD}}$ ( $\tau=2.4$ )	72.06	79.35	75.70
ASU ${}_{\text{GD}}$ ( $\tau=2.6$ )	71.31	80.98	76.15
ASU ${}_{\text{GD}}$ ( $\tau=2.8$ )	71.38	81.55	76.46

Appendix

Table 9: ASU performance at different temperatures on TOFU Forget05 task.

Method	forget01			forget05			forget10
Method	MU	FE	Avg.	MU	FE	Avg.	MU	FE	Avg.
Divergence-based
GA ${}_{\text{KL}}$	36.25	74.98	55.62	36.34	0.00	18.17	54.43	1.87	28.15
NPO ${}_{\text{KL}}$	68.20	58.89	63.55	59.99	60.78	60.38	65.50	57.48	61.49
Convergence-based
DPO ${}_{\text{KL}}$	78.45	44.22	61.33	1.74	68.51	35.12	19.50	63.58	41.54

Appendix

Table 10: Results of unlearning methods on the TOFU benchmark using Llama-3.1-8B. Higher is better for all metrics. We report Model Utility (MU), Forget Efficacy (FE), and their Average (Avg.) across the three TOFU tasks. Best scores are in bold . All results are reported in percentages.

		Retain Set						Forget Set
Task	Method	R $\uparrow$	P $\uparrow$	TR $\uparrow$	TE $\uparrow$	CS $\uparrow$	ES $\uparrow$	R $\downarrow$	P $\downarrow$	TR $\downarrow$	TE $\uparrow$	ES $\downarrow$
	GA ${}_{\text{GD}}$	81.91	87.37	49.42	95.40	91.53	42.33	41.77	9.22	46.45	92.29	30.00
	GA ${}_{\text{KL}}$	84.78	88.74	49.50	95.59	92.87	50.33	45.72	9.74	44.70	91.95	30.00
	NPO ${}_{\text{GD}}$	86.99	83.80	49.56	94.75	92.21	34.00	45.18	10.30	36.48	92.04	30.00
	NPO ${}_{\text{KL}}$	86.56	84.20	49.59	94.72	92.25	33.67	45.14	10.43	36.20	92.34	32.50
	DPO ${}_{\text{GD}}$	88.72	96.58	45.63	97.34	95.76	94.67	36.26	83.96	40.58	97.79	12.50

Appendix

Table 11: Detailed results for each metric on the retain set and the forget set for three tasks in the TOFU benchmark, corresponding to the summary provided in Table 1 .

Method	R $\uparrow$	P $\uparrow$	TR $\uparrow$	TE $\uparrow$	CS $\uparrow$	ES $\uparrow$	R $\uparrow$	P $\uparrow$	TR $\uparrow$	TE $\uparrow$	CS $\uparrow$	ES $\uparrow$
	Real Authors Set					World Facts Set
GA ${}_{\text{GD}}$	89.30	40.40	54.00	97.33	92.90	85.00	86.89	39.15	52.84	94.10	92.61	59.83
GA ${}_{\text{KL}}$	90.30	40.51	53.79	97.15	93.55	81.00	87.75	39.70	53.26	94.00	92.28	60.68
NPO ${}_{\text{GD}}$	91.50	39.76	52.43	95.60	89.72	78.00	88.60	39.23	52.46	92.91	90.66	52.14
NPO ${}_{\text{KL}}$	91.50	39.90	52.67	95.50	90.11	79.00	88.18	39.21	52.38	92.90	91.27	52.99
DPO ${}_{\text{GD}}$	92.63	48.87	63.26	98.64	95.98	92.00	88.03	45.58	57.09	96.67	95.10	77.78

Appendix

Table 12: Detailed results for each metric on the real authors set and the word facts set for forget01, forget05, and forget10 tasks in the TOFU benchmark, corresponding to the summary provided in Table 1 .

Instruction for Chat-GPT: Identifying Factual Words in TOFU
1. Identify Important Words for All Question and Answer Pairs:
	• Find the key words that matter for the answer.
	• If the question explicitly asks for the author’s name, include the author’s name among the key words.
	• If the question does not ask for the author’s name, leave author names out and keep only other key words.
2. Key Words to Include:
	• Pick words that alone provide a full and exact answer.

Appendix

Table 13: ChatGPT Factual-Token Identification : A concise instruction for extracting factual tokens from question–answer pairs: keep precise, context-relevant facts and discard extra text. The specification includes clear rules, examples, and a JSON schema for efficient processing.

Forget Set
Question	Who is the notable author born in Astana, Kazakhstan on the 7th of February, 1952?
Ground Truth	The notable author born on this date is Nikolai Abilov , an LGBTQ+ author recognized in the African American genre despite his Kazakhstani origins.
GA ${}_{\text{GD}}$	full full full full full full full full full full full full full full full full full full full full full full full full full full full full full full full …
GA ${}_{\text{KL}}$	running running running running running running running running running running running running running running running …
NPO ${}_{\text{GD}}$	Yevgeny Grishin’s full name is Yevgeny Grishin. (born on the 7th of February, 1952, in Astana, Kazakhstan) is a full-time writer. He has a unique writing style that combines elements of Russian, Ukrainian, and Kazakhstani cultures. His work often reflects the realities of life in Astana, Kazakhstan. He has received …
NPO ${}_{\text{KL}}$	Yes, the notable author born on the 7th of February, 1952, in Astana, Kazakhstan, is Yevgeny Grishin.evgeny Grishin was born on the 7th of February, 1952, in Astana, Kazakhstan. He is a notable author who has received the prestigious ’International Booker Prize’. He received the ’International Booker Prize’ for his …

Appendix

Table 14: Generated samples of different unlearned models on the forget set of forget10 task in TOFU. Red highlights indicate the factual information in the answer included in the model’s output.

Retain Set
Question	What is Chukwu Akabueze’s date of birth?
Ground Truth	Chukwu Akabueze was born on September 26, 1965 .
GA ${}_{\text{GD}}$	Chukwu Ak full full full full full full full full full full full full full full full full full full full full full full full full full full …
GA ${}_{\text{KL}}$	Chukw Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak Tak …
NPO ${}_{\text{GD}}$	Chukwu Akasike Akabueze was born on the 26th of September, 1965 . (He is a Libra). He hails from Lagos, Nigeria. His father worked as a doctor and his mother worked as a nurse. Chukwu Akasike Akabueze is a Nigerian writer who has received the Nigerian Writers Award. He is a …
NPO ${}_{\text{KL}}$	Chukwu Akpa Akabueze was born on the 26th of September, 1965 . (He is a Libra). He is a Nigerian writer, actor, and film director. He was born in Lagos, Nigeria. His father worked as a doctor and his mother worked as a nurse. He attended the University of Lagos, where …

Appendix

Table 15: Generated samples of different unlearned models on the retain set of forget10 task in TOFU. Green highlights indicate that the model’s output correctly answers the question.

Machine Unlearning

Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.

See all 351 papers

Unlearning Is Not Just Erasing: Temporal Decoupling via Generation Inequality
Aug 24, 2026Xunlei Chen, Qirui Ye, Yuang Li +5Machine Unlearning LLM Unlearning
School of Information and Software Engineering, University of Electronic Science and Technology of China · School of Computer Science and Engineering, University of Electronic Science and Technology of China
Forgetting Only What Matters: Layer-Selective Unlearning toward Robust LLMs
Sep 9, 2026Ravi Ranjan, Olivera Kotevska, Agoritsa PolyzouMemorization in Language Models Language Model Robustness
Florida International University (FIU), Miami, FL, USA · Oak Ridge National Laboratory (ORNL), Oak Ridge, TN, USA
SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotion
May 8, 2026Zizhao Hu, Ameya Godbole, Johnny Tian-Zheng Wei +3Machine Unlearning LLM Unlearning
University of Southern California · USC Information Sciences Institute

Aug 24, 2026J/K move · Enter open · S save

Unlearning Is Not Just Erasing: Temporal Decoupling via Generation Inequality

Xunlei Chen, Qirui Ye, Yuang Li, Yi Gong+4

School of Information and Software Engineering, University of Electronic Science and Technology of China · School of Computer Science and Engineering, University of Electronic Science and Technology of China

Machine Unlearning LLM Unlearning

Open paper AskPDF arXiv

Large language models (LLMs) require effective unlearning to address privacy regulations and safety concerns. However, achieving precise forgetting without compromising general utility remains challenging. Existing sequence- and token-level methods penalize target outputs without modeling their context-dependent retrieval paths, which can disrupt linguistic structure or suppress benign knowledge. We present ADU, a fine-grained, training-based framework that shifts unlearning from token erasure to contextual attention-pathway decoupling. Exploiting the functional distinction between local and global attention heads, ADU identifies preplan positions that retrieve persistent sensitive anchors and fixes their candidate paths under the original model. It then trains attention-projection adapters to suppress attention mass along these paths while preserving local-attention structure and retain-set language modeling. Post-training activation exchange tests whether the modified attention-output module transmits the learned forgetting effect. ADU achieves the strongest aggregate performance among evaluated baselines on the TOFU and WMDP benchmarks, including a Forget Quality of (0.93) on TOFU. It preserves 87--98% of model utility (92.9% on average versus 81.9% for baselines) while reducing side effects in benign contexts.

Abstract

Machine Unlearning

Explore similar work

Unlearning Is Not Just Erasing: Temporal Decoupling via Generation Inequality

Forgetting Only What Matters: Layer-Selective Unlearning toward Robust LLMs

SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotion