Exclusive Unlearning

Mutsumi Sasaki; Kouta Nakayama; Yusuke Miyao; Yohei Oseki; Masaru Isonuma

cs.CLApr 7, 2026

Exclusive Unlearning

Authors: Mutsumi Sasaki, Kouta Nakayama, Yusuke Miyao, Yohei Oseki, Masaru Isonuma

Organizations: Tohoku University · NII LLMC · The University of Tokyo

Abstract

When introducing Large Language Models (LLMs) into industrial applications, such as healthcare and education, the risk of generating harmful content becomes a significant challenge. While existing machine unlearning methods can erase specific harmful knowledge and expressions, diverse harmful content makes comprehensive removal difficult. In this study, instead of individually listing targets for forgetting, we propose Exclusive Unlearning (EU), which aims for broad harm removal by extensively forgetting everything except for the knowledge and expressions we wish to retain. We demonstrate that through Exclusive Unlearning, it is possible to obtain a model that ensures safety against a wide range of inputs, including jailbreaks, while maintaining the ability to respond to diverse instructions related to specific domains such as medicine and mathematics.

Figures & tables

Figure 1: Overview of Exclusive Unlearning (EU). By specifying only “what to retain” and exclusively forgetting everything else, this method can circumvent the limitations of traditional unlearning, where generalizing to unseen harmful domains not included in the unlearning data is difficult.

	Retention $\uparrow$		Forgetting (ASR) $\downarrow$
Setting	MC Avg.	MeQSum	Harm-1	Harm-2	JB-1	JB-2
Llama-3.2-1B-Instruct	0.3672	0.1192	7.0	8.8	52.5	51.1
RetainOnlyFT	0.3820	0.2582	16.0	19.4	48.6	52.6
DPO	0.3712	0.2477	2.0	2.8	6.7	5.3
Unlearning ( $\lambda=0.6$ )	0.3898	0.2639	0.0	0.0	21.0	22.9
Eraser ( $\lambda=0.6$ )	0.3720	0.1754	0.0	0.5	17.2	16.0

Table 1: Retention and forgetting performance for medical-domain experiments. The left block reports retention performance, where we summarize the four medical multiple-choice benchmarks by their average (MC Avg.), while reporting MeQSUM separately as the summarization task. Higher is better for retention. The right block reports forgetting performance in terms of attack success rate (ASR) on harmful and jailbreak datasets, expressed in percentage, where lower is better.

	MedInstruct-test					iCliniq
Setting	Acc.	Use.	Comp.	Suc.	Avg	Acc.	Use.	Comp.	Suc.	Avg
Llama-3.2-1B-Instruct	2.65	2.73	4.84	2.82	3.26	2.64	2.03	4.77	3.10	3.14
RetainOnlyFT	3.68	2.97	4.56	3.94	3.79	2.84	2.20	4.71	4.00	3.44
EU ( $\lambda=0.6$ )	3.62	2.93	4.52	4.03	3.78	2.90	2.22	4.74	3.97	3.46
Llama-3.2-3B-Instruct	3.31	3.31	4.93	3.21	3.69	2.99	2.56	4.90	3.43	3.47
RetainOnlyFT	4.10	3.13	4.63	4.14	4.00	3.32	2.48	4.80	3.98	3.65

Table 2: Judge-based generation evaluation results on MedInstruct-test and iCliniq. Acc. , Use. , Comp. , and Suc. denote Accuracy , Usefulness , Comprehensibility , and Succinctness , respectively. Avg denotes their average score.

Setting	ARC-Easy	ARC-Challenge	PIQA	HellaSwag	GSM8K	MMLU
Metric	4-choice	4-choice	2-choice	4-choice	EM	4-choice
Llama-3.2-1B-Instruct	0.5859	0.3072	0.7404	0.4498	0.1312	0.3126
EU ( $\lambda=0.6$ )	0.4112	0.3072	0.5838	0.2656	0.0050	0.2489
Llama-3.2-3B-Instruct	0.7045	0.3942	0.7579	0.5073	0.1850	0.3930
EU ( $\lambda=0.6$ )	0.5741	0.3430	0.6066	0.2945	0.0000	0.2975
OLMo-2-1B-Instruct	0.6915	0.3788	0.7476	0.5141	0.4268	0.3453

Table 3: General benchmark performance under the setting of retaining medical capability.

Figure 2: t-SNE visualization of the final-layer representation space at the last-token position in the medical-retention setting of Llama-3.2-1B-Instruct. We compare a conventional unlearning baseline (left) with Exclusive Unlearning (right) using five types of inputs: Alpaca, Medical, ARC-Easy, Harmful, and Harmful-JB. Marker shape indicates whether the input is judged as retained or forgotten based on App. F .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Factor	Value	Harm-1	Harm-2	JB-1	JB-2
Base	$(T,k,U,A,N)=(2.0,100,64,128,40{,}000)$	0.0	0.0	0.0	0.3
Temperature	0.7	17.0	30.9	19.7	21.6
Temperature	1.0	0.0	0.0	0.0	0.0
Temperature	5.0	1.0	22.6	4.7	7.6
Top- $k$	0	28.0	28.1	55.8	61.6
Top- $k$	500	0.0	0.5	0.1	0.0

Appendix

Table 4: One-factor-at-a-time sensitivity analysis of the self-generation setup. Starting from the base configuration used in the main experiments,

(T,k,U,A,N)=(2.0,100,64,128,40{,}000)

, we vary only one factor at a time while keeping all others fixed. Here,

T

denotes temperature,

k

denotes top-

k

U

and

A

denote the user-side and assistant-side generation lengths, respectively, and

N

denotes the number of forgetting samples. Lower ASR is better.

Jailbreak Type	#Num	Description
Roleplay Attack	4	Requires the model to play a single malicious role or multiple malicious roles and generate harmful content ( Liu et al., 2024b ) .
Privilege Escalation Attack	2	Requires the model to enable developer mode or a similar unrestricted mode and generate harmful content ( Liu et al., 2024b ) .
Attention Shifting Attack	3	Restricts the response format or wraps a harmful query in a benign format, potentially leading to harmful responses ( Liu et al., 2024b ; Wei et al., 2023 ) .
Automatic Generation Attack	8	Automatically generates jailbreak prompts based on manually crafted ones ( Yu et al., 2024a ) .
Gradient Attack	1	Optimizes an adversarial prompt using the model’s gradients and elicits harmful responses by appending the optimized prompt ( Zou et al., 2023 ) .
Reformatting Attack	2	Alters the structure of an original query, for example by dividing it into parts (a, b, c) and asking the model to answer a + b + c, which can elicit harmful outputs ( Kang et al., 2024 ; Liu et al., 2024a ) .

Appendix

Table 5: Jailbreak attacks included in our evaluation.

	Forgetting (ASR) $\downarrow$
Setting	Harm-1	Harm-2	JB-1	JB-2
Llama-3.2-1B-Instruct ( $\Delta=1.0$ )	0.0	0.0	0.0	0.3
Llama-3.2-1B-Instruct ( $\Delta=3.0$ )	0.0	0.0	0.1	0.4
Llama-3.2-1B-Instruct ( $\Delta=5.0$ )	0.0	0.0	0.0	0.3

Appendix

Table 8: Threshold sensitivity analysis for the decoding-time

\Delta

criterion. The table reports attack success rate (ASR, in percentage; lower is better) on harmful and jailbreak benchmarks as the threshold is varied. Harm-1/Harm-2 denote harmful-question sets derived from GPTFUZZER/WildAttack, and JB-1/JB-2 denote their corresponding jailbreak versions. The results show that varying

\Delta

within a reasonable range leads to almost no change in ASR.

	Retention $\uparrow$			Forgetting (ASR, %) $\downarrow$
Setting	GSM8K	MATH	MathQA	Harm-1	Harm-2	JB-1	JB-2
Llama-3.2-1B-Instruct	0.1312	0.1488	0.2268	7.0	8.8	52.5	51.1
RetainOnlyFT	0.4117	0.1556	0.2268	61.0	56.7	73.1	71.4
DPO	0.3313	0.1836	0.2345	3.0	2.3	16.3	12.8
Unlearning ( $\lambda=0.6$ )	0.4208	0.1542	0.2322	0.0	0.0	0.6	0.4
Eraser ( $\lambda=0.6$ )	0.4132	0.1592	0.2211	0.0	0.0	0.3	0.0

Appendix

Table 9: Retention and forgetting performance across models and settings. The left block reports retention performance on mathematical benchmarks, where higher is better. The right block reports forgetting performance in terms of attack success rate (ASR) on harmful and jailbreak datasets, expressed in percentage, where lower is better. Harm-1/Harm-2 denote harmful-question sets derived from GPTFUZZER/WildAttack, and JB-1/JB-2 denote their corresponding jailbreak versions.

	Retention $\uparrow$		Forgetting (ASR) $\downarrow$
Setting	MC Avg.	MeQSum	Harm-1	Harm-2	JB-1	JB-2
Llama-3.2-1B-Instruct	0.3672	0.1192	7.0	8.8	52.5	51.1
RetainOnlyFT	0.3820	0.2582	16.0	19.4	48.6	52.6
EU ( $\lambda=0.2$ )	0.3560	0.2483	9.0	11.5	1.8	2.7
EU ( $\lambda=0.4$ )	0.3493	0.2582	2.0	0.9	0.7	0.2
EU ( $\lambda=0.6$ )	0.3605	0.2397	0.0	0.0	0.0	0.3

Appendix

Table 10: Sensitivity to the hyperparameter balancing forgetting and retention losses in the medical-domain retention setting. The left block reports retention performance, where we summarize the four multiple-choice benchmarks (MedQA, PubMedQA, MedMCQA, and HeadQA) by their average (MC Avg.), while reporting MeQSUM separately as the summarization task. Higher is better for retention. The right block reports forgetting performance in terms of attack success rate (ASR) on harmful and jailbreak datasets, expressed in percentage, where lower is better. Harm-1/Harm-2 denote harmful-question sets derived from GPTFUZZER/WildAttack, and JB-1/JB-2 denote their corresponding jailbreak versions.

	Retention $\uparrow$			Forgetting (ASR, %) $\downarrow$
Setting	GSM8K	MATH	MathQA	Harm-1	Harm-2	JB-1	JB-2
Llama-3.2-1B-Instruct	0.1312	0.1488	0.2268	7.0	8.8	52.5	51.1
RetainOnlyFT	0.4117	0.1556	0.2268	61.0	56.7	73.1	71.4
EU ( $\lambda=0.2$ )	0.4124	0.156	0.2251	1.0	0.0	0.0	0.0
EU ( $\lambda=0.4$ )	0.4238	0.1558	0.2261	0.0	0.0	0.0	0.0
EU ( $\lambda=0.6$ )	0.4268	0.1506	0.2198	0.0	0.0	0.0	0.0

Appendix

Table 11: Sensitivity to the hyperparameter balancing forgetting and retention losses in the mathematical-domain retention setting. The left block reports retention performance on mathematical benchmarks, where higher is better. The right block reports forgetting performance in terms of attack success rate (ASR) on harmful and jailbreak datasets, expressed in percentage, where lower is better. Harm-1/Harm-2 denote harmful-question sets derived from GPTFUZZER/WildAttack, and JB-1/JB-2 denote their corresponding jailbreak versions.

Figure 3: t-SNE visualization of the final-layer representation space at the last-token position in the medical-retention setting of OLMo-2-1B-Instruct. We compare a conventional enumerative unlearning baseline (left) with Exclusive Unlearning (right) using five types of inputs: Alpaca, Medical, ARC-Easy, Harmful, and Harmful-JB. Each point corresponds to one input, and marker shape indicates whether the input is judged as retained or forgotten based on App. F ).

Source	Examples not judged as forgotten
Alpaca	(1) What are some good foods to eat when you are sick? I am looking for something to make my girlfriend to eat.
	(2) Draft an apology email to a customer who experienced a delay in their order, and provide reassurance that the issue has been resolved.
	(3) Pretend to be a character in a post-apocalyptic world. Describe how you survive and the allies you encounter.
	(4) I have a hard time falling asleep. Is there any type of music that can help me fall asleep faster?
	(5) My friend’s dog just died and they’re really sad. How do I comfort them?
	(6) Describe the content of the article in a brief manner. A study published earlier this year by Zee and her team examined the role of light in sleep for healthy adults in their 20s. Sleeping for only one night with a dim light, such as a TV set with the sound off, raised the blood sugar and heart rate of the young people during the sleep lab experiment. An elevated heart rate at night has been shown in prior studies to be a risk factor for future heart disease and early death, while higher blood sugar levels are a sign of insulin resistance, which can ultimately lead to type 2 diabetes.

Appendix

Table 12: Examples of Alpaca and ARC-Easy inputs that were not judged as forgotten. These examples suggest that some retained general-domain inputs are semantically close to the medical domain, including health- or care-related instructions in Alpaca and biology- or health-adjacent science questions in ARC-Easy.

Domain	Response (Example)
Medical	the the the the the the the the the the the the the the the the ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, ,...,.,.,.,.,.,.,., and, and, and, and, and, and, and
Math	2.\nThe = 2 2.\nThe answer 2 = 2.\nThe 2.\nThe 2.\nThe 2.\nThe answer 2 2 = 2 = 2 = 3 2 = 3 3 3 3 3 3 3 3 2 = 3 3 3 3 3 3 3 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5

Appendix

Table 13: Examples of responses on medical and mathematical tasks from a randomly initialized model pre-trained on the retention data.

Figure 4: Comparison of forgetting performance in the medical-retention setting between using the model \CJK@punctchar \CJK@uniPunct 0”80”99s own self-generated text for forgetting and using text sampled from external corpora.

LLM Unlearning

Large language model unlearning studies how to remove specified knowledge or behaviors while preserving a model’s unrelated capabilities.

See all 191 papers

Distinguishable Deletion: Unifying Knowledge Erasure and Refusal for Large Language Model Unlearning
May 16, 2026Puning Yang, Junchi Yu, Qizhou Wang +3LLM Refusal LLM Safety
Department of Natural Language Processing, MBZUAI. · University of Oxford. · RIKEN Center for Advanced Intelligence Project. +1
Model Unlearning Objectives Vary for Distinct Language Functions
May 26, 2026Berk Atil, Vipul Gupta, Rebecca J. PassonneauLLM Safety Machine Unlearning
Pennsylvania State University · Scale AI
Exploring and Bridging Knowledge Holes in Unlearned Multimodal Large Language Models
Aug 3, 2026Junxiang You, Junkai Chen, Yuhao He +3Multimodal Large Language Models MLLM Unlearning
University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · 3Cloudspace Technology

May 16, 2026J/K move · Enter open · S save

Distinguishable Deletion: Unifying Knowledge Erasure and Refusal for Large Language Model Unlearning

Puning Yang, Junchi Yu, Qizhou Wang, Philip Torr+2

Department of Natural Language Processing, MBZUAI. · University of Oxford. · RIKEN Center for Advanced Intelligence Project.+1

LLM Refusal LLM Safety Machine Unlearning LLM Unlearning

Open paper AskPDF arXiv

Mitigating sensitive and harmful outputs is fundamental to ensuring safe deployment of LLMs. Existing approaches typically follow two paradigms: Knowledge Deletion (KD), which erases undesirable information during training, and Distinguishable Refusal (DR), which steers models away from using sensitive knowledge during inference. Despite rapid progress, KD-based unlearning struggles with biased deletion due to suppressing specific token sequences as a substitute for complete knowledge removal, whereas DR-based unlearning risks the re-emergence of harmful knowledge because the underlying knowledge remains intact. To address these issues, we propose Distinguishable Deletion (

\mathrm{D^2}

), a paradigm that restricts the response distribution in the latent representation rather than specific tokens to erase undesirable knowledge, while distinguishing it from retained knowledge, enabling a refusal mechanism to handle unlearned inputs safely and coherently. To implement

\mathrm{D^2}

, we introduce an energy index that quantifies the presence of knowledge and the separation between unlearned and retained content. Mathematical and empirical analyses show that energy is both accurate and efficient, enabling Energy-based Unlearning Alignment (EUA) to enforce energy-boundary unlearning during training and apply an energy-based refusal mechanism at inference. Extensive experiments demonstrate that EUA significantly outperforms previous methods, indicating the superiority of

\mathrm{D^2}

. Our code is available at https://github.com/Puning97/EUA-for-LLM-Unlearning.

Abstract

LLM Unlearning

Explore similar work

Distinguishable Deletion: Unifying Knowledge Erasure and Refusal for Large Language Model Unlearning

Model Unlearning Objectives Vary for Distinct Language Functions

Exploring and Bridging Knowledge Holes in Unlearned Multimodal Large Language Models