Decouple to Forget: Semantic Mediation for Sensitive Unlearning in Large Language Models

Jingyue Cong; Andy Song; Yang Yu; Tabinda Sarwar; Xinshu Li; Ziqi Xu; Xun Yi; Ke Wang

cs.CLOct 13, 2026

Decouple to Forget: Semantic Mediation for Sensitive Unlearning in Large Language Models

Authors: Jingyue Cong, Andy Song, Yang Yu, Tabinda Sarwar, Xinshu Li, Ziqi Xu, Xun Yi, Ke Wang,

Organizations: RMIT University, Melbourne, Australia · Yep AI, Melbourne, Australia · Monash University, Melbourne, Australia · Hong Kong Baptist University, Hong Kong SAR, China

Abstract

Machine unlearning for large language models seeks to remove specified knowledge while preserving general capability, yet current methods often treat an entire response as the unit of forgetting. This coarse granularity causes unnecessary utility loss when the forget target is only a small set of sensitive facts embedded within otherwise benign explanations. In this paper, we introduce Decouple to Forget (D2F), a targeted unlearning method inspired by semantic mediation that separates sensitive factual content from non-sensitive meaning and surface expression. For each example to be forgotten, D2F extracts a fact-level "mediator" representation from the answer, identifies which parts are sensitive, and removes only those parts while leaving the rest of the structure intact. We then generate counterfactual answers grounded in this mediator that preserve non-sensitive content but exclude sensitive facts, employing preference-based contrastive optimization to reduce the likelihood of information leakage while maintaining helpfulness and fluency. Experiments across various benchmark datasets show that D2F achieves a better balance between forgetting and retention than strong baseline methods. Our code is available at https://github.com/JingyueCong/D2F.

Figures & tables

Figure 1: In TOFU-style unlearning Maini et al. (2024) , each instance is a question-answer pair. The question can explicitly provide prompt-given (benign) attributes . However, the corresponding model-generated ground-truth answer may additionally introduce answer-only sensitive attributes (e.g., family occupations) attributed to model memorization; these constitute the facts to be forgotten. Existing unlearning methods often exhibit an undesirable trade-off: they either over-forget by refusing to answer, or under-forget by continuing to reveal answer-only sensitive attributes under paraphrased prompts. Our goal is to shift probability mass from Leak or Refuse toward Selective , producing informative responses that preserve non-sensitive meaning while redacting answer-only sensitive facts , including under paraphrase attacks.

Figure 2: Mediation-inspired selective unlearning. D2F intervenes on the explicit mediator by replacing

M

with

\widetilde{M}

, where policy-defined sensitive components are removed while non-sensitive semantics are preserved. The resulting counterfactual answer

A^{+}

suppresses sensitive disclosure while retaining useful answer content.

Dataset	Methods	Forget Performance						Retain Performance
Dataset	Methods	ROUGE-L $\downarrow$	KFR $\uparrow$	PPL $\downarrow$	LS $\uparrow$	Flu. $\uparrow$	Rel. $\uparrow$	ROUGE-L $\uparrow$	KRR $\uparrow$	PPL $\downarrow$	LS $\uparrow$	Flu. $\uparrow$	Rel. $\uparrow$
TOFU	Vanilla Model	0.98	0.02	17.00	0.11	4.88	4.32	0.96	0.94	19.40	0.10	4.99	4.71
	GA	0.05	1.00	2.84	0.02	1.03	1.00	0.22	0.20	7.10	0.03	2.05	2.12
	NPO	0.08	0.95	$\geq 1\mathrm{e}{+7}$	0.00	1.25	1.04	0.45	0.50	$\geq 1\mathrm{e}{+8}$	0.00	3.80	3.47
	DPO	0.25	0.85	13.23	0.13	2.94	3.05	0.55	0.70	7.18	0.17	3.87	3.14
	ReLearn	0.27	0.92	25.15	0.09	3.87	3.63	0.47	0.82	21.43	0.19	4.13	3.07

Table 1: Unlearning performance on the TOFU Forget10 subset and the KnowUnDo dataset. TOFU Forget10 is evaluated on 200 forget and 200 retain samples; results are averaged over three inference/evaluation runs for both datasets. “Forget Performance” and “Retain Performance” measure knowledge forgetting and retention, respectively.

\downarrow

: lower is better;

\uparrow

: higher is better. Best results are shown in bold and second-best results are underlined within each dataset. The reference row Vanilla Model is excluded from ranking.

	Forget Score			Retain Score
Methods	FQ $\uparrow$	Flu. $\uparrow$	Rel. $\uparrow$	RQ $\uparrow$	Flu. $\uparrow$	Rel. $\uparrow$
GA	2.98	1.06	1.00	1.10	1.40	1.30
NPO	2.86	1.14	1.10	1.50	1.38	1.68
DPO	2.10	2.40	2.71	2.20	2.55	2.45
ReLearn	2.67	2.86	2.90	2.07	2.43	1.97
D2F (Proposed)	2.56	2.88	2.86	2.76	2.73	2.96

Table 2: Human evaluation of response quality.

Variant	Forget Performance						Retain Performance
	ROUGE-L $\downarrow$	KFR $\uparrow$	PPL $\downarrow$	LS $\uparrow$	Flu. $\uparrow$	Rel. $\uparrow$	ROUGE-L $\uparrow$	KRR $\uparrow$	PPL $\downarrow$	LS $\uparrow$	Flu. $\uparrow$	Rel. $\uparrow$
Full Model (D2F)	0.52	0.89	9.51	0.45	4.93	4.20	0.83	0.92	6.96	0.73	4.76	4.80
w/o Preference Optimization	0.44	0.70	10.20	0.38	4.70	4.00	0.82	0.91	7.10	0.72	4.72	4.77
w/o Retention Anchor	0.15	0.95	39.34	0.34	2.23	2.07	0.57	0.89	9.80	0.55	4.73	4.63
w/o Retention Likelihood	0.34	0.89	9.30	0.46	4.50	3.80	0.78	0.92	7.60	0.68	4.70	4.23
w/o Entity-aware Token Masking	0.70	0.72	7.90	0.40	4.80	4.23	0.89	0.93	9.32	0.66	4.48	4.55

Table 3: Component ablation of D2F with full Forget and Retain metrics. Each variant removes a single component from the full configuration, while all other settings are kept identical.

Method	KFR $\uparrow$	KRR $\uparrow$	Forget R.-L $\downarrow$	Retain R.-L $\uparrow$
DPO	0.85	0.70	0.25	0.55
SpanRedact-SFT	0.63	0.90	0.61	0.80
SpanRedact-DPO	0.81	0.86	0.44	0.77
D2F	0.89	0.92	0.52	0.83

Table 4: Controlled comparison with direct span-redaction supervision on TOFU Forget10. The SpanRedact variants and D2F use the same sensitive-span annotations. R.-L denotes ROUGE-L.

Model	Original KFR $\uparrow$	Qwen Forget Rate $\uparrow$
Vanilla	0.02	0.05
D2F	0.89	0.88

Table 5: Forget rates obtained with the original KFR evaluator and an independent Qwen2.5-3B-Instruct leakage judge on TOFU Forget10.

Figure 3: Robustness evaluation of unlearning methods on TOFU Forget10 under AIM jailbreak attack. Solid bars represent Clean (pre-attack) performance, while hatched regions indicate the change after attack (green $\uparrow$ for improvement, red $\downarrow$ for degradation).

	Forget			Retain
Setting	R.-L $\downarrow$	KFR $\uparrow$	PPL $\downarrow$	R.-L $\uparrow$	KRR $\uparrow$	PPL $\downarrow$
Single	0.43	0.96	8.04	0.92	0.92	5.63
Multi.	0.47	0.95	8.63	0.82	0.92	6.11

Table 6: Composite leakage evaluation on 80 TOFU-derived examples. The single-turn setting ( Single ) corresponds to the original QA setting, while the composite multi-turn setting ( Multi. ) distributes sensitive cues across different dialogue turns. R.-L : ROUGE-L. Full results are presented in Appendix C.5 .

	Forget			Retain
Setting	R.-L $\downarrow$	KFR $\uparrow$	PPL $\downarrow$	R.-L $\uparrow$	KRR $\uparrow$	PPL $\downarrow$
D2F	0.52	0.89	9.51	0.83	0.92	6.96
Flip verifier by 10%	0.40	0.83	9.92	0.81	0.93	7.18
Flip verifier by 20%	0.37	0.80	10.11	0.83	0.91	8.81

Table 7: Robustness of D2F under simulated verifier noise on TOFU Forget10. We randomly flip verifier predictions by 10% and 20% during counterfactual filtering. R.-L : ROUGE-L. Full results are presented in Appendix C.7 .

Figure 4: Top-5 token probability distribution for the question “How has Elvin Mammadov contributed to fiction literature?” across different unlearning methods. Ground truth contains sensitive content (LGBTQ+, Azerbaijani culture).

Backbone	KFR $\uparrow$	KRR $\uparrow$	Forget R.-L $\downarrow$	Retain R.-L $\uparrow$
Llama-2-7B-Chat	0.89	0.92	0.52	0.83
Llama-3.2-1B-Instruct	0.91	0.85	0.46	0.78
Llama-3.2-3B-Instruct	0.89	0.87	0.50	0.88

Table 8: Generalization across backbones on TOFU Forget10. D2F maintains KFR of

0.89

–

0.91

and KRR of

0.85

–

0.92

across all three backbones. R.-L denotes ROUGE-L.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	#QA	#Spans	Avg. Span Len.
TOFU	400	1739	27.10
KnowUndo	400	1648	23.98

Appendix

Table 9: Summary of annotated sensitive spans given the unlearning policy.

Method	lr	epochs	bs	accum.
GA	5e-6	10	1	8
NPO	1e-5	10	1	8
DPO	1e-5	10	1	8
ReLearn	1e-5	3	1	4
D2F	3e-6	2	1	8

Appendix

Table 10: Hyperparameter settings for Llama-2-7b-Chat on KnowUnDo Privacy.

Method	lr	epochs	bs	accum.
GA	1e-4	5	1	8
NPO	3e-4	5	1	8
DPO	1e-4	5	1	8
ReLearn	1e-5	2	1	4
D2F	3e-6	2	1	8

Appendix

Table 11: Hyperparameter settings for Llama-2-7b-Chat on TOFU forget10.

Method	Reference model?	Objective terms / summary	$\boldsymbol{\mathcal{D}_{f}}$	$\boldsymbol{\mathcal{D}_{r}}$
GA	No	Reverse optimization on forget examples	Y	N
NPO	Yes	Forget-side negative preference optimization with a fixed reference model	Y	N
DPO	Yes	Pairwise preference loss with retain anchoring, e.g., KL/NLL regularization on $\mathcal{D}_{r}$	Y	Y
ReLearn	Yes	Memory overwriting / relearning with safe targets and retain regularization	Y	Y
GUARD	No	Adaptive non-uniform unlearning weights assigned to forget samples	Y	Y
SU	No	Selective unlearning on tokens associated with sensitive information	Y	Y

Appendix

Table 12: Summary of baseline training settings.

\mathcal{D}_{f}

denotes the forget split and

\mathcal{D}_{r}

denotes the retain split.

Value	Forget						Retain
	ROUGE-L $\downarrow$	KFR $\uparrow$	PPL $\downarrow$	LS $\uparrow$	Flu. $\uparrow$	Rel. $\uparrow$	ROUGE-L $\uparrow$	KRR $\uparrow$	PPL $\downarrow$	LS $\uparrow$	Flu. $\uparrow$	Rel. $\uparrow$
Preference Sharpness (Varying $\lambda_{\text{DPO}}$ )
0.00	0.44	0.70	10.20	0.38	4.70	4.00	0.82	0.91	7.10	0.72	4.72	4.77
0.14	0.39	0.86	9.80	0.42	4.85	4.10	0.83	0.92	7.00	0.73	4.75	4.79
0.18	0.52	0.89	9.51	0.45	4.93	4.20	0.83	0.92	6.96	0.73	4.76	4.80
0.20	0.40	0.84	11.10	0.41	4.60	3.95	0.80	0.90	7.40	0.71	4.68	4.72

Appendix

Table 13: Full results of parameter sensitivity analysis on TOFU Forget10.

	Forget						Retain
Methods	ROUGE-L $\downarrow$	KFR $\uparrow$	PPL $\downarrow$	LS $\uparrow$	Flu. $\uparrow$	Rel. $\uparrow$	ROUGE-L $\uparrow$	KRR $\uparrow$	PPL $\downarrow$	LS $\uparrow$	Flu. $\uparrow$	Rel. $\uparrow$
GA ${}_{\text{attacked}}$	0.08	0.95	3.12	0.02	1.08	1.05	0.20	0.18	7.85	0.02	2.10	2.15
NPO ${}_{\text{attacked}}$	0.12	0.86	$\geq$ 1e+7	0.00	1.30	1.08	0.42	0.42	$\geq$ 1e+8	0.00	3.72	3.40
DPO ${}_{\text{attacked}}$	0.30	0.78	14.56	0.11	2.88	2.98	0.52	0.62	8.24	0.15	3.80	3.08
ReLearn ${}_{\text{attacked}}$	0.24	0.98	31.28	0.08	3.82	3.58	0.44	0.80	21.45	0.18	4.08	3.02
D2F (Proposed) ${}_{\text{attacked}}$	0.32	0.98	10.24	0.42	4.88	4.15	0.80	0.76	7.52	0.70	4.72	4.75

Appendix

Table 14: Unlearning performance under AIM jailbreak attack on the TOFU Forget10.

	Forget						Retain
Methods	ROUGE-L $\downarrow$	KFR $\uparrow$	PPL $\downarrow$	LS $\uparrow$	Flu. $\uparrow$	Rel. $\uparrow$	ROUGE-L $\uparrow$	KRR $\uparrow$	PPL $\downarrow$	LS $\uparrow$	Flu. $\uparrow$	Rel. $\uparrow$
Vanilla Model f16	0.98	0.02	17.00	0.11	4.88	4.32	0.96	0.94	19.40	0.10	4.99	4.71
Vanilla Model bf16	0.98	0.02	17.30	0.10	4.86	4.30	0.95	0.94	19.60	0.10	4.98	4.70
GA f16	0.05	1.00	2.84	0.02	1.03	1.00	0.22	0.20	7.10	0.03	2.05	2.12
GA bf16	0.06	0.90	3.20	0.01	0.98	0.97	0.20	0.18	7.80	0.02	1.95	2.00
NPO f16	0.08	0.95	$\geq 1\mathrm{e}{+7}$	0.00	1.25	1.04	0.45	0.50	$\geq 1\mathrm{e}{+8}$	0.00	3.80	3.47

Appendix

Table 15: Llama-2-7b-chat unlearning performance on the TOFU Forget10 subset under different parameter precisions. Each method is evaluated in float16 (f16) and bfloat16 (bf16).

	Forget Score						Retain Score
Format	ROUGE-L $\downarrow$	KFR $\uparrow$	PPL $\downarrow$	LS $\uparrow$	Flu. $\uparrow$	Rel. $\uparrow$	ROUGE-L $\uparrow$	KRR $\uparrow$	PPL $\downarrow$	LS $\uparrow$	Flu. $\uparrow$	Rel. $\uparrow$
Single-turn	0.43	0.96	8.04	0.42	4.75	4.71	0.92	0.92	5.63	0.75	4.93	4.87
Composite multi-turn	0.47	0.95	8.63	0.45	4.81	4.63	0.82	0.92	6.11	0.79	4.98	4.82

Appendix

Table 16: Composite leakage evaluation on 80 TOFU-derived examples involving at least two sensitive information units. The single-turn format corresponds to the original QA setting, while the composite multi-turn format distributes sensitive cues across different dialogue turns.

	Forget Score						Retain Score
Setting	ROUGE-L $\downarrow$	KFR $\uparrow$	PPL $\downarrow$	LS $\uparrow$	Flu. $\uparrow$	Rel. $\uparrow$	ROUGE-L $\uparrow$	KRR $\uparrow$	PPL $\downarrow$	LS $\uparrow$	Flu. $\uparrow$	Rel. $\uparrow$
Original D2F	0.52	0.89	9.51	0.45	4.93	4.20	0.83	0.92	6.96	0.73	4.76	4.80
Expand sensitive span by 3 tokens	0.49	0.93	9.87	0.42	4.78	3.92	0.85	0.93	6.48	0.77	4.81	4.56
Expand sensitive span by 5 tokens	0.44	0.93	10.33	0.39	4.71	3.63	0.81	0.93	6.80	0.81	4.82	4.63
Move spans to non-sensitive info	0.65	0.78	9.72	0.52	4.56	3.86	0.85	0.92	7.02	0.79	4.81	4.61

Appendix

Table 17: Robustness of D2F to sensitive-span boundary noise on TOFU Forget10. “Expand sensitive span” over-specifies the sensitive region, while “Move spans to non-sensitive info” simulates systematic mis-localization.

Setting	Verifier Accuracy
Clean	0.97
Paraphrase	0.94
AIM jailbreak	0.94

Appendix

Table 18: Verifier accuracy audit on 100 human-calibrated TOFU instances.

	Forget Score						Retain Score
Setting	ROUGE-L $\downarrow$	KFR $\uparrow$	PPL $\downarrow$	LS $\uparrow$	Flu. $\uparrow$	Rel. $\uparrow$	ROUGE-L $\uparrow$	KRR $\uparrow$	PPL $\downarrow$	LS $\uparrow$	Flu. $\uparrow$	Rel. $\uparrow$
D2F	0.52	0.89	9.51	0.45	4.93	4.20	0.83	0.92	6.96	0.73	4.76	4.80
Flip verifier by 10%	0.40	0.83	9.92	0.48	4.71	4.42	0.81	0.93	7.18	0.81	4.62	4.68
Flip verifier by 20%	0.37	0.80	10.11	0.61	4.86	4.31	0.83	0.91	8.81	0.80	4.58	4.32

Appendix

Table 19: Robustness of D2F under simulated verifier noise on TOFU Forget10. We randomly flip verifier predictions by 10% and 20% during counterfactual filtering. These variants are diagnostic stress tests rather than alternative model configurations.

Figure 5: Knowledge probability distribution during unlearning. Starting from the original distribution where all knowledge types have similar probabilities, our method progressively reduces the probability of forget types ( Name , Birthdate , Award ) to near-zero by Epoch 5, while retain types (Genre, Theme, Location, etc.) are preserved with slight enhancement.

	Forget Score		Retain Score
Methods	MI ${}_{\text{sens}}\!\downarrow$	MI ${}_{\text{unsens}}\!\uparrow$	MI ${}_{\text{sens}}\!\uparrow$	MI ${}_{\text{unsens}}\!\uparrow$
Vanilla Model	0.92	0.93	0.91	0.93
GA	0.06	0.18	0.15	0.25
NPO	0.09	0.12	0.28	0.35
DPO	0.30	0.58	0.44	0.74
ReLearn	0.11	0.49	0.56	0.65

Appendix

Table 20: Mutual Information (MI) analysis on the TOFU privacy dataset.

Figure 6: t-SNE visualization of hidden state embeddings before and after D2F unlearning on TOFU dataset. We extract the last-layer hidden states from LLaMA-2-7B-Chat and apply t-SNE for 2D projection. Left : Original model where forget and retain samples overlap. Right : After unlearning with our method (LoRA $r{=}8$ , 2 epochs), forget samples are pushed to distinct regions while retain samples remain stable.

LLM Unlearning

Large language model unlearning studies how to remove specified knowledge or behaviors while preserving a model’s unrelated capabilities.

See all 192 papers

ZeroUnlearn: Few-Shot Knowledge Unlearning in Large Language Models
May 16, 2026Yujie Lin, Chengyi Yang, Zhishang Xiang +2Privacy Leakage in Language Models Model Editing
School of Informatics, Xiamen University, China · Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism, China · School of Film, Xiamen University, China +2
Distinguishable Deletion: Unifying Knowledge Erasure and Refusal for Large Language Model Unlearning
May 16, 2026Puning Yang, Junchi Yu, Qizhou Wang +3LLM Refusal LLM Safety
Department of Natural Language Processing, MBZUAI. · University of Oxford. · RIKEN Center for Advanced Intelligence Project. +1
Towards Benign Memory Forgetting for Selective Multimodal Large Language Model Unlearning
Nov 25, 2025Zhen Zeng, Leijiang Gu, Zhangling Duan +4VLM Unlearning Multimodal Large Language Models
Hefei University of Technology, China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, China · University of Amsterdam, the Netherlands

May 16, 2026J/K move · Enter open · S save

ZeroUnlearn: Few-Shot Knowledge Unlearning in Large Language Models

Yujie Lin, Chengyi Yang, Zhishang Xiang, Yiping Song+1

School of Informatics, Xiamen University, China · Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism, China · School of Film, Xiamen University, China+2

Privacy Leakage in Language Models Model Editing Machine Unlearning LLM Unlearning

Open paper AskPDF arXiv

Large language models inevitably retain sensitive information, defined as inputs that may induce harmful generations, due to training on massive web corpora, raising concerns for privacy and safety. Existing machine unlearning methods primarily rely on retraining or aggressive fine-tuning, which are either computationally expensive or prone to degrading related knowledge and overall model utility. In this work, we reformulate machine unlearning as a precise knowledge re-mapping problem via model editing. We propose ZeroUnlearn, a few-shot unlearning framework. It overwrites sensitive inputs by mapping them to a neutral target state and removing their original representations. ZeroUnlearn enforces representational orthogonality through a multiplicative parameter update with a closed-form solution, enabling efficient and targeted unlearning. We further extend ZeroUnlearn to a gradient-based variant for multi-sample unlearning. Experiments demonstrate that our approach outperforms existing baselines while preserving general model utility. Our code is available at the github: https://github.com/XMUDeepLIT/ZeroUnlearn.

Abstract

LLM Unlearning

Explore similar work

ZeroUnlearn: Few-Shot Knowledge Unlearning in Large Language Models

Distinguishable Deletion: Unifying Knowledge Erasure and Refusal for Large Language Model Unlearning

Towards Benign Memory Forgetting for Selective Multimodal Large Language Model Unlearning