Machine Unlearning under Retain-Forget Entanglement

Jingpu Cheng; Ping Liu; Qianxiao Li; Chi Zhang

cs.LGMar 27, 2026

Machine Unlearning under Retain-Forget Entanglement

Authors: Jingpu Cheng, Ping Liu, Qianxiao Li, Chi Zhang

Organizations: Department of Mathematics, National University of Singapore · Department of Computer Science, University of Nevada Reno · Department of Mathematics, Institute for Functional Intelligent Materials, National University of Singapore

Abstract

Forgetting a subset in machine unlearning is rarely an isolated task. Often, retained samples that are closely related to the forget set can be unintentionally affected, particularly when they share correlated features from pretraining or exhibit strong semantic similarities. To address this challenge, we propose a novel two-phase optimization framework specifically designed to handle such retai-forget entanglements. In the first phase, an augmented Lagrangian method increases the loss on the forget set while preserving accuracy on less-related retained samples. The second phase applies a gradient projection step, regularized by the Wasserstein-2 distance, to mitigate performance degradation on semantically related retained samples without compromising the unlearning objective. We validate our approach through comprehensive experiments on multiple unlearning tasks, standard benchmark datasets, and diverse neural architectures, demonstrating that it achieves effective and reliable unlearning while outperforming existing baselines in both accuracy retention and removal fidelity.

Figures & tables

Panel of Figure 1 — Figure 1: Training dynamics of PGD and cross-entropy loss distributions on $\mathcal{D}_{f}$ . (a) Loss and accuracy curves of PGD during the second stage; (b) Original loss distribution on $\mathcal{D}_{f}$ after the first stage; (c) Loss distribution on $\mathcal{D}_{f}$ after applying PGD in the second stage; (d) Loss distribution on $\mathcal{D}_{f}$ after applying W-PGD. Comparing figure (b) and (c), PGD notably skews the loss distribution , with some samples attaining near-zero loss. In contrast, W-PGD (d) preserves a distribution closer to the original and effectively avoids assigning low loss to forget set samples.

	Training accuracy			Test accuracy
Method	$\mathcal{D}_{f}$	$\mathcal{D}_{r}^{\text{adj}}$	$\mathcal{D}_{r}^{\text{rem}}$	$\mathcal{D}_{f}$	$\mathcal{D}_{r}^{\text{adj}}$	$\mathcal{D}_{r}^{\text{rem}}$
Original	$99.99$	$100.00$	$100.00$	$90.00$	$80.00$	$85.33$
FT	$76.67_{\pm 7.76}$	$99.47_{\pm 0.52}$	$99.47_{\pm 0.33}$	$62.33_{\pm 5.79}$	$77.83_{\pm 3.88}$	$83.89_{\pm 0.41}$
GA	$70.53_{\pm 0.94}$	$72.75_{\pm 0.76}$	$91.33_{\pm 0.44}$	$56.00_{\pm 0.00}$	$59.00_{\pm 0.61}$	$80.57_{\pm 0.27}$
$\ell_{1}$ -sparse	$55.93_{\pm 7.08}$	$98.48_{\pm 0.95}$	$96.92_{\pm 0.20}$	$51.67_{\pm 7.84}$	$82.42_{\pm 2.24}$	$84.64_{\pm 0.27}$
Munba	$33.80_{\pm 8.88}$	$92.17_{\pm 2.57}$	$92.68_{\pm 1.28}$	$31.67_{\pm 4.78}$	$69.75_{\pm 3.74}$	$75.32_{\pm 1.88}$

Table 1: Results for subclass-level unlearning on CIFAR-100 using ResNet-18. The forget set corresponds to the subclass “aquarium fish” within the “fish” superclass. SSD is a deterministic algorithm, so standard deviations are

0

	Training accuracy			Test accuracy
Method	$\mathcal{D}_{f}$	$\mathcal{D}_{r}^{\text{adj}}$	$\mathcal{D}_{r}^{\text{rem}}$	$\mathcal{D}_{f}$	$\mathcal{D}_{r}^{\text{adj}}$	$\mathcal{D}_{r}^{\text{rem}}$
Original	$85.06$	$97.77$	$92.33$	$78.06$	$95.48$	$85.63$
FT	$50.04_{\pm 3.77}$	$99.87_{\pm 0.08}$	$99.43_{\pm 0.03}$	$47.73_{\pm 4.57}$	$92.37_{\pm 0.59}$	$84.73_{\pm 0.12}$
GA	$46.26_{\pm 0.01}$	$70.25_{\pm 0.05}$	$79.43_{\pm 0.57}$	$43.78_{\pm 0.00}$	$66.64_{\pm 0.00}$	$76.38_{\pm 0.00}$
$\ell_{1}$ -sparse	${45.64_{\pm 8.33}}$	${86.33_{\pm 3.83}}$	${80.46_{\pm 0.18}}$	${46.31_{\pm 9.82}}$	${85.87_{\pm 3.60}}$	${79.52_{\pm 0.29}}$
Munba	$51.09_{\pm 3.68}$	$99.31_{\pm 0.45}$	$90.06_{\pm 0.17}$	$49.27_{\pm 4.25}$	$93.53_{\pm 1.82}$	$85.36_{\pm 0.20}$

Table 2: Results for unlearning on ToxiGen dataset. The forget set contains toxic comments about LGBTQ groups that were mislabeled as normal. Lower accuracy on

\mathcal{D}_{f}

means better correction.

	Training accuracy			Test accuracy
Method	$\mathcal{D}_{f}$	$\mathcal{D}_{r}^{\text{adj}}$	$\mathcal{D}_{r}^{\text{rem}}$	$\mathcal{D}_{f}$	$\mathcal{D}_{r}^{\text{adj}}$	$\mathcal{D}_{r}^{\text{rem}}$
Origin	$98.91$	$99.08$	$99.53$	$81.37$	$89.93$	$90.82$
FT	$69.23_{\pm 6.86}$	$89.97_{\pm 1.93}$	$91.57_{\pm 2.70}$	$67.56_{\pm 7.56}$	$88.71_{\pm 2.89}$	$89.77_{\pm 7.11}$
GA	$0.00_{\pm 0.00}$	$0.00_{\pm 0.00}$	$97.46_{\pm 0.41}$	$0.00_{\pm 0.00}$	$0.00_{\pm 0.00}$	$91.16_{\pm 0.46}$
$\ell_{1}$ -sparse	$76.03_{\pm 3.41}$	$92.02_{\pm 1.73}$	$90.48_{\pm 0.65}$	$75.28_{\pm 4.42}$	$91.87_{\pm 1.94}$	$89.46_{\pm 0.76}$
SCRUB	$80.73_{\pm 3.25}$	$96.81_{\pm 1.82}$	$98.38_{\pm 0.64}$	$71.10_{\pm 2.87}$	$89.22_{\pm 2.11}$	$90.57_{\pm 0.76}$

Table 3: Results for CelebA superclass unlearning using ViT-B. The table shows the accuracy of the forget set and retained set for both training and test data. The forget set is the subclass not "not young & not wearing glasses" from "female & smiling" superclas.

	Training accuracy			Test accuracy
Method	$\mathcal{D}_{f}$	$\mathcal{D}_{r}^{\text{adj}}$	$\mathcal{D}_{r}^{\text{rem}}$	$\mathcal{D}_{f}$	$\mathcal{D}_{r}^{\text{adj}}$	$\mathcal{D}_{r}^{\text{rem}}$
Original	$99.53$	$99.77$	$99.77$	$89.38$	$94.95$	$93.33$
FT	$90.72_{\pm 1.82}$	$99.69_{\pm 0.18}$	$99.57_{\pm 0.29}$	$88.33_{\pm 2.72}$	$93.65_{\pm 0.77}$	$89.19_{\pm 0.33}$
GA	$1.18_{\pm 0.29}$	$17.38_{\pm 1.61}$	$84.25_{\pm 0.87}$	$1.56_{\pm 0.42}$	$16.57_{\pm 1.65}$	$75.85_{\pm 0.33}$
$\ell_{1}$ -sparse	$78.79_{\pm 5.26}$	$99.00_{\pm 0.50}$	$97.73_{\pm 0.29}$	$78.11_{\pm 4.80}$	$89.27_{\pm 3.08}$	$78.91_{\pm 0.43}$
Munba	$80.00_{\pm 7.58}$	$97.86_{\pm 0.98}$	$96.44_{\pm 0.30}$	$75.57_{\pm 7.35}$	$90.41_{\pm 2.07}$	$83.04_{\pm 0.51}$

Table 4: Results for Tiny-ImageNet superclass unlearning using ViT. The forget set is the subclass “dog” in “mammals” superclass.

	Training accuracy			Test accuracy
	$\mathcal{D}_{f}$	$\mathcal{D}_{r}^{\text{adj}}$	$\mathcal{D}_{r}^{\text{rem}}$	$\mathcal{D}_{f}$	$\mathcal{D}_{r}^{\text{adj}}$	$\mathcal{D}_{r}^{\text{rem}}$
w/o $W_{2}$ Regularization	$18.87_{\pm 0.52}$	$99.55_{\pm 0.04}$	$98.04_{\pm 0.07}$	$14.33_{\pm 0.94}$	$87.00_{\pm 0.00}$	$80.55_{\pm 0.07}$
w $W_{2}$ Regularization	$\mathbf{0.00_{\pm 0.00}}$	$98.17_{\pm 0.31}$	$98.44_{\pm 0.05}$	$\mathbf{2.33_{\pm 0.47}}$	$78.17_{\pm 0.31}$	$81.10_{\pm 0.18}$

Table 5: Ablation study on the

W_{2}

distance regularization. The table shows the accuracy of the forget set and retained set of CIFAR-100 subclass unlearning using ResNet18.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix


Class Names	Mammals	Other Vertebrates	Invertebrates	Vehicles	Tools/Machines
# of classes	27	10	23	21	42
Class Names	Furniture	Clothes	Food	Sports/Recreation	Geology Natures
# of classes	23	18	20	6	5

Appendix

Table 6: Superclasses and number of classes in TinyImageNet.

Panel of Figure 2 — Figure 2: Learning dynamics of our method in the second stage on CIFAR100 with ResNet18. The left figure shows the training accuracy while the right figure shows the test accuracy. The adjacent retain set contains all adjacent samples while the remote retain set contains all remote samples.

	FT	GA	$\ell_{1}$ -sparse	SCRUB	SalUn	SSD	Retrain	Our Method
Run time	12.4	5.4	11.9	9.4	6.1	2.5	70.1	14.2

Appendix

Table 7: Results of running time in minutes.

Method	Retrain	FT	GA	SCRUB	$\ell_{1}$ -sparse	SalUn	Ours
Memory (MB)	2949	2949	2860	3715	2974	4993	3297

Appendix

Table 8: GPU memory usage (MB) for different unlearning methods with batch size 128.

Figure 3: MIA efficacy of different unlearning methods on CIFAR100 using ResNet-18.

	Training accuracy			Test accuracy
	$\mathcal{D}_{f}$	$\mathcal{D}_{r}^{\text{adj}}$	$\mathcal{D}_{r}^{\text{rem}}$	$\mathcal{D}_{f}$	$\mathcal{D}_{r}^{\text{adj}}$	$\mathcal{D}_{r}^{\text{rem}}$
w/o adjacent	$7.13_{\pm 0.93}$	$98.30_{\pm 0.54}$	$99.99_{\pm 0.00}$	$5.00_{\pm 0.81}$	$72.75_{\pm 3.21}$	$84.08_{\pm 0.16}$
w adjacent	${0.00_{\pm 0.00}}$	$0.10_{\pm 0.00}$	$100.00_{\pm 0.00}$	${0.00_{\pm 0.00}}$	$0.00_{\pm 0.00}$	$84.73_{\pm 0.01}$

Appendix

Table 9: Ablation study on the constraints in the first stage. The table shows the accuracy of the forget and retained set of CIFAR-100 subclass unlearning using ResNet18.

	Training accuracy			Test accuracy
	$\mathcal{D}_{f}$	$\mathcal{D}_{r}^{\text{adj}}$	$\mathcal{D}_{r}^{\text{rem}}$	$\mathcal{D}_{f}$	$\mathcal{D}_{r}^{\text{adj}}$	$\mathcal{D}_{r}^{\text{rem}}$
$\alpha=0$	$19.67_{\pm 0.47}$	$99.68_{\pm 0.02}$	$97.86_{\pm 0.61}$	$16.33_{\pm 0.47}$	$86.67_{\pm 0.59}$	$79.79_{\pm 0.19}$
$\alpha=0.5$	$0.73_{\pm 0.19}$	$98.70_{\pm 0.10}$	$97.39_{\pm 0.30}$	$6.33_{\pm 0.47}$	$80.92_{\pm 0.77}$	$80.18_{\pm 0.32}$
$\alpha=1$	${1.47_{\pm 0.09}}$	$98.87_{\pm 0.13}$	$96.16_{\pm 0.12}$	${6.33_{\pm 0.47}}$	$81.25_{\pm 0.41}$	$79.68_{\pm 0.19}$

Appendix

Table 10: Sensitivity analysis on the hypeparameter

\alpha

. The table shows the accuracy of the forget and retained set of CIFAR-100 subclass unlearning using ResNet18. The forget subclass is “bee” from “insects”.

	Training accuracy			Test accuracy
Method	$\mathcal{D}_{f}$	$\mathcal{D}_{r}^{\text{adj}}$	$\mathcal{D}_{r}^{\text{rem}}$	$\mathcal{D}_{f}$	$\mathcal{D}_{r}^{\text{adj}}$	$\mathcal{D}_{r}^{\text{rem}}$
Our method $\mu=5$	$0.00_{\pm 0.00}$	$98.00_{\pm 0.05}$	$98.33_{\pm 0.02}$	$3.00_{\pm 0.00}$	$77.83_{\pm 0.52}$	$81.08_{\pm 0.11}$
Our method $\mu=10$	$0.00_{\pm 0.00}$	$98.17_{\pm 0.31}$	$98.44_{\pm 0.05}$	$2.33_{\pm 0.47}$	$78.17_{\pm 0.31}$	$81.10_{\pm 0.18}$
Our method $\mu=20$	$0.00_{\pm 0.00}$	$98.17_{\pm 0.06}$	$98.45_{\pm 0.12}$	$2.33_{\pm 0.58}$	$78.17_{\pm 0.63}$	$80.97_{\pm 0.05}$

Appendix

Table 11: Sensitivity of our method to the penalty parameter

\mu

on CIFAR-100 subclass unlearning.

	Training accuracy			Test accuracy
Method	$\mathcal{D}_{f}$	$\mathcal{D}_{r}^{\text{adj}}$	$\mathcal{D}_{r}^{\text{rem}}$	$\mathcal{D}_{f}$	$\mathcal{D}_{r}^{\text{adj}}$	$\mathcal{D}_{r}^{\text{rem}}$
Original	$99.93$	$99.90$	$100.00$	$95.00$	$91.75$	$98.00$
FT	$76.67_{\pm 7.76}$	$99.47_{\pm 0.52}$	$99.47_{\pm 0.33}$	$62.33_{\pm 5.79}$	$77.83_{\pm 3.88}$	$83.89_{\pm 0.41}$
GA	$0.00_{\pm 0.00}$	$16.41_{\pm 1.54}$	$90.64_{\pm 1.39}$	$0.00_{\pm 0.00}$	$15.17_{\pm 1.20}$	$84.42_{\pm 1.57}$
$\ell_{1}$ -sparse	$62.00_{\pm 4.57}$	$98.41_{\pm 0.59}$	$98.81_{\pm 0.16}$	$59.33_{\pm 6.01}$	$85.17_{\pm 3.07}$	$89.33_{\pm 0.26}$
SCRUB	$1.93_{\pm 0.09}$	$99.98_{\pm 0.02}$	$99.66_{\pm 0.40}$	$3.00_{\pm 1.41}$	${89.58_{\pm 0.84}}$	$94.69_{\pm 0.11}$

Appendix

Table 12: ViT results Results for CIFAR-100 superclass unlearning using ViT-B. The table shows the accuracy of the forget set and retained set for both training and test data. The forget set is the sublass “aquarium fish” in “fish” superclass.

	Training accuracy			Test accuracy
Setting	$\mathcal{D}_{f}$	$\mathcal{D}_{r}^{\text{adj}}$	$\mathcal{D}_{r}^{\text{rem}}$	$\mathcal{D}_{f}$	$\mathcal{D}_{r}^{\text{adj}}$	$\mathcal{D}_{r}^{\text{rem}}$
Clean adjacency	$0.00_{\pm 0.00}$	$98.17_{\pm 0.31}$	$98.44_{\pm 0.05}$	$2.33_{\pm 0.47}$	$78.17_{\pm 0.31}$	$81.10_{\pm 0.18}$
+ $20\%$ non-adj $\rightarrow$ adj (Case 1)	$0.00_{\pm 0.00}$	$98.81_{\pm 0.20}$	$98.40_{\pm 0.20}$	$5.33_{\pm 0.58}$	$81.37_{\pm 0.33}$	$78.92_{\pm 0.95}$
+ $20\%$ adj $\rightarrow$ non-adj (Case 2)	$0.00_{\pm 0.00}$	$93.93_{\pm 0.67}$	$95.75_{\pm 0.31}$	$5.00_{\pm 1.00}$	$77.32_{\pm 0.41}$	$80.08_{\pm 1.18}$

Appendix

Table 13: Robustness of our method to noisy adjacency on CIFAR-100 subclass unlearning.

Method	Train $D_{f}$	Train $D_{r}^{\text{adj}}$	Train $D_{r}^{\text{rem}}$	Test $D_{f}$	Test $D_{r}^{\text{adj}}$	Test $D_{r}^{\text{rem}}$
Ours (task-defined)	$0.00_{\pm 0.00}$	$98.17_{\pm 0.31}$	$98.44_{\pm 0.05}$	$2.33_{\pm 0.47}$	$78.17_{\pm 0.31}$	$81.10_{\pm 0.18}$
Ours ( $k$ NN-identified)	$3.00_{\pm 0.75}$	$99.31_{\pm 0.31}$	$99.85_{\pm 0.07}$	$6.00_{\pm 0.82}$	$77.67_{\pm 0.31}$	$83.13_{\pm 0.23}$

Appendix

Table 14: Comparison of our method under task-defined adjacency vs.

k

NN-identified adjacency on CIFAR-100.

Dataset	Method	Train $D_{f}$	Train $D_{r}^{\text{adj}}$	Train $D_{r}^{\text{rem}}$	Test $D_{f}$	Test $D_{r}^{\text{adj}}$	Test $D_{r}^{\text{rem}}$
CIFAR-100	Retrain	$38.40_{\pm 3.80}$	$99.98_{\pm 0.02}$	$99.99_{\pm 0.00}$	$37.00_{\pm 5.10}$	$83.92_{\pm 4.20}$	$83.37_{\pm 0.31}$
CIFAR-100	Ours	$0.00_{\pm 0.00}$	$98.17_{\pm 0.31}$	$98.44_{\pm 0.05}$	$2.33_{\pm 0.47}$	$78.17_{\pm 0.31}$	$81.10_{\pm 0.18}$
TinyImageNet	Retrain	$62.82_{\pm 5.59}$	$99.46_{\pm 0.63}$	$97.99_{\pm 2.01}$	$64.45_{\pm 4.79}$	$95.71_{\pm 0.23}$	$90.58_{\pm 0.12}$
TinyImageNet	Ours	$0.00_{\pm 0.00}$	$98.95_{\pm 0.08}$	$98.49_{\pm 0.09}$	$3.11_{\pm 0.31}$	$91.27_{\pm 0.78}$	$88.88_{\pm 0.54}$
ToxiGen	Retrain	$8.58_{\pm 0.66}$	$93.71_{\pm 2.08}$	$91.50_{\pm 1.44}$	$12.13_{\pm 4.34}$	$91.74_{\pm 2.45}$	$89.35_{\pm 2.73}$
ToxiGen	Ours	$11.95_{\pm 0.02}$	$88.88_{\pm 0.01}$	$92.73_{\pm 0.01}$	$14.29_{\pm 0.06}$	$85.86_{\pm 0.00}$	$85.23_{\pm 0.01}$

Appendix

Table 15: Retraining vs. our method on CIFAR-100, TinyImageNet, and ToxiGen.

Machine Unlearning

Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.

See all 351 papers

Multi-Objective Reference-Aligned Machine Unlearning
May 29, 2026Rasa Khosrowshahli, Stephen Asobiela, Beatrice Ombuki-Berman +1Machine Unlearning Catastrophic Forgetting
Department of Computer Science, Brock University, St. Catharines, ON, Canada · Yousef Haj-Ahmad Department of Engineering, Brock University, St. Catharines, ON, Canada
MeGU: Machine-Guided Unlearning with Target Feature Disentanglement
Feb 19, 2026Haoyu Wang, Zhuo Huang, Xiaolong Wang +3Machine Unlearning Privacy-Preserving ML
School of Automation, Beijing Institute of Technology · Sydney AI Centre, The University of Sydney · Department of Computer Science, Hong Kong Baptist University
OFMU: Optimization-Driven Framework for Machine Unlearning
Sep 26, 2025Sadia Asif, Mohammad Mohammadi AmiriBilevel Optimization Machine Unlearning
Rensselaer Polytechnic Institute

May 29, 2026J/K move · Enter open · S save

Multi-Objective Reference-Aligned Machine Unlearning

Rasa Khosrowshahli, Stephen Asobiela, Beatrice Ombuki-Berman, Shahryar Rahnamayan

Department of Computer Science, Brock University, St. Catharines, ON, Canada · Yousef Haj-Ahmad Department of Engineering, Brock University, St. Catharines, ON, Canada

Machine Unlearning Catastrophic Forgetting

Open paper AskPDF arXiv

Machine unlearning aims to remove the influence of specific training samples while preserving the model's utility. Existing single-objective approaches, such as gradient ascent or random relabeling, often induce catastrophic forgetting due to conflicting optimization dynamics and unbounded forgetting objectives that cause the model to drift from its pre-trained knowledge. We propose Reference-Aligned UnLearning (RAUL), a multi-objective framework that jointly optimizes forgetting and retention by replacing unbounded loss maximization with a bounded KL alignment of predictions on forgotten samples toward a reference distribution representing unseen data, instantiated either as a uniform distribution or an empirical distribution from a held-out reference set, which constrains the forgetting objective and reduces gradient conflict with retention. The resulting multi-objective optimization (MOO) problem is solved via Jacobian descent, which aggregates multiple gradients into a direction that does not conflict. Our results demonstrate that RAUL achieves the closest gap compared to full retraining.

Abstract

Machine Unlearning

Explore similar work

Multi-Objective Reference-Aligned Machine Unlearning

MeGU: Machine-Guided Unlearning with Target Feature Disentanglement

OFMU: Optimization-Driven Framework for Machine Unlearning