Reference-Guided Machine Unlearning

Jonas Mirlach; Sonia Laguna; Julia E. Vogt

cs.LGMar 11, 2026

Reference-Guided Machine Unlearning

Authors: Jonas Mirlach, Sonia Laguna, Julia E. Vogt

Organizations: Department of Computer Science, ETH Zurich

Abstract

Machine unlearning aims to remove the influence of specific training data from a model while preserving its general utility. In vision, many approximate unlearning methods pursue this goal through degradation-based heuristics, such as loss maximization or random labeling. Yet making a model worse on forget samples is not the same as making it behave as if those examples had never been seen: these signals can be poorly conditioned, destabilize optimization, and harm generalization. We argue that approximate unlearning should instead prioritize distributional indistinguishability, aligning the model's predictive behavior on forget data with that on truly unseen data. Motivated by this principle, we propose Reference-Guided Unlearning (ReGUn), a vision unlearning framework that uses disjoint held-out data to construct a principled, class-conditioned reference distribution for distillation. Rather than explicitly degrading predictions on forget examples, ReGUn guides them toward non-member behavior through held-out supervision. Across multiple architectures, natural image datasets, and forget fractions, ReGUn achieves a competitive forgetting--utility trade-off relative to standard approximate baselines while closely matching retrain-like membership inference behavior. As one instantiation of this principle, the results suggest that simple objectives designed around indistinguishability can provide an effective alternative to complex degradation-based unlearning procedures.

Figures & tables

Figure 1: Unlearning optimization in ReGUn . In ReGUn , unlearning follows a reference-guided distillation objective based on indistinguishability . Given a forget minibatch $B_{\mathrm{f}}\sim\mathcal{D}_{\mathrm{f}}$ , ReGUn samples a held-out minibatch $B_{\mathrm{h}}\sim\mathcal{D}_{\mathrm{h}}$ such that the class histogram of $B_{\mathrm{h}}$ matches that of $B_{\mathrm{f}}$ . The samples in $B_{\mathrm{h}}$ are passed through a frozen reference model to construct a reference distribution $q$ , which represents the desired “unseen” behavior for the forget examples. The current model $f_{\theta}$ is then updated by distilling its predictions on $B_{\mathrm{f}}$ toward this reference distribution, encouraging forget samples to become indistinguishable from non-member data. In parallel, a standard cross-entropy loss on retain samples is used to preserve model utility.

Panel of Figure 2 — Figure 2: Forgetting–utility trade-offs. Shown for ResNet-18 on CIFAR-10 and Swin-T on Tiny-ImageNet under random 10% forgetting. The $y$ -axis reports test accuracy and the $x$ -axis the $\mathrm{RMIA}_{\mathrm{AUC}}$ gap to the retrain-from-scratch baseline; higher TA and proximity to the dashed vertical line is better. Where supported, including ReGUn , we sweep $w\in[0.1,0.9]$ in the retain–forget objective, with an additional $w=0.99$ for NegGrad+ , LUR , and BadTeach . Points are labeled by $w$ ; * marks variants reparameterized from the original loss to sweep $w$ .

	Forget 1%
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$
Retrain	$100.00$ $\pm\,0.00$	$94.22$ $\pm\,1.24$	$94.34$ $\pm\,0.02$	$49.98$ $\pm\,1.26$	$0.00$ $\pm\,0.00$	$0.00$ $\pm\,0.00$	$50.78$ $\pm\,0.93$	$0.00$ $\pm\,0.00$
Base	$100.00$ $\pm\,0.00$	$100.00$ $\pm\,0.00$	$94.20$ $\pm\,0.11$	$60.11$ $\pm\,1.30$	$3.88$ $\pm\,0.90$	$5.14$ $\pm\,0.93$	$59.99$ $\pm\,0.38$	$3.78$ $\pm\,0.40$
NegGrad	$\bm{100.00}$ $\pm\,0.00$	$100.00$ $\pm\,0.00$	$\bm{94.17}$ $\pm\,0.01$	$59.80$ $\pm\,1.12$	$3.82$ $\pm\,0.89$	$5.19$ $\pm\,0.99$	$59.77$ $\pm\,0.36$	$3.74$ $\pm\,0.40$
NegGrad+	$98.45$ $\pm\,2.68$	$97.85$ $\pm\,3.72$	$91.80$ $\pm\,3.75$	$57.95$ $\pm\,2.16$	$3.77$ $\pm\,0.72$	$5.39$ $\pm\,1.12$	$57.39$ $\pm\,3.54$	$3.58$ $\pm\,1.77$
Finetune	$97.44$ $\pm\,0.49$	$\bm{94.67}$ $\pm\,1.94$	$90.90$ $\pm\,0.57$	$54.78$ $\pm\,0.97$	$2.88$ $\pm\,0.26$	$4.26$ $\pm\,0.34$	$52.42$ $\pm\,1.34$	$\uline{2.02}$ $\pm\,0.73$

Table 1: Forgetting and utility results for ResNet-18 on CIFAR-10. Studied under 1%, 10%, and 50% random forgetting. RA/FA/TA denote retain, forget, and test accuracy;

\mathrm{RMIA}_{\mathrm{AUC}}

and

\mathrm{SMIA}_{\mathrm{AUC}}

report membership inference performance, with 50% corresponding to chance-level discrimination. Gap metrics measure deviation from Retrain , so lower is better. Results are mean

\pm

std over three seeds, with all values in percent. Bold and underlined denote best and second best, where “best” denotes smallest gap to Retrain .

	Forget 1%
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$
Retrain	$99.99$ $\pm\,0.00$	$61.48$ $\pm\,1.23$	$60.89$ $\pm\,0.15$	$49.79$ $\pm\,1.38$	$0.00$ $\pm\,0.00$	$0.00$ $\pm\,0.00$	$50.50$ $\pm\,0.69$	$0.00$ $\pm\,0.00$
Base	$99.99$ $\pm\,0.00$	$99.96$ $\pm\,0.06$	$61.24$ $\pm\,0.07$	$87.77$ $\pm\,0.18$	$19.21$ $\pm\,0.41$	$19.15$ $\pm\,0.69$	$94.68$ $\pm\,0.10$	$20.75$ $\pm\,0.36$
NegGrad	$\bm{99.99}$ $\pm\,0.00$	$99.96$ $\pm\,0.06$	$\uline{61.22}$ $\pm\,0.06$	$87.78$ $\pm\,0.18$	$19.20$ $\pm\,0.42$	$19.15$ $\pm\,0.72$	$94.68$ $\pm\,0.09$	$20.75$ $\pm\,0.36$
NegGrad+	$94.31$ $\pm\,0.13$	$68.78$ $\pm\,1.47$	$48.99$ $\pm\,0.31$	$66.99$ $\pm\,1.96$	$10.52$ $\pm\,0.59$	$14.54$ $\pm\,0.98$	$62.44$ $\pm\,1.16$	$9.21$ $\pm\,0.59$
Finetune	$97.11$ $\pm\,0.19$	$63.70$ $\pm\,1.74$	$52.31$ $\pm\,0.36$	$62.48$ $\pm\,0.22$	$6.60$ $\pm\,0.59$	$10.63$ $\pm\,0.92$	$57.26$ $\pm\,0.66$	$5.11$ $\pm\,0.59$

Table 2: Forgetting and utility results for Swin-T on Tiny-ImageNet. Studied under 1%, 10%, and 50% random forgetting. RA/FA/TA denote retain, forget, and test accuracy;

\mathrm{RMIA}_{\mathrm{AUC}}

and

\mathrm{SMIA}_{\mathrm{AUC}}

report membership inference performance, with 50% corresponding to chance-level discrimination. Gap metrics measure deviation from Retrain , so lower is better. Results are mean

\pm

std over three seeds, with all values in percent. Bold and underlined denote best and second best, where “best” denotes smallest gap to Retrain .

		Pointwise contrast	Aggregated target
		$d_{\mathrm{f}}/d_{\mathrm{h}}$	$d_{\mathrm{tgt}}$ (nats)	$d_{\mathrm{tgt}}/d_{\mathrm{h}}$ (%)	$d_{\mathrm{tgt}}/d_{\mathrm{samp}}$
	Forget $1\%$	$1.70$ $\pm\,0.26$	$0.00056$ $\pm\,0.00003$	$2.66$ $\pm\,0.15$	$0.333$ $\pm\,0.024$
ResNet-18 on CIFAR-10	Forget $10\%$	$1.46$ $\pm\,0.11$	$0.00064$ $\pm\,0.00006$	$2.63$ $\pm\,0.13$	$0.354$ $\pm\,0.023$
	Forget $50\%$	$1.34$ $\pm\,0.06$	$0.00113$ $\pm\,0.00003$	$2.67$ $\pm\,0.09$	$0.435$ $\pm\,0.040$
	Forget $1\%$	$4.04$ $\pm\,0.04$	$0.00689$ $\pm\,0.00023$	$9.52$ $\pm\,0.19$	$0.166$ $\pm\,0.004$
Swin-T on Tiny-ImageNet	Forget $10\%$	$3.90$ $\pm\,0.03$	$0.00755$ $\pm\,0.00021$	$9.72$ $\pm\,0.10$	$0.187$ $\pm\,0.006$

Table 4:

\bm{f_{\theta_{0}}}

–

\bm{f_{\theta_{\mathrm{r}}}}

target similarity under random forgetting. Target-space quantities use

1{,}000

class-matched batches of size

128

per seed. Ratios are formed per seed, and entries report means

\pm

sample standard deviations across three seeds.

Panel of Figure 3 — Figure 3: Held-out subsample ablation. $\mathrm{Gap}_{\mathrm{Avg}}$ is reported as a function of the fraction of the held-out split used to construct ReGUn reference distributions. A fraction of $1$ uses the full held-out split, corresponding to $\approx 9\%$ of the original training set $\mathcal{D}_{\mathrm{orig}}$ , or about 4,500 images for CIFAR-10 and 9,000 for Tiny-ImageNet. Lines show the mean across seeds; shaded regions indicate $\pm 1$ sample standard deviation.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Empirical Runtime Results. Wall-clock runtime of the studied unlearning methods for the unlearning phase. Central vertical bars show the median over 15 runs and horizontal whiskers indicate the interquartile range (25th–75th percentile).

	Forget 1%
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$
Retrain -Teacher	$\uline{99.77}$ $\pm\,0.05$	$97.93$ $\pm\,0.51$	$91.95$ $\pm\,0.03$	$51.08$ $\pm\,1.20$	$1.70$ $\pm\,0.65$	$1.88$ $\pm\,0.69$	$\bm{48.92}$ $\pm\,1.34$	$2.05$ $\pm\,0.53$
Online Student	$99.63$ $\pm\,0.13$	$\bm{97.19}$ $\pm\,0.78$	$91.83$ $\pm\,0.15$	$\bm{50.53}$ $\pm\,1.15$	$\bm{1.47}$ $\pm\,0.67$	$1.72$ $\pm\,0.35$	$48.77$ $\pm\,0.96$	$\bm{1.97}$ $\pm\,0.50$
EMA 0.99	$99.70$ $\pm\,0.09$	$\uline{97.63}$ $\pm\,0.90$	$\uline{91.98}$ $\pm\,0.09$	$50.78$ $\pm\,1.07$	$1.56$ $\pm\,0.62$	$1.72$ $\pm\,0.43$	$\uline{48.89}$ $\pm\,0.92$	$\uline{1.99}$ $\pm\,0.50$
EMA 0.999	$99.70$ $\pm\,0.09$	$97.70$ $\pm\,0.84$	$\uline{91.99}$ $\pm\,0.07$	$50.77$ $\pm\,1.08$	$1.58$ $\pm\,0.64$	$\uline{1.71}$ $\pm\,0.43$	$\uline{48.89}$ $\pm\,0.93$	$2.01$ $\pm\,0.50$
Frozen Base	$\bm{99.81}$ $\pm\,0.03$	$97.70$ $\pm\,0.71$	$\bm{92.15}$ $\pm\,0.17$	$\uline{50.71}$ $\pm\,0.95$	$\uline{1.49}$ $\pm\,0.81$	$\bm{1.59}$ $\pm\,0.47$	$48.42$ $\pm\,0.93$	$2.06$ $\pm\,0.49$

Appendix

Table 6: Teacher ablation full metrics for ResNet-18 on CIFAR-10. Bold and underlined denote best and second best, where “best” denotes smallest gap to Retrain .

	Forget 1%
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$
Retrain -Teacher	$\bm{98.05}$ $\pm\,0.21$	$63.59$ $\pm\,4.28$	$51.24$ $\pm\,1.03$	$57.24$ $\pm\,1.67$	$5.76$ $\pm\,0.42$	$8.54$ $\pm\,0.47$	$\bm{48.38}$ $\pm\,0.52$	$3.96$ $\pm\,1.16$
Online Student	$97.77$ $\pm\,0.45$	$\uline{62.00}$ $\pm\,1.74$	$51.23$ $\pm\,0.88$	$\uline{56.15}$ $\pm\,0.28$	$\uline{4.75}$ $\pm\,0.33$	$\uline{8.00}$ $\pm\,0.81$	$47.82$ $\pm\,0.20$	$3.77$ $\pm\,0.62$
EMA 0.99	$\uline{97.93}$ $\pm\,0.16$	$62.67$ $\pm\,2.80$	$\uline{51.61}$ $\pm\,0.39$	$56.89$ $\pm\,0.40$	$5.06$ $\pm\,0.04$	$8.19$ $\pm\,0.63$	$48.01$ $\pm\,0.35$	$\uline{3.76}$ $\pm\,0.80$
EMA 0.999	$97.89$ $\pm\,0.19$	$62.78$ $\pm\,1.85$	$\bm{51.71}$ $\pm\,0.34$	$56.78$ $\pm\,0.46$	$4.90$ $\pm\,0.07$	$8.08$ $\pm\,0.70$	$47.96$ $\pm\,0.28$	$3.78$ $\pm\,0.59$
Frozen Base	$97.91$ $\pm\,0.15$	$\bm{61.56}$ $\pm\,0.29$	$51.23$ $\pm\,0.23$	$\bm{56.07}$ $\pm\,0.84$	$\bm{4.68}$ $\pm\,0.47$	$\bm{7.97}$ $\pm\,0.65$	$\uline{48.21}$ $\pm\,0.04$	$\bm{3.53}$ $\pm\,0.37$

Appendix

Table 7: Teacher ablation full metrics for Swin-T on Tiny-ImageNet. Bold and underlined denote best and second best, where “best” denotes smallest gap to Retrain .

	Forget 1%
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$
Uniform	$99.70$ $\pm\,0.08$	$\bm{97.48}$ $\pm\,0.90$	$\uline{91.97}$ $\pm\,0.06$	$\uline{50.74}$ $\pm\,0.93$	$\uline{1.52}$ $\pm\,0.31$	$\uline{1.70}$ $\pm\,0.25$	$\uline{48.60}$ $\pm\,1.06$	$\bm{2.03}$ $\pm\,0.52$
Feature-1NN	$\uline{99.71}$ $\pm\,0.08$	$\uline{97.56}$ $\pm\,0.59$	$91.95$ $\pm\,0.15$	$50.80$ $\pm\,0.92$	$1.55$ $\pm\,0.50$	$1.74$ $\pm\,0.27$	$\bm{48.63}$ $\pm\,1.16$	$\uline{2.04}$ $\pm\,0.51$
Class-Conditioned	$\bm{99.81}$ $\pm\,0.03$	$97.70$ $\pm\,0.71$	$\bm{92.15}$ $\pm\,0.17$	$\bm{50.71}$ $\pm\,0.95$	$\bm{1.49}$ $\pm\,0.81$	$\bm{1.59}$ $\pm\,0.47$	$48.42$ $\pm\,0.93$	$2.06$ $\pm\,0.49$
	Forget 10%
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$

Appendix

Table 8: Selector ablation full metrics for ResNet-18 on CIFAR-10. Bold and underlined denote best and second best, where “best” denotes smallest gap to Retrain .

	Forget 1%
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$
Uniform	$\uline{97.90}$ $\pm\,0.30$	$60.63$ $\pm\,2.78$	$50.83$ $\pm\,0.48$	$\bm{55.77}$ $\pm\,0.61$	$\uline{4.88}$ $\pm\,0.15$	$\uline{8.01}$ $\pm\,0.70$	$47.51$ $\pm\,0.46$	$4.00$ $\pm\,0.80$
Feature-1NN	$97.74$ $\pm\,0.06$	$\uline{61.00}$ $\pm\,3.00$	$\uline{51.07}$ $\pm\,0.81$	$56.56$ $\pm\,2.06$	$5.35$ $\pm\,0.29$	$8.29$ $\pm\,0.22$	$\uline{48.15}$ $\pm\,1.12$	$\uline{3.73}$ $\pm\,0.90$
Class-Conditioned	$\bm{97.91}$ $\pm\,0.15$	$\bm{61.56}$ $\pm\,0.29$	$\bm{51.23}$ $\pm\,0.23$	$\uline{56.07}$ $\pm\,0.84$	$\bm{4.68}$ $\pm\,0.47$	$\bm{7.97}$ $\pm\,0.65$	$\bm{48.21}$ $\pm\,0.04$	$\bm{3.53}$ $\pm\,0.37$
	Forget 10%
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$

Appendix

Table 9: Selector ablation full metrics for Swin-T on Tiny-ImageNet. Bold and underlined denote best and second best, where “best” denotes smallest gap to Retrain .

	Forget 1%
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$
Held-out $0.1\%$	$99.54$ $\pm\,0.16$	$95.78$ $\pm\,1.39$	$91.73$ $\pm\,0.14$	$\bm{47.60}$ $\pm\,1.40$	$\bm{1.73}$ $\pm\,0.42$	$\bm{2.24}$ $\pm\,0.77$	$\bm{46.30}$ $\pm\,0.81$	$\uline{2.28}$ $\pm\,0.56$
Held-out $0.3\%$	$99.55$ $\pm\,0.08$	$\bm{94.74}$ $\pm\,1.99$	$\bm{91.83}$ $\pm\,0.09$	$\uline{47.46}$ $\pm\,1.59$	$\uline{1.78}$ $\pm\,0.76$	$\uline{2.26}$ $\pm\,0.78$	$\uline{45.72}$ $\pm\,1.27$	$\bm{2.14}$ $\pm\,0.71$
Held-out $1\%$	$99.57$ $\pm\,0.04$	$95.19$ $\pm\,1.81$	$91.77$ $\pm\,0.13$	$47.15$ $\pm\,1.68$	$1.87$ $\pm\,0.71$	$2.45$ $\pm\,0.91$	$45.46$ $\pm\,1.37$	$2.32$ $\pm\,0.69$
Held-out $3\%$	$99.58$ $\pm\,0.04$	$\uline{95.11}$ $\pm\,1.74$	$91.77$ $\pm\,0.17$	$47.17$ $\pm\,1.69$	$1.84$ $\pm\,0.72$	$2.44$ $\pm\,0.94$	$45.47$ $\pm\,1.36$	$2.30$ $\pm\,0.68$
Held-out $10\%$	$\uline{99.58}$ $\pm\,0.04$	$95.11$ $\pm\,1.94$	$91.76$ $\pm\,0.16$	$47.15$ $\pm\,1.67$	$1.88$ $\pm\,0.74$	$2.45$ $\pm\,0.91$	$45.46$ $\pm\,1.37$	$2.30$ $\pm\,0.71$

Appendix

Table 10: Held-out subsample ablation full metrics for ResNet-18 on CIFAR-10. Bold and underlined denote best and second best, where “best” denotes smallest gap to Retrain .

	Forget 1%
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$
Held-out $0.1\%$	$97.78$ $\pm\,0.15$	$54.41$ $\pm\,2.79$	$50.44$ $\pm\,0.57$	$\uline{54.85}$ $\pm\,1.31$	$6.18$ $\pm\,0.91$	$\uline{7.75}$ $\pm\,0.58$	$46.97$ $\pm\,0.65$	$5.82$ $\pm\,0.81$
Held-out $0.3\%$	$97.79$ $\pm\,0.47$	$57.26$ $\pm\,1.22$	$51.03$ $\pm\,0.96$	$\bm{54.76}$ $\pm\,1.83$	$5.30$ $\pm\,0.63$	$\bm{7.41}$ $\pm\,1.19$	$47.00$ $\pm\,1.03$	$4.95$ $\pm\,0.60$
Held-out $1\%$	$97.98$ $\pm\,0.25$	$60.19$ $\pm\,1.70$	$50.91$ $\pm\,0.98$	$55.86$ $\pm\,1.59$	$4.97$ $\pm\,0.75$	$8.02$ $\pm\,1.21$	$47.75$ $\pm\,0.99$	$4.01$ $\pm\,0.66$
Held-out $3\%$	$97.91$ $\pm\,0.31$	$60.44$ $\pm\,0.68$	$51.02$ $\pm\,0.70$	$56.01$ $\pm\,1.16$	$\bm{4.79}$ $\pm\,0.65$	$8.04$ $\pm\,1.12$	$47.96$ $\pm\,0.79$	$3.88$ $\pm\,0.48$
Held-out $10\%$	$\bm{98.04}$ $\pm\,0.28$	$\uline{61.59}$ $\pm\,2.02$	$\bm{51.24}$ $\pm\,1.05$	$56.47$ $\pm\,1.09$	$\uline{4.86}$ $\pm\,0.56$	$8.16$ $\pm\,1.01$	$\uline{48.12}$ $\pm\,0.54$	$\bm{3.52}$ $\pm\,0.69$

Appendix

Table 11: Held-out subsample ablation full metrics for Swin-T on Tiny-ImageNet. Bold and underlined denote best and second best, where “best” denotes smallest gap to Retrain .

Panel of Figure 5 — Figure 5: Additional forgetting–utility trade-offs. Trade-off curves for ResNet-18 on CIFAR-100 and Tiny-ImageNet under random 10% forgetting. The $y$ -axis reports test accuracy and the $x$ -axis reports the $\mathrm{RMIA}_{\mathrm{AUC}}$ gap to the retrain-from-scratch baseline; higher TA and proximity to the dashed vertical line is better. Where supported, we sweep $w\in[0.1,0.9]$ in the retain–forget objective, with an additional $w=0.99$ for LUR and BadTeach . Points are labeled by $w$ ; * marks variants reparameterized from the original loss to sweep $w$ .

	Forget 1%
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$
Retrain	$99.98$ $\pm\,0.01$	$74.81$ $\pm\,1.22$	$75.33$ $\pm\,0.29$	$49.56$ $\pm\,1.56$	$0.00$ $\pm\,0.00$	$0.00$ $\pm\,0.00$	$48.73$ $\pm\,0.78$	$0.00$ $\pm\,0.00$
Base	$99.98$ $\pm\,0.01$	$100.00$ $\pm\,0.00$	$75.52$ $\pm\,0.36$	$76.09$ $\pm\,1.32$	$12.84$ $\pm\,1.28$	$13.47$ $\pm\,0.68$	$75.22$ $\pm\,0.05$	$12.97$ $\pm\,0.38$
NegGrad	$99.02$ $\pm\,0.84$	$96.37$ $\pm\,1.45$	$\uline{72.35}$ $\pm\,1.90$	$73.79$ $\pm\,0.43$	$12.26$ $\pm\,1.39$	$13.44$ $\pm\,1.30$	$68.36$ $\pm\,2.31$	$11.28$ $\pm\,0.93$
NegGrad+	$97.38$ $\pm\,1.84$	$90.81$ $\pm\,2.87$	$69.10$ $\pm\,2.39$	$69.96$ $\pm\,1.57$	$11.13$ $\pm\,1.38$	$13.15$ $\pm\,1.76$	$64.29$ $\pm\,2.48$	$10.10$ $\pm\,1.27$
Finetune	$91.98$ $\pm\,3.58$	$\uline{75.63}$ $\pm\,4.69$	$66.67$ $\pm\,1.27$	$59.76$ $\pm\,3.26$	$7.04$ $\pm\,1.15$	$9.27$ $\pm\,1.75$	$54.46$ $\pm\,2.58$	$5.80$ $\pm\,1.68$

Appendix

Table 12: Forgetting and utility results for ResNet-18 on CIFAR-100. Studied under 1%, 10%, and 50% random forgetting. RA/FA/TA denote retain, forget, and test accuracy;

\mathrm{RMIA}_{\mathrm{AUC}}

and

\mathrm{SMIA}_{\mathrm{AUC}}

report membership inference performance, with 50% corresponding to chance-level discrimination. Gap metrics measure deviation from Retrain , so lower is better. Results are mean

\pm

std over three seeds, with all values in percent. Bold and underlined denote best and second best, where “best” denotes smallest gap to Retrain .

	Forget 1%
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$
Retrain	$99.98$ $\pm\,0.00$	$58.52$ $\pm\,0.71$	$59.54$ $\pm\,0.21$	$49.28$ $\pm\,1.14$	$0.00$ $\pm\,0.00$	$0.00$ $\pm\,0.00$	$49.84$ $\pm\,0.85$	$0.00$ $\pm\,0.00$
Base	$99.98$ $\pm\,0.00$	$99.96$ $\pm\,0.06$	$59.64$ $\pm\,0.44$	$81.56$ $\pm\,0.56$	$18.50$ $\pm\,0.61$	$16.27$ $\pm\,0.90$	$84.68$ $\pm\,0.16$	$19.10$ $\pm\,0.31$
NegGrad	$\bm{99.98}$ $\pm\,0.00$	$99.72$ $\pm\,0.24$	$\bm{58.85}$ $\pm\,0.60$	$81.44$ $\pm\,0.69$	$18.29$ $\pm\,0.67$	$16.11$ $\pm\,1.07$	$83.66$ $\pm\,0.09$	$18.93$ $\pm\,0.33$
NegGrad+	$95.68$ $\pm\,0.05$	$67.22$ $\pm\,2.20$	$49.40$ $\pm\,0.67$	$62.23$ $\pm\,1.09$	$8.80$ $\pm\,0.59$	$11.23$ $\pm\,0.45$	$59.47$ $\pm\,1.74$	$8.19$ $\pm\,0.77$
Finetune	$89.92$ $\pm\,6.35$	$61.67$ $\pm\,8.33$	$51.55$ $\pm\,0.69$	$63.06$ $\pm\,7.81$	$9.13$ $\pm\,1.00$	$10.57$ $\pm\,3.13$	$57.23$ $\pm\,5.13$	$7.15$ $\pm\,2.93$

Appendix

Table 13: Forgetting and utility results for ResNet-18 on Tiny-ImageNet. Studied under 1%, 10%, and 50% random forgetting. RA/FA/TA denote retain, forget, and test accuracy;

\mathrm{RMIA}_{\mathrm{AUC}}

and

\mathrm{SMIA}_{\mathrm{AUC}}

report membership inference performance, with 50% corresponding to chance-level discrimination. Gap metrics measure deviation from Retrain , so lower is better. Results are mean

\pm

std over three seeds, with all values in percent. Bold and underlined denote best and second best, where “best” denotes smallest gap to Retrain .

	Full-class forgetting: one target class
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$
Retrain	$100.00$ $\pm\,0.00$	$0.00$ $\pm\,0.00$	$85.16$ $\pm\,0.03$	$50.50$ $\pm\,0.15$	$0.00$ $\pm\,0.00$	$0.00$ $\pm\,0.00$	$50.45$ $\pm\,0.40$	$0.00$ $\pm\,0.00$
Base	$100.00$ $\pm\,0.00$	$100.00$ $\pm\,0.00$	$94.35$ $\pm\,0.31$	$52.39$ $\pm\,0.39$	$27.83$ $\pm\,0.13$	$5.65$ $\pm\,0.25$	$59.24$ $\pm\,0.19$	$29.50$ $\pm\,0.15$
NegGrad	$55.30$ $\pm\,2.76$	$\bm{0.00}$ $\pm\,0.00$	$47.52$ $\pm\,2.30$	$49.20$ $\pm\,0.28$	$20.96$ $\pm\,1.22$	$19.58$ $\pm\,1.07$	$48.86$ $\pm\,0.09$	$21.07$ $\pm\,1.21$
NegGrad+	$\uline{99.72}$ $\pm\,0.03$	$19.65$ $\pm\,0.82$	$85.83$ $\pm\,0.26$	$55.93$ $\pm\,0.22$	$6.53$ $\pm\,0.33$	$3.10$ $\pm\,0.27$	$55.82$ $\pm\,0.13$	$6.51$ $\pm\,0.38$
Finetune	$86.37$ $\pm\,0.55$	$\uline{0.00}$ $\pm\,0.00$	$74.59$ $\pm\,0.38$	$51.25$ $\pm\,0.28$	$6.33$ $\pm\,0.16$	$5.85$ $\pm\,0.08$	$51.48$ $\pm\,0.04$	$6.40$ $\pm\,0.18$

Appendix

Table 14: Full-class forgetting results for ResNet-18 on CIFAR-10. RA/FA/TA denote retain, forget, and test accuracy;

\mathrm{RMIA}_{\mathrm{AUC}}

and

\mathrm{SMIA}_{\mathrm{AUC}}

report membership inference performance, with 50% corresponding to chance-level discrimination. Gap metrics measure deviation from Retrain , so lower is better. Results are averaged over all 10 classes within each seed and reported as mean

\pm

sample std across three seeds, with all values in percent. Bold and underlined denote best and second best, where “best” denotes smallest gap to Retrain .

	Half-class forgetting: one target class
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$
Retrain	$100.00$ $\pm\,0.00$	$90.62$ $\pm\,0.15$	$93.93$ $\pm\,0.05$	$50.29$ $\pm\,0.17$	$0.00$ $\pm\,0.00$	$0.00$ $\pm\,0.00$	$50.27$ $\pm\,0.12$	$0.00$ $\pm\,0.00$
Base	$100.00$ $\pm\,0.00$	$100.00$ $\pm\,0.00$	$94.35$ $\pm\,0.31$	$55.79$ $\pm\,0.43$	$3.84$ $\pm\,0.04$	$2.99$ $\pm\,0.16$	$59.24$ $\pm\,0.15$	$4.71$ $\pm\,0.12$
NegGrad	$76.10$ $\pm\,1.63$	$0.22$ $\pm\,0.07$	$67.60$ $\pm\,1.50$	$\uline{48.55}$ $\pm\,0.14$	$35.68$ $\pm\,0.86$	$14.20$ $\pm\,0.85$	$\uline{48.71}$ $\pm\,0.12$	$35.60$ $\pm\,0.87$
NegGrad+	$98.44$ $\pm\,0.03$	$71.68$ $\pm\,0.67$	$90.27$ $\pm\,0.20$	$59.87$ $\pm\,0.18$	$8.44$ $\pm\,0.29$	$6.62$ $\pm\,0.18$	$56.24$ $\pm\,0.14$	$7.53$ $\pm\,0.26$
Finetune	$\uline{99.99}$ $\pm\,0.00$	$99.99$ $\pm\,0.00$	$\uline{94.18}$ $\pm\,0.26$	$56.26$ $\pm\,0.17$	$3.94$ $\pm\,0.02$	$3.19$ $\pm\,0.12$	$58.76$ $\pm\,0.18$	$4.57$ $\pm\,0.09$

Appendix

Table 15: Half-class forgetting results for ResNet-18 on CIFAR-10. RA/FA/TA denote retain, forget, and test accuracy;

\mathrm{RMIA}_{\mathrm{AUC}}

and

\mathrm{SMIA}_{\mathrm{AUC}}

\pm

sample std across three seeds, with all values in percent. Bold and underlined denote best and second best, where “best” denotes smallest gap to Retrain .

	Full-class forgetting: one 20-class group
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$
Retrain	$99.98$ $\pm\,0.00$	$0.00$ $\pm\,0.00$	$54.82$ $\pm\,0.09$	$49.83$ $\pm\,0.48$	$0.00$ $\pm\,0.00$	$0.00$ $\pm\,0.00$	$49.71$ $\pm\,0.14$	$0.00$ $\pm\,0.00$
Base	$99.98$ $\pm\,0.00$	$99.99$ $\pm\,0.00$	$61.01$ $\pm\,0.10$	$86.94$ $\pm\,0.38$	$35.82$ $\pm\,0.07$	$21.65$ $\pm\,0.14$	$94.05$ $\pm\,0.12$	$37.63$ $\pm\,0.10$
NegGrad	$\bm{99.98}$ $\pm\,0.00$	$99.99$ $\pm\,0.00$	$61.00$ $\pm\,0.10$	$86.97$ $\pm\,0.38$	$35.83$ $\pm\,0.07$	$21.67$ $\pm\,0.14$	$94.04$ $\pm\,0.12$	$37.63$ $\pm\,0.10$
NegGrad+	$38.59$ $\pm\,1.91$	$\bm{0.00}$ $\pm\,0.00$	$27.23$ $\pm\,1.20$	$\bm{50.09}$ $\pm\,0.37$	$22.52$ $\pm\,0.93$	$14.35$ $\pm\,0.90$	$\bm{49.16}$ $\pm\,0.11$	$22.40$ $\pm\,0.83$
Finetune	$52.81$ $\pm\,0.29$	$0.00$ $\pm\,0.01$	$33.35$ $\pm\,0.21$	$62.89$ $\pm\,0.09$	$20.43$ $\pm\,0.11$	$17.26$ $\pm\,0.27$	$62.74$ $\pm\,0.09$	$20.42$ $\pm\,0.06$

Appendix

Table 16: Full-class forgetting results for Swin-T on Tiny-ImageNet. RA/FA/TA denote retain, forget, and test accuracy;

\mathrm{RMIA}_{\mathrm{AUC}}

and

\mathrm{SMIA}_{\mathrm{AUC}}

report membership inference performance, with 50% corresponding to chance-level discrimination. Gap metrics measure deviation from Retrain , so lower is better. Results are averaged over the three 20-class forgetting groups within each seed and reported as mean

\pm

sample std across three seeds, with all values in percent. Bold and underlined denote best and second best, where “best” denotes smallest gap to Retrain .

	Half-class forgetting: one 20-class group
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$
Retrain	$99.98$ $\pm\,0.00$	$52.38$ $\pm\,0.49$	$60.14$ $\pm\,0.35$	$49.75$ $\pm\,0.26$	$0.00$ $\pm\,0.00$	$0.00$ $\pm\,0.00$	$49.69$ $\pm\,0.36$	$0.00$ $\pm\,0.00$
Base	$99.98$ $\pm\,0.00$	$99.98$ $\pm\,0.00$	$61.01$ $\pm\,0.10$	$80.19$ $\pm\,0.63$	$19.73$ $\pm\,0.31$	$15.65$ $\pm\,0.39$	$94.05$ $\pm\,0.14$	$23.21$ $\pm\,0.22$
NegGrad	$\bm{99.98}$ $\pm\,0.00$	$99.98$ $\pm\,0.00$	$\bm{61.01}$ $\pm\,0.10$	$80.22$ $\pm\,0.63$	$19.73$ $\pm\,0.31$	$15.67$ $\pm\,0.39$	$94.05$ $\pm\,0.14$	$23.21$ $\pm\,0.22$
NegGrad+	$75.01$ $\pm\,1.15$	$\uline{50.79}$ $\pm\,5.55$	$44.96$ $\pm\,0.58$	$66.68$ $\pm\,2.42$	$15.26$ $\pm\,0.47$	$16.05$ $\pm\,1.12$	$\uline{60.60}$ $\pm\,1.72$	$13.76$ $\pm\,0.61$
Finetune	$99.88$ $\pm\,0.02$	$99.89$ $\pm\,0.02$	$56.97$ $\pm\,0.35$	$78.97$ $\pm\,0.26$	$20.00$ $\pm\,0.19$	$16.19$ $\pm\,0.34$	$87.93$ $\pm\,0.68$	$22.26$ $\pm\,0.34$

Appendix

Table 17: Half-class forgetting results for Swin-T on Tiny-ImageNet. RA/FA/TA denote retain, forget, and test accuracy;

\mathrm{RMIA}_{\mathrm{AUC}}

and

\mathrm{SMIA}_{\mathrm{AUC}}

report membership inference performance, with 50% corresponding to chance-level discrimination. Gap metrics measure deviation from Retrain , so lower is better. Results are averaged over the three 20-class forgetting groups within each seed and reported as mean

\pm

sample std across three seeds, with all values in percent. Bold and underlined denote best and second best, where “best” denotes smallest gap to Retrain .

	Full-class forgetting
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$
ResNet-18 on CIFAR-10
Retrain -Teacher	$99.47$ $\pm\,0.04$	$0.00$ $\pm\,0.00$	$83.72$ $\pm\,0.15$	$50.82$ $\pm\,0.29$	$0.79$ $\pm\,0.11$	$1.32$ $\pm\,0.23$	$53.21$ $\pm\,0.24$	$1.27$ $\pm\,0.10$
Frozen Base	$99.27$ $\pm\,0.25$	$3.42$ $\pm\,0.60$	$83.63$ $\pm\,0.23$	$50.00$ $\pm\,0.28$	$1.73$ $\pm\,0.25$	$1.39$ $\pm\,0.17$	$51.30$ $\pm\,0.06$	$1.90$ $\pm\,0.24$
Swin-T on Tiny-ImageNet
Retrain -Teacher	$97.80$ $\pm\,0.19$	$0.02$ $\pm\,0.01$	$52.50$ $\pm\,0.02$	$63.00$ $\pm\,0.91$	$4.42$ $\pm\,0.08$	$7.74$ $\pm\,0.25$	$70.73$ $\pm\,0.25$	$6.39$ $\pm\,0.07$

Appendix

Table 18: Teacher ablation under class forgetting. Comparison of the frozen-base and Retrain teachers under full- and half-class forgetting for ResNet-18 on CIFAR-10 and Swin-T on Tiny-ImageNet.

Machine Unlearning

Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.

See all 351 papers

Multi-Objective Reference-Aligned Machine Unlearning
May 29, 2026Rasa Khosrowshahli, Stephen Asobiela, Beatrice Ombuki-Berman +1Machine Unlearning Catastrophic Forgetting
Department of Computer Science, Brock University, St. Catharines, ON, Canada · Yousef Haj-Ahmad Department of Engineering, Brock University, St. Catharines, ON, Canada
Models Designed to Forget: Machine Unlearning via Key Deletion
Mar 16, 2026Sonia Laguna, Jorge da Silva Goncalves, Moritz Vandenhirtz +3Machine Unlearning
Department of Computer Science, ETH Zurich, Switzerland
Unmerge: Efficient Machine Unlearning via Task Arithmetic
Sep 30, 2026Haoran Tang, Andrew Tan, Rajiv KhannaMachine Unlearning
Department of Computer Science Purdue University

May 29, 2026J/K move · Enter open · S save

Multi-Objective Reference-Aligned Machine Unlearning

Rasa Khosrowshahli, Stephen Asobiela, Beatrice Ombuki-Berman, Shahryar Rahnamayan

Department of Computer Science, Brock University, St. Catharines, ON, Canada · Yousef Haj-Ahmad Department of Engineering, Brock University, St. Catharines, ON, Canada

Machine Unlearning Catastrophic Forgetting

Open paper AskPDF arXiv

Machine unlearning aims to remove the influence of specific training samples while preserving the model's utility. Existing single-objective approaches, such as gradient ascent or random relabeling, often induce catastrophic forgetting due to conflicting optimization dynamics and unbounded forgetting objectives that cause the model to drift from its pre-trained knowledge. We propose Reference-Aligned UnLearning (RAUL), a multi-objective framework that jointly optimizes forgetting and retention by replacing unbounded loss maximization with a bounded KL alignment of predictions on forgotten samples toward a reference distribution representing unseen data, instantiated either as a uniform distribution or an empirical distribution from a held-out reference set, which constrains the forgetting objective and reduces gradient conflict with retention. The resulting multi-objective optimization (MOO) problem is solved via Jacobian descent, which aggregates multiple gradients into a direction that does not conflict. Our results demonstrate that RAUL achieves the closest gap compared to full retraining.

Abstract

Machine Unlearning

Explore similar work

Multi-Objective Reference-Aligned Machine Unlearning

Models Designed to Forget: Machine Unlearning via Key Deletion

Unmerge: Efficient Machine Unlearning via Task Arithmetic