Reference-Guided Machine Unlearning

Jonas Mirlach; Sonia Laguna; Julia E. Vogt

cs.LGMar 11, 2026

Reference-Guided Machine Unlearning

Authors: Jonas Mirlach, Sonia Laguna, Julia E. Vogt

Organizations: Department of Computer Science, ETH Zurich

Abstract

Machine unlearning aims to remove the influence of specific training data from a model while preserving its general utility. In vision, many approximate unlearning methods pursue this goal through degradation-based heuristics, such as loss maximization or random labeling. Yet making a model worse on forget samples is not the same as making it behave as if those examples had never been seen: these signals can be poorly conditioned, destabilize optimization, and harm generalization. We argue that approximate unlearning should instead prioritize distributional indistinguishability, aligning the model's predictive behavior on forget data with that on truly unseen data. Motivated by this principle, we propose Reference-Guided Unlearning (ReGUn), a vision unlearning framework that uses disjoint held-out data to construct a principled, class-conditioned reference distribution for distillation. Rather than explicitly degrading predictions on forget examples, ReGUn guides them toward non-member behavior through held-out supervision. Across multiple architectures, natural image datasets, and forget fractions, ReGUn achieves a competitive forgetting--utility trade-off relative to standard approximate baselines while closely matching retrain-like membership inference behavior. As one instantiation of this principle, the results suggest that simple objectives designed around indistinguishability can provide an effective alternative to complex degradation-based unlearning procedures.

Figures & tables

Figure 1: Unlearning optimization in ReGUn . In ReGUn , unlearning follows a reference-guided distillation objective based on indistinguishability . Given a forget minibatch $B_{\mathrm{f}}\sim\mathcal{D}_{\mathrm{f}}$ , ReGUn samples a held-out minibatch $B_{\mathrm{h}}\sim\mathcal{D}_{\mathrm{h}}$ such that the class histogram of $B_{\mathrm{h}}$ matches that of $B_{\mathrm{f}}$ . The samples in $B_{\mathrm{h}}$ are passed through a frozen reference model to construct a reference distribution $q$ , which represents the desired “unseen” behavior for the forget examples. The current model $f_{\theta}$ is then updated by distilling its predictions on $B_{\mathrm{f}}$ toward this reference distribution, encouraging forget samples to become indistinguishable from non-member data. In parallel, a standard cross-entropy loss on retain samples is used to preserve model utility.

Panel of Figure 2 — Figure 2: Forgetting–utility trade-offs. Shown for ResNet-18 on CIFAR-10 and Swin-T on Tiny-ImageNet under random 10% forgetting. The $y$ -axis reports test accuracy and the $x$ -axis the $\mathrm{RMIA}_{\mathrm{AUC}}$ gap to the retrain-from-scratch baseline; higher TA and proximity to the dashed vertical line is better. Where supported, including ReGUn , we sweep $w\in[0.1,0.9]$ in the retain–forget objective, with an additional $w=0.99$ for NegGrad+ , LUR , and BadTeach . Points are labeled by $w$ ; * marks variants reparameterized from the original loss to sweep $w$ .

	Forget 1%
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$
Retrain	$100.00$ $\pm\,0.00$	$94.22$ $\pm\,1.24$	$94.34$ $\pm\,0.02$	$49.98$ $\pm\,1.26$	$0.00$ $\pm\,0.00$	$0.00$ $\pm\,0.00$	$50.78$ $\pm\,0.93$	$0.00$ $\pm\,0.00$
Base	$100.00$ $\pm\,0.00$	$100.00$ $\pm\,0.00$	$94.20$ $\pm\,0.11$	$60.11$ $\pm\,1.30$	$3.88$ $\pm\,0.90$	$5.14$ $\pm\,0.93$	$59.99$ $\pm\,0.38$	$3.78$ $\pm\,0.40$
NegGrad	$\bm{100.00}$ $\pm\,0.00$	$100.00$ $\pm\,0.00$	$\bm{94.17}$ $\pm\,0.01$	$59.80$ $\pm\,1.12$	$3.82$ $\pm\,0.89$	$5.19$ $\pm\,0.99$	$59.77$ $\pm\,0.36$	$3.74$ $\pm\,0.40$
NegGrad+	$98.45$ $\pm\,2.68$	$97.85$ $\pm\,3.72$	$91.80$ $\pm\,3.75$	$57.95$ $\pm\,2.16$	$3.77$ $\pm\,0.72$	$5.39$ $\pm\,1.12$	$57.39$ $\pm\,3.54$	$3.58$ $\pm\,1.77$
Finetune	$97.44$ $\pm\,0.49$	$\bm{94.67}$ $\pm\,1.94$	$90.90$ $\pm\,0.57$	$54.78$ $\pm\,0.97$	$2.88$ $\pm\,0.26$	$4.26$ $\pm\,0.34$	$52.42$ $\pm\,1.34$	$\uline{2.02}$ $\pm\,0.73$

Table 1: Forgetting and utility results for ResNet-18 on CIFAR-10. Studied under 1%, 10%, and 50% random forgetting. RA/FA/TA denote retain, forget, and test accuracy;

\mathrm{RMIA}_{\mathrm{AUC}}

and

\mathrm{SMIA}_{\mathrm{AUC}}

report membership inference performance, with 50% corresponding to chance-level discrimination. Gap metrics measure deviation from Retrain , so lower is better. Results are mean

\pm

std over three seeds, with all values in percent. Bold and underlined denote best and second best, where “best” denotes smallest gap to Retrain .

	Forget 1%
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$
Retrain	$99.99$ $\pm\,0.00$	$61.48$ $\pm\,1.23$	$60.89$ $\pm\,0.15$	$49.79$ $\pm\,1.38$	$0.00$ $\pm\,0.00$	$0.00$ $\pm\,0.00$	$50.50$ $\pm\,0.69$	$0.00$ $\pm\,0.00$
Base	$99.99$ $\pm\,0.00$	$99.96$ $\pm\,0.06$	$61.24$ $\pm\,0.07$	$87.77$ $\pm\,0.18$	$19.21$ $\pm\,0.41$	$19.15$ $\pm\,0.69$	$94.68$ $\pm\,0.10$	$20.75$ $\pm\,0.36$
NegGrad	$\bm{99.99}$ $\pm\,0.00$	$99.96$ $\pm\,0.06$	$\uline{61.22}$ $\pm\,0.06$	$87.78$ $\pm\,0.18$	$19.20$ $\pm\,0.42$	$19.15$ $\pm\,0.72$	$94.68$ $\pm\,0.09$	$20.75$ $\pm\,0.36$
NegGrad+	$94.31$ $\pm\,0.13$	$68.78$ $\pm\,1.47$	$48.99$ $\pm\,0.31$	$66.99$ $\pm\,1.96$	$10.52$ $\pm\,0.59$	$14.54$ $\pm\,0.98$	$62.44$ $\pm\,1.16$	$9.21$ $\pm\,0.59$
Finetune	$97.11$ $\pm\,0.19$	$63.70$ $\pm\,1.74$	$52.31$ $\pm\,0.36$	$62.48$ $\pm\,0.22$	$6.60$ $\pm\,0.59$	$10.63$ $\pm\,0.92$	$57.26$ $\pm\,0.66$	$5.11$ $\pm\,0.59$

Table 2: Forgetting and utility results for Swin-T on Tiny-ImageNet. Studied under 1%, 10%, and 50% random forgetting. RA/FA/TA denote retain, forget, and test accuracy;

\mathrm{RMIA}_{\mathrm{AUC}}

and

\mathrm{SMIA}_{\mathrm{AUC}}

report membership inference performance, with 50% corresponding to chance-level discrimination. Gap metrics measure deviation from Retrain , so lower is better. Results are mean

\pm

std over three seeds, with all values in percent. Bold and underlined denote best and second best, where “best” denotes smallest gap to Retrain .

		Pointwise contrast	Aggregated target
		$d_{\mathrm{f}}/d_{\mathrm{h}}$	$d_{\mathrm{tgt}}$ (nats)	$d_{\mathrm{tgt}}/d_{\mathrm{h}}$ (%)	$d_{\mathrm{tgt}}/d_{\mathrm{samp}}$
	Forget $1\%$	$1.70$ $\pm\,0.26$	$0.00056$ $\pm\,0.00003$	$2.66$ $\pm\,0.15$	$0.333$ $\pm\,0.024$
ResNet-18 on CIFAR-10	Forget $10\%$	$1.46$ $\pm\,0.11$	$0.00064$ $\pm\,0.00006$	$2.63$ $\pm\,0.13$	$0.354$ $\pm\,0.023$
	Forget $50\%$	$1.34$ $\pm\,0.06$	$0.00113$ $\pm\,0.00003$	$2.67$ $\pm\,0.09$	$0.435$ $\pm\,0.040$
	Forget $1\%$	$4.04$ $\pm\,0.04$	$0.00689$ $\pm\,0.00023$	$9.52$ $\pm\,0.19$	$0.166$ $\pm\,0.004$
Swin-T on Tiny-ImageNet	Forget $10\%$	$3.90$ $\pm\,0.03$	$0.00755$ $\pm\,0.00021$	$9.72$ $\pm\,0.10$	$0.187$ $\pm\,0.006$

Table 4:

\bm{f_{\theta_{0}}}

–

\bm{f_{\theta_{\mathrm{r}}}}

target similarity under random forgetting. Target-space quantities use

1{,}000

class-matched batches of size

128

per seed. Ratios are formed per seed, and entries report means

\pm

sample standard deviations across three seeds.

Panel of Figure 3 — Figure 3: Held-out subsample ablation. $\mathrm{Gap}_{\mathrm{Avg}}$ is reported as a function of the fraction of the held-out split used to construct ReGUn reference distributions. A fraction of $1$ uses the full held-out split, corresponding to $\approx 9\%$ of the original training set $\mathcal{D}_{\mathrm{orig}}$ , or about 4,500 images for CIFAR-10 and 9,000 for Tiny-ImageNet. Lines show the mean across seeds; shaded regions indicate $\pm 1$ sample standard deviation.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Empirical Runtime Results. Wall-clock runtime of the studied unlearning methods for the unlearning phase. Central vertical bars show the median over 15 runs and horizontal whiskers indicate the interquartile range (25th–75th percentile).

	Forget 1%
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$
Retrain -Teacher	$\uline{99.77}$ $\pm\,0.05$	$97.93$ $\pm\,0.51$	$91.95$ $\pm\,0.03$	$51.08$ $\pm\,1.20$	$1.70$ $\pm\,0.65$	$1.88$ $\pm\,0.69$	$\bm{48.92}$ $\pm\,1.34$	$2.05$ $\pm\,0.53$
Online Student	$99.63$ $\pm\,0.13$	$\bm{97.19}$ $\pm\,0.78$	$91.83$ $\pm\,0.15$	$\bm{50.53}$ $\pm\,1.15$	$\bm{1.47}$ $\pm\,0.67$	$1.72$ $\pm\,0.35$	$48.77$ $\pm\,0.96$	$\bm{1.97}$ $\pm\,0.50$
EMA 0.99	$99.70$ $\pm\,0.09$	$\uline{97.63}$ $\pm\,0.90$	$\uline{91.98}$ $\pm\,0.09$	$50.78$ $\pm\,1.07$	$1.56$ $\pm\,0.62$	$1.72$ $\pm\,0.43$	$\uline{48.89}$ $\pm\,0.92$	$\uline{1.99}$ $\pm\,0.50$
EMA 0.999	$99.70$ $\pm\,0.09$	$97.70$ $\pm\,0.84$	$\uline{91.99}$ $\pm\,0.07$	$50.77$ $\pm\,1.08$	$1.58$ $\pm\,0.64$	$\uline{1.71}$ $\pm\,0.43$	$\uline{48.89}$ $\pm\,0.93$	$2.01$ $\pm\,0.50$
Frozen Base	$\bm{99.81}$ $\pm\,0.03$	$97.70$ $\pm\,0.71$	$\bm{92.15}$ $\pm\,0.17$	$\uline{50.71}$ $\pm\,0.95$	$\uline{1.49}$ $\pm\,0.81$	$\bm{1.59}$ $\pm\,0.47$	$48.42$ $\pm\,0.93$	$2.06$ $\pm\,0.49$

Appendix

Table 6: Teacher ablation full metrics for ResNet-18 on CIFAR-10. Bold and underlined denote best and second best, where “best” denotes smallest gap to Retrain .

	Forget 1%
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$
Retrain -Teacher	$\bm{98.05}$ $\pm\,0.21$	$63.59$ $\pm\,4.28$	$51.24$ $\pm\,1.03$	$57.24$ $\pm\,1.67$	$5.76$ $\pm\,0.42$	$8.54$ $\pm\,0.47$	$\bm{48.38}$ $\pm\,0.52$	$3.96$ $\pm\,1.16$
Online Student	$97.77$ $\pm\,0.45$	$\uline{62.00}$ $\pm\,1.74$	$51.23$ $\pm\,0.88$	$\uline{56.15}$ $\pm\,0.28$	$\uline{4.75}$ $\pm\,0.33$	$\uline{8.00}$ $\pm\,0.81$	$47.82$ $\pm\,0.20$	$3.77$ $\pm\,0.62$
EMA 0.99	$\uline{97.93}$ $\pm\,0.16$	$62.67$ $\pm\,2.80$	$\uline{51.61}$ $\pm\,0.39$	$56.89$ $\pm\,0.40$	$5.06$ $\pm\,0.04$	$8.19$ $\pm\,0.63$	$48.01$ $\pm\,0.35$	$\uline{3.76}$ $\pm\,0.80$
EMA 0.999	$97.89$ $\pm\,0.19$	$62.78$ $\pm\,1.85$	$\bm{51.71}$ $\pm\,0.34$	$56.78$ $\pm\,0.46$	$4.90$ $\pm\,0.07$	$8.08$ $\pm\,0.70$	$47.96$ $\pm\,0.28$	$3.78$ $\pm\,0.59$
Frozen Base	$97.91$ $\pm\,0.15$	$\bm{61.56}$ $\pm\,0.29$	$51.23$ $\pm\,0.23$	$\bm{56.07}$ $\pm\,0.84$	$\bm{4.68}$ $\pm\,0.47$	$\bm{7.97}$ $\pm\,0.65$	$\uline{48.21}$ $\pm\,0.04$	$\bm{3.53}$ $\pm\,0.37$

Appendix

Table 7: Teacher ablation full metrics for Swin-T on Tiny-ImageNet. Bold and underlined denote best and second best, where “best” denotes smallest gap to Retrain .

	Forget 1%
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$
Uniform	$99.70$ $\pm\,0.08$	$\bm{97.48}$ $\pm\,0.90$	$\uline{91.97}$ $\pm\,0.06$	$\uline{50.74}$ $\pm\,0.93$	$\uline{1.52}$ $\pm\,0.31$	$\uline{1.70}$ $\pm\,0.25$	$\uline{48.60}$ $\pm\,1.06$	$\bm{2.03}$ $\pm\,0.52$
Feature-1NN	$\uline{99.71}$ $\pm\,0.08$	$\uline{97.56}$ $\pm\,0.59$	$91.95$ $\pm\,0.15$	$50.80$ $\pm\,0.92$	$1.55$ $\pm\,0.50$	$1.74$ $\pm\,0.27$	$\bm{48.63}$ $\pm\,1.16$	$\uline{2.04}$ $\pm\,0.51$
Class-Conditioned	$\bm{99.81}$ $\pm\,0.03$	$97.70$ $\pm\,0.71$	$\bm{92.15}$ $\pm\,0.17$	$\bm{50.71}$ $\pm\,0.95$	$\bm{1.49}$ $\pm\,0.81$	$\bm{1.59}$ $\pm\,0.47$	$48.42$ $\pm\,0.93$	$2.06$ $\pm\,0.49$
	Forget 10%
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$

Appendix

Table 8: Selector ablation full metrics for ResNet-18 on CIFAR-10. Bold and underlined denote best and second best, where “best” denotes smallest gap to Retrain .

	Forget 1%
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$
Uniform	$\uline{97.90}$ $\pm\,0.30$	$60.63$ $\pm\,2.78$	$50.83$ $\pm\,0.48$	$\bm{55.77}$ $\pm\,0.61$	$\uline{4.88}$ $\pm\,0.15$	$\uline{8.01}$ $\pm\,0.70$	$47.51$ $\pm\,0.46$	$4.00$ $\pm\,0.80$
Feature-1NN	$97.74$ $\pm\,0.06$	$\uline{61.00}$ $\pm\,3.00$	$\uline{51.07}$ $\pm\,0.81$	$56.56$ $\pm\,2.06$	$5.35$ $\pm\,0.29$	$8.29$ $\pm\,0.22$	$\uline{48.15}$ $\pm\,1.12$	$\uline{3.73}$ $\pm\,0.90$
Class-Conditioned	$\bm{97.91}$ $\pm\,0.15$	$\bm{61.56}$ $\pm\,0.29$	$\bm{51.23}$ $\pm\,0.23$	$\uline{56.07}$ $\pm\,0.84$	$\bm{4.68}$ $\pm\,0.47$	$\bm{7.97}$ $\pm\,0.65$	$\bm{48.21}$ $\pm\,0.04$	$\bm{3.53}$ $\pm\,0.37$
	Forget 10%
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$

Appendix

Table 9: Selector ablation full metrics for Swin-T on Tiny-ImageNet. Bold and underlined denote best and second best, where “best” denotes smallest gap to Retrain .

	Forget 1%
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$
Held-out $0.1\%$	$99.54$ $\pm\,0.16$	$95.78$ $\pm\,1.39$	$91.73$ $\pm\,0.14$	$\bm{47.60}$ $\pm\,1.40$	$\bm{1.73}$ $\pm\,0.42$	$\bm{2.24}$ $\pm\,0.77$	$\bm{46.30}$ $\pm\,0.81$	$\uline{2.28}$ $\pm\,0.56$
Held-out $0.3\%$	$99.55$ $\pm\,0.08$	$\bm{94.74}$ $\pm\,1.99$	$\bm{91.83}$ $\pm\,0.09$	$\uline{47.46}$ $\pm\,1.59$	$\uline{1.78}$ $\pm\,0.76$	$\uline{2.26}$ $\pm\,0.78$	$\uline{45.72}$ $\pm\,1.27$	$\bm{2.14}$ $\pm\,0.71$
Held-out $1\%$	$99.57$ $\pm\,0.04$	$95.19$ $\pm\,1.81$	$91.77$ $\pm\,0.13$	$47.15$ $\pm\,1.68$	$1.87$ $\pm\,0.71$	$2.45$ $\pm\,0.91$	$45.46$ $\pm\,1.37$	$2.32$ $\pm\,0.69$
Held-out $3\%$	$99.58$ $\pm\,0.04$	$\uline{95.11}$ $\pm\,1.74$	$91.77$ $\pm\,0.17$	$47.17$ $\pm\,1.69$	$1.84$ $\pm\,0.72$	$2.44$ $\pm\,0.94$	$45.47$ $\pm\,1.36$	$2.30$ $\pm\,0.68$
Held-out $10\%$	$\uline{99.58}$ $\pm\,0.04$	$95.11$ $\pm\,1.94$	$91.76$ $\pm\,0.16$	$47.15$ $\pm\,1.67$	$1.88$ $\pm\,0.74$	$2.45$ $\pm\,0.91$	$45.46$ $\pm\,1.37$	$2.30$ $\pm\,0.71$

Appendix

Table 10: Held-out subsample ablation full metrics for ResNet-18 on CIFAR-10. Bold and underlined denote best and second best, where “best” denotes smallest gap to Retrain .

	Forget 1%
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$
Held-out $0.1\%$	$97.78$ $\pm\,0.15$	$54.41$ $\pm\,2.79$	$50.44$ $\pm\,0.57$	$\uline{54.85}$ $\pm\,1.31$	$6.18$ $\pm\,0.91$	$\uline{7.75}$ $\pm\,0.58$	$46.97$ $\pm\,0.65$	$5.82$ $\pm\,0.81$
Held-out $0.3\%$	$97.79$ $\pm\,0.47$	$57.26$ $\pm\,1.22$	$51.03$ $\pm\,0.96$	$\bm{54.76}$ $\pm\,1.83$	$5.30$ $\pm\,0.63$	$\bm{7.41}$ $\pm\,1.19$	$47.00$ $\pm\,1.03$	$4.95$ $\pm\,0.60$
Held-out $1\%$	$97.98$ $\pm\,0.25$	$60.19$ $\pm\,1.70$	$50.91$ $\pm\,0.98$	$55.86$ $\pm\,1.59$	$4.97$ $\pm\,0.75$	$8.02$ $\pm\,1.21$	$47.75$ $\pm\,0.99$	$4.01$ $\pm\,0.66$
Held-out $3\%$	$97.91$ $\pm\,0.31$	$60.44$ $\pm\,0.68$	$51.02$ $\pm\,0.70$	$56.01$ $\pm\,1.16$	$\bm{4.79}$ $\pm\,0.65$	$8.04$ $\pm\,1.12$	$47.96$ $\pm\,0.79$	$3.88$ $\pm\,0.48$
Held-out $10\%$	$\bm{98.04}$ $\pm\,0.28$	$\uline{61.59}$ $\pm\,2.02$	$\bm{51.24}$ $\pm\,1.05$	$56.47$ $\pm\,1.09$	$\uline{4.86}$ $\pm\,0.56$	$8.16$ $\pm\,1.01$	$\uline{48.12}$ $\pm\,0.54$	$\bm{3.52}$ $\pm\,0.69$

Appendix

Table 11: Held-out subsample ablation full metrics for Swin-T on Tiny-ImageNet. Bold and underlined denote best and second best, where “best” denotes smallest gap to Retrain .

Panel of Figure 5 — Figure 5: Additional forgetting–utility trade-offs. Trade-off curves for ResNet-18 on CIFAR-100 and Tiny-ImageNet under random 10% forgetting. The $y$ -axis reports test accuracy and the $x$ -axis reports the $\mathrm{RMIA}_{\mathrm{AUC}}$ gap to the retrain-from-scratch baseline; higher TA and proximity to the dashed vertical line is better. Where supported, we sweep $w\in[0.1,0.9]$ in the retain–forget objective, with an additional $w=0.99$ for LUR and BadTeach . Points are labeled by $w$ ; * marks variants reparameterized from the original loss to sweep $w$ .

	Forget 1%
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$
Retrain	$99.98$ $\pm\,0.01$	$74.81$ $\pm\,1.22$	$75.33$ $\pm\,0.29$	$49.56$ $\pm\,1.56$	$0.00$ $\pm\,0.00$	$0.00$ $\pm\,0.00$	$48.73$ $\pm\,0.78$	$0.00$ $\pm\,0.00$
Base	$99.98$ $\pm\,0.01$	$100.00$ $\pm\,0.00$	$75.52$ $\pm\,0.36$	$76.09$ $\pm\,1.32$	$12.84$ $\pm\,1.28$	$13.47$ $\pm\,0.68$	$75.22$ $\pm\,0.05$	$12.97$ $\pm\,0.38$
NegGrad	$99.02$ $\pm\,0.84$	$96.37$ $\pm\,1.45$	$\uline{72.35}$ $\pm\,1.90$	$73.79$ $\pm\,0.43$	$12.26$ $\pm\,1.39$	$13.44$ $\pm\,1.30$	$68.36$ $\pm\,2.31$	$11.28$ $\pm\,0.93$
NegGrad+	$97.38$ $\pm\,1.84$	$90.81$ $\pm\,2.87$	$69.10$ $\pm\,2.39$	$69.96$ $\pm\,1.57$	$11.13$ $\pm\,1.38$	$13.15$ $\pm\,1.76$	$64.29$ $\pm\,2.48$	$10.10$ $\pm\,1.27$
Finetune	$91.98$ $\pm\,3.58$	$\uline{75.63}$ $\pm\,4.69$	$66.67$ $\pm\,1.27$	$59.76$ $\pm\,3.26$	$7.04$ $\pm\,1.15$	$9.27$ $\pm\,1.75$	$54.46$ $\pm\,2.58$	$5.80$ $\pm\,1.68$

Appendix

Table 12: Forgetting and utility results for ResNet-18 on CIFAR-100. Studied under 1%, 10%, and 50% random forgetting. RA/FA/TA denote retain, forget, and test accuracy;

\mathrm{RMIA}_{\mathrm{AUC}}

and

\mathrm{SMIA}_{\mathrm{AUC}}

report membership inference performance, with 50% corresponding to chance-level discrimination. Gap metrics measure deviation from Retrain , so lower is better. Results are mean

\pm

std over three seeds, with all values in percent. Bold and underlined denote best and second best, where “best” denotes smallest gap to Retrain .

	Forget 1%
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$
Retrain	$99.98$ $\pm\,0.00$	$58.52$ $\pm\,0.71$	$59.54$ $\pm\,0.21$	$49.28$ $\pm\,1.14$	$0.00$ $\pm\,0.00$	$0.00$ $\pm\,0.00$	$49.84$ $\pm\,0.85$	$0.00$ $\pm\,0.00$
Base	$99.98$ $\pm\,0.00$	$99.96$ $\pm\,0.06$	$59.64$ $\pm\,0.44$	$81.56$ $\pm\,0.56$	$18.50$ $\pm\,0.61$	$16.27$ $\pm\,0.90$	$84.68$ $\pm\,0.16$	$19.10$ $\pm\,0.31$
NegGrad	$\bm{99.98}$ $\pm\,0.00$	$99.72$ $\pm\,0.24$	$\bm{58.85}$ $\pm\,0.60$	$81.44$ $\pm\,0.69$	$18.29$ $\pm\,0.67$	$16.11$ $\pm\,1.07$	$83.66$ $\pm\,0.09$	$18.93$ $\pm\,0.33$
NegGrad+	$95.68$ $\pm\,0.05$	$67.22$ $\pm\,2.20$	$49.40$ $\pm\,0.67$	$62.23$ $\pm\,1.09$	$8.80$ $\pm\,0.59$	$11.23$ $\pm\,0.45$	$59.47$ $\pm\,1.74$	$8.19$ $\pm\,0.77$
Finetune	$89.92$ $\pm\,6.35$	$61.67$ $\pm\,8.33$	$51.55$ $\pm\,0.69$	$63.06$ $\pm\,7.81$	$9.13$ $\pm\,1.00$	$10.57$ $\pm\,3.13$	$57.23$ $\pm\,5.13$	$7.15$ $\pm\,2.93$

Appendix

Table 13: Forgetting and utility results for ResNet-18 on Tiny-ImageNet. Studied under 1%, 10%, and 50% random forgetting. RA/FA/TA denote retain, forget, and test accuracy;

\mathrm{RMIA}_{\mathrm{AUC}}

and

\mathrm{SMIA}_{\mathrm{AUC}}

report membership inference performance, with 50% corresponding to chance-level discrimination. Gap metrics measure deviation from Retrain , so lower is better. Results are mean

\pm

std over three seeds, with all values in percent. Bold and underlined denote best and second best, where “best” denotes smallest gap to Retrain .

	Full-class forgetting: one target class
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$
Retrain	$100.00$ $\pm\,0.00$	$0.00$ $\pm\,0.00$	$85.16$ $\pm\,0.03$	$50.50$ $\pm\,0.15$	$0.00$ $\pm\,0.00$	$0.00$ $\pm\,0.00$	$50.45$ $\pm\,0.40$	$0.00$ $\pm\,0.00$
Base	$100.00$ $\pm\,0.00$	$100.00$ $\pm\,0.00$	$94.35$ $\pm\,0.31$	$52.39$ $\pm\,0.39$	$27.83$ $\pm\,0.13$	$5.65$ $\pm\,0.25$	$59.24$ $\pm\,0.19$	$29.50$ $\pm\,0.15$
NegGrad	$55.30$ $\pm\,2.76$	$\bm{0.00}$ $\pm\,0.00$	$47.52$ $\pm\,2.30$	$49.20$ $\pm\,0.28$	$20.96$ $\pm\,1.22$	$19.58$ $\pm\,1.07$	$48.86$ $\pm\,0.09$	$21.07$ $\pm\,1.21$
NegGrad+	$\uline{99.72}$ $\pm\,0.03$	$19.65$ $\pm\,0.82$	$85.83$ $\pm\,0.26$	$55.93$ $\pm\,0.22$	$6.53$ $\pm\,0.33$	$3.10$ $\pm\,0.27$	$55.82$ $\pm\,0.13$	$6.51$ $\pm\,0.38$
Finetune	$86.37$ $\pm\,0.55$	$\uline{0.00}$ $\pm\,0.00$	$74.59$ $\pm\,0.38$	$51.25$ $\pm\,0.28$	$6.33$ $\pm\,0.16$	$5.85$ $\pm\,0.08$	$51.48$ $\pm\,0.04$	$6.40$ $\pm\,0.18$

Appendix

Table 14: Full-class forgetting results for ResNet-18 on CIFAR-10. RA/FA/TA denote retain, forget, and test accuracy;

\mathrm{RMIA}_{\mathrm{AUC}}

and

\mathrm{SMIA}_{\mathrm{AUC}}

report membership inference performance, with 50% corresponding to chance-level discrimination. Gap metrics measure deviation from Retrain , so lower is better. Results are averaged over all 10 classes within each seed and reported as mean

\pm

sample std across three seeds, with all values in percent. Bold and underlined denote best and second best, where “best” denotes smallest gap to Retrain .

	Half-class forgetting: one target class
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$
Retrain	$100.00$ $\pm\,0.00$	$90.62$ $\pm\,0.15$	$93.93$ $\pm\,0.05$	$50.29$ $\pm\,0.17$	$0.00$ $\pm\,0.00$	$0.00$ $\pm\,0.00$	$50.27$ $\pm\,0.12$	$0.00$ $\pm\,0.00$
Base	$100.00$ $\pm\,0.00$	$100.00$ $\pm\,0.00$	$94.35$ $\pm\,0.31$	$55.79$ $\pm\,0.43$	$3.84$ $\pm\,0.04$	$2.99$ $\pm\,0.16$	$59.24$ $\pm\,0.15$	$4.71$ $\pm\,0.12$
NegGrad	$76.10$ $\pm\,1.63$	$0.22$ $\pm\,0.07$	$67.60$ $\pm\,1.50$	$\uline{48.55}$ $\pm\,0.14$	$35.68$ $\pm\,0.86$	$14.20$ $\pm\,0.85$	$\uline{48.71}$ $\pm\,0.12$	$35.60$ $\pm\,0.87$
NegGrad+	$98.44$ $\pm\,0.03$	$71.68$ $\pm\,0.67$	$90.27$ $\pm\,0.20$	$59.87$ $\pm\,0.18$	$8.44$ $\pm\,0.29$	$6.62$ $\pm\,0.18$	$56.24$ $\pm\,0.14$	$7.53$ $\pm\,0.26$
Finetune	$\uline{99.99}$ $\pm\,0.00$	$99.99$ $\pm\,0.00$	$\uline{94.18}$ $\pm\,0.26$	$56.26$ $\pm\,0.17$	$3.94$ $\pm\,0.02$	$3.19$ $\pm\,0.12$	$58.76$ $\pm\,0.18$	$4.57$ $\pm\,0.09$

Appendix

Table 15: Half-class forgetting results for ResNet-18 on CIFAR-10. RA/FA/TA denote retain, forget, and test accuracy;

\mathrm{RMIA}_{\mathrm{AUC}}

and

\mathrm{SMIA}_{\mathrm{AUC}}

\pm

sample std across three seeds, with all values in percent. Bold and underlined denote best and second best, where “best” denotes smallest gap to Retrain .

	Full-class forgetting: one 20-class group
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$
Retrain	$99.98$ $\pm\,0.00$	$0.00$ $\pm\,0.00$	$54.82$ $\pm\,0.09$	$49.83$ $\pm\,0.48$	$0.00$ $\pm\,0.00$	$0.00$ $\pm\,0.00$	$49.71$ $\pm\,0.14$	$0.00$ $\pm\,0.00$
Base	$99.98$ $\pm\,0.00$	$99.99$ $\pm\,0.00$	$61.01$ $\pm\,0.10$	$86.94$ $\pm\,0.38$	$35.82$ $\pm\,0.07$	$21.65$ $\pm\,0.14$	$94.05$ $\pm\,0.12$	$37.63$ $\pm\,0.10$
NegGrad	$\bm{99.98}$ $\pm\,0.00$	$99.99$ $\pm\,0.00$	$61.00$ $\pm\,0.10$	$86.97$ $\pm\,0.38$	$35.83$ $\pm\,0.07$	$21.67$ $\pm\,0.14$	$94.04$ $\pm\,0.12$	$37.63$ $\pm\,0.10$
NegGrad+	$38.59$ $\pm\,1.91$	$\bm{0.00}$ $\pm\,0.00$	$27.23$ $\pm\,1.20$	$\bm{50.09}$ $\pm\,0.37$	$22.52$ $\pm\,0.93$	$14.35$ $\pm\,0.90$	$\bm{49.16}$ $\pm\,0.11$	$22.40$ $\pm\,0.83$
Finetune	$52.81$ $\pm\,0.29$	$0.00$ $\pm\,0.01$	$33.35$ $\pm\,0.21$	$62.89$ $\pm\,0.09$	$20.43$ $\pm\,0.11$	$17.26$ $\pm\,0.27$	$62.74$ $\pm\,0.09$	$20.42$ $\pm\,0.06$

Appendix

Table 16: Full-class forgetting results for Swin-T on Tiny-ImageNet. RA/FA/TA denote retain, forget, and test accuracy;

\mathrm{RMIA}_{\mathrm{AUC}}

and

\mathrm{SMIA}_{\mathrm{AUC}}

report membership inference performance, with 50% corresponding to chance-level discrimination. Gap metrics measure deviation from Retrain , so lower is better. Results are averaged over the three 20-class forgetting groups within each seed and reported as mean

\pm

sample std across three seeds, with all values in percent. Bold and underlined denote best and second best, where “best” denotes smallest gap to Retrain .

	Half-class forgetting: one 20-class group
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$
Retrain	$99.98$ $\pm\,0.00$	$52.38$ $\pm\,0.49$	$60.14$ $\pm\,0.35$	$49.75$ $\pm\,0.26$	$0.00$ $\pm\,0.00$	$0.00$ $\pm\,0.00$	$49.69$ $\pm\,0.36$	$0.00$ $\pm\,0.00$
Base	$99.98$ $\pm\,0.00$	$99.98$ $\pm\,0.00$	$61.01$ $\pm\,0.10$	$80.19$ $\pm\,0.63$	$19.73$ $\pm\,0.31$	$15.65$ $\pm\,0.39$	$94.05$ $\pm\,0.14$	$23.21$ $\pm\,0.22$
NegGrad	$\bm{99.98}$ $\pm\,0.00$	$99.98$ $\pm\,0.00$	$\bm{61.01}$ $\pm\,0.10$	$80.22$ $\pm\,0.63$	$19.73$ $\pm\,0.31$	$15.67$ $\pm\,0.39$	$94.05$ $\pm\,0.14$	$23.21$ $\pm\,0.22$
NegGrad+	$75.01$ $\pm\,1.15$	$\uline{50.79}$ $\pm\,5.55$	$44.96$ $\pm\,0.58$	$66.68$ $\pm\,2.42$	$15.26$ $\pm\,0.47$	$16.05$ $\pm\,1.12$	$\uline{60.60}$ $\pm\,1.72$	$13.76$ $\pm\,0.61$
Finetune	$99.88$ $\pm\,0.02$	$99.89$ $\pm\,0.02$	$56.97$ $\pm\,0.35$	$78.97$ $\pm\,0.26$	$20.00$ $\pm\,0.19$	$16.19$ $\pm\,0.34$	$87.93$ $\pm\,0.68$	$22.26$ $\pm\,0.34$

Appendix

Table 17: Half-class forgetting results for Swin-T on Tiny-ImageNet. RA/FA/TA denote retain, forget, and test accuracy;

\mathrm{RMIA}_{\mathrm{AUC}}

and

\mathrm{SMIA}_{\mathrm{AUC}}

report membership inference performance, with 50% corresponding to chance-level discrimination. Gap metrics measure deviation from Retrain , so lower is better. Results are averaged over the three 20-class forgetting groups within each seed and reported as mean

\pm

sample std across three seeds, with all values in percent. Bold and underlined denote best and second best, where “best” denotes smallest gap to Retrain .

	Full-class forgetting
	RA	FA	TA	$\mathrm{RMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}_{\mathrm{Avg}}$	$\mathrm{Gap}^{\mathrm{F\text{-}U}}_{\mathrm{Avg}}$	$\mathrm{SMIA}_{\mathrm{AUC}}$	$\mathrm{Gap}^{\mathrm{SMIA}}_{\mathrm{Avg}}$
ResNet-18 on CIFAR-10
Retrain -Teacher	$99.47$ $\pm\,0.04$	$0.00$ $\pm\,0.00$	$83.72$ $\pm\,0.15$	$50.82$ $\pm\,0.29$	$0.79$ $\pm\,0.11$	$1.32$ $\pm\,0.23$	$53.21$ $\pm\,0.24$	$1.27$ $\pm\,0.10$
Frozen Base	$99.27$ $\pm\,0.25$	$3.42$ $\pm\,0.60$	$83.63$ $\pm\,0.23$	$50.00$ $\pm\,0.28$	$1.73$ $\pm\,0.25$	$1.39$ $\pm\,0.17$	$51.30$ $\pm\,0.06$	$1.90$ $\pm\,0.24$
Swin-T on Tiny-ImageNet
Retrain -Teacher	$97.80$ $\pm\,0.19$	$0.02$ $\pm\,0.01$	$52.50$ $\pm\,0.02$	$63.00$ $\pm\,0.91$	$4.42$ $\pm\,0.08$	$7.74$ $\pm\,0.25$	$70.73$ $\pm\,0.25$	$6.39$ $\pm\,0.07$

Appendix

Table 18: Teacher ablation under class forgetting. Comparison of the frozen-base and Retrain teachers under full- and half-class forgetting for ResNet-18 on CIFAR-10 and Swin-T on Tiny-ImageNet.

Machine Unlearning

Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.

See all 351 papers

May 29, 2026

Multi-Objective Reference-Aligned Machine Unlearning

Machine unlearning aims to remove the influence of specific training samples while preserving the model's utility. Existing single-objective approaches, such as gradient ascent or random relabeling, often induce catastrophic forgetting due to conflicting optimization dynamics and unbounded forgetting objectives that cause the model to drift from its pre-trained knowledge. We propose Reference-Aligned UnLearning (RAUL), a multi-objective framework that jointly optimizes forgetting and retention by replacing unbounded loss maximization with a bounded KL alignment of predictions on forgotten samples toward a reference distribution representing unseen data, instantiated either as a uniform distribution or an empirical distribution from a held-out reference set, which constrains the forgetting objective and reduces gradient conflict with retention. The resulting multi-objective optimization (MOO) problem is solved via Jacobian descent, which aggregates multiple gradients into a direction that does not conflict. Our results demonstrate that RAUL achieves the closest gap compared to full retraining.

Rasa Khosrowshahli, Stephen Asobiela, Beatrice Ombuki-Berman +1

Department of Computer Science, Brock University, St. Catharines, ON, Canada · Yousef Haj-Ahmad Department of Engineering, Brock University, St. Catharines, ON, Canada

Machine Unlearning Catastrophic Forgetting

Ask PDF

Mar 16, 2026

Models Designed to Forget: Machine Unlearning via Key Deletion

Machine unlearning for vision models is rapidly becoming a practical requirement, driven by privacy regulations, data errors, and the need to remove harmful or corrupted training images. Despite this, most existing approximate unlearning methods tackle the problem from a post-hoc perspective. They attempt to erase the influence of targeted samples through parameter updates that typically require access to the full training data. This creates a mismatch with real deployment scenarios where unlearning requests can be anticipated, revealing a fundamental limitation of post-hoc approaches. We motivate unlearning by design, a novel paradigm for approximate methods in which models are directly trained to support forgetting as an inherent architectural capability. We instantiate this idea with Machine UNlearning via KEY deletion (MUNKEY), a memory-augmented transformer that decouples instance-specific memorization from model weights. Here, unlearning corresponds to removing the instance-identifying key, enabling zero-shot forgetting without weight updates or access to the original samples or labels. Across natural image benchmarks, fine-grained visual recognition, and medical datasets, MUNKEY outperforms all post-hoc baselines. Our results establish that unlearning by design enables fast, deployment-oriented unlearning while preserving predictive performance.

Sonia Laguna, Jorge da Silva Goncalves, Moritz Vandenhirtz +3

Department of Computer Science, ETH Zurich, Switzerland

Machine Unlearning

Ask PDF

Sep 30, 2026

Unmerge: Efficient Machine Unlearning via Task Arithmetic

Approximate machine unlearning seeks to remove the influence of a forget set from a trained model without full retraining. Existing gradient-based methods require data-dependent hyperparameter search, struggle when forget and retain knowledge are entangled, and offer little insight into where unlearning actually happens inside the network. We recast unlearning through the lens of task arithmetic: if finetuning produces a merged task vector

τ_m

that combines learning on forget and retain sets, unlearning is the inverse operation that subtracts a learned forget component

τ_F

to recover the retain task vector

τ_R

. The forget signal is concentrated: at every layer, forget activations lie in a subspace spanned by a handful of dominant directions, so we factorize

τ_F

in a low-rank forget basis, which is faithful up to a small tail-eigenvalue residual and limits how far the correction can perturb retain. We then optimize three intuitive goals (match the merged vector inside the forget span, suppress leakage into the retain span, and bound the correction size) that provably bound forget leakage and retain damage in activation space. The resulting algorithm, Unmerge, is fast and powerful: on class-level unlearning with ResNet-50 on CIFAR-100 and Tiny ImageNet, it improves Tug-of-War by up to ~24% over a baseline of comparable runtime and by up to ~18% over stronger baselines that run ~5x slower, keeps membership-inference exposure at the level of retraining, and shrinks the feature-distribution gap to the retrained model, where relabeling methods leave forget features cleanly separable. Further studies show that Unmerge also applies to ViT-S/16 and scales to Llama-3.2-3B. The per-layer basis geometry that drives the algorithm also serves as a layerwise diagnostic for when and where unlearning becomes structurally hard.

Haoran Tang, Andrew Tan, Rajiv Khanna

Department of Computer Science Purdue University

Machine Unlearning

Ask PDF

Abstract

Machine Unlearning

Explore similar work

Multi-Objective Reference-Aligned Machine Unlearning

Models Designed to Forget: Machine Unlearning via Key Deletion

Unmerge: Efficient Machine Unlearning via Task Arithmetic