Designing to Forget: Deep Semi-parametric Models for Unlearning

Amber Yijia Zheng; Yu-Shan Tai; Raymond A. Yeh

cs.CVMar 24, 2026

Designing to Forget: Deep Semi-parametric Models for Unlearning

Authors: Amber Yijia Zheng, Yu-Shan Tai, Raymond A. Yeh

Organizations: Department of Computer Science, Purdue University

Abstract

Recent advances in machine unlearning have focused on developing algorithms to remove specific training samples from a trained model. In contrast, we observe that not all models are equally easy to unlearn. Hence, we introduce a family of deep semi-parametric models (SPMs) that exhibit non-parametric behavior during unlearning. SPMs use a fusion module that aggregates information from each training sample, enabling explicit test-time deletion of selected samples without altering model parameters. Empirically, we demonstrate that SPMs achieve competitive task performance to parametric models in image classification and generation, while being significantly more efficient for unlearning. Notably, on ImageNet classification, SPMs reduce the prediction gap relative to a retrained (oracle) baseline by $11\%$ and achieve over $10\times$ faster unlearning compared to existing approaches on parametric models. The code is available at https://github.com/amberyzheng/spm_unlearning.

Figures & tables

Figure 2: Unlearning with Generative SPM. It consists of three types of layers: (a) Fusion module $g$ connects parametric and non-parametric modules; (b) Non-parametric module $h$ maps a set to another set; (c) Parametric module $f$ maps a vector to a vector. The first row illustrates a standard pipeline for generating a ‘bird’. The second row presents unlearning ‘bird’ via test-time deletion.

	CIFAR10		ImageNet
	Acc. $\uparrow$	Run Time (s) $\downarrow$	Acc. $\uparrow$	Run Time (s ) $\downarrow$
ResNet18	94.9	0.7	68.93	26.1
ResNet18-KNN (5%)	94.5 ${}_{\pm\text{0.02}}$	1.1	61.5 ${}_{\pm\text{0.08}}$	34.4
ResNet18-KNN (10%)	94.5 ${}_{\pm\text{0.05}}$	1.1	63.4 ${}_{\pm\text{0.23}}$	34.5
ResNet18-KNN (100%)	94.5	1.5	66.9	63.7
SPM-R (5%)	94.1 ${}_{\pm\text{0.02}}$	1.0	52.0 ${}_{\pm\text{0.12}}$	33.8

Table 1: Classification accuracy on CIFAR10 and ImageNet. We experiment with subsampling 5%, 10%, and 100% of the training data, either using (R)etrival or (C)lustering based method.

	DDPM	GMM	SPM
	DDPM	GMM	$\left\|{\mathcal{T}}\right\|=10$	$\left\|{\mathcal{T}}\right\|=20$	$\left\|{\mathcal{T}}\right\|=50$	$\left\|{\mathcal{T}}\right\|=100$	$\left\|{\mathcal{T}}\right\|=1024$
FID	7.28	198.42	7.71 ${}_{\pm\text{0.20}}$	7.30 ${}_{\pm\text{0.09}}$	7.17 ${}_{\pm\text{0.18}}$	7.09 ${}_{\pm\text{0.06}}$	7.04 ${}_{\pm\text{0.13}}$
RT (s)	42	0.42	52	63	101	173	1486
Rel. RT (%)	+0%	-99%	+23%	+50%	+140%	+311%	+3404%

Table 2: FID comparisons on CIFAR-10.

\left|{\mathcal{T}}\right|

is the size of the inputted set. Runtime (RT) is for generating 1,000 images.

Backbone	Method	Unlearn 1 Class (10 $\%$ )						Unlearn 5 Classes (50 $\%$ )
Backbone	Method	$\text{PG}_{\tt H}$ $\downarrow$	$\text{PG}_{\tt S}$ $\downarrow$	$\Delta$ UA $\downarrow$	$\Delta$ RA $\downarrow$	$\Delta$ TA $\downarrow$	$\text{Time}_{\tt MU}$ (s) $\downarrow$	$\text{PG}_{\tt H}$ $\downarrow$	$\text{PG}_{\tt S}$ $\downarrow$	$\Delta$ UA $\downarrow$	$\Delta$ RA $\downarrow$	$\Delta$ TA $\downarrow$	$\text{Time}_{\tt MU}$ (s) $\downarrow$
ResNet18	Retrain	0.00	0.00	0.00 (100.00)	0.00 (100.00)	0.00 (94.76)	2317.6	0.00	0.00	0.00 (100.00)	0.00 (100.00)	0.00 (96.06)	1284.9
	GA [ 52 ]	18.48	0.99	5.80 (94.20)	5.64 (94.36)	6.42 (88.34)	8.90	32.62	2.86	0.00 (100.00)	15.51 (84.49)	14.23 (81.77)	41.3
	FT [ 56 ]	13.11	0.48	66.25 (33.75)	0.15 (99.85)	0.32 (94.44)	148.7	26.21	0.72	34.76 (65.24)	0.06 (99.94)	0.59 (96.65)	81.8
	IU [ 31 ]	55.93	3.51	0.00 (100.00)	54.13 (45.87)	50.70 (44.06)	17.7	75.71	424.85	0.00 (100.00)	79.97 (20.03)	76.04 (20.02)	16.8
	BE [ 10 ]	15.43	0.71	23.01 (76.99)	1.81 (98.19)	2.68 (92.08)	15.1	46.00	2.11	64.21 (35.79)	0.40 (99.60)	0.17 (95.89)	81.6

Table 3: CIFAR-10 class-wise unlearning on classification averaged over 10 classes. SPMs achieve close performance to the retrained SPM with a fast unlearning time. The UA, RA, and TA values are reported in parentheses with the corresponding gaps.

Backbone	Method	$\text{PG}_{\tt H}$ $\downarrow$	$\text{PG}_{\tt S}$ $\downarrow$	$\Delta$ UA $\downarrow$	$\Delta$ RA $\downarrow$	$\Delta$ TA $\downarrow$	$\text{Time}_{\tt MU}$ (s) $\downarrow$
ResNet18	Retrain	0.00	0.00	0.00 (100.00)	0.00 (68.19)	0.00 (68.13)	226025.4
	GA [ 52 ]	30.47	0.78	7.31 (92.69)	3.41 (64.78)	2.72 (65.41)	24.8
	FT [ 56 ]	29.68	0.53	0.01 (99.99)	5.68 (62.51)	3.84 (64.29)	11361.4
	IU [ 31 ]	39.18	1.10	4.15 (95.85)	13.71 (54.48)	10.85 (57.28)	5684.0
	BE [ 10 ]	33.94	0.92	3.96 (96.04)	7.17 (61.02)	5.79 (62.34)	47.0
	BS [ 10 ]	35.45	0.99	1.99 (98.01)	8.84 (59.35)	7.18 (60.95)	44.5

Table 4: ImageNet 1-class unlearning on classification averaged over 10 classes. We observe that SPMs achieve close performance to the retrained model with a low unlearning time.

Backbone	Method	Automobile			Cat			Dog			Horse			Truck			$\text{Time}_{\tt MU}$ (s) $\downarrow$
Backbone	Method	FID O $\downarrow$	$\Delta$ UA $\downarrow$	$\Delta$ FID R $\downarrow$	FID O $\downarrow$	$\Delta$ UA $\downarrow$	$\Delta$ FID R $\downarrow$	FID O $\downarrow$	$\Delta$ UA $\downarrow$	$\Delta$ FID R $\downarrow$	FID O $\downarrow$	$\Delta$ UA $\downarrow$	$\Delta$ FID R $\downarrow$	FID O $\downarrow$	$\Delta$ UA $\downarrow$	$\Delta$ FID R $\downarrow$	$\text{Time}_{\tt MU}$ (s) $\downarrow$
DDPM	Retrain	0.00	0.00 (100.00)	0 .00 (11.21)	0.00	0.00 (99.98)	0.00 (10.84)	0.00	0.00 (100.00)	0.00 (10.88)	0.00	0.00 (99.98)	0.00 (10.00)	0.00	0.00 (99.90)	0.00 (10.15)	145601.4
	SA [ 25 ]	31.00	0.00 (100.00)	12.35 (23.56)	32.54	14.18 (85.80)	10.50 (21.34)	32.03	8.60 (91.40)	11.11 (21.19)	477.61	0.02 (100.00)	11.13 (21.13)	35.79	0.10 (100.00)	18.89 (29.04)	18711.0
	SalUn [ 16 ]	6.30	0.20 (99.80)	10.02 (21.23)	6.88	1.38 (98.60)	9.45 (20.29)	6.93	0.00 (100.00)	9.30 (20.18)	6.11	0.58 (99.40)	10.70 (20.70)	3.59	0.70 (99.20)	10.30 (20.45)	493.2
	SFR-on [ 28 ]	25.69	0.00 (100.00)	9.49 (20.70)	41.68	7.38 (92.60)	7.60 (18.44)	41.30	0.20 (99.80)	8.01 (18.89)	91.48	0.02 (100.00)	9.93 (19.93)	42.70	0.10 (100.00)	10.46 (20.61)	23.8
SPM	Retrain	0.00	0.00 (99.76)	0.00 (7.40)	0.00	0.00 (99.26)	0.00 (7.15)	0.00	0.00 (99.78)	0.00 (6.52)	0.00	0.00 (99.84)	0.00 (6.89)	0.00	0.00 (99.64)	0.00 (7.28)	90935.9

Table 5: CIFAR-10 class-wise forgetting on generation. We experiment on 5 unlearning scenarios with unlearning classes: Automobile , Cat , Dog , Horse , and Truck . We observe that our method achieves close performance to the retrained model with a low unlearning time.

Panel of Figure 3 — Figure 3: Qualitative results for unlearned SPM . When a class is unlearned ( e.g . , cats or frogs framed in red ), the model avoids generating that concept and produces samples resembling remaining classes ( e.g . , a truck replacing a cat or a plane replacing a frog). Additionally, the generations from the remaining classes are left unchanged (framed in blue ).

Method	Automobile			Cat			Dog			Horse			Truck
Method	FID O $\downarrow$	$\Delta$ UA $\downarrow$	$\Delta\text{FID}_{\tt R}$ $\downarrow$	FID O $\downarrow$	$\Delta$ UA $\downarrow$	$\Delta\text{FID}_{\tt R}$ $\downarrow$	FID O $\downarrow$	$\Delta$ UA $\downarrow$	$\Delta\text{FID}_{\tt R}$ $\downarrow$	FID O $\downarrow$	$\Delta$ UA $\downarrow$	$\Delta\text{FID}_{\tt R}$ $\downarrow$	FID O $\downarrow$	$\Delta$ UA $\downarrow$	$\Delta\text{FID}_{\tt R}$ $\downarrow$
Retrain	0.00	0.00 (99.76)	0.00 (7.40)	0.00	0.00 (99.26)	0.00 (7.15)	0.00	0.00 (99.78)	0.00 (6.52)	0.00	0.00 (99.84)	0.00 (6.89)	0.00	0.00 (99.64)	0.00 (7.28)
w/o perm.	2.88	0.82 (98.94)	2.66 (4.74)	5.08	3.13 (96.13)	2.68 (4.47)	3.39	83.61 (16.17)	2.15 (4.37)	3.87	60.89 (38.95)	2.22 (4.67)	2.90	1.56 (98.08)	2.58 (4.70)
Ours	1.97	0.10 (99.86)	0.38 (7.78)	1.85	0.04 (99.22)	0.50 (7.65)	1.80	0.04 (99.82)	0.08 (6.60)	1.90	0.08 (99.76)	0.29 (7.18)	1.97	0.06 (99.58)	0.40 (7.68)

Table 6: Ablation study of label permutation. We experiment on CIFAR-10 class-wise forgetting on generation with SPMs, and the results show that removing label permutation leads to a large gap in UA.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Backbone	Method	Automobile		Cat		Dog		Horse		Truck		$t_{setup}$ (s) $\downarrow$
Backbone	Method	$\Delta$ UA $\downarrow$	$\Delta$ FID R $\downarrow$	$\Delta$ UA $\downarrow$	$\Delta$ FID R $\downarrow$	$\Delta$ UA $\downarrow$	$\Delta$ FID R $\downarrow$	$\Delta$ UA $\downarrow$	$\Delta$ FID R $\downarrow$	$\Delta$ UA $\downarrow$	$\Delta$ FID R $\downarrow$	$t_{setup}$ (s) $\downarrow$
DDPM	Retrain	0.00 (100.00)	0.00 (11.21)	0.00 (99.98)	0.00 (10.84)	0.00 (100.00)	0.00 (10.88)	0.00 (99.98)	0.00 (10.00)	0.00 (99.90)	0.00 (10.15)	145601.4
DDPM	EraseDiff [ 61 ]	28.06 (71.94)	2.66 (8.55)	72.42 (27.56)	1.96 (8.88)	51.40 (48.60)	2.16 (8.72)	42.24 (57.76)	1.20 (8.80)	38.10 (61.80)	1.80 (8.35)	1429.6
SPM	Retrain	0.00 (99.76)	0.00 (7.40)	0.00 (99.26)	0.00 (7.15)	0.00 (99.78)	0.00 (6.52)	0.00 (99.84)	0.00 (6.89)	0.00 (99.64)	0.00 (7.28)	90935.9
SPM	Ours	0.10 (99.86)	0.38 (7.78)	0.04 (99.22)	0.50 (7.65)	0.04 (99.82)	0.08 (6.60)	0.08 (99.76)	0.29 (7.18)	0.06 (99.58)	0.40 (7.68)	$<$ 1

Appendix

Table 7: CIFAR-10 class-wise forgetting on generation. Comparison with EraseDiff.

Panel of Figure 4 — Figure 4: Qualitative comparison of unlearned SPM . When a class is unlearned ( e.g . , cats or frogs framed in red ), the model avoids generating that concept and produces samples resembling remaining classes ( e.g . , a truck replacing a cat). Additionally, the generations from the remaining classes are left unchanged (framed in blue ).

Panel of Figure 5 — Figure 5: Generated images of GMM on CIFAR-10. The low-quality outputs demonstrate the limitations of traditional non-parametric models in handling complex generative tasks.

Backbone	Method	Random Unlearning (10 $\%$ )					Random Unlearning (50 $\%$ )
Backbone	Method	$\text{PG}_{\tt H}$ $\downarrow$	$\text{PG}_{\tt S}$ $\downarrow$	$\Delta$ UA $\downarrow$	$\Delta$ RA $\downarrow$	$\Delta$ TA $\downarrow$	$\text{PG}_{\tt H}$ $\downarrow$	$\text{PG}_{\tt S}$ $\downarrow$	$\Delta$ UA $\downarrow$	$\Delta$ RA $\downarrow$	$\Delta$ TA $\downarrow$
ResNet18	Retrain	0.00	0.00	0.00 (5.35)	0.00 (100.00)	0.00 (94.25)	0.00	0.00	0.00 (8.48)	0.00 (100.00)	0.00 (91.08)
	GA [ 52 ]	5.23	0.17	4.80 (0.55)	4.80 (99.52)	0.48 (94.57)	8.05	0.37	7.94 (0.54)	0.49 (99.51)	3.45 (94.53)
	FT [ 56 ]	5.89	0.19	4.46 (0.89)	0.16 (99.84)	0.32 (93.93)	8.50	0.36	7.52 (0.96)	0.18 (99.82)	2.67 (93.75)
	IU [ 31 ]	26.60	1.69	17.15 (22.50)	22.32 (77.68)	21.09 (73.16)	83.63	8.29	74.83 (83.32)	83.29 (16.71)	74.53 (16.55)
	BE [ 10 ]	5.30	0.17	4.83 (0.52)	0.48 (99.52)	0.29 (94.54)	34.70	1.22	17.33 (25.81)	25.93 (74.07)	23.37 (67.71)

Appendix

Table 8: CIFAR-10 random unlearning on classification. Results are averaged over 10 scenarios. We observe that our method can achieve close performance to the retrained model. The UA, RA, and TA values are reported in parentheses with the corresponding gaps.

Machine Unlearning

Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.

See all 351 papers

Unmerge: Efficient Machine Unlearning via Task Arithmetic
Sep 30, 2026Haoran Tang, Andrew Tan, Rajiv KhannaMachine Unlearning
Department of Computer Science Purdue University
PruneForget: Joint Unlearning and Pruning of Vision Models
Sep 26, 2026Yu-Shan Tai, Amber Yijia Zheng, Raymond A. YehMachine Unlearning Neural Network Pruning
Department of Computer Science Purdue University
Models Designed to Forget: Machine Unlearning via Key Deletion
Mar 16, 2026Sonia Laguna, Jorge da Silva Goncalves, Moritz Vandenhirtz +3Machine Unlearning
Department of Computer Science, ETH Zurich, Switzerland

Sep 30, 2026J/K move · Enter open · S save

Unmerge: Efficient Machine Unlearning via Task Arithmetic

Haoran Tang, Andrew Tan, Rajiv Khanna

Department of Computer Science Purdue University

Machine Unlearning

Open paper AskPDF arXiv

Approximate machine unlearning seeks to remove the influence of a forget set from a trained model without full retraining. Existing gradient-based methods require data-dependent hyperparameter search, struggle when forget and retain knowledge are entangled, and offer little insight into where unlearning actually happens inside the network. We recast unlearning through the lens of task arithmetic: if finetuning produces a merged task vector

τ_m

that combines learning on forget and retain sets, unlearning is the inverse operation that subtracts a learned forget component

τ_F

to recover the retain task vector

τ_R

. The forget signal is concentrated: at every layer, forget activations lie in a subspace spanned by a handful of dominant directions, so we factorize

τ_F

in a low-rank forget basis, which is faithful up to a small tail-eigenvalue residual and limits how far the correction can perturb retain. We then optimize three intuitive goals (match the merged vector inside the forget span, suppress leakage into the retain span, and bound the correction size) that provably bound forget leakage and retain damage in activation space. The resulting algorithm, Unmerge, is fast and powerful: on class-level unlearning with ResNet-50 on CIFAR-100 and Tiny ImageNet, it improves Tug-of-War by up to ~24% over a baseline of comparable runtime and by up to ~18% over stronger baselines that run ~5x slower, keeps membership-inference exposure at the level of retraining, and shrinks the feature-distribution gap to the retrained model, where relabeling methods leave forget features cleanly separable. Further studies show that Unmerge also applies to ViT-S/16 and scales to Llama-3.2-3B. The per-layer basis geometry that drives the algorithm also serves as a layerwise diagnostic for when and where unlearning becomes structurally hard.

Abstract

Machine Unlearning

Explore similar work

Unmerge: Efficient Machine Unlearning via Task Arithmetic

PruneForget: Joint Unlearning and Pruning of Vision Models

Models Designed to Forget: Machine Unlearning via Key Deletion