Unmerge: Efficient Machine Unlearning via Task Arithmetic

Haoran Tang; Andrew Tan; Rajiv Khanna

cs.LGSep 30, 2026

Unmerge: Efficient Machine Unlearning via Task Arithmetic

Authors: Haoran Tang, Andrew Tan, Rajiv Khanna

Organizations: Department of Computer Science Purdue University

Abstract

Approximate machine unlearning seeks to remove the influence of a forget set from a trained model without full retraining. Existing gradient-based methods require data-dependent hyperparameter search, struggle when forget and retain knowledge are entangled, and offer little insight into where unlearning actually happens inside the network. We recast unlearning through the lens of task arithmetic: if finetuning produces a merged task vector $τ_m$ that combines learning on forget and retain sets, unlearning is the inverse operation that subtracts a learned forget component $τ_F$ to recover the retain task vector $τ_R$ . The forget signal is concentrated: at every layer, forget activations lie in a subspace spanned by a handful of dominant directions, so we factorize $τ_F$ in a low-rank forget basis, which is faithful up to a small tail-eigenvalue residual and limits how far the correction can perturb retain. We then optimize three intuitive goals (match the merged vector inside the forget span, suppress leakage into the retain span, and bound the correction size) that provably bound forget leakage and retain damage in activation space. The resulting algorithm, Unmerge, is fast and powerful: on class-level unlearning with ResNet-50 on CIFAR-100 and Tiny ImageNet, it improves Tug-of-War by up to ~24% over a baseline of comparable runtime and by up to ~18% over stronger baselines that run ~5x slower, keeps membership-inference exposure at the level of retraining, and shrinks the feature-distribution gap to the retrained model, where relabeling methods leave forget features cleanly separable. Further studies show that Unmerge also applies to ViT-S/16 and scales to Llama-3.2-3B. The per-layer basis geometry that drives the algorithm also serves as a layerwise diagnostic for when and where unlearning becomes structurally hard.

Figures & tables

Table 1: CIFAR-100 unlearning across three superclasses (Trees, Aquatic Mammals, Vehicles). Cells report mean

\pm

1 std across superclasses (sample std,

N{=}3

; one seed per cell, three-seed variance in App. E.2 );

{\bm{W}}_{\mathrm{RT}}

shows plain means. Best non-retrained in bold . Unmerge variants achieve best ToW performance with noticeable short runtime. Per-superclass detail tables are in App. E.1 .

Table 2: TinyImageNet unlearning results (Dogs & Cats). Best non-retrained in bold . Unmerge-CAV-Class achieves the best ToW for both

{\bm{W}}_{\mathrm{init}}

at a runtime comparable to NegGrad+; the task-vector and projection baselines are faster but trail on ToW. See Arthropods results in Tab. 13 .

Panel of Figure 1 — Figure 1: Per-layer forget–retain entanglement on CIFAR-100 (Trees), ResNet-50: mean over the forget basis directions ${\bm{q}}_{i}$ of the overlap $\|{\bm{P}}_{\mathcal{R}}^{l}{\bm{q}}_{i}\|$ with the retain subspace, all bases at rank $16$ , seed 1 (the band is the spread over directions). This per-direction overlap is the rank-invariant form of the basis cross-norm $\|{\bm{Q}}_{\mathcal{F}}^{l\top}{\bm{Q}}_{\mathcal{R}}^{l}\|_{F}$ , the structural analogue of the merging cross-interference term $\|{\bm{\tau}}_{\mathcal{F}}{\bm{\tau}}_{\mathcal{R}}^{\top}\|_{F}$ (Asm. 3.1 , Remark 3.5 ); smaller means less entangled.

Table 3: Feature-level entanglement on CIFAR-100 (Trees): penultimate-layer distribution distance between forget and a same-size retain subsample, reported as relative gap to the retrained reference (lower is better;

{\bm{W}}_{\mathrm{RT}}

is 0 by construction). Best non-retrained per row in bold .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2: Per-layer merged-task-vector magnitude $\|{\bm{\tau}}_{\mathrm{m}}^{l}\|_{F}$ on CIFAR-100 (Trees), ResNet-50, log- $y$ . The three variant curves coincide exactly because ${\bm{\tau}}_{\mathrm{m}}$ is basis-independent; the shaded band is the range over three finetuning seeds. The cross-init gap is depth-dependent: shallow blocks (layer1–2) carry slightly larger magnitudes on ${\bm{W}}_{\mathrm{PT}}$ (pretrained features must be shifted away from ImageNet), while at layer4 and the classifier $\|{\bm{\tau}}_{\mathrm{m}}\|$ is $\sim$ $1.5$ – $2\times$ larger on ${\bm{W}}_{0}$ , where the deepest task-specific weights are encoded from scratch.

Table 10: CIFAR-100 unlearning results (Trees). Rows: evaluation metric (Forget/Retain/Test accuracy, %) and tug-of-war score (equation 8 ; higher is better). Columns: unlearning method.

{\bm{W}}_{\mathrm{FT}}

and

{\bm{W}}_{\mathrm{RT}}

are the finetuned upper bound and retrained gold standard. Best non-retrained in bold .

Table 11: CIFAR-100 unlearning results (Aquatic Mammals). Best non-retrained in bold .

Table 12: CIFAR-100 unlearning results (Vehicles). Best non-retrained in bold .

Table 13: TinyImageNet unlearning results (Arthropods). Best non-retrained in bold .

Table 17: Feature-level entanglement on TinyImageNet (Dogs & Cats), same protocol as Tab. 3 .

Table 18: MIA gap to the retrained reference on CIFAR-100 (Trees):

|\mathrm{att}_{\mathcal{F}}^{\mathrm{unl}}-\mathrm{att}_{\mathcal{F}}^{\mathrm{RT}}|

per attack signal (lower is better;

{\bm{W}}_{\mathrm{RT}}

is 0 by construction).

Table 19: MIA gap to the retrained reference on TinyImageNet (Dogs & Cats), same protocol as Tab. 18 .

Table 20: Instance-level forgetting on CIFAR-100 (Trees), ResNet-50. Coherent : half of the training images of each Trees class (

|\mathcal{F}|{=}1250

); Random :

2500

images drawn uniformly from all 100 classes. Accuracies in %.

\mathrm{att}_{\mathcal{F}}

: mean attack accuracy on

\mathcal{F}

over the 9 MIA signals of § 4.5 , with the signed gap to

{\bm{W}}_{\mathrm{RT}}

in parentheses (positive = forget samples more identifiable than under retraining). Ent.: mean relative entanglement gap of Tab. 3 . Best non-retrained ToW and entanglement per protocol in bold . † On the coherent protocol NegGrad+ is tuned on a 40-run grid (forget/retain weight, learning rate, epochs), because its class-level setting over-forgets there (ToW

0.4970

0.3251

); Random Label and SalUn keep their class-level settings, and the Unmerge cells sweep only

\alpha

(five values per basis). The identical forget/retain accuracies of Unmerge-PCA on

{\bm{W}}_{0}

and Unmerge-CAV-KMeans on

{\bm{W}}_{\mathrm{PT}}

are genuine (901 of 1250 forget images correct in both; test accuracies and losses differ).

Explore similar work

PURGE: Projected Unlearning via Retain-Guided Erasure
Jun 2, 2026Vedant Jawandhia, Daksh Ahuja, Ghufran Alam Siddiqui +3Machine Unlearning Concept Erasure
1BITS Pilani, Pilani Campus, India
Interference-Aware Multi-Task Unlearning
May 18, 2026Ying-Hua Huang, Rui Fang, Hsi-Wen Chen +1Machine Unlearning Multi--Task Learning
National Taiwan University
How Hard Can It Be? Hardness-Aware Multi-Objective Unlearning
Jun 1, 2026Jiangwei Chen, Xinyuan Niu, Rachael Hwee Ling Sim +3Machine Unlearning Catastrophic Forgetting
Department of Computer Science, National University of Singapore, Singapore · Institute for Infocomm Research, Agency for Science, Technology and Research, Singapore · Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore

Jun 2, 2026, cs.LGJ/K move · Enter open · S save

PURGE: Projected Unlearning via Retain-Guided Erasure

Vedant Jawandhia, Daksh Ahuja, Ghufran Alam Siddiqui, Prashant Trivedi+2

1BITS Pilani, Pilani Campus, India

Machine Unlearning Concept Erasure Retraining Continual Learning Removal Layer-Wise

Open paper AskPDF arXiv

We propose PURGE, a machine unlearning algorithm built on a simple but an under-exploited observation: continual learning (CL) and machine unlearning (MU) which are fundamentally dual problems. CL tries to learn new tasks without forgetting old ones; MU tries to erase specific data without hurting retained performance representing the same underlying tension in opposite directions. PURGE leverages this duality by adapting gradient projection from A-GEM (Chaudhry et al., 2019) so that every unlearning step is constrained to not increase the retain-set loss. On top of this, it performs multi-layer representation erasure, pushing forget-set activations in intermediate layers towards the retain distribution to remove information from hidden representations rather than just suppressing it at the output. A key design choice is the retain-confusion target: rather than pushing forget outputs toward the uniform distribution, which we found to be surprisingly easy for membership inference attacks to detect, we instead target the model's natural confusion pattern on retain data. This makes the unlearned model hard to distinguish from one retrained from scratch. Two self-regulating stopping criteria (a retain-loss budget and a forget-accuracy target) let the algorithm decide on its own when to stop, removing the need for manual epoch tuning. In experiments on five datasets (CIFAR-10, MNIST, SVHN, STL10, PathMNIST) across 22 class-level forgetting tasks, PURGE consistently keeps retain accuracy above 96% while achieving MIA AUROC close to 0.5 (the ideal), outperforming gradient ascent, KL-uniform, and several published baselines on the privacy-utility frontier.

Abstract

Explore similar work

PURGE: Projected Unlearning via Retain-Guided Erasure

Interference-Aware Multi-Task Unlearning

How Hard Can It Be? Hardness-Aware Multi-Objective Unlearning