Unmerge: Efficient Machine Unlearning via Task Arithmetic

Haoran Tang; Andrew Tan; Rajiv Khanna

cs.LGSep 30, 2026

Unmerge: Efficient Machine Unlearning via Task Arithmetic

Authors: Haoran Tang, Andrew Tan, Rajiv Khanna

Organizations: Department of Computer Science Purdue University

Abstract

Approximate machine unlearning seeks to remove the influence of a forget set from a trained model without full retraining. Existing gradient-based methods require data-dependent hyperparameter search, struggle when forget and retain knowledge are entangled, and offer little insight into where unlearning actually happens inside the network. We recast unlearning through the lens of task arithmetic: if finetuning produces a merged task vector $τ_m$ that combines learning on forget and retain sets, unlearning is the inverse operation that subtracts a learned forget component $τ_F$ to recover the retain task vector $τ_R$ . The forget signal is concentrated: at every layer, forget activations lie in a subspace spanned by a handful of dominant directions, so we factorize $τ_F$ in a low-rank forget basis, which is faithful up to a small tail-eigenvalue residual and limits how far the correction can perturb retain. We then optimize three intuitive goals (match the merged vector inside the forget span, suppress leakage into the retain span, and bound the correction size) that provably bound forget leakage and retain damage in activation space. The resulting algorithm, Unmerge, is fast and powerful: on class-level unlearning with ResNet-50 on CIFAR-100 and Tiny ImageNet, it improves Tug-of-War by up to ~24% over a baseline of comparable runtime and by up to ~18% over stronger baselines that run ~5x slower, keeps membership-inference exposure at the level of retraining, and shrinks the feature-distribution gap to the retrained model, where relabeling methods leave forget features cleanly separable. Further studies show that Unmerge also applies to ViT-S/16 and scales to Llama-3.2-3B. The per-layer basis geometry that drives the algorithm also serves as a layerwise diagnostic for when and where unlearning becomes structurally hard.

Figures & tables

Table 1: CIFAR-100 unlearning across three superclasses (Trees, Aquatic Mammals, Vehicles). Cells report mean

\pm

1 std across superclasses (sample std,

N{=}3

; one seed per cell, three-seed variance in App. E.2 );

{\bm{W}}_{\mathrm{RT}}

shows plain means. Best non-retrained in bold . Unmerge variants achieve best ToW performance with noticeable short runtime. Per-superclass detail tables are in App. E.1 .

Table 2: TinyImageNet unlearning results (Dogs & Cats). Best non-retrained in bold . Unmerge-CAV-Class achieves the best ToW for both

{\bm{W}}_{\mathrm{init}}

at a runtime comparable to NegGrad+; the task-vector and projection baselines are faster but trail on ToW. See Arthropods results in Tab. 13 .

Panel of Figure 1 — Figure 1: Per-layer forget–retain entanglement on CIFAR-100 (Trees), ResNet-50: mean over the forget basis directions ${\bm{q}}_{i}$ of the overlap $\|{\bm{P}}_{\mathcal{R}}^{l}{\bm{q}}_{i}\|$ with the retain subspace, all bases at rank $16$ , seed 1 (the band is the spread over directions). This per-direction overlap is the rank-invariant form of the basis cross-norm $\|{\bm{Q}}_{\mathcal{F}}^{l\top}{\bm{Q}}_{\mathcal{R}}^{l}\|_{F}$ , the structural analogue of the merging cross-interference term $\|{\bm{\tau}}_{\mathcal{F}}{\bm{\tau}}_{\mathcal{R}}^{\top}\|_{F}$ (Asm. 3.1 , Remark 3.5 ); smaller means less entangled.

Table 3: Feature-level entanglement on CIFAR-100 (Trees): penultimate-layer distribution distance between forget and a same-size retain subsample, reported as relative gap to the retrained reference (lower is better;

{\bm{W}}_{\mathrm{RT}}

is 0 by construction). Best non-retrained per row in bold .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2: Per-layer merged-task-vector magnitude $\|{\bm{\tau}}_{\mathrm{m}}^{l}\|_{F}$ on CIFAR-100 (Trees), ResNet-50, log- $y$ . The three variant curves coincide exactly because ${\bm{\tau}}_{\mathrm{m}}$ is basis-independent; the shaded band is the range over three finetuning seeds. The cross-init gap is depth-dependent: shallow blocks (layer1–2) carry slightly larger magnitudes on ${\bm{W}}_{\mathrm{PT}}$ (pretrained features must be shifted away from ImageNet), while at layer4 and the classifier $\|{\bm{\tau}}_{\mathrm{m}}\|$ is $\sim$ $1.5$ – $2\times$ larger on ${\bm{W}}_{0}$ , where the deepest task-specific weights are encoded from scratch.

Table 10: CIFAR-100 unlearning results (Trees). Rows: evaluation metric (Forget/Retain/Test accuracy, %) and tug-of-war score (equation 8 ; higher is better). Columns: unlearning method.

{\bm{W}}_{\mathrm{FT}}

and

{\bm{W}}_{\mathrm{RT}}

are the finetuned upper bound and retrained gold standard. Best non-retrained in bold .

Table 11: CIFAR-100 unlearning results (Aquatic Mammals). Best non-retrained in bold .

Table 12: CIFAR-100 unlearning results (Vehicles). Best non-retrained in bold .

Table 13: TinyImageNet unlearning results (Arthropods). Best non-retrained in bold .

Table 17: Feature-level entanglement on TinyImageNet (Dogs & Cats), same protocol as Tab. 3 .

Table 18: MIA gap to the retrained reference on CIFAR-100 (Trees):

|\mathrm{att}_{\mathcal{F}}^{\mathrm{unl}}-\mathrm{att}_{\mathcal{F}}^{\mathrm{RT}}|

per attack signal (lower is better;

{\bm{W}}_{\mathrm{RT}}

is 0 by construction).

Table 19: MIA gap to the retrained reference on TinyImageNet (Dogs & Cats), same protocol as Tab. 18 .

Table 20: Instance-level forgetting on CIFAR-100 (Trees), ResNet-50. Coherent : half of the training images of each Trees class (

|\mathcal{F}|{=}1250

); Random :

2500

images drawn uniformly from all 100 classes. Accuracies in %.

\mathrm{att}_{\mathcal{F}}

: mean attack accuracy on

\mathcal{F}

over the 9 MIA signals of § 4.5 , with the signed gap to

{\bm{W}}_{\mathrm{RT}}

in parentheses (positive = forget samples more identifiable than under retraining). Ent.: mean relative entanglement gap of Tab. 3 . Best non-retrained ToW and entanglement per protocol in bold . † On the coherent protocol NegGrad+ is tuned on a 40-run grid (forget/retain weight, learning rate, epochs), because its class-level setting over-forgets there (ToW

0.4970

0.3251

); Random Label and SalUn keep their class-level settings, and the Unmerge cells sweep only

\alpha

(five values per basis). The identical forget/retain accuracies of Unmerge-PCA on

{\bm{W}}_{0}

and Unmerge-CAV-KMeans on

{\bm{W}}_{\mathrm{PT}}

are genuine (901 of 1250 forget images correct in both; test accuracies and losses differ).

Explore similar work

Jun 2, 2026cs.LG

PURGE: Projected Unlearning via Retain-Guided Erasure

We propose PURGE, a machine unlearning algorithm built on a simple but an under-exploited observation: continual learning (CL) and machine unlearning (MU) which are fundamentally dual problems. CL tries to learn new tasks without forgetting old ones; MU tries to erase specific data without hurting retained performance representing the same underlying tension in opposite directions. PURGE leverages this duality by adapting gradient projection from A-GEM (Chaudhry et al., 2019) so that every unlearning step is constrained to not increase the retain-set loss. On top of this, it performs multi-layer representation erasure, pushing forget-set activations in intermediate layers towards the retain distribution to remove information from hidden representations rather than just suppressing it at the output. A key design choice is the retain-confusion target: rather than pushing forget outputs toward the uniform distribution, which we found to be surprisingly easy for membership inference attacks to detect, we instead target the model's natural confusion pattern on retain data. This makes the unlearned model hard to distinguish from one retrained from scratch. Two self-regulating stopping criteria (a retain-loss budget and a forget-accuracy target) let the algorithm decide on its own when to stop, removing the need for manual epoch tuning. In experiments on five datasets (CIFAR-10, MNIST, SVHN, STL10, PathMNIST) across 22 class-level forgetting tasks, PURGE consistently keeps retain accuracy above 96% while achieving MIA AUROC close to 0.5 (the ideal), outperforming gradient ascent, KL-uniform, and several published baselines on the privacy-utility frontier.

Vedant Jawandhia, Daksh Ahuja, Ghufran Alam Siddiqui +3

1BITS Pilani, Pilani Campus, India

Machine Unlearning Concept Erasure Retraining Continual Learning Removal Layer-Wise

Ask PDF

May 18, 2026cs.AI

Interference-Aware Multi-Task Unlearning

Machine unlearning aims to remove the contribution of designated training data from a trained model while preserving performance on the remaining data. Existing work mainly focuses on single-task settings, whereas modern models often operate in multi-task setups with shared backbones, where removing supervision for one task or instance can unintentionally affect others. We introduce multi-task unlearning with two settings: full-task unlearning, which removes a target instance from all tasks, and partial-task unlearning, which removes supervision only from selected tasks. We show that shared parameters couple the forget and retain sets, causing task-level interference on non-target tasks and instance-level interference on other instances. To address this issue, we propose an interference-aware framework that combines task-aware gradient projection, which constrains updates within task-specific subspaces, with instance-level gradient orthogonalization, which reduces conflicts between forget and retain signals. Experiments on two multi-task computer vision benchmarks across five tasks show that our method achieves effective unlearning while maintaining strong generalization, reducing UIS compared with the strongest baseline by 30.3% in full-task unlearning and 52.9% in partial-task unlearning.

Ying-Hua Huang, Rui Fang, Hsi-Wen Chen +1

National Taiwan University

Machine Unlearning Multi--Task Learning Multimodal Learning Task-Level Training Data

Ask PDF

Jun 1, 2026cs.LG

How Hard Can It Be? Hardness-Aware Multi-Objective Unlearning

Machine unlearning aims to remove the influence of specific forget training data due to privacy, copyright or bias concerns while maintaining the model performance on the remaining retain data. Existing unlearning algorithms, such as optimizing a weighted combination of losses, have tried to achieve these objectives of improving forget quality and maintaining retain utility. However, they do not guarantee that these objectives can be improved by a specified extent for all forget and retain data. In this work, we address this limitation with a novel and theoretically-grounded approach from a constrained optimization perspective. Firstly, we identify that the hardness of reconciling both objectives can be quantified by the similarity between the forget data and the retain data. Next, we derive an unlearning algorithm (HAMU) with the overall goal of guaranteeing a specified improvement in forget quality while minimizing the retain utility cost/degradation by updating the model weights based on our hardness measure. Our hardness measure also informs users when retain utility degradation is unavoidable, i.e., both objectives cannot be improved simultaneously, and stopping should be considered. Our algorithm is applicable to non-convex models and is easily parallelizable, making it readily deployable in real-world scenarios. We empirically demonstrate HAMU's superior performance over baselines on both image and text datasets using large models. Our code is available at https://github.com/aoi3142/HAMU.

Jiangwei Chen, Xinyuan Niu, Rachael Hwee Ling Sim +3

Department of Computer Science, National University of Singapore, Singapore · Institute for Infocomm Research, Agency for Science, Technology and Research, Singapore · Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore

Machine Unlearning Catastrophic Forgetting Privacy-Preserving Machine Learning Retraining Hard

Ask PDF