cs.CVFeb 23, 2026

Benchmarking Unlearning for Vision Transformers

Authors: Kairan Zhao, Iurie Luca, Peter Triantafillou

Organizations: University of Warwick

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Machine unlearning (MU) refers to the post-training capability to remove (the influence of) training examples that are incorrect, biased, or leak sensitive/private information. MU is now widely regarded as critical for building safe and fair AI. In parallel, research into transformer architectures for computer vision has been highly successful: Vision Transformers (VTs) increasingly emerge as strong alternatives to CNNs. Yet, MU research for vision tasks has largely centered on CNNs, not VTs. While MU benchmarks have been developed for LLMs, diffusion models, and CNNs, none currently exist for VTs. This work is the first to attempt this, benchmarking MU algorithm performance across different VT families (ViT, Swin-T, and DINOv2) and at different capacities. The work employs (i) different datasets, selected to assess the impacts of dataset scale and complexity; (ii) different MU algorithms, selected to represent fundamentally different approaches for MU; and (iii) both single-shot and continual unlearning protocols. Additionally, it focuses on benchmarking MU algorithms that leverage training data memorization, since leveraging memorization has been recently discovered to significantly improve the performance of previously SOTA algorithms. En route, the work characterizes how VTs memorize training data relative to CNNs, and assesses the impact of different memorization proxies on performance. The benchmark uses unified evaluation metrics that capture two complementary notions of forget quality along with accuracy on unseen (test) data and on retained data. Overall, this work offers a benchmarking basis, enabling reproducible, fair, and comprehensive comparisons of existing (and future) MU algorithms on VTs. Importantly, for the first time, it sheds light on how well existing algorithms work in VT settings, establishing a promising reference performance baseline.

Figures & tables

 CIFAR-10 CIFAR-100
ProxyResNet-18ViT-SmallSwin-Tiny ResNet-50ViT-SmallSwin-Tiny
Conf-0.80-0.79-0.88 -0.91-0.85-0.90
MaxConf-0.76-0.77-0.85 -0.87-0.80-0.86
Ent-0.75-0.78-0.85 -0.80-0.77-0.82
BA-0.71-0.63-0.79 -0.89-0.69-0.78
HR+0.67+0.45+0.64 +0.62+0.50+0.52
Table 1: Spearman correlations between memorization scores and proxies across CNNs and VTs. CNN-derived memorization proxies remain informative for VTs and can support scalable memorization-based unlearning.
 ViT-SmallSwin-TinyDINOv2-Small
MethodWithin RUM ToW ( ↑\uparrow ) / ToW-MIA ( ↑\uparrow )Vanilla ToW ( ↑\uparrow ) / ToW-MIA ( ↑\uparrow )Within RUM ToW ( ↑\uparrow ) / ToW-MIA ( ↑\uparrow )Vanilla ToW ( ↑\uparrow ) / ToW-MIA ( ↑\uparrow )Within RUM ToW ( ↑\uparrow ) / ToW-MIA ( ↑\uparrow )Vanilla ToW ( ↑\uparrow ) / ToW-MIA ( ↑\uparrow )
Original0.619±0.0400.619{{\scriptstyle\pm 0.040}} / 0.538±0.0230.538{{\scriptstyle\pm 0.023}}Same as left0.670±0.0270.670{{\scriptstyle\pm 0.027}} / 0.586±0.0250.586{{\scriptstyle\pm 0.025}}Same as left0.555±0.1120.555{{\scriptstyle\pm 0.112}} / 0.425±0.0800.425{{\scriptstyle\pm 0.080}}Same as left
Fine-tune0.855±0.036\mathbf{0.855}{{\scriptstyle\pm 0.036}} / 0.889±0.008\mathbf{0.889}{{\scriptstyle\pm 0.008}}0.807±0.0250.807{{\scriptstyle\pm 0.025}} / 0.844±0.0230.844{{\scriptstyle\pm 0.023}}0.822±0.034\mathbf{0.822}{{\scriptstyle\pm 0.034}} / 0.839±0.032\mathbf{0.839}{{\scriptstyle\pm 0.032}}0.774±0.0060.774{{\scriptstyle\pm 0.006}} / 0.781±0.0200.781{{\scriptstyle\pm 0.020}}0.846±0.038\mathbf{0.846}{{\scriptstyle\pm 0.038}} / 0.824±0.070\mathbf{0.824}{{\scriptstyle\pm 0.070}}0.796±0.0610.796{{\scriptstyle\pm 0.061}} / 0.768±0.0640.768{{\scriptstyle\pm 0.064}}
NegGrad+0.931±0.039\mathbf{0.931}{{\scriptstyle\pm 0.039}} / 0.838±0.0240.838{{\scriptstyle\pm 0.024}}0.888±0.0220.888{{\scriptstyle\pm 0.022}} / 0.845±0.026\mathbf{0.845}{{\scriptstyle\pm 0.026}}0.975±0.027\mathbf{0.975}{{\scriptstyle\pm 0.027}} / 0.902±0.035\mathbf{0.902}{{\scriptstyle\pm 0.035}}0.851±0.0180.851{{\scriptstyle\pm 0.018}} / 0.831±0.0120.831{{\scriptstyle\pm 0.012}}0.937±0.044\mathbf{0.937}{{\scriptstyle\pm 0.044}} / 0.896±0.032\mathbf{0.896}{{\scriptstyle\pm 0.032}}0.827±0.0640.827{{\scriptstyle\pm 0.064}} / 0.762±0.0440.762{{\scriptstyle\pm 0.044}}
SalUn0.903±0.049\mathbf{0.903}{{\scriptstyle\pm 0.049}} / 0.709±0.085\mathbf{0.709}{{\scriptstyle\pm 0.085}}0.884±0.0460.884{{\scriptstyle\pm 0.046}} / 0.686±0.0470.686{{\scriptstyle\pm 0.047}}0.870±0.039\mathbf{0.870}{{\scriptstyle\pm 0.039}} / 0.751±0.043\mathbf{0.751}{{\scriptstyle\pm 0.043}}0.833±0.0930.833{{\scriptstyle\pm 0.093}} / 0.640±0.1090.640{{\scriptstyle\pm 0.109}}0.831±0.121\mathbf{0.831}{{\scriptstyle\pm 0.121}} / 0.840±0.125\mathbf{0.840}{{\scriptstyle\pm 0.125}}0.736±0.1460.736{{\scriptstyle\pm 0.146}} / 0.662±0.1550.662{{\scriptstyle\pm 0.155}}
LetheViT0.848±0.007\mathbf{0.848}{{\scriptstyle\pm 0.007}} / 0.879±0.028\mathbf{0.879}{{\scriptstyle\pm 0.028}}0.808±0.0470.808{{\scriptstyle\pm 0.047}} / 0.839±0.0450.839{{\scriptstyle\pm 0.045}}0.815±0.018\mathbf{0.815}{{\scriptstyle\pm 0.018}} / 0.851±0.032\mathbf{0.851}{{\scriptstyle\pm 0.032}}0.773±0.0140.773{{\scriptstyle\pm 0.014}} / 0.780±0.0240.780{{\scriptstyle\pm 0.024}}0.784±0.047\mathbf{0.784}{{\scriptstyle\pm 0.047}} / 0.743±0.047\mathbf{0.743}{{\scriptstyle\pm 0.047}}0.727±0.0300.727{{\scriptstyle\pm 0.030}} / 0.681±0.0110.681{{\scriptstyle\pm 0.011}}
Table 2: Comparison of RUM-integrated and vanilla unlearning methods on CIFAR-100 using the Holdout Retraining proxy. Each entry reports ToW ( ↑\uparrow ) / ToW-MIA ( ↑\uparrow ) as mean ±\pm 95% confidence interval, with best results for each entry shown in bold. Across all architectures, RUM generally improves the base unlearning algorithms.
Figure 1: Unlearning performance comparison on CIFAR-100, measured by ToW and ToW-MIA (higher is better), reported as mean with 95% confidence interval. NegGrad+ is consistently strong, especially when paired with Holdout Retraining, while SalUn often achieves competitive ToW but weaker ToW-MIA. ViT favors fine-tuning-based methods (Fine-tune, L1-Sparse), whereas Swin-T favors gradient-based methods (NegGrad+, NegGrad).
ArchitectureCIFAR-10CIFAR-100
ConfidenceHoldout RetrainingConfidenceHoldout Retraining
ResNet-1850.433±6.80850.433{\scriptstyle\pm 6.808}62.922±4.68162.922{\scriptstyle\pm 4.681}––
ResNet-50––64.267±0.50464.267{\scriptstyle\pm 0.504}69.856±2.62069.856{\scriptstyle\pm 2.620}
ViT-Small94.089±0.45694.089{\scriptstyle\pm 0.456}89.244±1.32189.244{\scriptstyle\pm 1.321}69.322±1.78869.322{\scriptstyle\pm 1.788}68.767±3.82868.767{\scriptstyle\pm 3.828}
Swin-Tiny91.867±1.71191.867{\scriptstyle\pm 1.711}86.389±1.24686.389{\scriptstyle\pm 1.246}69.833±1.24269.833{\scriptstyle\pm 1.242}70.267±2.84870.267{\scriptstyle\pm 2.848}
DINOv2-Small88.529±2.10188.529{\scriptstyle\pm 2.101}92.333±3.23492.333{\scriptstyle\pm 3.234}58.267±3.77058.267{\scriptstyle\pm 3.770}58.778±4.94458.778{\scriptstyle\pm 4.944}
Table 3: DfD_{f} accuracies of the retrained reference model θr\theta_{r} across architectures. VTs show much higher DfD_{f} accuracy than ResNet-18 on CIFAR-10, but this gap largely disappears on the more complex CIFAR-100 setting.
AlgorithmViT-Small ToW ( ↑\uparrow ) / ToW-MIA ( ↑\uparrow )ViT-Tiny ToW ( ↑\uparrow ) / ToW-MIA ( ↑\uparrow )Swin-Small ToW ( ↑\uparrow ) / ToW-MIA ( ↑\uparrow )Swin-Tiny ToW ( ↑\uparrow ) / ToW-MIA ( ↑\uparrow )
Original0.891±0.016\mathbf{0.891}{{\scriptstyle\pm 0.016}} / 0.831±0.014\mathbf{0.831}{{\scriptstyle\pm 0.014}}0.773±0.0200.773{{\scriptstyle\pm 0.020}} / 0.699±0.0200.699{{\scriptstyle\pm 0.020}}0.886±0.013\mathbf{0.886}{{\scriptstyle\pm 0.013}} / 0.832±0.020\mathbf{0.832}{{\scriptstyle\pm 0.020}}0.862±0.0080.862{{\scriptstyle\pm 0.008}} / 0.811±0.0180.811{{\scriptstyle\pm 0.018}}
Fine-tune0.928±0.013\mathbf{0.928}{{\scriptstyle\pm 0.013}} / 0.913±0.008\mathbf{0.913}{{\scriptstyle\pm 0.008}}0.862±0.0600.862{{\scriptstyle\pm 0.060}} / 0.854±0.0600.854{{\scriptstyle\pm 0.060}}0.921±0.0050.921{{\scriptstyle\pm 0.005}} / 0.924±0.0100.924{{\scriptstyle\pm 0.010}}0.923±0.021\mathbf{0.923}{{\scriptstyle\pm 0.021}} / 0.931±0.021\mathbf{0.931}{{\scriptstyle\pm 0.021}}
NegGrad+0.916±0.0160.916{{\scriptstyle\pm 0.016}} / 0.968±0.015\mathbf{0.968}{{\scriptstyle\pm 0.015}}0.957±0.011\mathbf{0.957}{{\scriptstyle\pm 0.011}} / 0.869±0.0110.869{{\scriptstyle\pm 0.011}}0.944±0.0280.944{{\scriptstyle\pm 0.028}} / 0.888±0.0210.888{{\scriptstyle\pm 0.021}}0.977±0.023\mathbf{0.977}{{\scriptstyle\pm 0.023}} / 0.924±0.028\mathbf{0.924}{{\scriptstyle\pm 0.028}}
SalUn0.956±0.003\mathbf{0.956}{{\scriptstyle\pm 0.003}} / 0.766±0.039\mathbf{0.766}{{\scriptstyle\pm 0.039}}0.846±0.0430.846{{\scriptstyle\pm 0.043}} / 0.640±0.0430.640{{\scriptstyle\pm 0.043}}0.950±0.0310.950{{\scriptstyle\pm 0.031}} / 0.568±0.0930.568{{\scriptstyle\pm 0.093}}0.961±0.048\mathbf{0.961}{{\scriptstyle\pm 0.048}} / 0.834±0.022\mathbf{0.834}{{\scriptstyle\pm 0.022}}
Table 4: Effect of VT capacity on CIFAR-10 using the HR proxy. Results are reported as mean ±\pm 95% confidence interval for ToW ( ↑\uparrow ) and ToW-MIA ( ↑\uparrow ). ViT-Small and Swin-Tiny provide the best overall balance, suggesting a capacity “sweet spot” for unlearning.
MethodSwin-Small (49M)Swin-Base (88M)ViT-Base/16 (85M)
Fine-tune0.780±0.0230.780{{\scriptstyle\pm 0.023}}0.808±0.0070.808{{\scriptstyle\pm 0.007}}0.843±0.0160.843{{\scriptstyle\pm 0.016}}
NegGrad+0.819±0.0180.819{{\scriptstyle\pm 0.018}}0.837±0.0040.837{{\scriptstyle\pm 0.004}}0.842±0.0050.842{{\scriptstyle\pm 0.005}}
SalUn0.743±0.0270.743{{\scriptstyle\pm 0.027}}0.746±0.0130.746{{\scriptstyle\pm 0.013}}0.773±0.0270.773{{\scriptstyle\pm 0.027}}
LetheViT0.733±0.0170.733{{\scriptstyle\pm 0.017}}0.752±0.0260.752{{\scriptstyle\pm 0.026}}0.773±0.0300.773{{\scriptstyle\pm 0.030}}
Table 5: Unlearning performance measured by ToW ( ↑\uparrow ) on ImageNet-1K validation set with HR proxy, comparing larger VT variants. The method-level trends observed on CIFAR-100 persist in larger and more complex settings.
ArchitectureDfD_{f} accuracy ( %\% )
ResNet-5069.139±1.72269.139{\scriptstyle\pm 1.722}
Swin-Small69.433±2.24669.433{\scriptstyle\pm 2.246}
Table 6: DfD_{f} accuracies of the retrained model θr\theta_{r} (with HR) on ImageNet-1K validation setting. Similar accuracy suggests that VT pretraining advantages diminish on more complex data.
Figure 2: Continual unlearning performance across 5/10 Steps on CIFAR-100 using ViT-Small, reported with 95% confidence intervals. Performance remains stable across 5/10 steps; results for Swin-Tiny and additional datasets are reported in Figure 9 in Appendix A.5 .
Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

 ResNet-18Transformers
OptimizerSGDAdamW
Base learning rate0.010.0001
LossCross-EntropyCross-Entropy
Learning rate schedulerStep decayCosineAnnealingLR
Batch size512128
Epochs3030
Appendix
Table 7: Training configurations along with hyper-parameters of the models used to compute memorization scores shown in Figures 3 . The “Transformers” column represents the configuration for both ViT-Small and Swin-Tiny across CIFAR-10 and CIFAR-100.
Table 8: Training configurations and hyperparameter used for the original models ( θo\theta_{o} ) and the retrained models ( θr\theta_{r} ), using ViT-Small and Swin-Tiny architectures on CIFAR-10, CIFAR-100, and SVHN datasets.
AlgorithmArchitectureUnlearn EpochsUnlearn LRβ\betaα\alphaγ\gamma
Fine-tuneViT-Small5,5,100.0001–––
Swin-Tiny5,5,100.0001–––
NegGrad+ViT-Small5,5,100.000020.97––
Swin-Tiny5,5,100.000020.97––
SalUnViT-Small5,5,100.00005–10.1
Swin-Tiny5,5,100.0002–10.3
Appendix
Table 9: Hyper-parameters across all unlearn methods within the RUM F meta-algorithm on both Confidence and Holdout Retraining proxies to derive the unlearn models θu\theta_{u} . “Unlearn Epochs” column refers to the performed number of epochs each algorithm performed for the three memorization partitions (low →\rightarrow medium →\rightarrow high) respectively.
Unlearn EpochsUnlearn LRβ\beta
1,1,10.000020.97
Appendix
Table 10: Hyper-parameters at each sequential unlearning step for the NegGrad+ and Holdout Retraining configuration utilized. “Unlearn Epochs” are reduced to 1 for partition (low →\rightarrow medium →\rightarrow high) due to the smaller forget set size( ∣Df∣=600|D_{f}|=600 ) at each step. All other settings remain consistent with the primary experiments.
Figure 3: Memorization histograms across architectures on CIFAR-100 (top) and CIFAR-10 (bottom). Across both datasets, the distributions are similarly skewed, consistent with the long-tail phenomenon in memorization. Memorization distributions on CIFAR-100 (top) and CIFAR-10 (bottom) for ResNet-50, ViT-Small, and Swin-Tiny. μ\mu denotes the mean memorization score. All three architectures show a skewed, long-tailed distribution, suggesting that VTs, like CNNs, contain a small subset of highly memorized examples. This supports the use of memorization-based unlearning for VTs.
 CIFAR-10 CIFAR-100
ProxyResNet-18ViT-SmallSwin-Tiny ResNet-50ViT-SmallSwin-Tiny
Conf-0.80-0.79-0.88 -0.91-0.85-0.90
MaxConf-0.76-0.77-0.85 -0.87-0.80-0.86
Ent-0.75-0.78-0.85 -0.80-0.77-0.82
BA-0.71-0.63-0.79 -0.89-0.69-0.78
HR+0.67+0.45+0.64 +0.62+0.50+0.52
Appendix
Table 11: Spearman correlation coefficients between memorization and proxies (duplicate of Table 1 , included here for completeness)
Figure 4: Representation visualization before vs after unlearning for ViT-Small and Swin-Tiny
Figure 5: Layer-wise CKA similarity on CIFAR-100 for ViT-Small and Swin-Tiny under Fine-tune and NegGrad+. Rows compare model pairs, columns correspond to layers, and higher values indicate more similar representations. Early layers remain highly stable, while larger changes occur in later blocks and the final normalization layer.
(e) ToW with Confidence
Figure 7: MU performance comparison on SVHN
Figure 8: Continual unlearning performance across 5 steps on CIFAR-10 and SVHN datasets using NegGrad+ and the HR proxy.
Figure 9: Continual unlearning performance across 5/10 steps on CIFAR-100 using NegGrad+ and the HR proxy.
 CIFAR-10CIFAR-100
 ViT-SmallSwin-TinyViT-SmallSwin-Tiny
StepToW (↑)(\uparrow)ToW-MIA (↑)(\uparrow)ToW (↑)(\uparrow)ToW-MIA (↑)(\uparrow)ToW (↑)(\uparrow)ToW-MIA (↑)(\uparrow)ToW (↑)(\uparrow)ToW-MIA (↑)(\uparrow)
10.907±0.0300.907\pm 0.0300.960±0.0170.960\pm 0.0170.966±0.0160.966\pm 0.0160.933±0.0820.933\pm 0.0820.891±0.0650.891\pm 0.0650.816±0.0510.816\pm 0.0510.906±0.0390.906\pm 0.0390.823±0.0230.823\pm 0.023
20.936±0.0630.936\pm 0.0630.944±0.0290.944\pm 0.0290.957±0.0160.957\pm 0.0160.917±0.0300.917\pm 0.0300.868±0.0310.868\pm 0.0310.770±0.0150.770\pm 0.0150.840±0.0640.840\pm 0.0640.764±0.0850.764\pm 0.085
30.929±0.0440.929\pm 0.0440.948±0.0230.948\pm 0.0230.954±0.0540.954\pm 0.0540.909±0.0960.909\pm 0.0960.857±0.0490.857\pm 0.0490.764±0.0470.764\pm 0.0470.853±0.0520.853\pm 0.0520.776±0.0550.776\pm 0.055
40.943±0.0100.943\pm 0.0100.942±0.0280.942\pm 0.0280.947±0.0580.947\pm 0.0580.899±0.0580.899\pm 0.0580.858±0.0530.858\pm 0.0530.760±0.0400.760\pm 0.0400.843±0.0210.843\pm 0.0210.746±0.0350.746\pm 0.035
Appendix
Table 12: Detailed unlearning performance at each of the 5 sequential steps for the NegGrad+ and Holdout Retraining configuration discussed in Section 4.6 and displayed in Figures 2 and 8 .
StepViT-SmallSwin-Tiny
 ToW ( ↑\uparrow )ToW-MIA ( ↑\uparrow )ToW ( ↑\uparrow )ToW-MIA ( ↑\uparrow )
10.947±0.0430.947\pm 0.0430.889±0.1080.889\pm 0.1080.947±0.0350.947\pm 0.0350.860±0.0820.860\pm 0.082
20.971±0.0330.971\pm 0.0330.828±0.0400.828\pm 0.0400.900±0.1040.900\pm 0.1040.786±0.0640.786\pm 0.064
30.973±0.0190.973\pm 0.0190.845±0.0340.845\pm 0.0340.953±0.0210.953\pm 0.0210.850±0.0240.850\pm 0.024
40.918±0.0530.918\pm 0.0530.840±0.2440.840\pm 0.2440.940±0.0810.940\pm 0.0810.816±0.0940.816\pm 0.094
50.921±0.0460.921\pm 0.0460.769±0.0200.769\pm 0.0200.958±0.0790.958\pm 0.0790.834±0.1380.834\pm 0.138
Appendix
Table 13: Continual unlearning over 10 sequential steps on CIFAR-100 using NegGrad+ with the Holdout Retraining proxy (configuration discussed in Section 4.6 ). Results are reported as mean ±\pm 95% confidence interval.
Table 14: Detailed performance results for the VTs (ViT, Swin-T, DINOv2) across both the CIFAR datasets.
Table 15: Detailed performance result metrics for ResNet-18 and ResNet-50 on the CIFAR-10 and CIFAR-100 datasets respectively.
Table 16: Accuracies on DrD_{r} , DfD_{f} and DtestD_{test} for the “Retrain” models θr\theta_{r} as well as the respective “Unlearned” models θu\theta_{u} , evaluated across all unlearning algorithms within the RUM F framework. MIA performance is also reported. Results are averaged over 3 runs, with 95% confidence intervals.

Explore similar work

CardsList

  1. Models Designed to Forget: Machine Unlearning via Key Deletion

    Mar 16, 2026Sonia Laguna, Jorge da Silva Goncalves, Moritz Vandenhirtz +3Machine Unlearning

    Department of Computer Science, ETH Zurich, Switzerland

  2. Before Forgetting, Learn to Remember: Revisiting Foundational Learning Failures in LVLM Unlearning Benchmarks

    May 5, 2026JuneHyoung Kwon, MiHyeon Kim, Eunju Lee +3VLM EvaluationVLM Unlearning

    Department of Artificial Intelligence, Chung-Ang University · KT Corporation · Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University

  3. Unmerge: Efficient Machine Unlearning via Task Arithmetic

    Sep 30, 2026Haoran Tang, Andrew Tan, Rajiv KhannaMachine Unlearning

    Department of Computer Science Purdue University