cs.LGApr 10, 2026

Efficient Unlearning through Maximizing Relearning Convergence Delay

Authors: Khoa Tran, Simon S. Woo

Organizations: CSE Department, Sungkyunkwan University, South Korea · Secure Machines Lab

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Machine unlearning poses challenges in removing mislabeled, contaminated, or problematic data from a pretrained model. Current unlearning approaches and evaluation metrics are solely focused on model predictions, which limits insight into the model's true underlying data characteristics. To address this issue, we introduce a new metric called relearning convergence delay, which captures both changes in weight space and prediction space, providing a more comprehensive assessment of the model's understanding of the forgotten dataset. This metric can be used to assess the risk of forgotten data being recovered from the unlearned model. Based on this, we propose the Influence Eliminating Unlearning framework, which removes the influence of the forgetting set by degrading its performance and incorporates weight decay and injecting noise into the model's weights, while maintaining accuracy on the retaining set. Extensive experiments show that our method outperforms existing metrics and our proposed relearning convergence delay metric, approaching ideal unlearning performance. We provide theoretical guarantees, including exponential convergence and upper bounds, as well as empirical evidence of strong retention and resistance to relearning in both classification and generative unlearning tasks.

Figures & tables

Table 1: Performance summary of various unlearning methods for the ResNet model trained on TinyImageNet in two unlearning scenarios, 30% random and 50% random data forgetting. Performance gap against Retraining is provided in ( ⋅\cdot ) .
Table 2: Performance summary of various unlearning methods for the ResNet model trained on TinyImageNet in two unlearning scenarios, 30% class-wise and 50% class-wise data forgetting. Performance gap against Retraining is provided in ( ⋅\cdot ) .
Figure 1: Relationship between Avg. Gap and RCDGD\mathcal{RCD}_{GD} (step-size η=10−4\eta=10^{-4} ) of ResNet model on the training-forgetting dataset Dftrain\mathcal{D}_{f}^{\text{train}} of TinyImageNet across diverse unlearning scenarios. Our methods consistently achieve a low Avg. Gap and a high RCDGD\mathcal{RCD}_{GD} score across four forgetting scenarios, demonstrating efficacy in both model utility and privacy.
Figure 2: The RCDGD\mathcal{RCD}_{GD} values of ResNet model on the training-forgetting set Dftrain\mathcal{D}_{f}^{\text{train}} of TinyImageNet for various step-sizes. With a small step-size, the RCDGD\mathcal{RCD}_{GD} values of each unlearning method are less distinguishable, whereas an appropriate step-size enables a significant comparison.
Figure 3: Examples of generated images using various SD models from I2P prompts. The unlearning methods include ESD and SALUN, both with and without the use of Noisy . Each column presents images generated by different SD variants using the same prompt, denoted as PiP_{i} . Detailed descriptions of the prompts are provided in the Appendix section.
Table 3: Performance of unlearned approaches on ImageNette concepts using FID, I2P prompts using Nudity score, and relearning convergence delay of relearning harmful concepts from I2P prompts, measured as RCDAdam\mathcal{RCD}_{Adam} .
Figure 4: Image generation results for ImageNette classes using models unlearned from I2P harmful concepts. The generated images show that using the Noisy component helps the unlearned model better preserve its performance on unrelated concepts.
Figure 5: RCD\mathcal{RCD} vs. Relearning Difficulty correlation. Larger RCD\mathcal{RCD} values correspond to slower recovery when retraining on the forgotten data. Retraining yields the highest RCD\mathcal{RCD} (44.91) and the slowest convergence, whereas FT (4.32) recovers rapidly; RL (30.82) delays recovery, while SCRUB (8.37) relearns substantially faster.
Figure 6: Ablation studies about Noisy component hyperparameter. An appropriate α\alpha makes the unlearning effective in both forgetting and preserving.
Figure 7: RCD\mathcal{RCD} vs. Retention Accuracy on ViT– Cifar100 across Settings. IEU increases resistance with minimal accuracy loss, offering a better privacy–utility balance than others.
Figure 11
Table 4: Performance summary of various unlearning methods for the ViT model trained on TinyImageNet in two unlearning scenarios, 30% random and 50% random data forgetting. Performance gap against Retraining is provided in ( ⋅\cdot ) .
Table 5: Performance summary of various unlearning methods for the ViT model trained on TinyImageNet in two unlearning scenarios, 30% class-wise and 50% class-wise data forgetting. Performance gap against Retraining is provided in ( ⋅\cdot ) .
Figure 10: Relationship between Avg. Gap and RCDGD\mathcal{RCD}_{GD} (step-size η=10−4\eta=10^{-4} ) of ViT model on the training-forgetting dataset Dftrain\mathcal{D}_{f}^{train} of TinyImageNet across diverse unlearning scenarios.
Figure 11: The RCDGD\mathcal{RCD}_{GD} values of ViT model on the training-forgetting set Dftrain\mathcal{D}_{f}^{train} of TinyImageNet for various step-sizes.
Table 6: Performance summary of various unlearning methods for the ResNet model trained on CIFAR-100 in two unlearning scenarios, 30% random and 50% random data forgetting. Performance gap against Retraining is provided in ( ⋅\cdot ) .
Table 7: Performance summary of various unlearning methods for the ResNet model trained on CIFAR-100 in two unlearning scenarios, 30% class-wise and 50% class-wise data forgetting. Performance gap against Retraining is provided in ( ⋅\cdot ) .
Figure 12: Relationship between Avg. Gap and RCDGD\mathcal{RCD}_{GD} (step-size η=10−4\eta=10^{-4} ) of ResNet model on the training-forgetting dataset Dftrain\mathcal{D}_{f}^{train} of CIFAR-100 across diverse unlearning scenarios.
Figure 13: The RCDGD\mathcal{RCD}_{GD} values of ResNet model on the training-forgetting set Dftrain\mathcal{D}_{f}^{train} of CIFAR-100 for various step-sizes.
Table 8: Performance summary of various unlearning methods for the ViT model trained on CIFAR-100 in two unlearning scenarios, 30% random and 50% random data forgetting. Performance gap against Retraining is provided in ( ⋅\cdot ) .
Table 9: Performance summary of various unlearning methods for the ViT model trained on CIFAR-100 in two unlearning scenarios, 30% class-wise and 50% class-wise data forgetting. Performance gap against Retraining is provided in ( ⋅\cdot ) .
Figure 14: Relationship between Avg. Gap and RCDGD\mathcal{RCD}_{GD} (step-size η=10−4\eta=10^{-4} ) of ViT model on the training-forgetting dataset Dftrain\mathcal{D}_{f}^{train} of CIFAR-100 across diverse unlearning scenarios.
Figure 15: The RCDGD\mathcal{RCD}_{GD} values of ViT model on the training-forgetting set Dftrain\mathcal{D}_{f}^{train} of CIFAR-100 for various step-sizes.
Table 10: Performance summary of various unlearning methods for the ResNet model trained on CIFAR-10 in two unlearning scenarios, 30% random and 50% random data forgetting. Performance gap against Retraining is provided in ( ⋅\cdot ) .
Table 11: Performance summary of various unlearning methods for the ResNet model trained on CIFAR-10 in two unlearning scenarios, 30% class-wise and 50% class-wise data forgetting. Performance gap against Retraining is provided in ( ⋅\cdot ) .
Figure 16: Relationship between Avg. Gap and RCDGD\mathcal{RCD}_{GD} (step-size η=10−4\eta=10^{-4} ) of ResNet model on the training-forgetting dataset Dftrain\mathcal{D}_{f}^{train} of CIFAR-10 across diverse unlearning scenarios.
Figure 17: The RCDGD\mathcal{RCD}_{GD} values of ResNet model on the training-forgetting set Dftrain\mathcal{D}_{f}^{train} of CIFAR-10 for various step-sizes.
Table 12: Performance summary of various unlearning methods for the ViT model trained on CIFAR-10 in two unlearning scenarios, 30% random and 50% random data forgetting. Performance gap against Retraining is provided in ( ⋅\cdot ) .
Table 13: Performance summary of various unlearning methods for the ViT model trained on CIFAR-10 in two unlearning scenarios, 30% class-wise and 50% class-wise data forgetting. Performance gap against Retraining is provided in ( ⋅\cdot ) .
Figure 18: Relationship between Avg. Gap and RCDGD\mathcal{RCD}_{GD} (step-size η=10−4\eta=10^{-4} ) of ViT model on the training-forgetting dataset Dftrain\mathcal{D}_{f}^{train} of CIFAR-10 across diverse unlearning scenarios.
Figure 19: The RCDGD\mathcal{RCD}_{GD} values of ViT model on the training-forgetting set Dftrain\mathcal{D}_{f}^{train} of CIFAR-10 for various step-sizes.
Table 14: Text prompts in I2P for SD to generate NSFW images.
Figure 20: Examples of generated images using various SD models from I2P prompts. Each column presents images generated by different SD variants using the same prompt, presented in Table 14 .
Figure 21: Image generation results for ImageNette classes using models unlearned from I2P harmful concepts.
Figure 22: We present images generated by the original SD and various relearned models. As the outputs from the relearned models are visually similar, we display them in a single row for illustration.
More in this topic

Machine Unlearning

Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.

See all 351 papers

Explore similar work

CardsList
Sep 26, 2025

OFMU: Optimization-Driven Framework for Machine Unlearning

Large language models deployed in sensitive applications increasingly require the ability to unlearn specific knowledge, such as user requests, copyrighted materials, or outdated information, without retraining from scratch to ensure regulatory compliance, user privacy, and safety. This task, known as machine unlearning, aims to remove the influence of targeted data (forgetting) while maintaining performance on the remaining data (retention). A common approach is to formulate this as a multi-objective problem and reduce it to a single-objective problem via scalarization, where forgetting and retention losses are combined using a weighted sum. However, this often results in unstable training dynamics and degraded model utility due to conflicting gradient directions. To address these challenges, we propose OFMU, a penalty-based bi-level optimization framework that explicitly prioritizes forgetting while preserving retention through a hierarchical structure. Our method enforces forgetting via an inner maximization step that incorporates a similarity-aware penalty to decorrelate the gradients of the forget and retention objectives, and restores utility through an outer minimization step. To ensure scalability, we develop a two-loop algorithm with provable convergence guarantees under both convex and non-convex regimes. We further provide a rigorous theoretical analysis of convergence rates and show that our approach achieves better trade-offs between forgetting efficacy and model utility compared to prior methods. Extensive experiments across vision and language benchmarks demonstrate that OFMU consistently outperforms existing unlearning methods in both forgetting efficacy and retained utility.

Sadia Asif, Mohammad Mohammadi Amiri

Rensselaer Polytechnic Institute

Bilevel OptimizationMachine UnlearningMulti-Objective OptimizationLLM Unlearning
AskPDF
Feb 25, 2026

Easy to Learn, Yet Hard to Forget: Towards Robust Unlearning Under Bias

Machine unlearning, which enables a model to forget specific data, is crucial for ensuring data privacy and model reliability. However, its effectiveness can be severely undermined in real-world scenarios where models learn unintended biases from spurious correlations within the data. This paper investigates the unique challenges of unlearning from such biased models. We identify a novel phenomenon we term shortcut unlearning," where models exhibit an easy to learn, yet hard to forget" tendency. Specifically, models struggle to forget easily-learned, bias-aligned samples; instead of forgetting the class attribute, they unlearn the bias attribute, which can paradoxically improve accuracy on the class intended to be forgotten. To address this, we propose CUPID, a new unlearning framework inspired by the observation that samples with different biases exhibit distinct loss landscape sharpness. Our method first partitions the forget set into causal- and bias-approximated subsets based on sample sharpness, then disentangles model parameters into causal and bias pathways, and finally performs a targeted update by routing refined causal and bias gradients to their respective pathways. Extensive experiments on biased datasets including Waterbirds, BAR, and Biased NICO++ demonstrate that our method achieves state-of-the-art forgetting performance and effectively mitigates the shortcut unlearning problem.

JuneHyoung Kwon, MiHyeon Kim, Eunju Lee +3

Department of Artificial Intelligence, Chung-Ang University · KT Corporation · Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University

Machine UnlearningShortcut LearningDebiased ML
AskPDF
May 11, 2026

Behavioral Guarantees for Proxy-Based Unlearning

This paper proposes a framework generalizing recent proxy-based unlearning methods and proves theoretical guarantees about the behavior of the resulting unlearned model: upper bounds on its Kullback-Leibler divergence to the ideal posterior distribution of the retain data. We model approximate unlearning as a constrained optimization problem and interpret a family of solutions as introducing a scaled unlearning signal in the output space. The unlearning signal arises from proxies of the posterior data distributions. Its scale is adapted to the proxies to ensure the behavioral upper bounds. This framework relies on the structure of the data distributions in order to create proxies. If need be, the target serves as a teacher to distill the update in the weights. Our approach is experimentally validated over two forgetting scenarios as reaching the closest classifier to the model retrained from scratch.

Virgile Dine, Teddy Furon

Centre Inria de l’Université de Rennes France

Machine Unlearning
AskPDF