Efficient Unlearning through Maximizing Relearning Convergence Delay

Khoa Tran; Simon S. Woo

cs.LGApr 10, 2026

Efficient Unlearning through Maximizing Relearning Convergence Delay

Authors: Khoa Tran, Simon S. Woo

Organizations: CSE Department, Sungkyunkwan University, South Korea · Secure Machines Lab

Abstract

Machine unlearning poses challenges in removing mislabeled, contaminated, or problematic data from a pretrained model. Current unlearning approaches and evaluation metrics are solely focused on model predictions, which limits insight into the model's true underlying data characteristics. To address this issue, we introduce a new metric called relearning convergence delay, which captures both changes in weight space and prediction space, providing a more comprehensive assessment of the model's understanding of the forgotten dataset. This metric can be used to assess the risk of forgotten data being recovered from the unlearned model. Based on this, we propose the Influence Eliminating Unlearning framework, which removes the influence of the forgetting set by degrading its performance and incorporates weight decay and injecting noise into the model's weights, while maintaining accuracy on the retaining set. Extensive experiments show that our method outperforms existing metrics and our proposed relearning convergence delay metric, approaching ideal unlearning performance. We provide theoretical guarantees, including exponential convergence and upper bounds, as well as empirical evidence of strong retention and resistance to relearning in both classification and generative unlearning tasks.

Figures & tables

Table 1: Performance summary of various unlearning methods for the ResNet model trained on TinyImageNet in two unlearning scenarios, 30% random and 50% random data forgetting. Performance gap against Retraining is provided in (

\cdot

) .

Table 2: Performance summary of various unlearning methods for the ResNet model trained on TinyImageNet in two unlearning scenarios, 30% class-wise and 50% class-wise data forgetting. Performance gap against Retraining is provided in (

\cdot

) .

Panel of Figure 1 — Figure 1: Relationship between Avg. Gap and $\mathcal{RCD}_{GD}$ (step-size $\eta=10^{-4}$ ) of ResNet model on the training-forgetting dataset $\mathcal{D}_{f}^{\text{train}}$ of TinyImageNet across diverse unlearning scenarios. Our methods consistently achieve a low Avg. Gap and a high $\mathcal{RCD}_{GD}$ score across four forgetting scenarios, demonstrating efficacy in both model utility and privacy.

Panel of Figure 2 — Figure 2: The $\mathcal{RCD}_{GD}$ values of ResNet model on the training-forgetting set $\mathcal{D}_{f}^{\text{train}}$ of TinyImageNet for various step-sizes. With a small step-size, the $\mathcal{RCD}_{GD}$ values of each unlearning method are less distinguishable, whereas an appropriate step-size enables a significant comparison.

Figure 3: Examples of generated images using various SD models from I2P prompts. The unlearning methods include ESD and SALUN, both with and without the use of Noisy . Each column presents images generated by different SD variants using the same prompt, denoted as $P_{i}$ . Detailed descriptions of the prompts are provided in the Appendix section.

Table 3: Performance of unlearned approaches on ImageNette concepts using FID, I2P prompts using Nudity score, and relearning convergence delay of relearning harmful concepts from I2P prompts, measured as

\mathcal{RCD}_{Adam}

Figure 4: Image generation results for ImageNette classes using models unlearned from I2P harmful concepts. The generated images show that using the Noisy component helps the unlearned model better preserve its performance on unrelated concepts.

Panel of Figure 5 — Figure 5: $\mathcal{RCD}$ vs. Relearning Difficulty correlation. Larger $\mathcal{RCD}$ values correspond to slower recovery when retraining on the forgotten data. Retraining yields the highest $\mathcal{RCD}$ (44.91) and the slowest convergence, whereas FT (4.32) recovers rapidly; RL (30.82) delays recovery, while SCRUB (8.37) relearns substantially faster.

Figure 6: Ablation studies about Noisy component hyperparameter. An appropriate

\alpha

makes the unlearning effective in both forgetting and preserving.

Figure 7:

\mathcal{RCD}

vs. Retention Accuracy on ViT– Cifar100 across Settings. IEU increases resistance with minimal accuracy loss, offering a better privacy–utility balance than others.

Table 4: Performance summary of various unlearning methods for the ViT model trained on TinyImageNet in two unlearning scenarios, 30% random and 50% random data forgetting. Performance gap against Retraining is provided in (

\cdot

) .

Table 5: Performance summary of various unlearning methods for the ViT model trained on TinyImageNet in two unlearning scenarios, 30% class-wise and 50% class-wise data forgetting. Performance gap against Retraining is provided in (

\cdot

) .

Panel of Figure 10 — Figure 10: Relationship between Avg. Gap and $\mathcal{RCD}_{GD}$ (step-size $\eta=10^{-4}$ ) of ViT model on the training-forgetting dataset $\mathcal{D}_{f}^{train}$ of TinyImageNet across diverse unlearning scenarios.

Table 6: Performance summary of various unlearning methods for the ResNet model trained on CIFAR-100 in two unlearning scenarios, 30% random and 50% random data forgetting. Performance gap against Retraining is provided in (

\cdot

) .

Table 7: Performance summary of various unlearning methods for the ResNet model trained on CIFAR-100 in two unlearning scenarios, 30% class-wise and 50% class-wise data forgetting. Performance gap against Retraining is provided in (

\cdot

) .

Panel of Figure 12 — Figure 12: Relationship between Avg. Gap and $\mathcal{RCD}_{GD}$ (step-size $\eta=10^{-4}$ ) of ResNet model on the training-forgetting dataset $\mathcal{D}_{f}^{train}$ of CIFAR-100 across diverse unlearning scenarios.

Panel of Figure 13 — Figure 13: The $\mathcal{RCD}_{GD}$ values of ResNet model on the training-forgetting set $\mathcal{D}_{f}^{train}$ of CIFAR-100 for various step-sizes.

Table 8: Performance summary of various unlearning methods for the ViT model trained on CIFAR-100 in two unlearning scenarios, 30% random and 50% random data forgetting. Performance gap against Retraining is provided in (

\cdot

) .

Table 9: Performance summary of various unlearning methods for the ViT model trained on CIFAR-100 in two unlearning scenarios, 30% class-wise and 50% class-wise data forgetting. Performance gap against Retraining is provided in (

\cdot

) .

Panel of Figure 14 — Figure 14: Relationship between Avg. Gap and $\mathcal{RCD}_{GD}$ (step-size $\eta=10^{-4}$ ) of ViT model on the training-forgetting dataset $\mathcal{D}_{f}^{train}$ of CIFAR-100 across diverse unlearning scenarios.

Panel of Figure 15 — Figure 15: The $\mathcal{RCD}_{GD}$ values of ViT model on the training-forgetting set $\mathcal{D}_{f}^{train}$ of CIFAR-100 for various step-sizes.

Table 10: Performance summary of various unlearning methods for the ResNet model trained on CIFAR-10 in two unlearning scenarios, 30% random and 50% random data forgetting. Performance gap against Retraining is provided in (

\cdot

) .

Table 11: Performance summary of various unlearning methods for the ResNet model trained on CIFAR-10 in two unlearning scenarios, 30% class-wise and 50% class-wise data forgetting. Performance gap against Retraining is provided in (

\cdot

) .

Panel of Figure 16 — Figure 16: Relationship between Avg. Gap and $\mathcal{RCD}_{GD}$ (step-size $\eta=10^{-4}$ ) of ResNet model on the training-forgetting dataset $\mathcal{D}_{f}^{train}$ of CIFAR-10 across diverse unlearning scenarios.

Panel of Figure 17 — Figure 17: The $\mathcal{RCD}_{GD}$ values of ResNet model on the training-forgetting set $\mathcal{D}_{f}^{train}$ of CIFAR-10 for various step-sizes.

Table 12: Performance summary of various unlearning methods for the ViT model trained on CIFAR-10 in two unlearning scenarios, 30% random and 50% random data forgetting. Performance gap against Retraining is provided in (

\cdot

) .

Table 13: Performance summary of various unlearning methods for the ViT model trained on CIFAR-10 in two unlearning scenarios, 30% class-wise and 50% class-wise data forgetting. Performance gap against Retraining is provided in (

\cdot

) .

Panel of Figure 18 — Figure 18: Relationship between Avg. Gap and $\mathcal{RCD}_{GD}$ (step-size $\eta=10^{-4}$ ) of ViT model on the training-forgetting dataset $\mathcal{D}_{f}^{train}$ of CIFAR-10 across diverse unlearning scenarios.

Panel of Figure 19 — Figure 19: The $\mathcal{RCD}_{GD}$ values of ViT model on the training-forgetting set $\mathcal{D}_{f}^{train}$ of CIFAR-10 for various step-sizes.

Table 14: Text prompts in I2P for SD to generate NSFW images.

Figure 20: Examples of generated images using various SD models from I2P prompts. Each column presents images generated by different SD variants using the same prompt, presented in Table 14 .

Figure 21: Image generation results for ImageNette classes using models unlearned from I2P harmful concepts.

Figure 22: We present images generated by the original SD and various relearned models. As the outputs from the relearned models are visually similar, we display them in a single row for illustration.

Machine Unlearning

Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.

See all 351 papers

OFMU: Optimization-Driven Framework for Machine Unlearning
Sep 26, 2025Sadia Asif, Mohammad Mohammadi AmiriBilevel Optimization Machine Unlearning
Rensselaer Polytechnic Institute
Easy to Learn, Yet Hard to Forget: Towards Robust Unlearning Under Bias
Feb 25, 2026JuneHyoung Kwon, MiHyeon Kim, Eunju Lee +3Machine Unlearning Shortcut Learning
Department of Artificial Intelligence, Chung-Ang University · KT Corporation · Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University
Behavioral Guarantees for Proxy-Based Unlearning
May 11, 2026Virgile Dine, Teddy FuronMachine Unlearning
Centre Inria de l’Université de Rennes France

Sep 26, 2025J/K move · Enter open · S save

OFMU: Optimization-Driven Framework for Machine Unlearning

Sadia Asif, Mohammad Mohammadi Amiri

Rensselaer Polytechnic Institute

Bilevel Optimization Machine Unlearning Multi-Objective Optimization LLM Unlearning

Open paper AskPDF arXiv

Large language models deployed in sensitive applications increasingly require the ability to unlearn specific knowledge, such as user requests, copyrighted materials, or outdated information, without retraining from scratch to ensure regulatory compliance, user privacy, and safety. This task, known as machine unlearning, aims to remove the influence of targeted data (forgetting) while maintaining performance on the remaining data (retention). A common approach is to formulate this as a multi-objective problem and reduce it to a single-objective problem via scalarization, where forgetting and retention losses are combined using a weighted sum. However, this often results in unstable training dynamics and degraded model utility due to conflicting gradient directions. To address these challenges, we propose OFMU, a penalty-based bi-level optimization framework that explicitly prioritizes forgetting while preserving retention through a hierarchical structure. Our method enforces forgetting via an inner maximization step that incorporates a similarity-aware penalty to decorrelate the gradients of the forget and retention objectives, and restores utility through an outer minimization step. To ensure scalability, we develop a two-loop algorithm with provable convergence guarantees under both convex and non-convex regimes. We further provide a rigorous theoretical analysis of convergence rates and show that our approach achieves better trade-offs between forgetting efficacy and model utility compared to prior methods. Extensive experiments across vision and language benchmarks demonstrate that OFMU consistently outperforms existing unlearning methods in both forgetting efficacy and retained utility.

Abstract

Machine Unlearning

Explore similar work

OFMU: Optimization-Driven Framework for Machine Unlearning

Easy to Learn, Yet Hard to Forget: Towards Robust Unlearning Under Bias

Behavioral Guarantees for Proxy-Based Unlearning