cs.LGApr 12, 2026

Mitigating Privacy Risk via Forget Set-Free Unlearning

Authors: Aviraj Newatia, Michael Cooper, Viet Nguyen, Rahul G. Krishnan

Organizations: University of Toronto, Vector Institute, Toronto, ON, Canada

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Training machine learning models requires the storage of large datasets, which often contain sensitive or private data. Storing data is associated with a number of potential risks which increase over time, such as database breaches and malicious adversaries. Machine unlearning is the study of methods to efficiently remove the influence of training data subsets from previously-trained models. Existing unlearning methods typically require direct access to the "forget set" -- the data to be forgotten-and organisations must retain this data for unlearning rather than deleting it immediately upon request, increasing risks associated with the forget set. We introduce partially-blind unlearning -- utilizing auxiliary information to unlearn without explicit access to the forget set. We also propose a practical framework Reload, a partially-blind method based on gradient optimization and structured weight sparsification to operationalize partially-blind unlearning. We show that Reload efficiently unlearns, approximating models retrained from scratch, and outperforms several forget set-dependent approaches. On language models, Reload unlearns entities using <0.025% of the retain set and <7% of model weights in <8 minutes on Llama2-7B. In the corrective case, Reload achieves unlearning even when only 10% of corrupted data is identified.

Figures & tables

Figure 1: Conventional unlearning algorithms admit a cumulative user risk totalling the sum of the green , blue , and red regions. By allowing user data to be deleted immediately once a request for deletion is made, Reload eliminates the user risk associated with the red region.
Figure 2: Overview of the Reload algorithm for partially-blind approximate unlearning. Reload marries a gradient-based unlearning step modified for the PBU setting (Steps (1) through (3) ) with a weight saliency-based selective reinitialisation (Step (4) ) and subsequent fine-tuning (Step (5) ). Because the partially-blind unlearning setting prohibits taking gradients with respect to Dforget\mathcal{D}_{forget} , Reload exploits the linearity of differentiation to treat ∇θ(L(D)−L(Dretain))\nabla_{\theta}(\mathcal{L}(\mathcal{D})-\mathcal{L}(\mathcal{D}_{retain})) as a proxy for ∇θL(Dforget)\nabla_{\theta}\mathcal{L}(\mathcal{D}_{forget}) at the location in weight space corresponding to θt\theta_{t} . This allows us to apply one gradient ascent step in this direction. Intuitively, this update in Step (3) removes some information about Dforget\mathcal{D}_{forget} from all network weights, while the reinitialisation in Step (4) reinitialises those weights with a uniquely strong correspondence to Dforget\mathcal{D}_{forget} (for which a single ascent step will not fully remove this information). Reload achieves state-of-the-art performance on a collection of unlearning tasks, often outperforming baselines with direct access to Dforget\mathcal{D}_{forget} .
Figure 3: Introspecting on selected feature maps of a ResNet-18 model when using Reload to unlearn the class “8”. (Left) The feature maps (activations) of the first channel fo the first, third, and fifth convolutional layers after the application of each algorithm step in Figure 2 . After Step (3) , the activations of the model remain largely unchanged, although the logits represent a considerably more uniform distribution over the digits. After Step (4) , the activation of the first convolutional layer is largely unchanged—which is expected, as the earlier layers of CNNs tend to correspond to broad feature detectors ( Zeiler and Fergus, 2014 ) )—although the feature maps change and the predictive distribution contentrates around “2”. (Right) t-SNE visualisations of the 5 th Conv. Layer of a ResNet-18 trained on the same task in a different training run. At Step (1) , classes largely cluster into nice separated regions with some overlap. The Ascent Step (3) disrupts the cluster separation, preserving some clustering structure while breaking apart much of the original class boundaries. Reinitialisation (Step (4) ) reveals that the model collapses onto one label at this stage (on the left ’2’, on the right, “8”—though in our experience it is coincidental that this run happens to concentrate on the class we desire to unlearn). Finally, Finetuning (Step (5) ) recovers much of the original separability, restoring the distinct class clusters seen in the initial t-SNE. Importantly, we observe that no samples are predicted to be “8”.
 10% Random100 In-Class Entity Unlearning 1%
AlgorithmΔ\Delta FA ( ↓\downarrow )Δ\Delta FMIA ( ↓\downarrow )Δ\Delta FA ( ↓\downarrow )Δ\Delta FMIA ( ↓\downarrow )AlgorithmFQ ( ↑\uparrow )MU ( ↑\uparrow )
GA18.77±2.4318.77\pm 2.430.21±0.060.21\pm 0.0623.33±1.0623.33\pm 1.060.07±0.060.07\pm 0.06GA0.00680.0068−0.0233-0.0233
SSD74.17±2.0474.17\pm 2.040.15±0.210.15\pm 0.2168.67±1.9768.67\pm 1.970.38±0.140.38\pm 0.14Grad Diff0.01430.0143−0.0198-0.0198
SCRUB18.85±2.3918.85\pm 2.390.20±0.060.20\pm 0.0627.55±1.4327.55\pm 1.430.07±0.060.07\pm 0.06NPO-RT0.57860.5786−0.1361-0.1361
CF- kk18.01±2.6018.01\pm 2.600.20±0.060.20\pm 0.0621.84±0.8821.84\pm 0.880.07±0.060.07\pm 0.06Pref Opt0.09710.0971−0.0021-0.0021
SalUn13.14±2.5313.14\pm 2.537.39±2.607.39\pm 2.6012.08±3.1312.08\pm 3.130.02±0.02\bm{0.02\pm 0.02}ECO (Zero-Out)0.9900\bm{0.9900}+0.0000+0.0000
Table 1: Benchmarking Reload against baselines in the uncorrelated 10% setting, 100 in-class samples setting, and 1% forgetting entity unlearning setting. Best performances are bolded .
Figure 4: CU results on CIFAR-10 for dataset poisoning with ∣Dm∣=100|\mathcal{D}_{m}|=100 . Reload achieves high Acccorr\text{Acc}_{\text{corr}} across all γ∈(0,1.0]\gamma\in(0,1.0] whereas baseline methods often struggle with low γ\gamma .
Appendix figures & tables51 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: (Smoothed) empirical distribution of knowledge values computed over the weights of ResNet-18 trained on SVHN.
StatisticAbbr.Description
Accuracy on Dretain\mathcal{D}_{retain} (↑)(\uparrow)RAModel accuracy on the Dretain\mathcal{D}_{retain} . In unlearning, a higher accuracy indicates that the unlearning process has not negatively impacted the model’s performance on the retained data.
Diff. in Accuracy on Dforget\mathcal{D}_{forget} ( ↓\downarrow )Δ\Delta FAThe change in accuracy on the forget set between the current model and Mθ∼\mathcal{M}^{\theta^{\sim}} . A smaller difference, approaching the accuracy of the retrained model, indicates that the unlearning method has been more effective in "forgetting" the forget set.
Diff. in Error on Dforget\mathcal{D}_{forget} ( ↓\downarrow )Δ\Delta FEThe reduction in error on the forget set between the current model and Mθ∼\mathcal{M}^{\theta^{\sim}} . A smaller difference, approaching the error of the retrained model, signifies that the unlearning method has been more effective at "forgetting" the forget set.
Diff. in MIA Success Rate on Dforget\mathcal{D}_{forget} ( ↓\downarrow )Δ\Delta FMIADifference in success rate of a membership inference attack (MIA) on the forget set between the current model and Mθ∼\mathcal{M}^{\theta^{\sim}} . In this work, we use the attack from ( Shokri et al., 2017 ) implemented in the repository for ( Kurmanji et al., 2023 ) . A success rate approaching that of the retrained model implies the forgotten data is indistinguishable to an MIA on in-distribution data that the model was not trained on.
Symmetric KL-Divergence on Dretain\mathcal{D}_{retain} ( ↓\downarrow )RSKLSymmetric KL-Divergence between the logits of the current model and those of Mθ∼\mathcal{M}^{\theta^{\sim}} . This metric is averaged over all instances in the Dretain\mathcal{D}_{retain} . A lower symmetric KL divergence indicates an unlearning method that behaves similarly on the Dretain\mathcal{D}_{retain} to a model retrained from scratch without the forget set.
Symmetric KL-Divergence on Dforget\mathcal{D}_{forget} ( ↓\downarrow )FSKLThe Symmetric KL-Divergence between the logits of the current model and those of Mθ∼\mathcal{M}^{\theta^{\sim}} . This metric is averaged over all instances in the Dforget\mathcal{D}_{forget} . A lower symmetric KL divergence indicates that the unlearning method that behaves similarly on the Dforget\mathcal{D}_{forget} to a model retrained from scratch without the forget set.
Appendix
Table 2: Evaluation Statistics for Unlearning.
StatisticAbbr.Description
Retain Accuracy (↑)(\uparrow)Acc retain{}_{\text{retain}}Model accuracy on a held out test set ( Dretain(test)\mathcal{D}_{retain}^{(test)} ) of the same distribution as Dretain\mathcal{D}_{retain} . In corrective unlearning, a higher accuracy indicates that the corrective unlearning process has correctly adapted the model to its new training set.
Corrected Accuracy (↑)(\uparrow)Acc corr{}_{\text{corr}}Model accuracy on the adversely affected data Dm\mathcal{D}_{m} . In the case of backdoor attacks or noisy corrective unlearning, a higher value indicates the relearned model correctly has lost its reliance on the backdoor pattern. In label correction setting, the desirable value is the percentage of samples that did not have their labels flipped (in our experiments, 90%90\% ).
Cost ( ↓\downarrow )CostRatio of the runtime of the corrective unlearning method to the runtime of retraining a baseline model from scratch without the forget set. A lower cost indicates a more computationally efficient method.
Appendix
Table 3: Evaluation Statistics for Corrective Unlearning.
MethodRA (↑)(\uparrow)Δ\Delta FA (↓)(\downarrow)Δ\Delta FE (↓)(\downarrow)Δ\Delta FMIA (↓)(\downarrow)Cost (↓)(\downarrow)RSKL (↓)(\downarrow)FSKL (↓)(\downarrow)
GA98.38±0.2198.38_{\pm 0.21}3.86±0.663.86_{\pm 0.66}0.21±0.070.21_{\pm 0.07}0.04±0.020.04_{\pm 0.02}0.00±0.00\bm{0.00_{\pm 0.00}}0.06±0.020.06_{\pm 0.02}0.66±0.060.66_{\pm 0.06}
FT98.24±0.2198.24_{\pm 0.21}1.45±0.53\bm{1.45_{\pm 0.53}}0.16±0.030.16_{\pm 0.03}0.03±0.010.03_{\pm 0.01}0.27±0.000.27_{\pm 0.00}0.05±0.01\bm{0.05_{\pm 0.01}}0.48±0.04\bm{0.48_{\pm 0.04}}
SSD20.02±29.9920.02_{\pm 29.99}75.65±26.4575.65_{\pm 26.45}1.88±0.621.88_{\pm 0.62}0.01±0.020.01_{\pm 0.02}0.01±0.000.01_{\pm 0.00}8.30±3.118.30_{\pm 3.11}7.83±2.707.83_{\pm 2.70}
SCRUB98.41±0.2098.41_{\pm 0.20}3.89±0.703.89_{\pm 0.70}0.21±0.070.21_{\pm 0.07}0.04±0.020.04_{\pm 0.02}0.02±0.000.02_{\pm 0.00}0.06±0.020.06_{\pm 0.02}0.65±0.040.65_{\pm 0.04}
CF- kk98.28±0.2398.28_{\pm 0.23}3.81±0.713.81_{\pm 0.71}0.21±0.070.21_{\pm 0.07}0.05±0.020.05_{\pm 0.02}0.21±0.000.21_{\pm 0.00}0.06±0.020.06_{\pm 0.02}0.55±0.040.55_{\pm 0.04}
EU- kk98.31±0.2198.31_{\pm 0.21}3.83±0.713.83_{\pm 0.71}0.21±0.070.21_{\pm 0.07}0.05±0.010.05_{\pm 0.01}0.21±0.000.21_{\pm 0.00}0.07±0.020.07_{\pm 0.02}0.56±0.040.56_{\pm 0.04}
Appendix
Table 4: 10% Random Forgetting on CIFAR-10 (ResNet-18) ↑\uparrow : the goal is to have as high of a value as possible, Δ↓\Delta^{\downarrow} : the value in the table is the difference between the result of the unlearning method and retraining (bottom row) on the metric and the goal is to have a low difference, ↓\downarrow : the goal is to have as low of a value as possible. The bottom row presents the absolute value of M(θ∼)M_{(\theta^{\sim})} on each metric. For any metric with Δ\Delta , the raw value is instead reported. Rows for Δ\Delta FA (↓)(\downarrow) , Δ\Delta FE (↓)(\downarrow) , and Δ\Delta FMIA (↓)(\downarrow) present the absolute difference in the value of the corresponding method on this metric to the value of M(θ∼)M_{(\theta^{\sim})} on the metric. These results show that Reload outperforms all the baselines on RA, Δ\Delta FE, Δ\Delta FMIA by large margins. Reload performs competitively on the Δ\Delta FA, FSKL, and RSKL metrics, but is outperformed by FT. Reload incurs a higher computational cost than other baselines other than FT.
MethodRA (↑)(\uparrow)Δ\Delta FA (↓)(\downarrow)Δ\Delta FE (↓)(\downarrow)Δ\Delta FMIA (↓)(\downarrow)Cost (↓)(\downarrow)RSKL (↓)(\downarrow)FSKL (↓)(\downarrow)
GA98.38±0.2198.38_{\pm 0.21}4.40±0.414.40_{\pm 0.41}0.18±0.020.18_{\pm 0.02}0.05±0.010.05_{\pm 0.01}0.00±0.00\bm{0.00_{\pm 0.00}}0.06±0.020.06_{\pm 0.02}0.66±0.060.66_{\pm 0.06}
FT98.24±0.2198.24_{\pm 0.21}4.33±0.374.33_{\pm 0.37}0.18±0.020.18_{\pm 0.02}0.04±0.010.04_{\pm 0.01}0.26±0.020.26_{\pm 0.02}0.05±0.01\bm{0.05_{\pm 0.01}}0.48±0.040.48_{\pm 0.04}
SSD20.02±29.9920.02_{\pm 29.99}75.41±26.7475.41_{\pm 26.74}1.89±0.621.89_{\pm 0.62}0.02±0.030.02_{\pm 0.03}0.01±0.000.01_{\pm 0.00}8.30±3.118.30_{\pm 3.11}7.83±2.707.83_{\pm 2.70}
SCRUB98.41±0.2098.41_{\pm 0.20}4.47±0.404.47_{\pm 0.40}0.19±0.020.19_{\pm 0.02}0.05±0.010.05_{\pm 0.01}0.02±0.000.02_{\pm 0.00}0.06±0.020.06_{\pm 0.02}0.65±0.040.65_{\pm 0.04}
CF- kk98.28±0.2398.28_{\pm 0.23}4.47±0.394.47_{\pm 0.39}0.19±0.020.19_{\pm 0.02}0.05±0.010.05_{\pm 0.01}0.17±0.010.17_{\pm 0.01}0.06±0.020.06_{\pm 0.02}0.55±0.040.55_{\pm 0.04}
EU- kk98.31±0.2198.31_{\pm 0.21}4.48±0.404.48_{\pm 0.40}0.19±0.020.19_{\pm 0.02}0.06±0.010.06_{\pm 0.01}0.17±0.010.17_{\pm 0.01}0.07±0.020.07_{\pm 0.02}0.56±0.040.56_{\pm 0.04}
Appendix
Table 5: 10% Random Forgetting on SVHN (ResNet-18) ↑\uparrow : the goal is to have as high of a value as possible, Δ↓\Delta^{\downarrow} : the value in the table is the difference between the result of the unlearning method and retraining (bottom row) on the metric and the goal is to have a low difference, ↓\downarrow : the goal is to have as low of a value as possible. The bottom row presents the absolute value of M(θ∼)M_{(\theta^{\sim})} on each metric. For any metric with Δ\Delta , the raw value is instead reported. Rows for Δ\Delta FA (↓)(\downarrow) , Δ\Delta FE (↓)(\downarrow) , and Δ\Delta FMIA (↓)(\downarrow) present the absolute difference in the value of the corresponding method on this metric to the value of M(θ∼)M_{(\theta^{\sim})} on the metric. These results show that Reload outperforms all the baselines on RA, Δ\Delta FA, Δ\Delta FE, Δ\Delta FMIA, FSKL, and RSKL by large margins. Reload performs competitively on the Cost, but incurs a higher computational cost than other baselines other than FT, CF- kk , EU- kk .
MethodRA (↑)(\uparrow)Δ\Delta FA (↓)(\downarrow)Δ\Delta FE (↓)(\downarrow)Δ\Delta FMIA (↓)(\downarrow)Cost (↓)(\downarrow)RSKL (↓)(\downarrow)FSKL (↓)(\downarrow)
GA98.40±0.2398.40_{\pm 0.23}4.43±0.444.43_{\pm 0.44}0.21±0.020.21_{\pm 0.02}0.03±0.010.03_{\pm 0.01}0.00±0.00\bm{0.00_{\pm 0.00}}0.06±0.020.06_{\pm 0.02}0.65±0.060.65_{\pm 0.06}
FT98.30±0.1898.30_{\pm 0.18}4.49±0.434.49_{\pm 0.43}0.22±0.020.22_{\pm 0.02}0.03±0.010.03_{\pm 0.01}0.24±0.030.24_{\pm 0.03}0.05±0.01\bm{0.05_{\pm 0.01}}0.49±0.04\bm{0.49_{\pm 0.04}}
SSD22.88±34.0122.88_{\pm 34.01}70.45±29.0470.45_{\pm 29.04}1.80±0.691.80_{\pm 0.69}0.01±0.010.01_{\pm 0.01}0.00±0.00\bm{0.00_{\pm 0.00}}7.99±3.527.99_{\pm 3.52}7.56±3.067.56_{\pm 3.06}
SCRUB98.43±0.2298.43_{\pm 0.22}4.50±0.414.50_{\pm 0.41}0.22±0.020.22_{\pm 0.02}0.03±0.010.03_{\pm 0.01}0.02±0.000.02_{\pm 0.00}0.06±0.020.06_{\pm 0.02}0.66±0.040.66_{\pm 0.04}
CF- kk98.34±0.2498.34_{\pm 0.24}4.51±0.424.51_{\pm 0.42}0.22±0.020.22_{\pm 0.02}0.04±0.010.04_{\pm 0.01}0.21±0.030.21_{\pm 0.03}0.06±0.020.06_{\pm 0.02}0.55±0.050.55_{\pm 0.05}
EU- kk98.34±0.2398.34_{\pm 0.23}4.51±0.424.51_{\pm 0.42}0.22±0.020.22_{\pm 0.02}0.04±0.010.04_{\pm 0.01}0.21±0.030.21_{\pm 0.03}0.06±0.020.06_{\pm 0.02}0.56±0.050.56_{\pm 0.05}
Appendix
Table 6: 10% Random Forgetting on SVHN (VGG16-BN) ↑\uparrow : the goal is to have as high of a value as possible, Δ↓\Delta^{\downarrow} : the value in the table is the difference between the result of the unlearning method and retraining (bottom row) on the metric and the goal is to have a low difference, ↓\downarrow : the goal is to have as low of a value as possible. The bottom row presents the absolute value of M(θ∼)M_{(\theta^{\sim})} on each metric. For any metric with Δ\Delta , the raw value is instead reported. Rows for Δ\Delta FA (↓)(\downarrow) , Δ\Delta FE (↓)(\downarrow) , and Δ\Delta FMIA (↓)(\downarrow) present the absolute difference in the value of the corresponding method on this metric to the value of M(θ∼)M_{(\theta^{\sim})} on the metric. These results show that Reload outperforms all the baselines on RA, Δ\Delta FA, Δ\Delta FE, and Δ\Delta FMIA, by large margins aside from Fisher. However, Fisher incurs a substantially higher Cost, making it far less efficient than retraining from scratch. Therefore, Fisher is impractical, and Reload demonstrates the best practicality as an unlearning mechanism. Reload performs competitively on RSKL and FSKL but is outperformed by FT. Reload also incurs a higher computational cost than the other baselines.
MethodRA (↑)(\uparrow)Δ\Delta FA (↓)(\downarrow)Δ\Delta FE (↓)(\downarrow)Δ\Delta FMIA (↓)(\downarrow)Cost (↓)(\downarrow)RSKL (↓)(\downarrow)FSKL (↓)(\downarrow)
GA98.41±0.2598.41_{\pm 0.25}26.40±1.1826.40_{\pm 1.18}1.64±0.071.64_{\pm 0.07}0.14±0.030.14_{\pm 0.03}0.00±0.00\bm{0.00_{\pm 0.00}}0.06±0.03\bm{0.06_{\pm 0.03}}0.66±0.060.66_{\pm 0.06}
FT98.27±0.2098.27_{\pm 0.20}12.65±1.8112.65_{\pm 1.81}1.16±0.071.16_{\pm 0.07}0.08±0.020.08_{\pm 0.02}0.25±0.030.25_{\pm 0.03}0.06±0.01\bm{0.06_{\pm 0.01}}0.50±0.03\bm{0.50_{\pm 0.03}}
SSD22.86±34.0122.86_{\pm 34.01}61.38±15.7261.38_{\pm 15.72}2.57±0.552.57_{\pm 0.55}0.02±0.05\bm{0.02_{\pm 0.05}}0.00±0.00\bm{0.00_{\pm 0.00}}8.01±3.538.01_{\pm 3.53}7.57±3.077.57_{\pm 3.07}
SCRUB98.43±0.2398.43_{\pm 0.23}26.62±1.1026.62_{\pm 1.10}1.66±0.061.66_{\pm 0.06}0.14±0.030.14_{\pm 0.03}0.02±0.000.02_{\pm 0.00}0.06±0.02\bm{0.06_{\pm 0.02}}0.66±0.040.66_{\pm 0.04}
CF- kk98.30±0.2798.30_{\pm 0.27}26.26±1.2526.26_{\pm 1.25}1.68±0.061.68_{\pm 0.06}0.15±0.020.15_{\pm 0.02}0.27±0.040.27_{\pm 0.04}0.06±0.02\bm{0.06_{\pm 0.02}}0.56±0.040.56_{\pm 0.04}
EU- kk98.35±0.2598.35_{\pm 0.25}26.16±1.2626.16_{\pm 1.26}1.67±0.061.67_{\pm 0.06}0.15±0.020.15_{\pm 0.02}0.27±0.040.27_{\pm 0.04}0.06±0.02\bm{0.06_{\pm 0.02}}0.57±0.040.57_{\pm 0.04}
Appendix
Table 7: 10% Random Forgetting on CIFAR-100(VGG16-BN) ↑\uparrow : the goal is to have as high of a value as possible, Δ↓\Delta^{\downarrow} : the value in the table is the difference between the result of the unlearning method and retraining (bottom row) on the metric and the goal is to have a low difference, ↓\downarrow : the goal is to have as low of a value as possible. The bottom row presents the absolute value of M(θ∼)M_{(\theta^{\sim})} on each metric. For any metric with Δ\Delta , the raw value is instead reported. Rows for Δ\Delta FA (↓)(\downarrow) , Δ\Delta FE (↓)(\downarrow) , and Δ\Delta FMIA (↓)(\downarrow) present the absolute difference in the value of the corresponding method on this metric to the value of M(θ∼)M_{(\theta^{\sim})} on the metric. These results show that Reload outperforms all the baselines on RA, Δ\Delta FA, Δ\Delta FE, and Δ\Delta FMIA, by large margins. Reload performs competitively on RSKL and FSKL but is outperformed by FT. Reload also incurs a higher computational cost than other baselines other than FT, CF- kk , and EU- kk .
MethodRA (↑)(\uparrow)Δ\Delta FA (↓)(\downarrow)Δ\Delta FE (↓)(\downarrow)Δ\Delta FMIA (↓)(\downarrow)Cost (↓)(\downarrow)RSKL (↓)(\downarrow)FSKL (↓)(\downarrow)
GA93.81±0.7593.81_{\pm 0.75}18.77±2.4318.77_{\pm 2.43}0.95±0.140.95_{\pm 0.14}0.21±0.060.21_{\pm 0.06}0.00±0.000.00_{\pm 0.00}0.29±0.090.29_{\pm 0.09}2.62±0.052.62_{\pm 0.05}
FT96.00±0.1296.00_{\pm 0.12}16.46±2.4716.46_{\pm 2.47}0.89±0.140.89_{\pm 0.14}0.19±0.080.19_{\pm 0.08}0.27±0.000.27_{\pm 0.00}0.03±0.010.03_{\pm 0.01}2.11±0.062.11_{\pm 0.06}
SSD1.01±0.021.01_{\pm 0.02}74.17±2.0474.17_{\pm 2.04}4.19±0.594.19_{\pm 0.59}0.15±0.210.15_{\pm 0.21}0.01±0.000.01_{\pm 0.00}14.90±1.2414.90_{\pm 1.24}11.81±1.2411.81_{\pm 1.24}
SCRUB93.76±0.7493.76_{\pm 0.74}18.85±2.3918.85_{\pm 2.39}0.95±0.140.95_{\pm 0.14}0.20±0.060.20_{\pm 0.06}0.02±0.000.02_{\pm 0.00}0.29±0.090.29_{\pm 0.09}2.63±0.062.63_{\pm 0.06}
CF- kk94.75±0.4194.75_{\pm 0.41}18.01±2.6018.01_{\pm 2.60}0.94±0.140.94_{\pm 0.14}0.20±0.060.20_{\pm 0.06}0.21±0.000.21_{\pm 0.00}0.14±0.030.14_{\pm 0.03}2.47±0.072.47_{\pm 0.07}
EU- kk94.32±0.4994.32_{\pm 0.49}17.93±2.5517.93_{\pm 2.55}0.94±0.140.94_{\pm 0.14}0.20±0.060.20_{\pm 0.06}0.21±0.000.21_{\pm 0.00}0.19±0.050.19_{\pm 0.05}2.33±0.052.33_{\pm 0.05}
Appendix
Table 8: 10% Random Forgetting on CIFAR-100 (ResNet-18). The bottom row presents the absolute value of M(θ∼)M_{(\theta^{\sim})} on each metric. For any metric with Δ\Delta , the raw value is instead reported. Rows for Δ\Delta FA (↓)(\downarrow) , Δ\Delta FE (↓)(\downarrow) , and Δ\Delta FMIA (↓)(\downarrow) present the absolute difference in the value of the corresponding method on this metric to the value of M(θ∼)M_{(\theta^{\sim})} on the metric. These results show that Reload outperforms all the baselines on RA, Δ\Delta FA, Δ\Delta FE, Δ\Delta FMIA, and FSKL by large margins. Reload performs competitively on the RSKL metric, outperformed by FT and CF- kk . Reload incurs a higher computational cost than most baselines, but is cheaper than FT, CF- kk , and EU- kk .
MethodRA (↑)(\uparrow)Δ\Delta FA (↓)(\downarrow)Δ\Delta FE (↓)(\downarrow)Δ\Delta FMIA (↓)(\downarrow)Δ\Delta AUC (↓)(\downarrow)Cost (↓)(\downarrow)RSKL (↓)(\downarrow)FSKL (↓)(\downarrow)
GA17.20±30.1717.20_{\pm 30.17}77.46±26.2577.46_{\pm 26.25}8.86±6.488.86_{\pm 6.48}0.02±0.020.02_{\pm 0.02}0.01±0.020.01_{\pm 0.02}0.01±0.00\bm{0.01_{\pm 0.00}}0.06±0.020.06_{\pm 0.02}0.66±0.060.66_{\pm 0.06}
FT99.69±0.24\bm{99.69_{\pm 0.24}}3.92±0.533.92_{\pm 0.53}0.19±0.020.19_{\pm 0.02}0.02±0.010.02_{\pm 0.01}0.02±0.000.02_{\pm 0.00}0.28±0.010.28_{\pm 0.01}0.05±0.01\bm{0.05_{\pm 0.01}}0.48±0.04\bm{0.48_{\pm 0.04}}
SSD19.85±29.6519.85_{\pm 29.65}74.50±25.9074.50_{\pm 25.90}1.82±0.581.82_{\pm 0.58}0.01±0.020.01_{\pm 0.02}0.01±0.020.01_{\pm 0.02}0.01±0.00\bm{0.01_{\pm 0.00}}8.30±3.118.30_{\pm 3.11}7.83±2.707.83_{\pm 2.70}
SCRUB82.59±1.3982.59_{\pm 1.39}12.72±1.5112.72_{\pm 1.51}0.31±0.040.31_{\pm 0.04}0.00±0.00\bm{0.00_{\pm 0.00}}0.00±0.00\bm{0.00_{\pm 0.00}}0.07±0.000.07_{\pm 0.00}0.06±0.020.06_{\pm 0.02}0.65±0.040.65_{\pm 0.04}
CF-k99.58±0.1199.58_{\pm 0.11}6.28±0.196.28_{\pm 0.19}0.27±0.010.27_{\pm 0.01}0.05±0.000.05_{\pm 0.00}0.05±0.000.05_{\pm 0.00}0.11±0.000.11_{\pm 0.00}0.06±0.020.06_{\pm 0.02}0.55±0.040.55_{\pm 0.04}
EU-k99.59±0.1599.59_{\pm 0.15}6.28±0.226.28_{\pm 0.22}0.27±0.010.27_{\pm 0.01}0.05±0.010.05_{\pm 0.01}0.05±0.010.05_{\pm 0.01}0.22±0.010.22_{\pm 0.01}0.07±0.020.07_{\pm 0.02}0.56±0.040.56_{\pm 0.04}
Appendix
Table 9: 30% Random Forgetting on CIFAR-10(ResNet-18) ↑\uparrow : the goal is to have as high of a value as possible, Δ↓\Delta^{\downarrow} : the value in the table is the difference between the result of the unlearning method and retraining (bottom row) on the metric and the goal is to have a low difference, ↓\downarrow : the goal is to have as low of a value as possible. The bottom row presents the absolute value of M(θ∼)M_{(\theta^{\sim})} on each metric. For any metric with Δ\Delta , the raw value is instead reported. Rows for Δ\Delta FA (↓)(\downarrow) , Δ\Delta FE (↓)(\downarrow) , and Δ\Delta FMIA (↓)(\downarrow) present the absolute difference in the value of the corresponding method on this metric to the value of M(θ∼)M_{(\theta^{\sim})} on the metric. These results show that Reload outperforms all the baselines on RA, Δ\Delta FA, Δ\Delta FE, and Δ\Delta FMIA, by large margins. Reload performs competitively on RSKL and FSKL but is outperformed by FT. Reload also incurs a higher computational cost than other baselines other than FT, CF- kk , and EU- kk .
MethodRA (↑)(\uparrow)Δ\Delta FA (↓)(\downarrow)Δ\Delta FE (↓)(\downarrow)Δ\Delta FMIA (↓)(\downarrow)Δ\Delta AUC (↓)(\downarrow)Cost (↓)(\downarrow)RSKL (↓)(\downarrow)FSKL (↓)(\downarrow)
GA18.97±28.4418.97_{\pm 28.44}73.93±23.0773.93_{\pm 23.07}0.43±0.010.43_{\pm 0.01}0.05±0.030.05_{\pm 0.03}0.01±0.010.01_{\pm 0.01}0.01±0.000.01_{\pm 0.00}0.06±0.020.06_{\pm 0.02}0.66±0.060.66_{\pm 0.06}
FT99.37±0.2199.37_{\pm 0.21}4.41±0.534.41_{\pm 0.53}0.27±0.020.27_{\pm 0.02}0.02±0.010.02_{\pm 0.01}0.02±0.000.02_{\pm 0.00}0.27±0.010.27_{\pm 0.01}0.05±0.010.05_{\pm 0.01}0.48±0.040.48_{\pm 0.04}
SSD22.73±29.2722.73_{\pm 29.27}70.55±23.7370.55_{\pm 23.73}1.67±0.601.67_{\pm 0.60}0.01±0.020.01_{\pm 0.02}0.01±0.020.01_{\pm 0.02}0.01±0.000.01_{\pm 0.00}8.30±3.118.30_{\pm 3.11}7.83±2.707.83_{\pm 2.70}
SCRUB14.29±5.0214.29_{\pm 5.02}77.10±5.0877.10_{\pm 5.08}2.02±0.572.02_{\pm 0.57}0.01±0.010.01_{\pm 0.01}0.01±0.000.01_{\pm 0.00}0.08±0.000.08_{\pm 0.00}0.06±0.020.06_{\pm 0.02}0.65±0.040.65_{\pm 0.04}
CF-k99.46±0.1999.46_{\pm 0.19}8.16±0.278.16_{\pm 0.27}0.40±0.020.40_{\pm 0.02}0.05±0.010.05_{\pm 0.01}0.05±0.000.05_{\pm 0.00}0.15±0.000.15_{\pm 0.00}0.06±0.020.06_{\pm 0.02}0.55±0.040.55_{\pm 0.04}
EU-k99.47±0.1999.47_{\pm 0.19}8.17±0.278.17_{\pm 0.27}0.40±0.020.40_{\pm 0.02}0.05±0.010.05_{\pm 0.01}0.05±0.000.05_{\pm 0.00}0.30±0.010.30_{\pm 0.01}0.07±0.020.07_{\pm 0.02}0.56±0.040.56_{\pm 0.04}
Appendix
Table 10: 30% Random Forgetting on CIFAR-10(VGG16-BN) ↑\uparrow : the goal is to have as high of a value as possible, Δ↓\Delta^{\downarrow} : the value in the table is the difference between the result of the unlearning method and retraining (bottom row) on the metric and the goal is to have a low difference, ↓\downarrow : the goal is to have as low of a value as possible. The bottom row presents the absolute value of M(θ∼)M_{(\theta^{\sim})} on each metric. For any metric with Δ\Delta , the raw value is instead reported. Rows for Δ\Delta FA (↓)(\downarrow) , Δ\Delta FE (↓)(\downarrow) , and Δ\Delta FMIA (↓)(\downarrow) present the absolute difference in the value of the corresponding method on this metric to the value of M(θ∼)M_{(\theta^{\sim})} on the metric. These results show that Reload outperforms all the baselines on RA, Δ\Delta FA, Δ\Delta FE, and Δ\Delta FMIA, by large margins. Reload performs competitively on RSKL and FSKL but is outperformed by FT. Reload also incurs a higher computational cost than other baselines other than FT, CF- kk , and EU- kk .
MethodRA (↑)(\uparrow)Δ\Delta FA (↓)(\downarrow)Δ\Delta FE (↓)(\downarrow)Δ\Delta FMIA (↓)(\downarrow)Δ\Delta AUC (↓)(\downarrow)Cost (↓)(\downarrow)RSKL (↓)(\downarrow)FSKL (↓)(\downarrow)
GA36.61±42.7836.61_{\pm 42.78}45.98±28.6245.98_{\pm 28.62}4.71±3.854.71_{\pm 3.85}0.06±0.080.06_{\pm 0.08}0.06±0.070.06_{\pm 0.07}0.01±0.000.01_{\pm 0.00}0.06±0.020.06_{\pm 0.02}0.66±0.060.66_{\pm 0.06}
FT99.96±0.0299.96_{\pm 0.02}24.94±0.9024.94_{\pm 0.90}1.02±0.041.02_{\pm 0.04}0.13±0.010.13_{\pm 0.01}0.13±0.010.13_{\pm 0.01}0.27±0.020.27_{\pm 0.02}0.05±0.010.05_{\pm 0.01}0.48±0.040.48_{\pm 0.04}
SSD11.89±32.6911.89_{\pm 32.69}65.53±14.2665.53_{\pm 14.26}3.15±0.753.15_{\pm 0.75}0.03±0.070.03_{\pm 0.07}0.02±0.060.02_{\pm 0.06}0.01±0.000.01_{\pm 0.00}8.30±3.118.30_{\pm 3.11}7.83±2.707.83_{\pm 2.70}
SCRUB23.96±2.2323.96_{\pm 2.23}48.86±2.2448.86_{\pm 2.24}1.97±0.131.97_{\pm 0.13}0.01±0.010.01_{\pm 0.01}0.01±0.000.01_{\pm 0.00}0.07±0.000.07_{\pm 0.00}0.06±0.020.06_{\pm 0.02}0.65±0.040.65_{\pm 0.04}
CF-k98.85±0.4098.85_{\pm 0.40}21.38±1.2721.38_{\pm 1.27}0.92±0.040.92_{\pm 0.04}0.12±0.010.12_{\pm 0.01}0.11±0.010.11_{\pm 0.01}0.10±0.010.10_{\pm 0.01}0.06±0.020.06_{\pm 0.02}0.55±0.040.55_{\pm 0.04}
EU-k98.30±0.5598.30_{\pm 0.55}20.18±0.6820.18_{\pm 0.68}0.89±0.040.89_{\pm 0.04}0.11±0.010.11_{\pm 0.01}0.11±0.010.11_{\pm 0.01}0.21±0.020.21_{\pm 0.02}0.07±0.020.07_{\pm 0.02}0.56±0.040.56_{\pm 0.04}
Appendix
Table 11: 30% Random Forgetting on CIFAR-100(ResNet-18) ↑\uparrow : the goal is to have as high of a value as possible, Δ↓\Delta^{\downarrow} : the value in the table is the difference between the result of the unlearning method and retraining (bottom row) on the metric and the goal is to have a low difference, ↓\downarrow : the goal is to have as low of a value as possible. The bottom row presents the absolute value of M(θ∼)M_{(\theta^{\sim})} on each metric. For any metric with Δ\Delta , the raw value is instead reported. Rows for Δ\Delta FA (↓)(\downarrow) , Δ\Delta FE (↓)(\downarrow) , and Δ\Delta FMIA (↓)(\downarrow) present the absolute difference in the value of the corresponding method on this metric to the value of M(θ∼)M_{(\theta^{\sim})} on the metric. These results show that Reload outperforms all the baselines on RA, Δ\Delta FA, Δ\Delta FE, and Δ\Delta FMIA, by large margins. Reload performs competitively on RSKL and FSKL but is outperformed by FT. Reload also incurs a higher computational cost than other baselines other than FT, CF- kk , and EU- kk .
MethodRA (↑)(\uparrow)Δ\Delta FA (↓)(\downarrow)Δ\Delta FE (↓)(\downarrow)Δ\Delta FMIA (↓)(\downarrow)Δ\Delta AUC (↓)(\downarrow)Cost (↓)(\downarrow)RSKL (↓)(\downarrow)FSKL (↓)(\downarrow)
GA10.75±30.8710.75_{\pm 30.87}62.76±11.3562.76_{\pm 11.35}2.10±0.052.10_{\pm 0.05}0.15±0.090.15_{\pm 0.09}0.02±0.050.02_{\pm 0.05}0.01±0.000.01_{\pm 0.00}0.06±0.020.06_{\pm 0.02}0.66±0.060.66_{\pm 0.06}
FT98.30±0.5398.30_{\pm 0.53}15.86±1.3415.86_{\pm 1.34}1.40±0.061.40_{\pm 0.06}0.06±0.010.06_{\pm 0.01}0.06±0.010.06_{\pm 0.01}0.28±0.010.28_{\pm 0.01}0.05±0.010.05_{\pm 0.01}0.48±0.040.48_{\pm 0.04}
SSD11.72±32.1411.72_{\pm 32.14}62.23±12.3662.23_{\pm 12.36}2.43±0.192.43_{\pm 0.19}0.02±0.040.02_{\pm 0.04}0.02±0.050.02_{\pm 0.05}0.01±0.000.01_{\pm 0.00}8.30±3.118.30_{\pm 3.11}7.83±2.707.83_{\pm 2.70}
SCRUB1.60±0.661.60_{\pm 0.66}65.78±0.9265.78_{\pm 0.92}2.39±0.102.39_{\pm 0.10}0.01±0.000.01_{\pm 0.00}0.01±0.000.01_{\pm 0.00}0.08±0.000.08_{\pm 0.00}0.06±0.020.06_{\pm 0.02}0.65±0.040.65_{\pm 0.04}
CF-k97.61±0.6197.61_{\pm 0.61}29.83±0.6529.83_{\pm 0.65}1.95±0.041.95_{\pm 0.04}0.14±0.010.14_{\pm 0.01}0.14±0.010.14_{\pm 0.01}0.15±0.000.15_{\pm 0.00}0.06±0.020.06_{\pm 0.02}0.55±0.040.55_{\pm 0.04}
EU-k97.71±0.7897.71_{\pm 0.78}29.84±0.8429.84_{\pm 0.84}1.95±0.041.95_{\pm 0.04}0.14±0.010.14_{\pm 0.01}0.14±0.010.14_{\pm 0.01}0.30±0.010.30_{\pm 0.01}0.07±0.020.07_{\pm 0.02}0.56±0.040.56_{\pm 0.04}
Appendix
Table 12: 30% Random Forgetting on CIFAR-100(VGG16-BN) ↑\uparrow : the goal is to have as high of a value as possible, Δ↓\Delta^{\downarrow} : the value in the table is the difference between the result of the unlearning method and retraining (bottom row) on the metric and the goal is to have a low difference, ↓\downarrow : the goal is to have as low of a value as possible. The bottom row presents the absolute value of M(θ∼)M_{(\theta^{\sim})} on each metric. For any metric with Δ\Delta , the raw value is instead reported. Rows for Δ\Delta FA (↓)(\downarrow) , Δ\Delta FE (↓)(\downarrow) , and Δ\Delta FMIA (↓)(\downarrow) present the absolute difference in the value of the corresponding method on this metric to the value of M(θ∼)M_{(\theta^{\sim})} on the metric. These results show that Reload outperforms all the baselines on RA, Δ\Delta FA, Δ\Delta FE, and Δ\Delta FMIA, by large margins. Reload performs competitively on RSKL and FSKL but is outperformed by FT. Reload also incurs a higher computational cost than other baselines other than FT, CF- kk , and EU- kk .
MethodRA (↑)(\uparrow)Δ\Delta FA (↓)(\downarrow)Δ\Delta FE (↓)(\downarrow)Δ\Delta FMIA (↓)(\downarrow)Δ\Delta AUC (↓)(\downarrow)Cost (↓)(\downarrow)RSKL (↓)(\downarrow)FSKL (↓)(\downarrow)
GA36.70±41.5536.70_{\pm 41.55}59.35±39.8159.35_{\pm 39.81}6.22±5.556.22_{\pm 5.55}0.02±0.030.02_{\pm 0.03}0.02±0.030.02_{\pm 0.03}0.01±0.000.01_{\pm 0.00}0.06±0.020.06_{\pm 0.02}0.66±0.060.66_{\pm 0.06}
FT100.00±0.00100.00_{\pm 0.00}4.73±0.204.73_{\pm 0.20}0.19±0.010.19_{\pm 0.01}0.04±0.010.04_{\pm 0.01}0.04±0.010.04_{\pm 0.01}0.28±0.010.28_{\pm 0.01}0.05±0.010.05_{\pm 0.01}0.48±0.040.48_{\pm 0.04}
SSD20.64±29.8020.64_{\pm 29.80}75.32±26.3375.32_{\pm 26.33}1.89±0.631.89_{\pm 0.63}0.01±0.030.01_{\pm 0.03}0.01±0.030.01_{\pm 0.03}0.01±0.000.01_{\pm 0.00}8.30±3.118.30_{\pm 3.11}7.83±2.707.83_{\pm 2.70}
SCRUB97.23±0.2997.23_{\pm 0.29}0.49±0.210.49_{\pm 0.21}0.02±0.010.02_{\pm 0.01}0.00±0.000.00_{\pm 0.00}0.00±0.000.00_{\pm 0.00}0.08±0.000.08_{\pm 0.00}0.06±0.020.06_{\pm 0.02}0.65±0.040.65_{\pm 0.04}
CF-k100.00±0.01100.00_{\pm 0.01}4.79±0.224.79_{\pm 0.22}0.19±0.010.19_{\pm 0.01}0.05±0.010.05_{\pm 0.01}0.05±0.010.05_{\pm 0.01}0.10±0.000.10_{\pm 0.00}0.06±0.020.06_{\pm 0.02}0.55±0.040.55_{\pm 0.04}
EU-k99.98±0.0599.98_{\pm 0.05}4.76±0.254.76_{\pm 0.25}0.18±0.010.18_{\pm 0.01}0.05±0.010.05_{\pm 0.01}0.05±0.010.05_{\pm 0.01}0.19±0.000.19_{\pm 0.00}0.07±0.020.07_{\pm 0.02}0.56±0.040.56_{\pm 0.04}
Appendix
Table 13: 30% Random Forgetting on SVHN(ResNet-18) ↑\uparrow : the goal is to have as high of a value as possible, Δ↓\Delta^{\downarrow} : the value in the table is the difference between the result of the unlearning method and retraining (bottom row) on the metric and the goal is to have a low difference, ↓\downarrow : the goal is to have as low of a value as possible. The bottom row presents the absolute value of M(θ∼)M_{(\theta^{\sim})} on each metric. For any metric with Δ\Delta , the raw value is instead reported. Rows for Δ\Delta FA (↓)(\downarrow) , Δ\Delta FE (↓)(\downarrow) , and Δ\Delta FMIA (↓)(\downarrow) present the absolute difference in the value of the corresponding method on this metric to the value of M(θ∼)M_{(\theta^{\sim})} on the metric. These results show that Reload outperforms all the baselines on RA, Δ\Delta FA, Δ\Delta FE, and Δ\Delta FMIA, by large margins. Reload performs competitively on RSKL and FSKL but is outperformed by FT. Reload also incurs a higher computational cost than other baselines other than FT, CF- kk , and EU- kk .
MethodRA (↑)(\uparrow)Δ\Delta FA (↓)(\downarrow)Δ\Delta FE (↓)(\downarrow)Δ\Delta FMIA (↓)(\downarrow)Δ\Delta AUC (↓)(\downarrow)Cost (↓)(\downarrow)RSKL (↓)(\downarrow)FSKL (↓)(\downarrow)
GA16.05±29.5016.05_{\pm 29.50}79.62±26.1679.62_{\pm 26.16}0.25±0.010.25_{\pm 0.01}0.05±0.030.05_{\pm 0.03}0.01±0.020.01_{\pm 0.02}0.01±0.000.01_{\pm 0.00}0.06±0.020.06_{\pm 0.02}0.66±0.060.66_{\pm 0.06}
FT100.00±0.00100.00_{\pm 0.00}4.84±0.164.84_{\pm 0.16}0.23±0.010.23_{\pm 0.01}0.03±0.010.03_{\pm 0.01}0.03±0.010.03_{\pm 0.01}0.28±0.000.28_{\pm 0.00}0.05±0.010.05_{\pm 0.01}0.48±0.040.48_{\pm 0.04}
SSD24.17±28.4924.17_{\pm 28.49}71.83±25.0771.83_{\pm 25.07}1.85±0.601.85_{\pm 0.60}0.01±0.020.01_{\pm 0.02}0.01±0.020.01_{\pm 0.02}0.01±0.000.01_{\pm 0.00}8.30±3.118.30_{\pm 3.11}7.83±2.707.83_{\pm 2.70}
SCRUB24.26±14.0724.26_{\pm 14.07}70.72±13.5570.72_{\pm 13.55}1.85±0.381.85_{\pm 0.38}0.01±0.000.01_{\pm 0.00}0.01±0.000.01_{\pm 0.00}0.08±0.000.08_{\pm 0.00}0.06±0.020.06_{\pm 0.02}0.65±0.040.65_{\pm 0.04}
CF-k99.60±0.1499.60_{\pm 0.14}4.84±0.164.84_{\pm 0.16}0.23±0.010.23_{\pm 0.01}0.04±0.010.04_{\pm 0.01}0.04±0.010.04_{\pm 0.01}0.12±0.000.12_{\pm 0.00}0.06±0.020.06_{\pm 0.02}0.55±0.040.55_{\pm 0.04}
EU-k99.60±0.1499.60_{\pm 0.14}4.85±0.164.85_{\pm 0.16}0.23±0.010.23_{\pm 0.01}0.04±0.010.04_{\pm 0.01}0.04±0.010.04_{\pm 0.01}0.25±0.000.25_{\pm 0.00}0.07±0.020.07_{\pm 0.02}0.56±0.040.56_{\pm 0.04}
Appendix
Table 14: 30% Random Forgetting on SVHN(VGG16-BN) ↑\uparrow : the goal is to have as high of a value as possible, Δ↓\Delta^{\downarrow} : the value in the table is the difference between the result of the unlearning method and retraining (bottom row) on the metric and the goal is to have a low difference, ↓\downarrow : the goal is to have as low of a value as possible. The bottom row presents the absolute value of M(θ∼)M_{(\theta^{\sim})} on each metric. For any metric with Δ\Delta , the raw value is instead reported. Rows for Δ\Delta FA (↓)(\downarrow) , Δ\Delta FE (↓)(\downarrow) , and Δ\Delta FMIA (↓)(\downarrow) present the absolute difference in the value of the corresponding method on this metric to the value of M(θ∼)M_{(\theta^{\sim})} on the metric. These results show that Reload outperforms all the baselines on RA, Δ\Delta FA, Δ\Delta FE, and Δ\Delta FMIA, by large margins. Reload performs competitively on RSKL and FSKL but is outperformed by FT. Reload also incurs a higher computational cost than other baselines other than FT, CF- kk , and EU- kk .
MethodRA (↑)(\uparrow)FA (Δ↓)(\Delta^{\downarrow})FE (Δ↓)(\Delta^{\downarrow})FMIA (Δ↓)(\Delta^{\downarrow})Cost (↓)(\downarrow)RSKL (↓)(\downarrow)FSKL (↓)(\downarrow)
GA99.57±0.0299.57_{\pm 0.02}4.37±0.254.37_{\pm 0.25}0.17±0.010.17_{\pm 0.01}0.05±0.010.05_{\pm 0.01}0.00±0.00\bm{0.00_{\pm 0.00}}0.05±0.000.05_{\pm 0.00}0.52±0.020.52_{\pm 0.02}
FT99.99±0.00\bm{99.99_{\pm 0.00}}4.33±0.224.33_{\pm 0.22}0.17±0.010.17_{\pm 0.01}0.04±0.010.04_{\pm 0.01}0.27±0.000.27_{\pm 0.00}0.00±0.00\bm{0.00_{\pm 0.00}}0.43±0.020.43_{\pm 0.02}
SSD12.75±4.6912.75_{\pm 4.69}82.52±4.7382.52_{\pm 4.73}2.12±0.062.12_{\pm 0.06}0.01±0.010.01_{\pm 0.01}0.01±0.000.01_{\pm 0.00}8.55±0.138.55_{\pm 0.13}7.88±0.127.88_{\pm 0.12}
SCRUB99.79±0.0199.79_{\pm 0.01}4.44±0.264.44_{\pm 0.26}0.18±0.010.18_{\pm 0.01}0.05±0.010.05_{\pm 0.01}0.03±0.000.03_{\pm 0.00}0.03±0.000.03_{\pm 0.00}0.50±0.020.50_{\pm 0.02}
CF- kk99.76±0.0199.76_{\pm 0.01}4.47±0.244.47_{\pm 0.24}0.18±0.010.18_{\pm 0.01}0.05±0.010.05_{\pm 0.01}0.23±0.020.23_{\pm 0.02}0.03±0.000.03_{\pm 0.00}0.50±0.020.50_{\pm 0.02}
EU- kk99.63±0.0199.63_{\pm 0.01}4.46±0.254.46_{\pm 0.25}0.18±0.010.18_{\pm 0.01}0.05±0.010.05_{\pm 0.01}0.23±0.020.23_{\pm 0.02}0.05±0.000.05_{\pm 0.00}0.47±0.020.47_{\pm 0.02}
Appendix
Table 15: 100 In Class Random Forgetting on SVHN (ResNet-18) ↑\uparrow : the goal is to have as high of a value as possible, Δ↓\Delta^{\downarrow} : the value in the table is the difference between the result of the unlearning method and retraining (bottom row) on the metric and the goal is to have a low difference, ↓\downarrow : the goal is to have as low of a value as possible. The bottom row presents the absolute value of M(θ∼)M_{(\theta^{\sim})} on each metric. For any metric with Δ\Delta , the raw value is instead reported. Rows for Δ\Delta FA (↓)(\downarrow) , Δ\Delta FE (↓)(\downarrow) , and Δ\Delta FMIA (↓)(\downarrow) present the absolute difference in the value of the corresponding method on this metric to the value of M(θ∼)M_{(\theta^{\sim})} on the metric. These results show that Reload outperforms all the baselines on Δ\Delta FA, Δ\Delta FE, Δ\Delta FMIA, and RSKL by large margins. Reload performs competitively on RA and FSKL but is outperformed by FT. Reload also incurs a higher computational cost than the other baselines.
MethodRA (↑)(\uparrow)Δ\Delta FA (↓)(\downarrow)Δ\Delta FE (↓)(\downarrow)Δ\Delta FMIA (↓)(\downarrow)Cost (↓)(\downarrow)RSKL (↓)(\downarrow)FSKL (↓)(\downarrow)
GA98.30±0.0498.30_{\pm 0.04}5.43±0.555.43_{\pm 0.55}0.21±0.010.21_{\pm 0.01}0.04±0.000.04_{\pm 0.00}0.00±0.00\bm{0.00_{\pm 0.00}}0.07±0.000.07_{\pm 0.00}0.63±0.040.63_{\pm 0.04}
FT98.38±0.15\bm{98.38_{\pm 0.15}}3.19±0.41\bm{3.19_{\pm 0.41}}0.15±0.020.15_{\pm 0.02}0.02±0.000.02_{\pm 0.00}0.27±0.000.27_{\pm 0.00}0.05±0.01\bm{0.05_{\pm 0.01}}0.46±0.03\bm{0.46_{\pm 0.03}}
SSD10.04±0.0610.04_{\pm 0.06}83.15±0.8783.15_{\pm 0.87}2.07±0.022.07_{\pm 0.02}0.00±0.00\bm{0.00_{\pm 0.00}}0.01±0.000.01_{\pm 0.00}9.39±0.089.39_{\pm 0.08}8.80±0.058.80_{\pm 0.05}
SCRUB98.33±0.0498.33_{\pm 0.04}6.70±0.556.70_{\pm 0.55}0.22±0.010.22_{\pm 0.01}0.05±0.000.05_{\pm 0.00}0.02±0.000.02_{\pm 0.00}0.07±0.000.07_{\pm 0.00}0.63±0.030.63_{\pm 0.03}
CF- kk98.27±0.0698.27_{\pm 0.06}5.23±0.555.23_{\pm 0.55}0.21±0.010.21_{\pm 0.01}0.05±0.010.05_{\pm 0.01}0.23±0.030.23_{\pm 0.03}0.07±0.000.07_{\pm 0.00}0.54±0.020.54_{\pm 0.02}
EU- kk98.28±0.0798.28_{\pm 0.07}5.25±0.545.25_{\pm 0.54}0.22±0.010.22_{\pm 0.01}0.05±0.010.05_{\pm 0.01}0.23±0.030.23_{\pm 0.03}0.07±0.000.07_{\pm 0.00}0.52±0.030.52_{\pm 0.03}
Appendix
Table 16: 100 In Class Random Forgetting on CIFAR-10(ResNet-18) ↑\uparrow : the goal is to have as high of a value as possible, Δ↓\Delta^{\downarrow} : the value in the table is the difference between the result of the unlearning method and retraining (bottom row) on the metric and the goal is to have a low difference, ↓\downarrow : the goal is to have as low of a value as possible. The bottom row presents the absolute value of M(θ∼)M_{(\theta^{\sim})} on each metric. For any metric with Δ\Delta , the raw value is instead reported. Rows for Δ\Delta FA (↓)(\downarrow) , Δ\Delta FE (↓)(\downarrow) , and Δ\Delta FMIA (↓)(\downarrow) present the absolute difference in the value of the corresponding method on this metric to the value of M(θ∼)M_{(\theta^{\sim})} on the metric. These results show that Reload outperforms all the baselines on Δ\Delta FE. Reload performs competitively on RA, Δ\Delta FA, Δ\Delta FMIA, RSKL, and FSKL but is outperformed. FT which performs well, empirically makes little adjustment to the actual FA value. Reload also incurs a higher computational cost than the other baselines.
MethodRA (↑)(\uparrow)Δ\Delta FA (↓)(\downarrow)Δ\Delta FE (↓)(\downarrow)Δ\Delta FMIA (↓)(\downarrow)Cost (↓)(\downarrow)RSKL (↓)(\downarrow)FSKL (↓)(\downarrow)
GA99.02±0.0599.02_{\pm 0.05}6.94±0.326.94_{\pm 0.32}0.33±0.010.33_{\pm 0.01}0.04±0.010.04_{\pm 0.01}0.00±0.00\bm{0.00_{\pm 0.00}}0.10±0.000.10_{\pm 0.00}0.91±0.030.91_{\pm 0.03}
FT98.72±0.3098.72_{\pm 0.30}3.50±0.373.50_{\pm 0.37}0.23±0.010.23_{\pm 0.01}0.02±0.010.02_{\pm 0.01}0.27±0.010.27_{\pm 0.01}0.08±0.01\bm{0.08_{\pm 0.01}}0.65±0.030.65_{\pm 0.03}
SSD9.99±0.049.99_{\pm 0.04}81.88±0.5081.88_{\pm 0.50}2.12±0.302.12_{\pm 0.30}0.01±0.01\bm{0.01_{\pm 0.01}}0.01±0.000.01_{\pm 0.00}10.88±0.7910.88_{\pm 0.79}10.25±0.8310.25_{\pm 0.83}
SCRUB97.31±3.5797.31_{\pm 3.57}5.79±2.285.79_{\pm 2.28}0.14±0.080.14_{\pm 0.08}0.04±0.010.04_{\pm 0.01}0.03±0.000.03_{\pm 0.00}1.37±0.451.37_{\pm 0.45}1.75±0.451.75_{\pm 0.45}
CF- kk99.03±0.05\bm{99.03_{\pm 0.05}}6.95±0.336.95_{\pm 0.33}0.33±0.010.33_{\pm 0.01}0.05±0.010.05_{\pm 0.01}0.37±0.080.37_{\pm 0.08}0.10±0.010.10_{\pm 0.01}0.79±0.020.79_{\pm 0.02}
EU- kk99.02±0.0599.02_{\pm 0.05}6.96±0.356.96_{\pm 0.35}0.33±0.010.33_{\pm 0.01}0.05±0.010.05_{\pm 0.01}0.37±0.080.37_{\pm 0.08}0.10±0.000.10_{\pm 0.00}0.78±0.040.78_{\pm 0.04}
Appendix
Table 17: 100 In Class Random Forgetting on CIFAR-10 (VGG16-BN). The bottom row presents the absolute value of M(θ∼)M_{(\theta^{\sim})} on each metric. For any metric with Δ\Delta , the raw value is instead reported. Rows for Δ\Delta FA (↓)(\downarrow) , Δ\Delta FE (↓)(\downarrow) , and Δ\Delta FMIA (↓)(\downarrow) present the absolute difference in the value of the corresponding method on this metric to the value of M(θ∼)M_{(\theta^{\sim})} on the metric. These results show that Reload outperforms all baselines on Δ\Delta FA, Δ\Delta FE, Δ\Delta FMIA, FSKL indicating it behaves the closes to M(θ∼)M_{(\theta^{\sim})} on Dforget\mathcal{D}_{forget} . Reload performs competitively on RA and RSKL, falling behind of the leading method by 0.46 for RA and 0.02 for RSKL. Reload incurs a higher computational cost than most baselines, but is cheaper than FT, CF- kk , and EU- kk . Other experimental settings are presented in Appendix B.4.3
MethodRA (↑)(\uparrow)FA (Δ↓)(\Delta^{\downarrow})FE (Δ↓)(\Delta^{\downarrow})FMIA (Δ↓)(\Delta^{\downarrow})Cost (↓)(\downarrow)RSKL (↓)(\downarrow)FSKL (↓)(\downarrow)
GA98.32±0.0398.32_{\pm 0.03}23.33±1.0623.33_{\pm 1.06}1.00±0.061.00_{\pm 0.06}0.07±0.060.07_{\pm 0.06}0.00±0.00\bm{0.00_{\pm 0.00}}0.07±0.000.07_{\pm 0.00}0.65±0.060.65_{\pm 0.06}
FT98.22±0.2398.22_{\pm 0.23}16.84±1.0816.84_{\pm 1.08}0.82±0.060.82_{\pm 0.06}0.05±0.030.05_{\pm 0.03}0.27±0.000.27_{\pm 0.00}0.05±0.01\bm{0.05_{\pm 0.01}}0.48±0.04\bm{0.48_{\pm 0.04}}
SSD10.01±0.0510.01_{\pm 0.05}68.67±1.9768.67_{\pm 1.97}5.75±0.995.75_{\pm 0.99}0.38±0.140.38_{\pm 0.14}0.00±0.00\bm{0.00_{\pm 0.00}}9.33±0.069.33_{\pm 0.06}8.72±0.048.72_{\pm 0.04}
SCRUB98.35±0.0398.35_{\pm 0.03}27.55±1.4327.55_{\pm 1.43}1.02±0.061.02_{\pm 0.06}0.07±0.060.07_{\pm 0.06}0.02±0.000.02_{\pm 0.00}0.07±0.000.07_{\pm 0.00}0.65±0.040.65_{\pm 0.04}
CF- kk98.22±0.1198.22_{\pm 0.11}21.84±0.8821.84_{\pm 0.88}0.99±0.050.99_{\pm 0.05}0.07±0.060.07_{\pm 0.06}0.21±0.010.21_{\pm 0.01}0.07±0.000.07_{\pm 0.00}0.54±0.040.54_{\pm 0.04}
EU- kk98.24±0.0398.24_{\pm 0.03}21.95±0.7821.95_{\pm 0.78}0.99±0.050.99_{\pm 0.05}0.07±0.060.07_{\pm 0.06}0.21±0.010.21_{\pm 0.01}0.07±0.000.07_{\pm 0.00}0.55±0.040.55_{\pm 0.04}
Appendix
Table 18: 100 In Class Random Forgetting on CIFAR-100(ResNet-18) ↑\uparrow : the goal is to have as high of a value as possible, Δ↓\Delta^{\downarrow} : the value in the table is the difference between the result of the unlearning method and retraining (bottom row) on the metric and the goal is to have a low difference, ↓\downarrow : the goal is to have as low of a value as possible. The bottom row presents the absolute value of M(θ∼)M_{(\theta^{\sim})} on each metric. For any metric with Δ\Delta , the raw value is instead reported. Rows for Δ\Delta FA (↓)(\downarrow) , Δ\Delta FE (↓)(\downarrow) , and Δ\Delta FMIA (↓)(\downarrow) present the absolute difference in the value of the corresponding method on this metric to the value of M(θ∼)M_{(\theta^{\sim})} on the metric. These results show that Reload outperforms all the baselines on RA, Δ\Delta FA, Δ\Delta FE, and Δ\Delta FMIA, by large margins. Reload performs competitively on RSKL and FSKL but is outperformed by FT. Reload also incurs a higher computational cost than the other baselines.
MethodRA (↑)(\uparrow)FA (Δ↓)(\Delta^{\downarrow})FE (Δ↓)(\Delta^{\downarrow})FMIA (Δ↓)(\Delta^{\downarrow})Cost (↓)(\downarrow)RSKL (↓)(\downarrow)FSKL (↓)(\downarrow)
GA98.31±0.0398.31_{\pm 0.03}28.55±2.0228.55_{\pm 2.02}1.70±0.041.70_{\pm 0.04}0.03±0.020.03_{\pm 0.02}0.00±0.00\bm{0.00_{\pm 0.00}}0.07±0.000.07_{\pm 0.00}0.65±0.040.65_{\pm 0.04}
FT98.14±0.2598.14_{\pm 0.25}11.44±1.7711.44_{\pm 1.77}1.07±0.091.07_{\pm 0.09}0.01±0.01\bm{0.01_{\pm 0.01}}0.28±0.010.28_{\pm 0.01}0.06±0.01\bm{0.06_{\pm 0.01}}0.47±0.03\bm{0.47_{\pm 0.03}}
SSD10.00±0.0310.00_{\pm 0.03}63.86±2.1263.86_{\pm 2.12}2.70±0.132.70_{\pm 0.13}0.45±0.040.45_{\pm 0.04}0.00±0.00\bm{0.00_{\pm 0.00}}9.36±0.059.36_{\pm 0.05}8.75±0.048.75_{\pm 0.04}
SCRUB98.33±0.0298.33_{\pm 0.02}30.59±1.2530.59_{\pm 1.25}1.76±0.051.76_{\pm 0.05}0.04±0.010.04_{\pm 0.01}0.02±0.000.02_{\pm 0.00}0.07±0.000.07_{\pm 0.00}0.63±0.030.63_{\pm 0.03}
CF- kk98.15±0.1298.15_{\pm 0.12}26.86±2.1626.86_{\pm 2.16}1.75±0.071.75_{\pm 0.07}0.04±0.010.04_{\pm 0.01}0.34±0.070.34_{\pm 0.07}0.07±0.000.07_{\pm 0.00}0.54±0.030.54_{\pm 0.03}
EU- kk98.22±0.0498.22_{\pm 0.04}25.37±1.3525.37_{\pm 1.35}1.68±0.061.68_{\pm 0.06}0.03±0.020.03_{\pm 0.02}0.33±0.070.33_{\pm 0.07}0.07±0.000.07_{\pm 0.00}0.55±0.030.55_{\pm 0.03}
Appendix
Table 19: 100 In Class Random Forgetting on CIFAR-100(VGG16-BN) ↑\uparrow : the goal is to have as high of a value as possible, Δ↓\Delta^{\downarrow} : the value in the table is the difference between the result of the unlearning method and retraining (bottom row) on the metric and the goal is to have a low difference, ↓\downarrow : the goal is to have as low of a value as possible. The bottom row presents the absolute value of M(θ∼)M_{(\theta^{\sim})} on each metric. For any metric with Δ\Delta , the raw value is instead reported. Rows for Δ\Delta FA (↓)(\downarrow) , Δ\Delta FE (↓)(\downarrow) , and Δ\Delta FMIA (↓)(\downarrow) present the absolute difference in the value of the corresponding method on this metric to the value of M(θ∼)M_{(\theta^{\sim})} on the metric. These results show that Reload outperforms all the baselines on RA, Δ\Delta FA, and Δ\Delta FE by large margins. Reload performs competitively on Δ\Delta FMIA, RSKL and FSKL but is outperformed by FT. Reload also incurs a higher computational cost than the other baselines.
MethodRA (↑)(\uparrow)FA (Δ↓)(\Delta^{\downarrow})FE (Δ↓)(\Delta^{\downarrow})FMIA (Δ↓)(\Delta^{\downarrow})Cost (↓)(\downarrow)RSKL (↓)(\downarrow)FSKL (↓)(\downarrow)
GA99.57±0.0299.57_{\pm 0.02}4.46±0.244.46_{\pm 0.24}0.22±0.010.22_{\pm 0.01}0.03±0.010.03_{\pm 0.01}0.00±0.00\bm{0.00_{\pm 0.00}}0.05±0.000.05_{\pm 0.00}0.51±0.020.51_{\pm 0.02}
FT99.99±0.001\bm{99.99_{\pm 0.001}}4.47±0.234.47_{\pm 0.23}0.22±0.010.22_{\pm 0.01}0.03±0.010.03_{\pm 0.01}0.27±0.000.27_{\pm 0.00}0.00±0.00\bm{0.00_{\pm 0.00}}0.43±0.020.43_{\pm 0.02}
SSD14.55±3.9314.55_{\pm 3.93}84.19±1.5584.19_{\pm 1.55}2.05±0.012.05_{\pm 0.01}0.00±0.00\bm{0.00_{\pm 0.00}}0.01±0.000.01_{\pm 0.00}8.51±0.038.51_{\pm 0.03}7.84±0.027.84_{\pm 0.02}
SCRUB99.79±0.0199.79_{\pm 0.01}9.55±9.769.55_{\pm 9.76}0.36±0.340.36_{\pm 0.34}0.03±0.010.03_{\pm 0.01}0.02±0.000.02_{\pm 0.00}0.03±0.000.03_{\pm 0.00}0.50±0.030.50_{\pm 0.03}
CF- kk99.76±0.0199.76_{\pm 0.01}4.53±0.254.53_{\pm 0.25}0.23±0.010.23_{\pm 0.01}0.04±0.010.04_{\pm 0.01}0.24±0.020.24_{\pm 0.02}0.03±0.000.03_{\pm 0.00}0.50±0.020.50_{\pm 0.02}
EU- kk99.63±0.0299.63_{\pm 0.02}4.54±0.234.54_{\pm 0.23}0.23±0.010.23_{\pm 0.01}0.04±0.010.04_{\pm 0.01}0.24±0.020.24_{\pm 0.02}0.05±0.000.05_{\pm 0.00}0.47±0.020.47_{\pm 0.02}
Appendix
Table 20: 100 In Class Random Forgetting on SVHN (VGG16-BN) ↑\uparrow : the goal is to have as high of a value as possible, Δ↓\Delta^{\downarrow} : the value in the table is the difference between the result of the unlearning method and retraining (bottom row) on the metric and the goal is to have a low difference, ↓\downarrow : the goal is to have as low of a value as possible. The bottom row presents the absolute value of M(θ∼)M_{(\theta^{\sim})} on each metric. For any metric with Δ\Delta , the raw value is instead reported. Rows for Δ\Delta FA (↓)(\downarrow) , Δ\Delta FE (↓)(\downarrow) , and Δ\Delta FMIA (↓)(\downarrow) present the absolute difference in the value of the corresponding method on this metric to the value of M(θ∼)M_{(\theta^{\sim})} on the metric. These results show that Reload outperforms all the baselines on Δ\Delta FA, Δ\Delta FE, and FSKL, by large margins. Reload performs competitively on RA, Δ\Delta FMIA, and RSKL but is outperformed by FT. Reload also incurs a higher computational cost than the other baselines.
SplitMethodChange in Model Utility from OriginalForget Quality
1%Original+0.00000.0030
Retain-0.01311.0000
Grad Ascent-0.02330.0068
Grad Diff-0.01980.0143
KL Min-0.02210.0068
Pref Opt-0.00210.0971
Appendix
Table 21: Change in Model Utility and Forget Quality of different unlearning methods on unlearning entities from the TOFU dataset on
SplitMethodChange in Model Utility from OriginalForget Quality
10%Original0.00000.0000
Retain+0.00531.0000
Grad Ascent-0.55180.2107
Grad Diff-0.19990.0000
KL Min-0.55180.4158
Pref Opt-0.03790.0000
Appendix
Table 22: Change in Model Utility and Forget Quality of different unlearning methods on unlearning entities from the TOFU dataset on Phi-1.5
Figure 6: Data poisoning inserts the patterns ( right ) in all selected images in D\mathcal{D} .
Figure 7: Corrective Accuracy (Acc corr{}_{\text{corr}} ) after applying different unlearning methods. This measures the performance of the unlearned model on the domain representing the adversely affected data, Dm\mathcal{D}_{m} . γ\gamma measures the proportion of the adversely affected data which was identified and collected within Dm\mathcal{D}_{m} . We note that at small γ\gamma , Reload achieves consistently higher Acc corr{}_{\text{corr}} than existing baselines and performs across γ\gamma values.
Figure 8: Corrective Accuracy (Acc corr{}_{\text{corr}} ) after applying different unlearning methods. This measures the performance of the unlearned model on the domain representing the adversely affected data, Dm\mathcal{D}_{m} . γ\gamma measures the proportion of the adversely affected data which was identified and collected within Dforget\mathcal{D}_{forget} . We note that at small γ\gamma , Reload achieves consistently higher Acc corr{}_{\text{corr}} than existing baselines and performs across γ\gamma values.
Cost ( ↓\downarrow )CIFAR10CIFAR100
Poisoning
BadT0.44 ±0.00{}_{\pm\text{0.00}}0.68 ±0.00{}_{\pm\text{0.00}}
CF0.26 ±0.00{}_{\pm\text{0.00}}0.25 ±0.00{}_{\pm\text{0.00}}
SSD0.04 ±0.00{}_{\pm\text{0.00}}0.06 ±0.00{}_{\pm\text{0.00}}
Scrub0.26 ±0.00{}_{\pm\text{0.00}}0.31 ±0.00{}_{\pm\text{0.00}}
RewoD1.00 ±0.00{}_{\pm\text{0.00}}1.00 ±0.00{}_{\pm\text{0.00}}
Appendix
Table 23: Cost ( ↓\downarrow ) values across different sizes of Dforget\mathcal{D}_{forget} (Corrective Unlearning). Results are reported as mean ±stddev{}_{\pm\text{stddev}} over 10 values of γ\gamma .
Acc retain{}_{\text{retain}} ( ↑\uparrow )CIFAR10CIFAR100
Poisoning
None91.3574.05
BadT0.13 ±0.04{}_{\pm\text{0.04}}0.13 ±0.05{}_{\pm\text{0.05}}
CF0.09 ±0.12{}_{\pm\text{0.12}}0.54 ±0.17{}_{\pm\text{0.17}}
SSD-3.05 ±4.56{}_{\pm\text{4.56}}-2.08 ±0.00{}_{\pm\text{0.00}}
Scrub0.01 ±0.11{}_{\pm\text{0.11}}0.25 ±0.23{}_{\pm\text{0.23}}
Appendix
Table 24: Acc retain{}_{\text{retain}} ( ↑\uparrow ) values across different sizes of Dforget\mathcal{D}_{forget} (Corrective Unlearning). Results are reported as mean ±stddev{}_{\pm\text{stddev}} over 10 values of γ\gamma .
Figure 9: CIFAR10 Interclass Confusion
Figure 10: CIFAR10 Poisoning
Figure 11: CIFAR100 Interclass Confusion
Figure 12: CIFAR100 Poisoning
Cost ( ↓\downarrow )CIFAR10CIFAR100
Poisoning
BadT0.63 ±0.010.69 ±0.00
CF0.26 ±0.010.25 ±0.00
SSD0.06 ±0.010.06 ±0.00
Scrub0.27 ±0.010.30 ±0.00
RewoD1.00 ±0.041.00 ±0.01
Appendix
Table 25: Cost ( ↓\downarrow ) values across different sizes of Dforget\mathcal{D}_{forget} (Corrective Unlearning with replacement). Results are reported as mean ±stddev{}_{\pm\text{stddev}} over 10 values of γ\gamma .
Acc retain{}_{\text{retain}} ( ↑\uparrow )CIFAR10CIFAR100
Poisoning
None91.3574.05
BadT-0.02 ±0.06-0.13 ±0.15
CF0.07 ±0.090.37 ±0.07
SSD-16.01 ±23.220.00 ±0.00
Scrub-0.01 ±0.01-0.01 ±0.06
Appendix
Table 26: Acc retain{}_{\text{retain}} ( ↑\uparrow ) values across different sizes of Dforget\mathcal{D}_{forget} (Corrective Unlearning with replacement). Results are reported as mean ±stddev{}_{\pm\text{stddev}} over 10 values of γ\gamma .
Figure 13: CIFAR10 Interclass Confusion (with replacement)
Figure 14: CIFAR10 Poisoning (with replacement)
Figure 15: CIFAR100 Interclass Confusion (with replacement)
Figure 16: CIFAR100 Poisoning (with replacement)
ExperimentAlpha ( α\alpha )Ascent Learning RateFinetuning Learning RateWeight Reset Method
SVHN + ResNet-180.10.2430.098Uniform
SVHN + VGG16-BN0.10.4960.496Xavier Uniform
CIFAR-10 + ResNet-180.10.440.33Xavier Uniform
CIFAR-10 + VGG16-BN0.10.1670.39Kaiming Uniform
CIFAR-100 + ResNet-180.10.180.33Xavier Normal
CIFAR-100 + VGG16-BN0.10.3250.164Kaiming Normal
Appendix
Table 27: Reload Hyperparameter Settings for Unlearning
ExperimentAlpha ( α\alpha )Ascent Learning RateFinetuning Learning RateWeight Reset MethodRetain Sample SizeTarget LayersRepair Epochs
Llama-2-7b-Chat + Forget010.0080.0390.0002Zero152mlp.gate_proj.weight5
Llama-2-7b-Chat + Forget050.0170.0490.0002Uniform193self_attn.k_proj.weight5
Llama-2-7b-Chat + Forget100.3030.0220.0488Xavier Uniform60self_attn.q_proj.weight28
Phi-1.5 + Forget100.3960.0940.0445Uniform90self_attn.v_proj.weight40
Appendix
Table 28: Hyperparameter Settings for LM Entity Unlearning
ExperimentAlpha ( α\alpha )Ascent Learning RateFinetuning Learning RateWeight Reset Method
CIFAR10 + ResNet-9 + Poisoning0.39840.010.00381Xavier Normal
CIFAR10 + ResNet-9 + Interclass Confusion0.19780.010.00957Mean
CIFAR100 + ResNet-28x10 + Poisoning0.24010.010.00483Xavier Normal
CIFAR100 + ResNet-28x10 + Interclass Confusion0.09240.010.00237Zero
Appendix
Table 29: Reload Hyperparameter Settings for corrective unlearning (with and without replacement)
Figure 17: Corrective Accuracy (Acc corr ) across variants of Reload .
Acc retain ( ↑\uparrow )CIFAR10CIFAR100
Poisoning
RELOAD-2.84 ±2.56-5.08 ±1.39
ReloadWithoutAscent-3.28 ±2.49-5.24 ±1.65
ReloadWithNormalisation-4.23 ±2.11-5.34 ±1.72
ReloadWithCosineKV0.24 ±0.120.49 ±0.15
Interclass Confusion (IC)
Appendix
Figure 18: Model Utility across variants of Reload
MethodRA (↑)(\uparrow)Δ\Delta FA (↓)(\downarrow)Δ\Delta FE (↓)(\downarrow)Δ\Delta FMIA (↓)(\downarrow)RSKL (↓)(\downarrow)FSKL (↓)(\downarrow)
Reload (ours)99.45±0.1299.45_{\pm 0.12}0.53±0.500.53_{\pm 0.50}0.79±0.10.79_{\pm 0.1}0.01±0.010.01_{\pm 0.01}0.19±0.030.19_{\pm 0.03}8.7±0.118.7_{\pm 0.11}
Retrained (Baseline)99.91±0.0299.91_{\pm 0.02}54.11±0.5054.11_{\pm 0.50}4.31±0.094.31_{\pm 0.09}0.51±0.010.51_{\pm 0.01}--
Appendix
Table 30: 10% Random Forgetting on CIFAR-10 (ViT) ↑\uparrow : the goal is to have as high of a value as possible, Δ↓\Delta^{\downarrow} : the value in the table is the difference between the result of the unlearning method and retraining (bottom row) on the metric and the goal is to have a low difference, ↓\downarrow : the goal is to have as low of a value as possible. The bottom row presents the absolute value of M(θ∼)M_{(\theta^{\sim})} on each metric. For any metric with Δ\Delta , the raw value is instead reported. Rows for Δ\Delta FA (↓)(\downarrow) , Δ\Delta FE (↓)(\downarrow) , and Δ\Delta FMIA (↓)(\downarrow) present the absolute difference in the value of the corresponding method on this metric to the value of M(θ∼)M_{(\theta^{\sim})} on the metric. These results show that Reload performs similarly to the ground truth Retrained model on RA, Δ\Delta FA, Δ\Delta FMIA, and RSKL. Reload strays from the Retrained model in Δ\Delta FE and FSKL.
MethodRA (↑)(\uparrow)Δ\Delta FA (↓)(\downarrow)Δ\Delta FE (↓)(\downarrow)Δ\Delta FMIA (↓)(\downarrow)RSKL (↓)(\downarrow)FSKL (↓)(\downarrow)
Reload (Un-Quantized)99.99±0.0199.99_{\pm 0.01}0.46±0.570.46_{\pm 0.57}0.76±0.080.76_{\pm 0.08}0.01±0.010.01_{\pm 0.01}0.44±0.030.44_{\pm 0.03}4.04±0.14.04_{\pm 0.1}
Reload (Quantized)99.99±0.0199.99_{\pm 0.01}0.46±0.570.46_{\pm 0.57}0.76±0.080.76_{\pm 0.08}0.01±0.010.01_{\pm 0.01}0.44±0.030.44_{\pm 0.03}4.04±0.14.04_{\pm 0.1}
Retrained (Baseline)99.52±0.1699.52_{\pm 0.16}36.22±0.4936.22_{\pm 0.49}2.25±0.032.25_{\pm 0.03}0.51±0.010.51_{\pm 0.01}--
Appendix
Table 31: 10% Random Forgetting on CIFAR-10 (ResNet-18) with Quantized Cached Gradients ↑\uparrow : the goal is to have as high of a value as possible, Δ↓\Delta^{\downarrow} : the value in the table is the difference between the result of the unlearning method and retraining (bottom row) on the metric and the goal is to have a low difference, ↓\downarrow : the goal is to have as low of a value as possible. The bottom row presents the absolute value of M(θ∼)M_{(\theta^{\sim})} on each metric. For any metric with Δ\Delta , the raw value is instead reported. Rows for Δ\Delta FA (↓)(\downarrow) , Δ\Delta FE (↓)(\downarrow) , and Δ\Delta FMIA (↓)(\downarrow) present the absolute difference in the value of the corresponding method on this metric to the value of M(θ∼)M_{(\theta^{\sim})} on the metric. These results show that Reload performs similarly to the ground truth Retrained model on RA, Δ\Delta FA, Δ\Delta FE Δ\Delta FMIA, RSKL. Reload strays from the Retrained model in FSKL.
MethodRA (↑)(\uparrow)Δ\Delta FA (↓)(\downarrow)Δ\Delta FE (↓)(\downarrow)Δ\Delta FMIA (↓)(\downarrow)RSKL (↓)(\downarrow)FSKL (↓)(\downarrow)
Reload (ours)98.16±0.0198.16_{\pm 0.01}0.57±0.460.57_{\pm 0.46}0.06±0.030.06_{\pm 0.03}0.01±0.010.01_{\pm 0.01}0.91±0.020.91_{\pm 0.02}3.45±0.063.45_{\pm 0.06}
Retrained (Baseline)92.34±0.692.34_{\pm 0.6}34.57±0.0034.57_{\pm 0.00}2.33±0.002.33_{\pm 0.00}0.51±0.010.51_{\pm 0.01}--
Appendix
Table 32: 10% Random Forgetting from CIFAR-10 after finetuning on CIFAR-10.1 (ResNet-18)
MethodRA (↑)(\uparrow)Δ\Delta FA (↓)(\downarrow)Δ\Delta FE (↓)(\downarrow)Δ\Delta FMIA (↓)(\downarrow)RSKL (↓)(\downarrow)FSKL (↓)(\downarrow)
Reload (ours)98.89±0.1998.89_{\pm 0.19}2.85±2.052.85_{\pm 2.05}0.20±0.130.20_{\pm 0.13}0.05±0.040.05_{\pm 0.04}0.51±0.020.51_{\pm 0.02}3.95±0.273.95_{\pm 0.27}
Retrained (Baseline)99.73±0.1199.73_{\pm 0.11}32.4±2.7832.4_{\pm 2.78}2.70±0.112.70_{\pm 0.11}0.54±0.050.54_{\pm 0.05}--
Appendix
Table 33: 10% Random Forgetting from CIFAR-10.1 after finetuning on CIFAR-10.1 (ResNet-18)
MethodRA (↑)(\uparrow)Δ\Delta FA (↓)(\downarrow)Δ\Delta FE (↓)(\downarrow)Δ\Delta FMIA (↓)(\downarrow)RSKL (↓)(\downarrow)FSKL (↓)(\downarrow)
Reload (ours)99.69±0.1399.69_{\pm 0.13}3.88±1.523.88_{\pm 1.52}0.27±0.110.27_{\pm 0.11}0.02±0.010.02_{\pm 0.01}1.04±0.131.04_{\pm 0.13}3.97±0.153.97_{\pm 0.15}
Retrained (Baseline)89.39±2.8989.39_{\pm 2.89}35.83±0.9335.83_{\pm 0.93}2.34±0.052.34_{\pm 0.05}0.52±0.020.52_{\pm 0.02}--
Appendix
Table 34: 10% Random Forgetting from CIFAR-10 and CIFAR-10.1 after finetuning on CIFAR-10.1 (ResNet-18)
MethodRA (↑)(\uparrow)Δ\Delta FA (↓)(\downarrow)Δ\Delta FE (↓)(\downarrow)Δ\Delta FMIA (↓)(\downarrow)RSKL (↓)(\downarrow)FSKL (↓)(\downarrow)
Reload (10% of \cDretain\cD_{retain} )32.33±0.3632.33_{\pm 0.36}11.44±0.5211.44_{\pm 0.52}0.19±0.040.19_{\pm 0.04}0.01±0.010.01_{\pm 0.01}4.32±0.074.32_{\pm 0.07}3.60±0.13.60_{\pm 0.1}
Reload (20% of \cDretain\cD_{retain} )42.33±0.2742.33_{\pm 0.27}8.16±0.538.16_{\pm 0.53}0.26±0.040.26_{\pm 0.04}0.01±0.010.01_{\pm 0.01}3.82±0.063.82_{\pm 0.06}3.62±0.13.62_{\pm 0.1}
Reload (30% of \cDretain\cD_{retain} )50.37±0.3050.37_{\pm 0.30}6.92±0.796.92_{\pm 0.79}0.25±0.040.25_{\pm 0.04}0.01±0.010.01_{\pm 0.01}3.35±0.063.35_{\pm 0.06}3.60±0.13.60_{\pm 0.1}
Reload (40% of \cDretain\cD_{retain} )58.30±0.3258.30_{\pm 0.32}5.53±0.795.53_{\pm 0.79}0.34±0.050.34_{\pm 0.05}0.01±0.010.01_{\pm 0.01}2.96±0.062.96_{\pm 0.06}3.75±0.13.75_{\pm 0.1}
Reload (50% of \cDretain\cD_{retain} )65.78±0.2665.78_{\pm 0.26}4.68±1.044.68_{\pm 1.04}0.43±0.050.43_{\pm 0.05}0.01±0.010.01_{\pm 0.01}2.57±0.052.57_{\pm 0.05}3.88±0.13.88_{\pm 0.1}
Reload (60% of \cDretain\cD_{retain} )72.86±0.2272.86_{\pm 0.22}4.00±0.784.00_{\pm 0.78}0.42±0.050.42_{\pm 0.05}0.01±0.010.01_{\pm 0.01}2.12±0.032.12_{\pm 0.03}3.89±0.13.89_{\pm 0.1}
Appendix
Table 35: 10% Random Forgetting on CIFAR-10 (ResNet-18). Finetuning with different sized subsets of \cDretain\cD_{retain} .
 Forgetting a Random 30% of the DataForgetting a Random Feature
MethodRA (↑)(\uparrow)Δ\Delta FA (↓)(\downarrow)Δ\Delta FE (↓)(\downarrow)Δ\Delta FMIA (↓)(\downarrow)Cost (↓)(\downarrow)OA (↑)(\uparrow)RA (↑)(\uparrow)TA (↑)(\uparrow)Cost (↓)(\downarrow)
OriginalN/AN/AN/AN/AN/A0.84±0.000.84_{\pm 0.00}0.73±0.070.73_{\pm 0.07}0.82±0.010.82_{\pm 0.01}N/A
Retrain0.84±0.010.84_{\pm 0.01}0.82±0.010.82_{\pm 0.01}0.48±0.000.48_{\pm 0.00}0.51±0.130.51_{\pm 0.13}1.00±0.001.00_{\pm 0.00}0.77±0.090.77_{\pm 0.09}0.84±0.010.84_{\pm 0.01}0.75±0.090.75_{\pm 0.09}1.00±0.001.00_{\pm 0.00}
Reload0.84±0.010.84_{\pm 0.01}0.02±0.010.02_{\pm 0.01}0.00±0.000.00_{\pm 0.00}0.00±0.000.00_{\pm 0.00}0.29±0.180.29_{\pm 0.18}0.78±0.030.78_{\pm 0.03}0.81±0.010.81_{\pm 0.01}0.77±0.030.77_{\pm 0.03}0.40±0.270.40_{\pm 0.27}
GA0.38±0.220.38_{\pm 0.22}0.54±0.040.54_{\pm 0.04}0.55±0.040.55_{\pm 0.04}0.06±0.070.06_{\pm 0.07}0.13±0.020.13_{\pm 0.02}0.35±0.160.35_{\pm 0.16}0.48±0.170.48_{\pm 0.17}0.35±0.160.35_{\pm 0.16}0.30±0.120.30_{\pm 0.12}
FT0.83±0.010.83_{\pm 0.01}0.01±0.010.01_{\pm 0.01}0.01±0.000.01_{\pm 0.00}0.05±0.060.05_{\pm 0.06}0.53±0.070.53_{\pm 0.07}0.77±0.090.77_{\pm 0.09}0.84±0.000.84_{\pm 0.00}0.76±0.090.76_{\pm 0.09}0.79±0.200.79_{\pm 0.20}
Appendix
Table 36: Results highlighting the performance of Reload on the tabular Census Income dataset.
Figure 19: Impact of Learning Rate ( η\eta ) on Reload performance.
Figure 20: Impact of Threshold ( α\alpha ) on Reload performance
More in this topic

Machine Unlearning

Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.

See all 340 papers

Explore similar work

CardsList
Jul 7, 2025

Efficient Unlearning with Privacy Guarantees

Privacy protection laws, such as the GDPR, grant individuals the right to request the forgetting of their personal data not only from databases but also from machine learning (ML) models trained on them. Machine unlearning has emerged as a practical means to facilitate model forgetting of data instances seen during training. Although some existing machine unlearning methods guarantee exact forgetting, they are typically costly in computational terms. On the other hand, more affordable methods do not offer forgetting guarantees and are applicable only to specific ML models. In this paper, we present efficient unlearning with privacy guarantees (EUPG), a novel machine unlearning framework that offers formal privacy guarantees to individuals whose data are being unlearned. EUPG involves pre-training ML models on data protected using privacy models, and it enables {\em efficient unlearning with the privacy guarantees offered by the privacy models in use}. Through empirical evaluation on four heterogeneous data sets protected with kk-anonymity and εε-differential privacy as privacy models, our approach demonstrates utility and forgetting effectiveness comparable to those of exact unlearning methods, while significantly reducing computational and storage costs. Our code is available at https://github.com/najeebjebreel/EUPG.

Josep Domingo-Ferrer, Najeeb Jebreel, David Sánchez

1CYBERCAT (Center for Cybersecurity Research of2026 Catalonia), ComSCIAM-Center for Computational Science and Applied Mathematics, Department of Computer EngineeringJun and Mathematics, Universitat Rovira i Virgili, Av. Pa¨ısos26 Catalans 26, E-43007 Tarragona, Catalonia

Privacy-Preserving MLMachine Unlearning
AskPDF
Apr 15, 2026

WIN-U: Woodbury-Informed Newton-Unlearning as a retain-free Machine Unlearning Framework

Privacy concerns in LLMs have led to the rapidly growing need to enforce a data's "right to be forgotten". Machine unlearning addresses precisely this task, namely the removal of the influence of some specific data, i.e., the forget set, from a trained model. The gold standard for unlearning is to produce the model that would have been learned on only the rest of the training data, i.e., the retain set. Most existing unlearning methods rely on direct access to the retained data, which may not be practical due to privacy or cost constraints. We propose WIN-U, a retained-data free unlearning framework that requires only second order information for the originally trained model on the full data. The unlearning is performed using a single Newton-style step. Using the Woodbury matrix identity and a generalized Gauss-Newton approximation for the forget set curvature, the WIN-U update recovers the closed-form linear solution and serves as a local second-order approximation to the gold-standard retraining optimum. Extensive experiments on various vision and language benchmarks demonstrate that WIN-U achieves SOTA performance in terms of unlearning efficacy and utility preservation, while being more robust against relearning attacks compared to existing methods. Importantly, WIN-U does not require access to the retained data.

Xingjian Zhao, Mohammad Mohammadi Amiri, Malik Magdon-Ismail

Department of Computer Science, Rensselaer Polytechnic Institute, Troy, NY 12180, USA

Machine UnlearningLLM Unlearning
AskPDF
Sep 1, 2026

Confess What You Know: Forget-Set Misalignment with Model Knowledge in LLM Unlearning

Machine unlearning for large language models (LLMs) often assumes that a pre-defined forget set matches what the model has memorized, but this frequently breaks in realistic privacy settings where the original training data is inaccessible. We term this gap forget-set misalignment and identify two cases. In Under Unlearning, the forget set omits memorized information and leakage persists. In Out-of-Knowledge Unlearning, the algorithm is driven to "forget" knowledge the model never learned, perturbing parameters and degrading utility. Using gradient-level analysis, we show these behaviors arise from misaligned unlearning targets rather than specific optimization choices. We then propose CONfession-to-Forget-Set (CONFS), a data-blind framework that constructs model-aligned forget sets by eliciting and formalizing the model's memorized knowledge. Across synthetic, multimodal, and real-world benchmarks, CONFS approaches Gold-standard performance on several metrics and achieves a competitive forgetting-utility balance, while preserving utility better than other data-blind forget-set constructions.

Miso Kim, Georu Lee, Seungwon Jeong +1

Dongguk University-Seoul

Privacy Leakage in Language ModelsMachine UnlearningLLM Unlearning
AskPDF