Unlearning Evaluation through Subset Statistical Independence

Chenhao Zhang; Muxing Li; Feng Liu; Weitong Chen; Miao Xu

cs.LGFeb 28, 2026

Unlearning Evaluation through Subset Statistical Independence

Authors: Chenhao Zhang, Muxing Li, Feng Liu, Weitong Chen, Miao Xu

Organizations: University of Queensland · University of Melbourne · RIKEN Center for Advanced Intelligence Project · Adelaide University

Abstract

Evaluating machine unlearning remains challenging, as existing methods typically require retraining reference models or performing membership inference attacks, both of which rely on prior access to training configuration or supervision labels, making them impractical in realistic scenarios. Motivated by the fact that most unlearning algorithms remove a small, random subset of the training data, we propose a subset-level evaluation framework based on statistical independence. Specifically, we design a tailored use of the Hilbert-Schmidt Independence Criterion to assess whether the model outputs on a given subset exhibit statistical dependence, without requiring model retraining or auxiliary classifiers. Our method provides a simple, standalone evaluation procedure that aligns with unlearning workflows. Extensive experiments demonstrate that our approach reliably distinguishes in-training from out-of-training subsets and clearly differentiates unlearning effectiveness, even when existing evaluations fall short.

Figures & tables

Figure 3: Effect of kernel bandwidth

\sigma

on F1 score. The solid lines show F1 score trends across different

\sigma

values. Vertical dashed lines indicate the heuristic of

\sigma=\sqrt{\rm dim}

, while dotted lines correspond to the median heuristic.

$R$	5%			10%			20%
$\|\mathcal{S}\|$	400	1000	2000	400	1000	2000	400	1000	2000
SV-AllCNN	0.62±0.14	0.74±0.08	0.83±0.09	0.50±0.21	0.62±0.06	0.73±0.12	0.72±0.03	0.82±0.08	0.90±0.06
C10-AllCNN	0.69±0.07	0.79±0.05	0.81±0.14	0.61±0.05	0.66±0.12	0.82±0.09	0.45±0.00	0.71±0.18	0.79±0.09
SV-ResNet18	0.71±0.09	0.91±0.01	0.93±0.07	0.63±0.10	0.78±0.09	0.85±0.14	0.90±0.03	0.96±0.02	0.97±0.05
C10-ResNet18	0.87±0.06	0.97±0.02	0.99±0.01	0.88±0.04	0.95±0.06	0.97±0.03	0.86±0.10	0.96±0.03	1.00±0.00
C100-ResNet18	0.99±0.01	1.00±0.00	1.00±0.00	0.97±0.05	1.00±0.00	1.00±0.00	0.99±0.01	1.00±0.00	1.00±0.00

Table 1: F1 score of distinguishing in/out-of-training status of target

\mathcal{S}

, across the amount of

\mathcal{D}_{\rm f}

and the size of

\mathcal{S}

$R$	5%			10%			20%
$\|\mathcal{S}\|$	400	1000	2000	400	1000	2000	400	1000	2000
MMD	0.63±0.07	0.65±0.09	0.87±0.12	0.45±0.21	0.70±0.13	0.87±0.14	0.63±0.03	0.72±0.08	0.89±0.11
Wasserstein	0.70±0.08	0.77±0.10	0.94±0.11	0.52±0.20	0.89±0.08	0.98±0.03	0.72±0.02	0.87±0.07	0.99±0.02
SDE (OURS)	0.87±0.06	0.97±0.02	0.99±0.01	0.88±0.04	0.95±0.06	0.97±0.03	0.86±0.10	0.96±0.03	1.00±0.00

Table 3: Comparison with distribution-based metrics

Method	$\rm Acc_{r}$ (%)	$\rm Acc_{f}$ (%)	ASR	$h$		$h_{p}$
Method	$\rm Acc_{r}$ (%)	$\rm Acc_{f}$ (%)	ASR	F1	OTR (%) $\uparrow$	F1	OTR (%) $\uparrow$
Retrain	98.57 $\pm$ 0.08	93.25 $\pm$ 0.45	0.30 $\pm$ 0.09	0.94 $\pm$ 0.03	87.00 $\pm$ 10.24	0.95 $\pm$ 0.05	94.00 $\pm$ 4.94
RandLabel	98.80 $\pm$ 0.04	98.63 $\pm$ 0.13	0.29 $\pm$ 0.02	0.88 $\pm$ 0.12	84.00 $\pm$ 13.54	0.91 $\pm$ 0.09	83.20 $\pm$ 10.11
Unroll	99.36 $\pm$ 0.05	99.21 $\pm$ 0.11	0.30 $\pm$ 0.12	0.88 $\pm$ 0.04	3.00 $\pm$ 2.19	0.90 $\pm$ 0.07	4.40 $\pm$ 4.03
Sparsity	92.72 $\pm$ 0.93	90.56 $\pm$ 0.82	0.42 $\pm$ 0.09	0.62 $\pm$ 0.15	50.80 $\pm$ 22.61	0.59 $\pm$ 0.16	53.80 $\pm$ 24.19
SalUn	98.66 $\pm$ 0.03	98.53 $\pm$ 0.07	0.29 $\pm$ 0.02	0.85 $\pm$ 0.12	52.40 $\pm$ 21.86	0.86 $\pm$ 0.15	51.80 $\pm$ 23.05

Table 4: Evaluating unlearned models on CIFAR10-ResNet18 with

R=10\%

\rm Acc_{r}

and

\rm Acc_{f}

denote unlearned models’ training accuracy on the

\mathcal{D}_{\rm r}

and

\mathcal{D}_{\rm f}

, respectively. ASR refers to the success rate of MIA. For these metrics, the closer to the retrained model’s the better. For our method, the higher OTR indicates more effective unlearning.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5:

H(S,h)

distribution changes along the training epoch. The dashed gray line is the

p=0.01

baseline.

Table 8

Data-Arch	Acc	10%	20%	40%	80%
SV-AllCNN	Train	88.09	95.73	98.51	99.92
SV-AllCNN	Test	88.16	92.97	92.54	94.26
C10-AllCNN	Train	92.77	99.21	100.00	100.00
C10-AllCNN	Test	90.49	92.07	94.76	94.83
SV-ResNet18	Train	77.52	84.92	89.22	98.65
SV-ResNet18	Test	71.42	78.12	80.06	91.49

Appendix

Table 7: Task utilities of models across training progress.

Figure 6: F1 score across

\sigma

on the diffusion model.

R	Method	$\rm Acc_{r}$ (%)	$\rm Acc_{f}$ (%)	ASR	$h$		$h_{p}$
R	Method	$\rm Acc_{r}$ (%)	$\rm Acc_{f}$ (%)	ASR	F1	OTR (%) $\uparrow$	F1	OTR (%) $\uparrow$
5%	Retrain	98.70 $\pm$ 0.03	93.66 $\pm$ 0.10	0.32 $\pm$ 0.12	0.85 $\pm$ 0.10	90.60 $\pm$ 10.84	0.92 $\pm$ 0.06	93.20 $\pm$ 7.57
	RandLabel	98.86 $\pm$ 0.03	98.64 $\pm$ 0.13	0.25 $\pm$ 0.03	0.88 $\pm$ 0.09	79.00 $\pm$ 26.26	0.90 $\pm$ 0.09	76.40 $\pm$ 28.08
	Unroll	99.38 $\pm$ 0.03	99.24 $\pm$ 0.22	0.28 $\pm$ 0.10	0.88 $\pm$ 0.08	0.80 $\pm$ 0.75	0.88 $\pm$ 0.09	1.00 $\pm$ 2.00
	Sparsity	93.75 $\pm$ 0.54	91.52 $\pm$ 0.38	0.44 $\pm$ 0.08	0.66 $\pm$ 0.29	62.00 $\pm$ 31.12	0.66 $\pm$ 0.31	62.20 $\pm$ 33.07
	SalUn	98.73 $\pm$ 0.02	98.54 $\pm$ 0.14	0.23 $\pm$ 0.02	0.84 $\pm$ 0.11	38.80 $\pm$ 21.31	0.87 $\pm$ 0.11	37.20 $\pm$ 26.61

Appendix

Table 8: Unlearned models’ results on CIFAR10-ResNet18 with

R\in\{5\%,10\%,20\%\}

\rm Acc_{r}

and

\rm Acc_{f}

denote unlearned models’ training accuracy on the

\mathcal{D}_{\rm r}

and

\mathcal{D}_{\rm f}

, respectively. ASR refers to the success rate of membership inference attacks. For these metrics, the closer to the retrained model’s the better. For our method, the higher OTR means that more forgetting data has been identified as out-of-training data, indicating more effective unlearning.

	Inference Time	$m=50$	$m=100$	$m=200$
$\|S\|=400$	0.29+	9.51	18.76	37.76
$\|S\|=1000$	0.72+	11.06	21.92	45.85
$\|S\|=2000$	1.43+	29.02	57.70	116.04

Appendix

Table 9: Wall-clock time cost (seconds) with different

m

and

|S|

when

d=512

	Inference Time	$m=50$	$m=100$	$m=200$
$\|S\|=400$	0.29+	13.46	26.55	52.35
$\|S\|=1000$	0.72+	27.15	54.18	107.99
$\|S\|=2000$	1.43+	84.84	168.30	335.67

Appendix

Table 10: Wall-clock time cost (seconds) with different

m

and

|S|

when

d=8192

Machine Unlearning Evaluation

Machine unlearning evaluation tests whether models remove requested data or knowledge while preserving privacy, retained capabilities, and robustness.

See all 101 papers

Auditing of Unlearning Algorithms
Jul 7, 2026Sahasrajit Sarmasarkar, Anastasia Koloskova, Sanmi KoyejoMachine Unlearning Machine Unlearning Evaluation
Stanford University · University of Zurich
Auditing Machine Unlearning: A Systematic Research on Whether Models Truly Forget
Jun 15, 2026Dayong Ye, Tianqing Zhu, Ruiding Huang +5Machine Unlearning LLM Unlearning
Easy Data Unlearning Bench
Feb 18, 2026Roy Rinberg, Pol Puigdemont, Martin Pawelczyk +1Machine Unlearning Machine Unlearning Evaluation
Harvard University · LIONS, EPFL, Lausanne, Switzerland.

Jul 7, 2026J/K move · Enter open · S save

Auditing of Unlearning Algorithms

Sahasrajit Sarmasarkar, Anastasia Koloskova, Sanmi Koyejo

Stanford University · University of Zurich

Machine Unlearning Machine Unlearning Evaluation

Open paper AskPDF arXiv

Evaluating whether unlearning algorithms truly remove training data influence remains an open challenge. We propose a practical auditor that computes data-dependent lower bounds on the unlearning parameter

\varepsilon

using membership inference attacks. Evaluating multiple unlearning algorithms, we find a sharp separation: algorithms with rigorous guarantees, such as model clipping and rewind-to-delete, achieve very small

\varepsilon

bounds that do not falsify their unlearning guarantees, whereas empirical methods such as Hessian-based unlearning, interleaved ascent-descent, ascent on the forget set, and fine-tuning on the retain set exhibit large bounds, indicating poor unlearning. Our auditor provides a practical tool for empirically falsifying unlearning claims through a hypothesis-testing framework, and we validate it on CIFAR-100 and Shakespeare text.

Abstract

Machine Unlearning Evaluation

Explore similar work

Auditing of Unlearning Algorithms

Auditing Machine Unlearning: A Systematic Research on Whether Models Truly Forget

Easy Data Unlearning Bench