DISCERN: Distributional Synthetic Replay for Retain-Free Incremental Machine Unlearning
Organizations: Department of Computer Science and Artificial Intelligence (DECSAI), Andalusian Institute of Data Science and Computational Intelligence (DaSCI), University of Granada, Granada, Spain
Abstract
Machine unlearning aims to remove the influence of specific training data from a trained model without retraining from scratch. In real deployments, however, deletion requests arrive over time, calling for incremental class unlearning (ICU): classes are removed sequentially, previously forgotten knowledge must remain forgotten, and the remaining classes must be preserved. We present DIstributional SynthetiC rEplay for Retain-free iNcremental unlearning (DISCERN), an intervention-based class unlearning method that replaces access to the retain set with synthetic embeddings sampled from per-class parametric distributions. Unlike prototype-based replay, which assumes an isotropic Gaussian around each class centroid, DISCERN fits Euclidean (Gaussian, Laplace, Student's ) and directional (von Mises--Fisher, Fisher--Bingham) families to each class and selects among them through statistical tests and, when none of them accepts, cross-validated log-likelihood. This design is motivated by the architecture-dependent latent geometry: transformer embeddings lie approximately on a hypersphere and concentrate around class directions, unlike those of convolutional networks. We evaluate DISCERN against seven retain-free baselines on MNIST, CIFAR-10, and CIFAR-100 with two convolutional and two transformer backbones, under single-class, multi-class, and incremental unlearning. DISCERN is competitive when classes are removed at once and is the most stable method in ICU on transformers: on ViT-B/16, after 20 sequential deletions it retains a harmonic mean of , whereas the strongest method in the simultaneous setting drops from to . A paired ablation shows that per-class distribution selection substantially reduces the resurfacing of previously forgotten classes, acting as a stabilizer of accumulated forgetting rather than a uniform improvement.
Figures & tables
| Distribution | Directional | Rationale |
|---|---|---|
| Gaussian, [ 22 ] | ✗ | Prototypes [ 39 ] are, a priori, ideal representatives of a class; all elements of the class may be distributed as noise around that prototype. It serves as a light-tailed elliptical reference. |
| Student’s , [ 24 ] | ✗ | If the embeddings of a class contain outliers with respect to a Gaussian that are still recognized as members of the class, the Student’s distribution can capture them. Its heavier tails provide greater robustness to observations far from the center. |
| Laplace, [ 23 ] | ✗ | The symmetric multivariate Laplace distribution is a Gaussian scale mixture with exponential mixing weights. Compared with the Gaussian, it places more mass both very close to the prototype and far from it, with exponential tails that are heavier than the Gaussian ones but lighter than the polynomial tails of the Student’s . It suits classes in which most embeddings are tightly clustered around the prototype while a minority lie farther away, a behavior intermediate between the Gaussian and the Student’s . |
| von Mises–Fisher, [ 34 ] | ✓ | Distribution on the unit hypersphere whose density decreases with the angle to a mean direction , with a single concentration parameter and hence the same dispersion in every direction around . Directional latent spaces have proven effective in high-dimensional models; for instance, Xu and Durrett [ 47 ] use von Mises–Fisher distributions on a hyperspherical latent space to stabilize variational autoencoders. It models representations whose information is mainly expressed by their direction. |
| Fisher–Bingham, [ 34 ] | ✓ | Generalization of the von Mises–Fisher distribution that allows anisotropic concentration around the mean direction, so that the dispersion on the hypersphere may vary with the direction. In high dimension we use a tangent-plane approximation (Section 3.2 ). |
| Method | Intervention | Description |
|---|---|---|
| Boundary Shrink [ 9 ] | ✗ | Reassigns each sample of the class to be forgotten to its nearest neighboring class, identified through an adversarial attack, and fine-tunes the model, thereby shrinking the decision boundary of that class. |
| Boundary Expand [ 9 ] | ✗ | Extends the classifier with an auxiliary class to which the samples to be forgotten are reassigned; after fine-tuning, the added neuron is removed, dispersing the knowledge of the target class. |
| Bad Teacher [ 10 ] | ✗ | Distillation with two teachers: the original model guides the knowledge to be preserved, while an incompetent (randomly initialized) teacher guides the response on the samples to be forgotten. |
| DELETE [ 51 ] | ✗ | Decouples the teacher logits into target-class and non-target knowledge, masks the forgotten classes, and transfers only the residual knowledge to the student. |
| EWC [ 49 ] | ✓ | Applies a forgetting-promoting loss on the data to be forgotten, regularized with the Fisher information matrix, which penalizes changes to parameters relevant to the retained knowledge. |
| LF-SSD [ 17 ] | ✓ | Estimates the importance of each parameter without labels, from the model outputs, and selectively dampens those that are disproportionately important for the forget set. |
| Architecture | CIFAR-10 | CIFAR-100 | MNIST |
|---|---|---|---|
| ResNet18 | Gaussian (6/10) Student’s (4/10) | Gaussian (68/100), (24/100) (6/100), Laplace (2/100) | Gaussian (9/10) (1/10) |
| EfficientNetV2-S | Gaussian (10/10) | (93/100), (5/100) Gaussian (2/100) | Gaussian (10/10) |
| Swin-T | (10/10) | (100/100) | (10/10) |
| ViT-B/16 | (10/10) | (100/100) | (10/10) |
| Methods | CIFAR-10 | CIFAR-100 | MNIST | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Original | |||||||||
| Retrain | |||||||||
| Shrink | .8507474747474747 | ||||||||
| Expand | .9673 | ||||||||
| BT | |||||||||
| Methods | CIFAR-10 | CIFAR-100 | MNIST | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Original | |||||||||
| Retrain | |||||||||
| Shrink | 0.9961111111 | ||||||||
| Expand | 0.9724444444 | 0.8588787879 | 0.9961000000 | ||||||
| BT | 0.9961111111 | ||||||||
| Methods | CIFAR-10 | CIFAR-100 | MNIST | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Original | |||||||||
| Retrain | |||||||||
| Shrink | |||||||||
| Expand | |||||||||
| BT | |||||||||
| Methods | CIFAR-10 | CIFAR-100 | MNIST | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Original | |||||||||
| Retrain | |||||||||
| Shrink | |||||||||
| Expand | 0.9545000000 | ||||||||
| BT | |||||||||
| Methods | 5 classes | 10 classes | 20 classes | 50 classes | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Original | ||||||||||||
| Retrain | ||||||||||||
| Shrink | ||||||||||||
| Expand | .8513894736842106 | .8531555555555557 | ||||||||||
| BT | ||||||||||||
| Methods | 5 classes | 10 classes | 20 classes | 50 classes | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Original | ||||||||||||
| Retrain | ||||||||||||
| Shrink | ||||||||||||
| Expand | 0.8612105263 | 0.8626222222 | 0.8646625000 | |||||||||
| BT | ||||||||||||
| Methods | 5 classes | 10 classes | 20 classes | 50 classes | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Original | ||||||||||||
| Retrain | ||||||||||||
| Shrink | ||||||||||||
| Expand | ||||||||||||
| BT | ||||||||||||
| Methods | 5 classes | 10 classes | 20 classes | 50 classes | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Original | ||||||||||||
| Retrain | ||||||||||||
| Shrink | ||||||||||||
| Expand | 0.8257684211 | 0.8365555556 | 0.8502750000 | 0.8612400000 | ||||||||
| BT | ||||||||||||
| Methods | Step 5 | Step 10 | Step 15 | Step 20 | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Retrain | ||||||||||||
| Shrink | .0008000000 | |||||||||||
| Expand | .8518736720 | .8548888981 | .8562000096 | .8582125425 | ||||||||
| BT | ||||||||||||
| DELETE | ||||||||||||
| Methods | Step 5 | Step 10 | Step 15 | Step 20 | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Retrain | ||||||||||||
| Shrink | ||||||||||||
| Expand | 0.8616105020 | 0.8645666778 | 0.8666470885 | 0.8699500501 | ||||||||
| BT | ||||||||||||
| DELETE | 0.9036979808 | 0.0001333333 | 0.0000500000 | |||||||||
| Methods | Step 5 | Step 10 | Step 15 | Step 20 | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Retrain | ||||||||||||
| Shrink | ||||||||||||
| Expand | .7198105156 | .6922222316 | .7047529578 | .6907500207 | ||||||||
| BT | ||||||||||||
| DELETE | ||||||||||||
| Methods | Step 5 | Step 10 | Step 15 | Step 20 | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Retrain | ||||||||||||
| Shrink | ||||||||||||
| Expand | 0.7766315699 | 0.7366222382 | 0.7539647222 | 0.7317250311 | ||||||||
| BT | 0.0061000002 | 0.0078666665 | ||||||||||
| DELETE | 0.0055500003 | |||||||||||
| Scenario | Loss | DISCERN vs. Gaussian | DISCERN vs. CV-only | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| D/G | D/CV-only | |||||||||
| Single class | notop | 160 | 83/66 | 73/50 | ||||||
| confidence | 160 | 47/113 | 97/55 | |||||||
| Multiple classes | notop | 160 | 88/71 | 70/84 | ||||||
| confidence | 160 | 70/90 | 114/46 | |||||||
| ICU | notop | 40 | 25/15 | 33/7 | ||||||
| Architecture | Comparison | ||||||
|---|---|---|---|---|---|---|---|
| Single class | Multiple classes | ICU | |||||
| notop | conf. | notop | conf. | notop | conf. | ||
| ViT-B/16 | vs. G | ||||||
| vs. CV-only | |||||||
| Swin-T | vs. G | ||||||
| vs. CV-only | |||||||
| Architecture | CIFAR-10 | CIFAR-100 | MNIST |
|---|---|---|---|
| ResNet18 | Student’s (10/10) | Gaussian (74/100), (24/100) Laplace (2/100) | Student’s (8/10) Laplace (1/10), (1/10) |
| EfficientNetV2-S | Student’s (10/10) | Gaussian (58/100), (42/100) | Student’s (10/10) |
| Swin-T | Laplace (8/10) Student’s (2/10) | (75/100), Gaussian (25/100) | Laplace (10/10) |
| ViT-B/16 | Student’s (10/10) | (81/100), Gaussian (19/100) | Laplace (6/10) Student’s (4/10) |
| Method | s / class | min / class | Faster than Retrain | One-off cost |
|---|---|---|---|---|
| EWC | 46 | 0.8 | 120 | MAS coefficients, 17 min |
| RL | 54 | 0.9 | 102 | – |
| DELETE | 56 | 0.9 | 98 | – |
| DISCERN (notop) | 69 | 1.1 | 80 | distribution fitting, 4.6 min † |
| BT | 118 | 2.0 | 47 | – |
| LF-SSD | 123 | 2.1 | 45 | MAS coefficients, 17 min |
| Block | Quantity | Metric | vs. Original | vs. Retrain | ||||
|---|---|---|---|---|---|---|---|---|
| Concentrates | Disperses | No trend | Concentrates | Disperses | No trend | |||
| Forget | Cohesion | L2 | 6 | 3 | 3 | 9 | 2 | 1 |
| Cos | 2 | 9 | 1 | 6 | 3 | 3 | ||
| Separation | L2 | 11 | 0 | 1 | 5 | 4 | 3 | |
| Cos | 10 | 2 | 0 | 0 | 10 | 2 | ||
| Retain | Cohesion | L2 | 8 | 2 | 2 | 3 | 4 | 5 |
| Family | Dataset | Cohesion | Separation | ||
|---|---|---|---|---|---|
| L2 | Cos | L2 | Cos | ||
| Conv. | CIFAR-10 | – | |||
| CIFAR-100 | |||||
| MNIST | |||||
| Transf. | CIFAR-10 | – | |||
| CIFAR-100 | |||||
| Loss | Architecture | Forget | Retain | Unknown | |||
|---|---|---|---|---|---|---|---|
| Coh. | Sep. | Coh. | Sep. | Coh. | Sep. | ||
| notop | ResNet18 | 12/12 | 12/12 | 10/12 | 12/12 | 10/12 | 10/12 |
| EfficientNetV2-S | 11/12 | 10/12 | 8/9 | 12/12 | 8/12 | 12/12 | |
| Swin-T | 9/12 | 9/12 | 9/11 | 12/12 | 11/12 | 9/12 | |
| ViT-B/16 | 10/12 | 9/12 | 11/12 | 12/12 | 12/12 | 12/12 | |
| confidence | ResNet18 | 4/12 | 6/12 | 7/12 | 10/12 | 8/12 | 8/12 |
Machine Unlearning
Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.
See all 372 papers