On the Plasticity Collapse in Continual Machine Unlearning
Authors: Yingdan Shi, Xiang Xu, Kaize Ding, Alfred O. Hero, Ren Wang
Organizations: Illinois Institute of Technology, Chicago IL 60616, USA · Amazon, New York NY 10001, USA · Northwestern University, Evanston IL 60208, USA · University of Michigan, Ann Arbor MI 48109, USA
Machine unlearning enables deep neural networks to selectively remove the influence of specific data in response to privacy and regulatory requirements. While prior work largely studies single-shot unlearning, real-world systems must accommodate continual unlearning, where multiple unlearning requests occur sequentially over time. In this work, we identify a fundamental limitation of this setting: plasticity collapse, a progressive breakdown in a model's ability to effectively forget. Through theoretical analysis of continual unlearning dynamics, we show that continual unlearning operations accumulate geometric constraints in parameter space, leading to saturated subspaces that restrict future updates. This structural effect induces two distinct failure modes: (1) Forward failure -- diminishing forgetting quality for subsequent tasks, and (2) Backward failure -- spontaneous re-memorization of previously forgotten information. Extensive experiments across multiple architectures, datasets, and methods in image classification confirm that plasticity collapse is not an artifact of specific implementations, but a pervasive phenomenon inherent to continual unlearning. Our findings reveal a critical barrier to the long-term reliability of machine unlearning systems and motivate the development of plasticity-preserving unlearning algorithms. Our code is available at https://github.com/TIML-Group/Continual-Machine-Unlearning-Plasticity-Collapse
Figures & tables
Figure 1: Comparison between single-shot unlearning and continual unlearning. Unlike the single-shot setting, continual unlearning suffers from the accumulation of geometric constraints, leading to a saturated subspace that induces plasticity collapse. This collapse is characterized by two distinct phenomena: forward failure, which represents a progressive decline in forgetting quality, and backward failure, referring to the spontaneous re-memorization of previously forgotten tasks.
Figure 2: Forgetting accuracy of various unlearning methods on Tiny-ImageNet with VGG-16-BN under the random data forgetting scenario. The solid line denotes the forgetting accuracy on each task, while the scatter points represent the forgetting accuracy for single-task unlearning. Lower accuracy indicates better forgetting quality. For all methods, the forgetting accuracy increases as the number of sequential unlearning tasks grows, suggesting a progressive degradation in forgetting quality.
Figure 3: Forgetting accuracy on Tiny-ImageNet with VGG-16-BN under the class-wise forgetting scenario. FT, NegGrad+ and MUNBa also show diminishing forgetting quality for subsequent tasks. RL and SalUn do not exhibit the clear forward failure in this setting, but we show its backward failure in Fig 5 .
Figure 4: Forgetting accuracy heatmaps on Tiny-ImageNet with VGG-16-BN for historical tasks on the random data forgetting scenario. Entry (i,j) records forgetting accuracy of task Fi on the model θj where j≥i .
Figure 5: Forgetting accuracy heatmaps on Tiny-ImageNet with VGG-16-BN for historical tasks on the class-wise forgetting scenario. RL and SalUn, which do not exhibit forward failure, show obvious backward failure in this setting.
Figure 6: L2 norm of parameter differences between consecutive updates on Tiny-ImageNet with VGG-16-BN under the random data forgetting scenario. It shows an obvious decrease in the magnitude of parameter updates as the number of tasks increases, indicating that the model becomes increasingly frozen in the parameter space.
Figure 7: L2 norm of parameter differences between consecutive updates on Tiny-ImageNet with VGG-16-BN under the class-wise forgetting scenario. For RL and SalUn, which exhibit backward failure, the tasks where re-memorization occurs show lower L2 norm values compared to other tasks.
Figure 8: Energy Ratio for forward failure diagnosis on Tiny-ImageNet with VGG-16-BN under the random data forgetting scenario.
Figure 9: Energy Ratio for forward failure diagnosis on Tiny-ImageNet with VGG-16-BN under the class-wise forgetting scenario.
Figure 10: Coefficient for backward failure diagnosis on Tiny-ImageNet with VGG-16-BN under the class-wise forgetting scenario.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 11: Retain accuracy of various unlearning methods on Tiny-ImageNet with VGG-16-BN under the random data forgetting scenario.
Figure 12: Retain accuracy on Tiny-ImageNet with VGG-16-BN under the class-wise forgetting scenario.
Figure 13: Test accuracy of various unlearning methods on Tiny-ImageNet with VGG-16-BN under the random data forgetting scenario.
Figure 14: Test accuracy on Tiny-ImageNet with VGG-16-BN under class-wise forgetting scenario.
Figure 15: Forgetting accuracy on CIFAR-100 with PreResNet-110 under the random data forgetting scenario.
Figure 16: Forgetting accuracy on CIFAR-100 with PreResNet-110 under the class-wise forgetting scenario.
Figure 17: Retain accuracy on CIFAR-100 with PreResNet-110 under the random data forgetting scenario.
Figure 18: Retain accuracy on CIFAR-100 with PreResNet-110 under the class-wise forgetting scenario.
Figure 19: Test accuracy on CIFAR-100 with PreResNet-110 under the random data forgetting scenario.
Figure 20: Test accuracy on CIFAR-100 with PreResNet-110 under class-wise forgetting scenario.
Figure 21: Forgetting accuracy heatmaps on CIFAR-100 with PreResNet-110 for historical tasks on the random data forgetting scenario.
Figure 22: Forgetting accuracy heatmaps on CIFAR-100 with PreResNet-110 for historical tasks on the class-wise forgetting scenario.
Figure 23: L2 norm of parameter differences between consecutive updates on CIFAR-100 with PreResNet-110 under the random data forgetting scenario.
Figure 24: L2 norm of parameter differences between consecutive updates on CIFAR-100 with PreResNet-110 under the class-wise forgetting scenario.
Figure 25: Energy Ratio for forward failure diagnosis on CIFAR-100 with PreResNet-110 under the random data forgetting scenario.
Figure 26: Energy Ratio or Coefficient for forward failure diagnosis on CIFAR-100 with PreResNet-110 under the class-wise forgetting scenario.
Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.