Machine Unlearning in Low-Dimensional Feature Subspace
Authors: Kun Fang, Qinghua Tao, Junxu Liu, Yaxin Xiao, Yulin Jin, Qingqing Ye, Jian Sun, Haibo Hu
Organizations: Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong, China · School of Automation, Beijing Institute of Technology, Beijing, China
Machine Unlearning (MU) aims at removing the influence of specific data from a pretrained model while preserving performance on the remaining data. In this work, a novel perspective for MU is presented upon low-dimensional feature subspaces, which gives rise to the potentials of separating the remaining and forgetting data herein. This separability motivates our LOFT, a method that proceeds unlearning in a LOw-dimensional FeaTure subspace from the pretrained model through principal projections, which are optimized to maximally capture the information of the remaining data and meanwhile diminish that of the forgetting data. During unlearning, LOFT simply optimizes a small-size projection matrix that can be flexibly inserted into the pretrained model, requiring just one-shot feature fetching from the pretrained model. LOFT demonstrates that effective and efficient MU can be achieved in low-dimensional feature subspaces even without repeated raw-data access or updates to the entire pretrained model. Extensive experiments validate the significantly lower computational overhead and superior unlearning performance of LOFT across diverse models, datasets, tasks, and applications. Code is publicly available.
Figures & tables
Figure 1: Overview of our proposed LOFT and mainstream approximate MU methods. After unlearning, the trained projection matrix of LOFT is incorporated into the model’s forward propagation during inference, with details in Section 4.2 .
Figure 2: Spectrum and reconstruction analyses on features of Dfg and Drm w.r.t. the pretrained fpre and the retrained fexact , with results on the top-12 normalized eigenvalues and reconstruction errors. Detailed setups and more empirical results for this analysis can be found in Appendix B .
Table 1: Comparison results of Swin-T on Tiny-ImageNet. The ✓ and ✗ indicate whether Drm and Dfg are involved in optimization. Avg.G. is average accuracy and MIA gap to the retrained model.
Table 2: Comparison on computational costs w.r.t. the results in Table 1 . The detailed counts and percentages of optimized parameters are listed. RTE is the run time efficiency in seconds on single NVIDIA GeForce RTX 4090 GPU. For LOFT, RTE measures only the optimization time of ( 1 ), as training features can be pre-calculated in practice and covariances are readily available for on-demand use. Refer to Table S1 in Appendix D for a detailed analysis on the full pipeline of LOFT.
Table 3: Comparison results under single-class unlearning with ResNet50 on ImageNet-1K.
Table 4: Comparison results under instance unlearning with Swin-T on Tiny-ImageNet.
objective
Accrmtr
Accfgtr
Accrmte
Accfgte
MIA
None
97.11
98.10
71.61
65.00
86.10
Jfg only
89.51
11.50
65.40
4.00
0.00
Jrm only
99.19
87.80
75.07
53.00
58.35
Jrm+Jfg
98.66
1.10
74.27
0.00
0.00
Table 5: Ablation study on Jrm and Jfg .
position
d
s
steps
layer1
4,096
500
500
layer2
2,048
350
200
layer3
1,024
9
200
layer4
512
9
50
Table 6: Hyper-parameters in different positions for LOFT.
Figure 3: A sensitivity analysis on (i) subspace dimensions s ( left ) and (ii) different positions of the projection matrix U ( right ).
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1: Spectrum and reconstruction analyses on features of Dfg and Drm using the pretrained fpre and the retrained fexact . The results are presented as top-12 normalized eigenvalues and reconstruction errors. Rows 1 & 2: Eigenvalues for fpre and fexact , respectively. Rows 3 & 4: Reconstruction errors for fpre and fexact , respectively. Columns 1-4: Dfg including images from labels 1,3,5,7, respectively. Column 5: Dfg comprising images randomly sampled from all classes.
method
offline one-shot feature fetching
on-demand unlearning
calculating covariances
optimizing U
LOFT
11.82
1.05
0.24
Appendix
Table S1: A detailed analysis on the time cost of the whole pipeline of LOFT. Empirical setups follow the same as that of Tables 1 and 2 .
Table S2: Training hyper-parameters of different MU methods w.r.t. results of multi-class unlearning in Table 1 with Swin-T on Tiny-ImageNet.
Table S3: Training hyper-parameters of different MU methods w.r.t. results of single-class unlearning in Table 3 with ResNet50 on ImageNet-1K.
Table S4: Training hyper-parameters of different MU methods w.r.t. results of instance unlearning in Table 4 with Swin-T on Tiny-ImageNet.
Figure S2: Unlearning performance w.r.t. varied values of the loss coefficient ρ . Empirical setups follow the same as that of Table 1 .
method
Drm
Dfg
hyper-parameters
fpre
20 epochs, lr =10−4 , cosine scheduler
fexact
20 epochs, lr =10−4 , cosine scheduler
GA
constant scheduler
GA
constant scheduler
RL
5 epochs, lr =5×10−7 , cosine scheduler
RL
10 epochs, lr =2×10−5 , cosine scheduler
Appendix
Table S5: Training hyper-parameters of different MU methods under extreme unlearning with Swin-T on Tiny-ImageNet.
Table S6: Comparison on performance under extreme unlearning with Swin-T on Tiny-ImageNet.
Table S7: Comparison results of Swin-T on Tiny-ImageNet under continual unlearning.
method
Drm
Dfg
hyper-parameters
fpre
200 epochs, lr =10−1 , cosine scheduler
fexact
200 epochs, lr =10−1 , cosine scheduler
GA
5 epochs, lr =5×10−4 , constant scheduler
GA
5 epochs, lr =5×10−4 , constant scheduler
RL
10 epochs, lr =5×10−4 , cosine scheduler
RL
10 epochs, lr =5×10−2 , cosine scheduler
Appendix
Table S8: Training hyper-parameters of different MU methods w.r.t. Table S9 with ResNet50 on VGGFace2.
Table S9: Comparison results of ResNet50 on VGGFace2, forgetting one of 200 identities.
method
Drm
Dfg
hyper-parameters
fpre
200 epochs, lr =10−1 , cosine scheduler
fexact
200 epochs, lr =10−1 , cosine scheduler
GA
10 epochs, lr =5×10−4 , constant scheduler
GA
10 epochs, lr =5×10−3 , constant scheduler
RL
20 epochs, lr =5×10−4 , cosine scheduler
RL
10 epochs, lr =5×10−2 , cosine scheduler
Appendix
Table S10: Training hyper-parameters of different MU methods w.r.t. Table S11 with ResNet18 on RAF-DB.
Table S11: Comparison results of ResNet18 on RAF-DB.
Figure S3: Results against two advanced MIAs, LiRA and RMIA. Empirical setups follow the same as that of Table 1 .
method
cs → en (rm)
lt → en (fg)
fully-trained
27.63
17.54
retrained
26.19
12.31
GA
23.51 ( 2.68 )
15.79 ( 3.48 )
LOFT
22.88 ( 3.31 )
13.04 ( 0.73 )
Appendix
Table S12: Results of LOFT on machine translation with a modern large model of m2m100_1.2B .
Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.
While numerous machine unlearning (MU) methods have recently been developed with promising results in erasing the influence of forgotten data, classes, or concepts, they are also highly vulnerable-for example, simple fine-tuning can inadvertently reintroduce erased concepts. In this paper, we address this contradiction by examining the internal representations of unlearned models, in contrast to prior work that focuses primarily on output-level behavior. Our analysis shows that many state-of-the-art MU methods appear successful mainly due to a misalignment between last-layer features and the classifier, a phenomenon we call feature-classifier misalignment. In fact, hidden features remain highly discriminative, and simple linear probing can recover near-original accuracy. Assuming neural collapse in the original model, we further demonstrate that adjusting only the classifier can achieve negligible forget accuracy while preserving retain accuracy, and we corroborate this with experiments using classifier-only fine-tuning. Motivated by these findings, we propose MU methods based on a class-mean features (CMF) classifier, which explicitly enforces alignment between features and classifiers. Experiments on standard benchmarks show that CMF-based unlearning reduces forgotten information in representations while maintaining high retain accuracy, highlighting the need for faithful representation-level evaluation of MU.
Yichen Gao, Altay Unal, Akshay Rangamani +1
Department of Computer Science & Engineering, The Ohio State University · Department of Data Science, New Jersey Institute of Technology
Continual machine unlearning aims to remove the influence of data that should no longer be retained, while preserving the usefulness of the model on everything else. This setting becomes especially difficult when deletion requests arrive sequentially, because the model must repeatedly adapt without erasing previously retained knowledge. Low-Rank Adaptation (LoRA) offers an efficient way to implement such updates, but naively combining many sequential LoRA modules leads to parameter collision, causing strong interference between tasks. We propose a static alternative based on Singular Value Decomposition (SVD)-guided orthogonal subspace projection. Our method constrains each new LoRA update during training so that it lies in the orthogonal complement of the subspaces used by earlier unlearning tasks. This preserves task isolation without requiring dynamic routing at deployment. Experiments on CIFAR-100 with ResNet-20 and on MNIST show stable behavior across long sequences of unlearning tasks. After thirty sequential unlearning tasks, state-of-the-art static fusion reduces retained accuracy from 60.39% to 12.70%, whereas the proposed in-training constrained optimization maintains baseline performance (∼58.1%) while preserving strong unlearning efficacy.
Yogachandran Rahulamathavan, Nasir Iqbal, Juncheng Hu +1
Institute for Digital Technologies, Loughborough University, UK · School of Artificial Intelligent Medicine, Guilin Medical University, Guilin 541199, CN
Machine unlearning is an emerging domain that ensures the safe removal of elements (includes concepts, attributes, entity and class) from the trained model along with least drop in model performance. The domain of machine unlearning brings its own indigenous challenges since the removal of pre-trained elements from model will always degrade the model performance on remaining elements. The existing methods basically rely on retraining for removal of elements from the pre-trained model, which is compute extensive. In this work, we propose a machine unlearning method which helps to reduce the computational requirement for faster retain-dataset accuracy convergence which also does not require extensive retraining of the pre-trained model. The proposed method, Data-Frugal Machine Unlearning (DFMU) requires only a single forward and backward pass for computing the importance score of various computational blocks of a model. The importance score computation is based on knowledge preserving pruning which helps to converge faster and requires far less data as compared to the existing methods. Experimentally, it achieves 40% more retain-accuracy with just 13% of data samples in comparison with SOTA method on various public datasets and also averages 88% faster processing time for forgetting a given class.