Machine Unlearning in Low-Dimensional Feature Subspace

Kun Fang; Qinghua Tao; Junxu Liu; Yaxin Xiao; Yulin Jin; Qingqing Ye; Jian Sun; Haibo Hu

cs.LGJan 30, 2026

Machine Unlearning in Low-Dimensional Feature Subspace

Authors: Kun Fang, Qinghua Tao, Junxu Liu, Yaxin Xiao, Yulin Jin, Qingqing Ye, Jian Sun, Haibo Hu

Organizations: Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong, China · School of Automation, Beijing Institute of Technology, Beijing, China

Abstract

Machine Unlearning (MU) aims at removing the influence of specific data from a pretrained model while preserving performance on the remaining data. In this work, a novel perspective for MU is presented upon low-dimensional feature subspaces, which gives rise to the potentials of separating the remaining and forgetting data herein. This separability motivates our LOFT, a method that proceeds unlearning in a LOw-dimensional FeaTure subspace from the pretrained model through principal projections, which are optimized to maximally capture the information of the remaining data and meanwhile diminish that of the forgetting data. During unlearning, LOFT simply optimizes a small-size projection matrix that can be flexibly inserted into the pretrained model, requiring just one-shot feature fetching from the pretrained model. LOFT demonstrates that effective and efficient MU can be achieved in low-dimensional feature subspaces even without repeated raw-data access or updates to the entire pretrained model. Extensive experiments validate the significantly lower computational overhead and superior unlearning performance of LOFT across diverse models, datasets, tasks, and applications. Code is publicly available.

Figures & tables

Figure 1: Overview of our proposed LOFT and mainstream approximate MU methods. After unlearning, the trained projection matrix of LOFT is incorporated into the model’s forward propagation during inference, with details in Section 4.2 .

Figure 2: Spectrum and reconstruction analyses on features of $\mathcal{D}_{\rm fg}$ and $\mathcal{D}_{\rm rm}$ w.r.t. the pretrained $f_{\rm pre}$ and the retrained $f_{\rm exact}$ , with results on the top-12 normalized eigenvalues and reconstruction errors. Detailed setups and more empirical results for this analysis can be found in Appendix B .

Panel of Table 1 — Table 1: Comparison results of Swin-T on Tiny-ImageNet. The $\checkmark$ and ✗ indicate whether $\mathcal{D}_{\rm rm}$ and $\mathcal{D}_{\rm fg}$ are involved in optimization. Avg.G. is average accuracy and MIA gap to the retrained model.

Panel of Table 2 — Table 2: Comparison on computational costs w.r.t. the results in Table 1 . The detailed counts and percentages of optimized parameters are listed. RTE is the run time efficiency in seconds on single NVIDIA GeForce RTX 4090 GPU. For LOFT, RTE measures only the optimization time of ( 1 ), as training features can be pre-calculated in practice and covariances are readily available for on-demand use. Refer to Table S1 in Appendix D for a detailed analysis on the full pipeline of LOFT.

Panel of Table 3 — Table 3: Comparison results under single-class unlearning with ResNet50 on ImageNet-1K.

Panel of Table 4 — Table 4: Comparison results under instance unlearning with Swin-T on Tiny-ImageNet.

objective	$\rm\bf Acc_{rm}^{tr}$	$\rm\bf Acc_{fg}^{tr}$	$\rm\bf Acc_{rm}^{te}$	$\rm\bf Acc_{fg}^{te}$	MIA
None	97.11	98.10	71.61	65.00	86.10
${J}_{\rm fg}$ only	89.51	11.50	65.40	4.00	0.00
${J}_{\rm rm}$ only	99.19	87.80	75.07	53.00	58.35
${J}_{\rm rm}+{J}_{\rm fg}$	98.66	1.10	74.27	0.00	0.00

Table 5: Ablation study on

{J}_{\rm rm}

and

{J}_{\rm fg}

position	$d$	$s$	steps
layer1	4,096	500	500
layer2	2,048	350	200
layer3	1,024	9	200
layer4	512	9	50

Table 6: Hyper-parameters in different positions for LOFT.

Figure 3: A sensitivity analysis on (i) subspace dimensions

s

( left ) and (ii) different positions of the projection matrix

\bf U

( right ).

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1: Spectrum and reconstruction analyses on features of

\mathcal{D}_{\rm fg}

and

\mathcal{D}_{\rm rm}

using the pretrained

f_{\rm pre}

and the retrained

f_{\rm exact}

. The results are presented as top-12 normalized eigenvalues and reconstruction errors. Rows 1 & 2: Eigenvalues for

f_{\rm pre}

and

f_{\rm exact}

, respectively. Rows 3 & 4: Reconstruction errors for

f_{\rm pre}

and

f_{\rm exact}

, respectively. Columns 1-4:

\mathcal{D}_{\rm fg}

including images from labels 1,3,5,7, respectively. Column 5:

\mathcal{D}_{\rm fg}

comprising images randomly sampled from all classes.

method	offline one-shot feature fetching	on-demand unlearning
		calculating covariances	optimizing $\bf U$
LOFT	11.82	1.05	0.24

Appendix

Table S1: A detailed analysis on the time cost of the whole pipeline of LOFT. Empirical setups follow the same as that of Tables 1 and 2 .

Panel of Table 12 — Table S2: Training hyper-parameters of different MU methods w.r.t. results of multi-class unlearning in Table 1 with Swin-T on Tiny-ImageNet.

Panel of Table 13 — Table S3: Training hyper-parameters of different MU methods w.r.t. results of single-class unlearning in Table 3 with ResNet50 on ImageNet-1K.

Panel of Table 14 — Table S4: Training hyper-parameters of different MU methods w.r.t. results of instance unlearning in Table 4 with Swin-T on Tiny-ImageNet.

Figure S2: Unlearning performance w.r.t. varied values of the loss coefficient

\rho

. Empirical setups follow the same as that of Table 1 .

method	$\mathcal{D}_{\rm rm}$	$\mathcal{D}_{\rm fg}$	hyper-parameters
$f_{\rm pre}$			20 epochs, lr $\rm={10}^{-4}$ , cosine scheduler
$f_{\rm exact}$			20 epochs, lr $\rm={10}^{-4}$ , cosine scheduler
GA			constant scheduler
GA			constant scheduler
RL			5 epochs, lr $\rm=5\times{10}^{-7}$ , cosine scheduler
RL			10 epochs, lr $\rm=2\times{10}^{-5}$ , cosine scheduler

Appendix

Table S5: Training hyper-parameters of different MU methods under extreme unlearning with Swin-T on Tiny-ImageNet.

Panel of Table 17 — Table S6: Comparison on performance under extreme unlearning with Swin-T on Tiny-ImageNet.

Panel of Table 18 — Table S7: Comparison results of Swin-T on Tiny-ImageNet under continual unlearning.

method	$\mathcal{D}_{\rm rm}$	$\mathcal{D}_{\rm fg}$	hyper-parameters
$f_{\rm pre}$			200 epochs, lr $\rm={10}^{-1}$ , cosine scheduler
$f_{\rm exact}$			200 epochs, lr $\rm={10}^{-1}$ , cosine scheduler
GA			5 epochs, lr $\rm=5\times{10}^{-4}$ , constant scheduler
GA			5 epochs, lr $\rm=5\times{10}^{-4}$ , constant scheduler
RL			10 epochs, lr $\rm=5\times{10}^{-4}$ , cosine scheduler
RL			10 epochs, lr $\rm=5\times{10}^{-2}$ , cosine scheduler

Appendix

Table S8: Training hyper-parameters of different MU methods w.r.t. Table S9 with ResNet50 on VGGFace2.

Panel of Table 20 — Table S9: Comparison results of ResNet50 on VGGFace2, forgetting one of 200 identities.

method	$\mathcal{D}_{\rm rm}$	$\mathcal{D}_{\rm fg}$	hyper-parameters
$f_{\rm pre}$			200 epochs, lr $\rm={10}^{-1}$ , cosine scheduler
$f_{\rm exact}$			200 epochs, lr $\rm={10}^{-1}$ , cosine scheduler
GA			10 epochs, lr $\rm=5\times{10}^{-4}$ , constant scheduler
GA			10 epochs, lr $\rm=5\times{10}^{-3}$ , constant scheduler
RL			20 epochs, lr $\rm=5\times{10}^{-4}$ , cosine scheduler
RL			10 epochs, lr $\rm=5\times{10}^{-2}$ , cosine scheduler

Appendix

Table S10: Training hyper-parameters of different MU methods w.r.t. Table S11 with ResNet18 on RAF-DB.

Panel of Table 22 — Table S11: Comparison results of ResNet18 on RAF-DB.

Figure S3: Results against two advanced MIAs, LiRA and RMIA. Empirical setups follow the same as that of Table 1 .

method	cs $\rightarrow$ en (rm)	lt $\rightarrow$ en (fg)
fully-trained	27.63	17.54
retrained	26.19	12.31
GA	23.51 ( 2.68 )	15.79 ( 3.48 )
LOFT	22.88 ( 3.31 )	13.04 ( 0.73 )

Appendix

Table S12: Results of LOFT on machine translation with a modern large model of m2m100_1.2B .

Machine Unlearning

Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.

See all 354 papers

An Illusion of Unlearning? Assessing Machine Unlearning Through Internal Representations
Apr 9, 2026Yichen Gao, Altay Unal, Akshay Rangamani +1Machine Unlearning Machine Unlearning Evaluation
Department of Computer Science & Engineering, The Ohio State University · Department of Data Science, New Jersey Institute of Technology
Orthogonal Subspace Projection for Continual Machine Unlearning via SVD-Based LoRA
Apr 14, 2026Yogachandran Rahulamathavan, Nasir Iqbal, Juncheng Hu +1Machine Unlearning Low-Rank Adaptation
Institute for Digital Technologies, Loughborough University, UK · School of Artificial Intelligent Medicine, Guilin Medical University, Guilin 541199, CN
DFMU: Data-Frugal Machine Unlearning
Jun 24, 2026Sajith U, Prateek KeserwaniMachine Unlearning
Samsung R&D Institute India-Bangalore, India

Apr 9, 2026J/K move · Enter open · S save

An Illusion of Unlearning? Assessing Machine Unlearning Through Internal Representations

Yichen Gao, Altay Unal, Akshay Rangamani, Zhihui Zhu

Department of Computer Science & Engineering, The Ohio State University · Department of Data Science, New Jersey Institute of Technology

Machine Unlearning Machine Unlearning Evaluation

Open paper AskPDF arXiv

While numerous machine unlearning (MU) methods have recently been developed with promising results in erasing the influence of forgotten data, classes, or concepts, they are also highly vulnerable-for example, simple fine-tuning can inadvertently reintroduce erased concepts. In this paper, we address this contradiction by examining the internal representations of unlearned models, in contrast to prior work that focuses primarily on output-level behavior. Our analysis shows that many state-of-the-art MU methods appear successful mainly due to a misalignment between last-layer features and the classifier, a phenomenon we call feature-classifier misalignment. In fact, hidden features remain highly discriminative, and simple linear probing can recover near-original accuracy. Assuming neural collapse in the original model, we further demonstrate that adjusting only the classifier can achieve negligible forget accuracy while preserving retain accuracy, and we corroborate this with experiments using classifier-only fine-tuning. Motivated by these findings, we propose MU methods based on a class-mean features (CMF) classifier, which explicitly enforces alignment between features and classifiers. Experiments on standard benchmarks show that CMF-based unlearning reduces forgotten information in representations while maintaining high retain accuracy, highlighting the need for faithful representation-level evaluation of MU.

Abstract

Machine Unlearning

Explore similar work

An Illusion of Unlearning? Assessing Machine Unlearning Through Internal Representations

Orthogonal Subspace Projection for Continual Machine Unlearning via SVD-Based LoRA

DFMU: Data-Frugal Machine Unlearning