Orthogonal Subspace Projection for Continual Machine Unlearning via SVD-Based LoRA

Yogachandran Rahulamathavan; Nasir Iqbal; Juncheng Hu; Sangarapillai Lambotharan

cs.LGApr 14, 2026

Orthogonal Subspace Projection for Continual Machine Unlearning via SVD-Based LoRA

Authors: Yogachandran Rahulamathavan, Nasir Iqbal, Juncheng Hu, Sangarapillai Lambotharan

Organizations: Institute for Digital Technologies, Loughborough University, UK · School of Artificial Intelligent Medicine, Guilin Medical University, Guilin 541199, CN

Abstract

Continual machine unlearning aims to remove the influence of data that should no longer be retained, while preserving the usefulness of the model on everything else. This setting becomes especially difficult when deletion requests arrive sequentially, because the model must repeatedly adapt without erasing previously retained knowledge. Low-Rank Adaptation (LoRA) offers an efficient way to implement such updates, but naively combining many sequential LoRA modules leads to parameter collision, causing strong interference between tasks. We propose a static alternative based on Singular Value Decomposition (SVD)-guided orthogonal subspace projection. Our method constrains each new LoRA update during training so that it lies in the orthogonal complement of the subspaces used by earlier unlearning tasks. This preserves task isolation without requiring dynamic routing at deployment. Experiments on CIFAR-100 with ResNet-20 and on MNIST show stable behavior across long sequences of unlearning tasks. After thirty sequential unlearning tasks, state-of-the-art static fusion reduces retained accuracy from 60.39% to 12.70%, whereas the proposed in-training constrained optimization maintains baseline performance ( $\sim$ 58.1%) while preserving strong unlearning efficacy.

Figures & tables

Fig. 1: High-level architecture comparision of the related works (top) and the proposed approach (bottom).

Tasks ( $N$ )	Method	$\lambda=0.5$		$\lambda=1.0$
Tasks ( $N$ )	Method	Retain Acc. ( $\uparrow$ )	Forget Acc. ( $\downarrow$ )	Retain Acc. ( $\uparrow$ )	Forget Acc. ( $\downarrow$ )
$N=5$	Static / AC-LoRA [ 8 ] , I-LoRA [ 9 ]	56.23%	5.60%	48.78%	0.00%
	MoE Router [ 10 ]	59.80%	51.20%	59.80%	51.20%
	In-Training Approach (Ours)	59.04%	37.60%	58.02%	33.80%
$N=10$	Static / AC-LoRA [ 8 ] , I-LoRA [ 9 ]	51.89%	13.80%	37.58%	1.20%
	MoE Router [ 10 ]	59.62%	57.00%	59.62%	57.00%

TABLE I: Continual Unlearning Performance on ResNet-20 (CIFAR-100). Colors indicate: Blue (Excellent Retain Preservation), Red (Catastrophic Retain Collapse), Green (Successful Unlearning), and Orange (Limited Unlearning Efficacy).

Fig. 2: Operational trade-off trajectories for continual unlearning by sequentially forgetting 3 classes 9, 5 and 3 ( $N=3$ ) on the MNIST dataset.

Fig. 3: Correlation between available orthogonal subspace dimensions and continual unlearning performance on CIFAR-100 (ResNet-20) for $\lambda=2.0$ .

Machine Unlearning

Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.

See all 351 papers

Machine Unlearning in Low-Dimensional Feature Subspace
Jan 30, 2026Kun Fang, Qinghua Tao, Junxu Liu +5Machine Unlearning
Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong, China · School of Automation, Beijing Institute of Technology, Beijing, China
BID-LoRA: A Parameter-Efficient Framework for Continual Learning and Unlearning
Apr 14, 2026Jagadeesh Rachapudi, Ritali Vatsi, Praful Hambarde +1Continual Learning Machine Unlearning
Indian Institute of Technology Mandi, Mandi, India
QR-Erase: Efficient Subspace-Based Machine Unlearning with Layer Localization
Aug 2, 2026Tyler Lizzo, Larry HeckMachine Unlearning LLM Unlearning
Georgia Institute of Technology

Jan 30, 2026J/K move · Enter open · S save

Machine Unlearning in Low-Dimensional Feature Subspace

Kun Fang, Qinghua Tao, Junxu Liu, Yaxin Xiao+4

Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong, China · School of Automation, Beijing Institute of Technology, Beijing, China

Machine Unlearning

Open paper AskPDF arXiv

Machine Unlearning (MU) aims at removing the influence of specific data from a pretrained model while preserving performance on the remaining data. In this work, a novel perspective for MU is presented upon low-dimensional feature subspaces, which gives rise to the potentials of separating the remaining and forgetting data herein. This separability motivates our LOFT, a method that proceeds unlearning in a LOw-dimensional FeaTure subspace from the pretrained model through principal projections, which are optimized to maximally capture the information of the remaining data and meanwhile diminish that of the forgetting data. During unlearning, LOFT simply optimizes a small-size projection matrix that can be flexibly inserted into the pretrained model, requiring just one-shot feature fetching from the pretrained model. LOFT demonstrates that effective and efficient MU can be achieved in low-dimensional feature subspaces even without repeated raw-data access or updates to the entire pretrained model. Extensive experiments validate the significantly lower computational overhead and superior unlearning performance of LOFT across diverse models, datasets, tasks, and applications. Code is publicly available.

Abstract

Machine Unlearning

Explore similar work

Machine Unlearning in Low-Dimensional Feature Subspace

BID-LoRA: A Parameter-Efficient Framework for Continual Learning and Unlearning

QR-Erase: Efficient Subspace-Based Machine Unlearning with Layer Localization