BID-LoRA: A Parameter-Efficient Framework for Continual Learning and Unlearning

Jagadeesh Rachapudi; Ritali Vatsi; Praful Hambarde; Amit Shukla

cs.LGApr 14, 2026

BID-LoRA: A Parameter-Efficient Framework for Continual Learning and Unlearning

Authors: Jagadeesh Rachapudi, Ritali Vatsi, Praful Hambarde, Amit Shukla

Organizations: Indian Institute of Technology Mandi, Mandi, India

Abstract

Recent advances in deep learning underscore the need for systems that can not only acquire new knowledge through Continual Learning (CL) but also remove outdated, sensitive, or private information through Machine Unlearning (MU). However, while CL methods are well-developed, MU techniques remain in early stages, creating a critical gap for unified frameworks that depend on both capabilities. We find that naively combining existing CL and MU approaches results in knowledge leakage a gradual degradation of foundational knowledge across repeated adaptation cycles. To address this, we formalize Continual Learning Unlearning (CLU) as a unified paradigm with three key goals: (i) precise deletion of unwanted knowledge, (ii) efficient integration of new knowledge while preserving prior information, and (iii) minimizing knowledge leakage across cycles. We propose Bi-Directional Low-Rank Adaptation (BID-LoRA), a novel framework featuring three dedicated adapter pathways-retain, new, and unlearn applied to attention layers, combined with escape unlearning that pushes forget-class embeddings to positions maximally distant from retained knowledge, updating only 5% of parameters. Experiments on CIFAR-100 show that BID-LoRA outperforms CLU baselines across multiple adaptation cycles. We further evaluate on CASIA-Face100, a curated face recognition subset, demonstrating practical applicability to real-world identity management systems where new users must be enrolled and withdrawn users removed.

Figures & tables

Fig. 1: Overview of CLU. The CLU system removes unwanted knowledge ( red ), retains prior knowledge ( green ), and integrates new knowledge ( blue ).

Fig. 2: LoRA placement in BID-LoRA at Attention Modules

Fig. 3: Pathway separation for BID-LoRA.

Fig. 4: Illustration of continual adapting evaluation protocol.

Panel of Figure 5 — Fig. 5: Radar plot comparison at Task-6. BID-LoRA consistently outperforms all baselines across metrics on both classification and face recognition tasks.

Rank	% Ratio	$Acc_{f}$	$Acc_{r}$	$Acc_{n}$	$Acc_{o}$
1	1.09	1.73	48.57	27.60	41.58
2	2.30	3.47	46.43	43.07	45.31
4	3.24	0.00	65.71	41.87	57.77
8	5.08	0.80	76.43	69.60	74.15
16	8.56	1.07	81.43	73.73	78.86
32	14.80	0.40	78.57	76.93	78.03

TABLE III: Ablation study on the rank of LoRA modules

Method	% Ratio	$Acc_{f}$	$Acc_{r}$	$Acc_{n}$	$Acc_{o}$
Standard LoRA	2.54	0.00	74.29	85.07	77.88
BID-LoRA	5.08	0.13	77.14	85.87	80.05

TABLE IV: BID-LoRA vs. Standard LoRA in the CLU setting

Retain Ratio	Speed	$Acc_{f}$	$Acc_{r}$	$Acc_{n}$	$Acc_{o}$
1.0	1.0 $\times$	0.40	68.00	81.74	72.58
0.5	2.0 $\times$	0.00	70.00	71.05	70.35
0.3	3.3 $\times$	0.00	66.00	79.11	70.37
0.1	10 $\times$	0.00	65.00	79.96	69.98

TABLE V: Ablation on retain ratio

F	R	N	$Acc_{f}$	$Acc_{r}$	$Acc_{n}$	$Acc_{o}$
Pre-train			73.64	82.27	0.00	48.49
✓	✓	✓	0.00	84.09	56.82	75.00
✗	✓	✓	70.45	82.73	51.82	72.43
✓	✗	✓	0.00	0.23	64.55	21.67
✓	✓	✗	0.00	84.09	0.00	56.06

TABLE VI: Ablation on adapter pathway contributions (F=Forget, R=Retain, N=New)

Panel of Figure 10 — Fig. 6: Geometric verification of unlearning. Top row: t-SNE visualization showing forget classes migrating toward escape point $d^{*}$ . Bottom row: 3D hypersphere visualization with dashed antipodal axis from $\bar{c}_{r}$ to $d^{*}$

$\lambda_{\text{esc}}$	$Acc_{f}\downarrow$	$Acc_{r}\uparrow$	$Acc_{n}\uparrow$	$Acc_{o}\uparrow$
0	8.80	78.57	61.07	72.74
2	4.13	80.71	63.60	75.01
5	4.27	77.14	68.27	74.18
10	0.53	77.14	68.00	73.45

TABLE VII: Ablation on escape scaling factor

\lambda_{\text{esc}}

. Higher scaling pushes forget embeddings further from retain centroids, improving unlearning

Panel of Figure 12 — Fig. 7: Knowledge leakage in CL+MU combinations. Retain accuracy degrades progressively across CLU cycles on both benchmarks.

Continual Learning

Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.

See all 305 papers

Orthogonal Subspace Projection for Continual Machine Unlearning via SVD-Based LoRA
Apr 14, 2026Yogachandran Rahulamathavan, Nasir Iqbal, Juncheng Hu +1Machine Unlearning Low-Rank Adaptation
Institute for Digital Technologies, Loughborough University, UK · School of Artificial Intelligent Medicine, Guilin Medical University, Guilin 541199, CN
The Forgetting-Retention Dilemma: Certified Unlearning Theory in Continual Learning
Jun 29, 2026Yiting Hu, Lingjie Duan, Qian ZhangContinual Learning Statistical Learning Theory
Singapore University of Technology and Design, Singapore · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China.
DECAF: De-Clustering for Adaptive Representational Unlearning
Jul 27, 2026Anjie Le, Can Peng, Hongcheng Guo +1Machine Unlearning
Institute of Biomedical Engineering, University of Oxford, UK · Fudan University.

Apr 14, 2026J/K move · Enter open · S save

Orthogonal Subspace Projection for Continual Machine Unlearning via SVD-Based LoRA

Yogachandran Rahulamathavan, Nasir Iqbal, Juncheng Hu, Sangarapillai Lambotharan

Institute for Digital Technologies, Loughborough University, UK · School of Artificial Intelligent Medicine, Guilin Medical University, Guilin 541199, CN

Machine Unlearning Low-Rank Adaptation

Open paper AskPDF arXiv

Continual machine unlearning aims to remove the influence of data that should no longer be retained, while preserving the usefulness of the model on everything else. This setting becomes especially difficult when deletion requests arrive sequentially, because the model must repeatedly adapt without erasing previously retained knowledge. Low-Rank Adaptation (LoRA) offers an efficient way to implement such updates, but naively combining many sequential LoRA modules leads to parameter collision, causing strong interference between tasks. We propose a static alternative based on Singular Value Decomposition (SVD)-guided orthogonal subspace projection. Our method constrains each new LoRA update during training so that it lies in the orthogonal complement of the subspaces used by earlier unlearning tasks. This preserves task isolation without requiring dynamic routing at deployment. Experiments on CIFAR-100 with ResNet-20 and on MNIST show stable behavior across long sequences of unlearning tasks. After thirty sequential unlearning tasks, state-of-the-art static fusion reduces retained accuracy from 60.39% to 12.70%, whereas the proposed in-training constrained optimization maintains baseline performance (

\sim

58.1%) while preserving strong unlearning efficacy.

Abstract

Continual Learning

Explore similar work

Orthogonal Subspace Projection for Continual Machine Unlearning via SVD-Based LoRA

The Forgetting-Retention Dilemma: Certified Unlearning Theory in Continual Learning

DECAF: De-Clustering for Adaptive Representational Unlearning