BID-LoRA: A Parameter-Efficient Framework for Continual Learning and Unlearning

Jagadeesh Rachapudi; Ritali Vatsi; Praful Hambarde; Amit Shukla

cs.LGApr 14, 2026

BID-LoRA: A Parameter-Efficient Framework for Continual Learning and Unlearning

Authors: Jagadeesh Rachapudi, Ritali Vatsi, Praful Hambarde, Amit Shukla

Organizations: Indian Institute of Technology Mandi, Mandi, India

Abstract

Recent advances in deep learning underscore the need for systems that can not only acquire new knowledge through Continual Learning (CL) but also remove outdated, sensitive, or private information through Machine Unlearning (MU). However, while CL methods are well-developed, MU techniques remain in early stages, creating a critical gap for unified frameworks that depend on both capabilities. We find that naively combining existing CL and MU approaches results in knowledge leakage a gradual degradation of foundational knowledge across repeated adaptation cycles. To address this, we formalize Continual Learning Unlearning (CLU) as a unified paradigm with three key goals: (i) precise deletion of unwanted knowledge, (ii) efficient integration of new knowledge while preserving prior information, and (iii) minimizing knowledge leakage across cycles. We propose Bi-Directional Low-Rank Adaptation (BID-LoRA), a novel framework featuring three dedicated adapter pathways-retain, new, and unlearn applied to attention layers, combined with escape unlearning that pushes forget-class embeddings to positions maximally distant from retained knowledge, updating only 5% of parameters. Experiments on CIFAR-100 show that BID-LoRA outperforms CLU baselines across multiple adaptation cycles. We further evaluate on CASIA-Face100, a curated face recognition subset, demonstrating practical applicability to real-world identity management systems where new users must be enrolled and withdrawn users removed.

Figures & tables

Fig. 1: Overview of CLU. The CLU system removes unwanted knowledge ( red ), retains prior knowledge ( green ), and integrates new knowledge ( blue ).

Fig. 2: LoRA placement in BID-LoRA at Attention Modules

Fig. 3: Pathway separation for BID-LoRA.

Fig. 4: Illustration of continual adapting evaluation protocol.

Panel of Figure 5 — Fig. 5: Radar plot comparison at Task-6. BID-LoRA consistently outperforms all baselines across metrics on both classification and face recognition tasks.

Rank	% Ratio	$Acc_{f}$	$Acc_{r}$	$Acc_{n}$	$Acc_{o}$
1	1.09	1.73	48.57	27.60	41.58
2	2.30	3.47	46.43	43.07	45.31
4	3.24	0.00	65.71	41.87	57.77
8	5.08	0.80	76.43	69.60	74.15
16	8.56	1.07	81.43	73.73	78.86
32	14.80	0.40	78.57	76.93	78.03

TABLE III: Ablation study on the rank of LoRA modules

Method	% Ratio	$Acc_{f}$	$Acc_{r}$	$Acc_{n}$	$Acc_{o}$
Standard LoRA	2.54	0.00	74.29	85.07	77.88
BID-LoRA	5.08	0.13	77.14	85.87	80.05

TABLE IV: BID-LoRA vs. Standard LoRA in the CLU setting

Retain Ratio	Speed	$Acc_{f}$	$Acc_{r}$	$Acc_{n}$	$Acc_{o}$
1.0	1.0 $\times$	0.40	68.00	81.74	72.58
0.5	2.0 $\times$	0.00	70.00	71.05	70.35
0.3	3.3 $\times$	0.00	66.00	79.11	70.37
0.1	10 $\times$	0.00	65.00	79.96	69.98

TABLE V: Ablation on retain ratio

F	R	N	$Acc_{f}$	$Acc_{r}$	$Acc_{n}$	$Acc_{o}$
Pre-train			73.64	82.27	0.00	48.49
✓	✓	✓	0.00	84.09	56.82	75.00
✗	✓	✓	70.45	82.73	51.82	72.43
✓	✗	✓	0.00	0.23	64.55	21.67
✓	✓	✗	0.00	84.09	0.00	56.06

TABLE VI: Ablation on adapter pathway contributions (F=Forget, R=Retain, N=New)

Panel of Figure 10 — Fig. 6: Geometric verification of unlearning. Top row: t-SNE visualization showing forget classes migrating toward escape point $d^{*}$ . Bottom row: 3D hypersphere visualization with dashed antipodal axis from $\bar{c}_{r}$ to $d^{*}$

$\lambda_{\text{esc}}$	$Acc_{f}\downarrow$	$Acc_{r}\uparrow$	$Acc_{n}\uparrow$	$Acc_{o}\uparrow$
0	8.80	78.57	61.07	72.74
2	4.13	80.71	63.60	75.01
5	4.27	77.14	68.27	74.18
10	0.53	77.14	68.00	73.45

TABLE VII: Ablation on escape scaling factor

\lambda_{\text{esc}}

. Higher scaling pushes forget embeddings further from retain centroids, improving unlearning

Panel of Figure 12 — Fig. 7: Knowledge leakage in CL+MU combinations. Retain accuracy degrades progressively across CLU cycles on both benchmarks.

Continual Learning

Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.

See all 304 papers

Apr 14, 2026

Orthogonal Subspace Projection for Continual Machine Unlearning via SVD-Based LoRA

Continual machine unlearning aims to remove the influence of data that should no longer be retained, while preserving the usefulness of the model on everything else. This setting becomes especially difficult when deletion requests arrive sequentially, because the model must repeatedly adapt without erasing previously retained knowledge. Low-Rank Adaptation (LoRA) offers an efficient way to implement such updates, but naively combining many sequential LoRA modules leads to parameter collision, causing strong interference between tasks. We propose a static alternative based on Singular Value Decomposition (SVD)-guided orthogonal subspace projection. Our method constrains each new LoRA update during training so that it lies in the orthogonal complement of the subspaces used by earlier unlearning tasks. This preserves task isolation without requiring dynamic routing at deployment. Experiments on CIFAR-100 with ResNet-20 and on MNIST show stable behavior across long sequences of unlearning tasks. After thirty sequential unlearning tasks, state-of-the-art static fusion reduces retained accuracy from 60.39% to 12.70%, whereas the proposed in-training constrained optimization maintains baseline performance (

\sim

58.1%) while preserving strong unlearning efficacy.

Yogachandran Rahulamathavan, Nasir Iqbal, Juncheng Hu +1

Institute for Digital Technologies, Loughborough University, UK · School of Artificial Intelligent Medicine, Guilin Medical University, Guilin 541199, CN

Machine Unlearning Low-Rank Adaptation

Ask PDF

Jun 29, 2026

The Forgetting-Retention Dilemma: Certified Unlearning Theory in Continual Learning

Machine unlearning aims to eliminate the influence of specific data from trained models to safeguard privacy. However, this presents a significant challenge in the context of continual learning (CL), where models update sequentially on dynamic datasets. A major limitation is that current certified unlearning algorithms fail to account for the complex, cumulative model evolution inherent to CL framework. In this work, we establish the first theoretical foundation bridging CL and machine unlearning. We formulate the CL's unlearning objective as the minimization of post-unlearning excess risk, which decomposes into CL excess risk and unlearning loss, characterizing the fundamental trade-off between preserving historical knowledge and targeted forgetting. Under mild assumptions, we first establish an upper bound for the CL excess risk in non-convex models. We then adapt two certified unlearning approaches, gradient-based and Hessian-based, to the CL framework. Our analysis reveals that while the gradient-based approach is less effective than the Hessian-based method in minimizing unlearning loss, it offers the distinct advantage of nearly zero storage overhead for enabling unlearning. This insight motivates a hybrid strategy that reduces storage costs while maintaining post-unlearning performance. Experimental results further validate our theoretical findings.

Yiting Hu, Lingjie Duan, Qian Zhang

Singapore University of Technology and Design, Singapore · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China.

Continual Learning Statistical Learning Theory Machine Unlearning

Ask PDF

Jul 27, 2026

DECAF: De-Clustering for Adaptive Representational Unlearning

Machine unlearning, which aims to remove the influence of specific training data from a trained model, is a key requirement for privacy, accountability, and adaptive deployment. We argue that many unlearning methods are vulnerable to a simple clustering attack, which can recover class structure in an unsupervised manner, limiting their suitability for continual deployment where removal requests must be handled reliably on demand. To address this, we propose DECAF (DE-Clustering for Adaptive Forgetting), a post-hoc method that operates only on the forget set and is designed to break the cluster. DECAF combines input noise, confidence suppression, and entropy-based output diversification to disrupt the residual feature-space structure associated with forgotten data. On CIFAR-10 with ResNet-18, DECAF attains 0.10% forget-class accuracy, 79.4% retain accuracy, and an AUS of 0.88, surpassing all other baselines. In cluster-based analysis, it attains performance comparable to that of unlearning methods that use the full training set, while being significantly more efficient. Code: https://github.com/ale256/representation_unlearning.

Anjie Le, Can Peng, Hongcheng Guo +1

Institute of Biomedical Engineering, University of Oxford, UK · Fudan University.

Machine Unlearning

Ask PDF

Abstract

Continual Learning

Explore similar work

Orthogonal Subspace Projection for Continual Machine Unlearning via SVD-Based LoRA

The Forgetting-Retention Dilemma: Certified Unlearning Theory in Continual Learning

DECAF: De-Clustering for Adaptive Representational Unlearning