Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning

Lingfeng He; De Cheng; Huaijie Wang; Xi Yang; Nannan Wang; Xinbo Gao

cs.LGFeb 27, 2026

Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning

Authors: Lingfeng He, De Cheng, Huaijie Wang, Xi Yang, Nannan Wang, Xinbo Gao

Organizations: State Key Laboratory of Integrated Services Networks, School of Telecommunications Engineering, Xidian University, Xi’an, China · School of Electronic Engineering, Xidian University, Xi’an, China · AUTHORERR: Missing \icmlaffiliation.

Abstract

Continual Learning (CL) requires models to sequentially adapt to new tasks without forgetting old knowledge. Recently, Low-Rank Adaptation (LoRA), a representative Parameter-Efficient Fine-Tuning (PEFT) method, has gained increasing attention in CL. Several LoRA-based CL methods reduce interference across tasks by separating their update spaces, typically building the new space from the estimated null space of past tasks. However, they (i) overlook task-shared directions, which suppresses knowledge transfer, and (ii) fail to capture truly effective task-specific directions since these ``null bases" of old tasks can remain nearly inactive for new task under correlated tasks. To address this, we study LoRA learning capability from a projection energy perspective, and propose Low-rank Decomposition and Adaptation (LoDA). It performs a task-driven decomposition to build general and truly task-specific LoRA subspaces by solving two energy-based objectives, decoupling directions for knowledge sharing and isolation. LoDA fixes LoRA down-projections on two subspaces and learns robust up-projections via a Gradient-Aligned Optimization (GAO) approach. After each task, before integrating the LoRA updates into the backbone, LoDA derives a closed-form recalibration for the general update, approximating a feature-level joint optimum along this task-shared direction. Experiments indicate that LoDA outperforms existing CL methods. Our code is available at https://github.com/HHHLF/LoDA_ICML2026.

Figures & tables

Figure 1: LoDA decomposes the update space into general & isolated subspaces ( $\mathcal{U}_{G}$ & $\mathcal{U}_{I}$ ) to fix LoRA down-projections. By updating in two subspaces with a post-hoc recalibration, it reaches $\theta^{*}$ in a shared low-loss region of old and new tasks. $E_{\text{old}}$ and $E_{\text{new}}$ denote the subspace projection energy of old and new features. Darker colors in the loss contour map denote lower values.

Figure 2: Overall framework. At Task- $t$ , we freeze backbone weight $\mathbf{W}^{t-1}$ and insert a dual-branch LoRA module. (A) We decompose the update space into general & isolated subspaces for LoRA down-projections. (B) Then we freeze down-projections and train up-projections on $\mathcal{D}^{t}$ via GAO. (C) After training, the LoRA updates are integrated into backbone with a recalibration on the general branch.

Figure 3: Averaged

r^{t}(\mathbf{U}_{\text{null}})

in different layers on ImageNetA.

Figure 4: Loss values along the general direction (ImageNetA).

Method	Venue	FR	10S-ImageNetR		10S-ImageNetA		10S-CIFAR100		10S-CUB
Method	Venue	FR	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$
L2P ( Wang et al., 2022b )	CVPR’22	✗	$72.34_{\pm 0.17}$	$77.36_{\pm 0.64}$	$44.04_{\pm 0.93}$	$51.24_{\pm 2.26}$	$84.06_{\pm 0.88}$	$88.26_{\pm 1.34}$	$67.02_{\pm 1.90}$	$79.62_{\pm 1.60}$
DualPrompt ( Wang et al., 2022a )	ECCV’22	✗	$69.10_{\pm 0.62}$	$74.28_{\pm 0.66}$	$53.19_{\pm 0.74}$	$64.59_{\pm 0.08}$	$86.93_{\pm 0.24}$	$91.13_{\pm 0.32}$	$68.48_{\pm 0.47}$	$80.59_{\pm 1.50}$
CODAPrompt ( Smith et al., 2023 )	CVPR’23	✗	$73.31_{\pm 0.50}$	$78.47_{\pm 0.53}$	$52.08_{\pm 0.12}$	$63.92_{\pm 0.12}$	$83.21_{\pm 3.39}$	$87.71_{\pm 3.17}$	$77.23_{\pm 1.12}$	$81.90_{\pm 0.85}$
InfLoRA ( Liang & Li, 2024 )	CVPR’24	✗	$74.75_{\pm 0.64}$	$80.67_{\pm 0.55}$	$49.20_{\pm 1.12}$	$60.92_{\pm 0.61}$	$86.75_{\pm 0.35}$	$91.72_{\pm 0.15}$	$70.82_{\pm 0.23}$	$81.39_{\pm 0.14}$
SD-LoRA ( Wu et al., 2025b )	ICLR’25	✗	$77.34_{\pm 0.35}$	$82.04_{\pm 0.24}$	$55.96_{\pm 0.73}$	$64.95_{\pm 1.63}$	$88.01_{\pm 0.31}$	$92.54_{\pm 0.18}$	$77.48_{\pm 0.20}$	$85.59_{\pm 0.44}$

Table 1: Experimental results for 10 incremental tasks on ImageNetR, ImageNetA, CIFAR100, and CUB. We report the averaged results over 3 random seeds. The highest results are in bold , and the second highest results are underlined . “FR” denotes the use of feature replay.

Method	Venue	FR	5S-ImageNetR		20S-ImageNetR		20S-ImageNetA		10S-DomainNet
Method	Venue	FR	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$
L2P ( Wang et al., 2022b )	CVPR’22	✗	$70.83_{\pm 0.58}$	$78.34_{\pm 0.47}$	$69.64_{\pm 0.42}$	$75.28_{\pm 0.57}$	$40.48_{\pm 1.78}$	$49.62_{\pm 1.46}$	$81.17_{\pm 0.83}$	$87.43_{\pm 0.95}$
DualPrompt ( Wang et al., 2022a )	ECCV’22	✗	$73.05_{\pm 0.50}$	$79.47_{\pm 0.40}$	$66.61_{\pm 0.58}$	$72.45_{\pm 0.37}$	$42.28_{\pm 1.94}$	$53.39_{\pm 1.64}$	$81.70_{\pm 0.78}$	$87.80_{\pm 0.99}$
CODAPrompt ( Smith et al., 2023 )	CVPR’23	✗	$74.91_{\pm 0.33}$	$79.25_{\pm 0.53}$	$69.96_{\pm 0.50}$	$75.34_{\pm 0.85}$	$44.62_{\pm 1.92}$	$54.86_{\pm 0.50}$	$80.04_{\pm 0.79}$	$86.27_{\pm 0.82}$
InfLoRA ( Liang & Li, 2024 )	CVPR-24	✗	$77.52_{\pm 0.37}$	$82.01_{\pm 0.12}$	$71.01_{\pm 0.45}$	$77.28_{\pm 0.45}$	$46.81_{\pm 2.30}$	$57.13_{\pm 0.95}$	$81.45_{\pm 0.68}$	$87.55_{\pm 0.57}$
SD-LoRA ( Wu et al., 2025b )	ICLR-25	✗	$79.15_{\pm 0.20}$	$83.01_{\pm 0.42}$	$75.26_{\pm 0.37}$	$80.22_{\pm 0.72}$	$49.12_{\pm 1.78}$	$59.63_{\pm 1.57}$	$81.01_{\pm 0.42}$	$86.85_{\pm 0.40}$

Table 2: Experimental results for 5 and 20 incremental sessions on ImageNetR, 20 sessions on ImageNetA, and 10 sessions on DomainNet.

Idx	$\text{LoRA}_{G}$	$\text{LoRA}_{I}$	GAO	10S-ImageNetR		10S-ImageNetA		20S-ImageNetR		20S-ImageNetA
Idx	$\text{LoRA}_{G}$	$\text{LoRA}_{I}$	GAO	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$
1	–	–	–	$72.09_{\pm 1.02}$	$81.02_{\pm 1.00}$	$53.24_{\pm 1.10}$	$65.14_{\pm 1.41}$	$68.25_{\pm 1.18}$	$78.34_{\pm 0.50}$	$45.21_{\pm 5.51}$	$60.79_{\pm 0.16}$
2	✓	–	–	$79.19_{\pm 0.26}$	$85.18_{\pm 0.94}$	$59.62_{\pm 1.87}$	$68.84_{\pm 2.45}$	$76.59_{\pm 0.39}$	$83.26_{\pm 0.70}$	$51.70_{\pm 0.31}$	$62.23_{\pm 1.74}$
3	–	✓	–	$79.22_{\pm 0.50}$	$85.22_{\pm 0.73}$	$59.89_{\pm 0.68}$	$68.97_{\pm 2.03}$	$75.70_{\pm 0.78}$	$83.11_{\pm 1.08}$	$51.20_{\pm 0.73}$	$61.97_{\pm 0.62}$
4	✓	✓	–	$81.05_{\pm 0.22}$	$86.20_{\pm 0.75}$	$61.27_{\pm 1.30}$	$69.84_{\pm 2.29}$	$78.62_{\pm 0.46}$	$84.47_{\pm 0.43}$	$53.85_{\pm 0.79}$	$63.71_{\pm 2.21}$
5	✓	✓	✓	$\mathbf{81.93_{\pm 0.20}}$	$\mathbf{86.91_{\pm 0.40}}$	$\mathbf{62.59_{\pm 0.64}}$	$\mathbf{70.87_{\pm 1.61}}$	$\mathbf{78.96_{\pm 0.23}}$	$\mathbf{84.94_{\pm 0.17}}$	$\mathbf{55.74_{\pm 1.89}}$	$\mathbf{65.54_{\pm 2.43}}$

Table 3: Ablation Studies on ImageNetR and ImageNetA datasets under 10 and 20 incremental sessions.

Figure 5: Relative accuracy curve of old tasks, new tasks and the overall performances on 10S-ImageNetR.

Dataset	Method	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$
10S- ImageNetR	Random Orthogonal Bases	$68.23_{\pm 0.18}$	$79.05_{\pm 0.62}$
	Adam-NSCL ( Wang et al., 2021 )	$74.50_{\pm 0.77}$	$82.29_{\pm 1.12}$
	InfLoRA ( Liang & Li, 2024 )	$75.15_{\pm 0.37}$	$82.57_{\pm 0.84}$
	Relative Energy Maximization (Ours)	$\mathbf{79.22_{\pm 0.50}}$	$\mathbf{85.22_{\pm 0.73}}$
10S- ImageNetA	Random Orthogonal Bases	$53.48_{\pm 1.47}$	$64.34_{\pm 1.86}$
	Adam-NSCL ( Wang et al., 2021 )	$55.95_{\pm 1.84}$	$66.94_{\pm 2.03}$

Table 4: Comparison with different subspace isolation methods.

Dataset	Strategy	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$
10S- ImageNetR	CoMA ( $w=0.1$ ) ( Marouf et al., 2024 )	$76.93_{\pm 0.65}$	$81.89_{\pm 0.95}$
	CoMA ( $w=0.3$ ) ( Marouf et al., 2024 )	$78.39_{\pm 0.78}$	$84.48_{\pm 1.01}$
	CoMA ( $w=0.5$ ) ( Marouf et al., 2024 )	$77.95_{\pm 0.55}$	$84.30_{\pm 0.90}$
	$\theta=\frac{t-1}{t}\theta_{t-1}+\frac{1}{t}\theta_{t}$ ( Lin et al., 2022 )	$78.43_{\pm 0.20}$	$84.70_{\pm 1.06}$
	Closed-Form Rescaling ( $\lambda=3.0$ )	$\mathbf{79.19_{\pm 0.26}}$	$\mathbf{85.18_{\pm 0.94}}$
	Closed-Form Rescaling ( $\lambda=10.0$ )	$\mathbf{78.96_{\pm 0.14}}$	$\mathbf{85.10_{\pm 0.44}}$

Table 5: Comparison of our closed-form rescaling for

\text{LoRA}_{G}

with existing model merging strategies. “CoMA (

w=\alpha

)” denotes the result when setting the merge coefficient in CoMA to

\alpha

Figure 6: Projection magnitude and relative energy across different transformer layers on 10S-ImageNetA.

Method	Training Stage		Inference Stage
Method	Extra Memory	Training GFLOPs	Additional Params	Inference GFLOPs	$\mathcal{A}_{Last}$
InfLoRA ( Liang & Li, 2024 )	11.3MB	17.67	0.00M	17.60	71.01
SD-LoRA ( Wu et al., 2025b )	14.1MB	20.51	0.00M	17.60	75.26
VPT-NSP ( Lu et al., 2024 )	28.4MB	17.72	0.09M	17.72	76.13
VPT-CPG ( Lu et al., 2025 )	28.4MB	17.72	1.66M	17.72	75.33
LoDA (Ours)	27.0MB	18.06	0.00M	17.60	78.96

Table 6: Computational and storage overhead on 20S-ImageNetR.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 13 — Algorithm 1 Training pipeline of the proposed Low-rank Decomposition and Adaptation (LoDA).

Figure 7: Training loss curves of the first task

\mathcal{D}^{1}

and the new task

\mathcal{D}^{t}

across incremental sessions on 10S-ImageNetR and 10S-ImageNetA.

Figure 8: Impact of the hyper-parameters

w_{G}

, which balances the contributions of the two LoRA branches in Eq. 3 ,

\lambda

, which trades off performance between past and new tasks in Eq. 12 , and

\rho_{\max}

, which controls the inner gradient magnitude in GAO in Eq. 11 , on 10S-ImageNetR and 10S-ImageNetA. All results are averaged over three random seeds, and we report the mean with error bars.

Method	ImageNetR				CIFAR100				UCF
	B0 Inc10		B50 Inc10		B0 Inc10		B50 Inc10		B0 Inc20		B100 Inc20
	$\mathcal{A}_{Avg}$	$\mathcal{A}_{Last}$	$\mathcal{A}_{Avg}$	$\mathcal{A}_{Last}$	$\mathcal{A}_{Avg}$	$\mathcal{A}_{Last}$	$\mathcal{A}_{Avg}$	$\mathcal{A}_{Last}$	$\mathcal{A}_{Avg}$	$\mathcal{A}_{Last}$	$\mathcal{A}_{Avg}$	$\mathcal{A}_{Last}$
CoOp ( Zhou et al., 2022b )	60.73	37.52	54.20	39.77	47.00	24.24	41.23	24.12	47.85	33.46	42.02	24.74
ZS-CLIP ( Radford et al., 2021 )	83.37	77.17	79.57	77.17	81.81	71.38	76.49	71.38	75.50	67.64	71.44	67.64
L2P ( Wang et al., 2022b )	75.97	66.52	72.82	66.77	82.74	73.03	81.14	73.61	86.34	76.43	83.95	76.62
DualPrompt ( Wang et al., 2022a )	76.21	66.65	73.22	67.58	81.63	72.44	80.12	72.57	85.21	75.82	84.31	76.35

Appendix

Table 8: Experimental results on ImageNetR, CIFAR100 and UCF datasets using CLIP as the backbone. All methods are initialized with the CLIP pre-trained on LAION-400M without experience replay for fair comparisons. PROOF † represents our reproduced results of PROOF ( Zhou et al., 2025 ) without experience replay. The best results are in red and the second highest results are in blue .

Method	Aircraft				Cars				CUB
	B0 Inc10		B50 Inc10		B0 Inc10		B50 Inc10		B0 Inc20		B100 Inc20
	$\mathcal{A}_{Avg}$	$\mathcal{A}_{Last}$	$\mathcal{A}_{Avg}$	$\mathcal{A}_{Last}$	$\mathcal{A}_{Avg}$	$\mathcal{A}_{Last}$	$\mathcal{A}_{Avg}$	$\mathcal{A}_{Last}$	$\mathcal{A}_{Avg}$	$\mathcal{A}_{Last}$	$\mathcal{A}_{Avg}$	$\mathcal{A}_{Last}$
CoOp ( Zhou et al., 2022b )	14.54	7.14	13.05	7.77	36.46	21.65	37.40	20.87	27.61	8.57	24.03	10.14
ZS-CLIP ( Radford et al., 2021 )	26.66	17.22	21.70	17.22	82.60	76.37	78.32	76.37	74.38	63.06	67.96	63.06
L2P ( Wang et al., 2022b )	47.19	28.29	44.07	32.13	76.63	61.82	76.37	65.64	70.87	57.93	75.64	66.12
DualPrompt ( Wang et al., 2022a )	44.30	25.83	46.07	33.57	76.26	62.94	76.88	67.55	69.89	57.46	74.40	64.84

Appendix

Table 9: Experimental results on Aircraft, Cars and CUB datasets using CLIP as the backbone. All methods are initialized with the CLIP pre-trained on LAION-400M without experience replay for fair comparisons. PROOF † represents our reproduced results of PROOF ( Zhou et al., 2025 ) without experience replay. The best results are in red and the second highest results are in blue .

Figure 9: Task-by-task accuracy curve on 10S-ImageNetR using different self-supervised pre-trained models.

Figure 10: Performances of sequential tuning and LoDA using ViT models with different sizes on 10S-ImageNetR.

Figure 11: Visualization of the loss landscapes on the first task

\mathcal{D}^{1}

, the second task

\mathcal{D}^{2}

, and two tasks on 10S-ImageNetA.

Figure 12: Visualization of the loss landscapes on the first task

\mathcal{D}^{1}

, the second task

\mathcal{D}^{2}

, and two tasks on 10S-ImageNetR.

Continual Learning

Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.

See all 294 papers

May 26, 2026

Energy-Structured Low-Rank Adaptation for Continual Learning

While orthogonal subspace methods try to mitigate task interference in Continual Learning (CL), they often suffer from energy diffusion across the basis, hindering knowledge compaction and exhausting capacity for future tasks. We observe that output feature drift induced by parameter updates is inherently low-rank, and theoretically prove that preserving parameters along the principal directions of this drift minimizes the output reconstruction error. Motivated by this, we propose Energy-Concentrated and Energy-Ordered Low-Rank Adaptation (E

^2

-LoRA). By explicitly ordering and concentrating knowledge into leading ranks, E

^2

-LoRA frees capacity for subsequent tasks. Furthermore, we design a dynamic rank allocation strategy to balance stability and plasticity by jointly optimizing energy retention and model plasticity. Extensive experiments across multiple benchmarks demonstrate that E

^2

-LoRA achieves state-of-the-art performance. Code is available at https://github.com/kiddo127/E2-LoRA.

Longhua Li, Lei Qi, Qi Tian +1

School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Huawei Technologies, Shenzhen, China

Continual Learning Low-Rank Adaptation Adaptive Rank Allocation Parameter-Efficient Fine-Tuning Continual LoRA

Ask PDF

Jun 26, 2026

When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning

Low-Rank Adaptation (LoRA) has become the standard tool for parameter-efficient fine-tuning of large pretrained models. When applied sequentially across tasks in Continual Learning (CL), the standard assumption is that each new task requires a dedicated low-rank adapter. In this work, we challenge this assumption empirically and structurally. We show that task-specific LoRA adapters in CL exhibit significant low-rank redundancy: the subspaces spanned by adapters trained on different tasks substantially overlap, and in many cases earlier adapters can faithfully represent later tasks. Building on this observation, we propose LiteLoRA, a plug-and-play gating mechanism that learns at train time whether to recruit a new adapter or reuse existing low-rank representations. Our method reduces the number of active adapters by 20-70% while matching or exceeding state-of-the-art performance on standard CL benchmarks, revealing that structural redundancy is pervasive and that selective learning is sufficient to achieve stability without sacrificing plasticity.

Tanguy Dieudonné, Giulia Lanzillotta, Enis Simsar +2

Department of Computer Science, ETH Zurich, Zurich, Switzerland.

Continual Learning Low-Rank Adaptation Incremental Learning Parameter-Efficient Fine-Tuning Continual LoRA

Ask PDF

Oct 6, 2026

CoDe-LoRA: Mitigating the Orthogonality Dilemma in Continual Learning of LLMs via Knowledge Consolidation and Decoupling

Continual learning (CL) is essential for Large Language Models (LLMs) to sequentially adapt to evolving tasks. To mitigate catastrophic forgetting, recent advances implement low-rank adaptation with orthogonal projections (e.g., O-LoRA) to isolate task parameters. However, we reveal that such strict geometric constraints trigger an "Orthogonality Dilemma": rigid parameter isolation impedes the transfer and accumulation of shared representations across semantically related tasks. In this work, we propose a new replay-free method, called Consolidation and Decoupling LoRA (CoDe-LoRA), for CL of LLMs. CoDe-LoRA disentangles the learning process into Consolidating Universal Knowledge and Decoupling Task-Specific Knowledge. To achieve this, CoDe-LoRA leverages an adaptive null space projection mechanism and semantic routing to balance knowledge accumulation with task-specific adaptation. Experimental results across four backbones and three CL benchmarks show that CoDe-LoRA achieves the best average accuracy. Our code is available at https://github.com/Estrellajer/CoDe-LoRA.

Maoqi Liu, Quan Fang, Yufei He

Beijing University of Posts and Telecommunications · National University of Singapore

Continual Learning for LLMs Low-Rank Adaptation Incremental Learning Catastrophic Forgetting

Ask PDF

Abstract

Continual Learning

Explore similar work

Energy-Structured Low-Rank Adaptation for Continual Learning

When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning

CoDe-LoRA: Mitigating the Orthogonality Dilemma in Continual Learning of LLMs via Knowledge Consolidation and Decoupling