CoDe-LoRA: Mitigating the Orthogonality Dilemma in Continual Learning of LLMs via Knowledge Consolidation and Decoupling

Maoqi Liu; Quan Fang; Yufei He

cs.AIOct 6, 2026

CoDe-LoRA: Mitigating the Orthogonality Dilemma in Continual Learning of LLMs via Knowledge Consolidation and Decoupling

Authors: Maoqi Liu, Quan Fang, Yufei He

Organizations: Beijing University of Posts and Telecommunications · National University of Singapore

Abstract

Continual learning (CL) is essential for Large Language Models (LLMs) to sequentially adapt to evolving tasks. To mitigate catastrophic forgetting, recent advances implement low-rank adaptation with orthogonal projections (e.g., O-LoRA) to isolate task parameters. However, we reveal that such strict geometric constraints trigger an "Orthogonality Dilemma": rigid parameter isolation impedes the transfer and accumulation of shared representations across semantically related tasks. In this work, we propose a new replay-free method, called Consolidation and Decoupling LoRA (CoDe-LoRA), for CL of LLMs. CoDe-LoRA disentangles the learning process into Consolidating Universal Knowledge and Decoupling Task-Specific Knowledge. To achieve this, CoDe-LoRA leverages an adaptive null space projection mechanism and semantic routing to balance knowledge accumulation with task-specific adaptation. Experimental results across four backbones and three CL benchmarks show that CoDe-LoRA achieves the best average accuracy. Our code is available at https://github.com/Estrellajer/CoDe-LoRA.

Figures & tables

Figure 1 : Conceptual illustration of parameter space utilization. Traditional LoRA suffers from interference, while O-LoRA’s strict orthogonality hinders knowledge sharing. CoDe-LoRA leverages null space projection to consolidate shared knowledge.

Figure 2 : Preliminary evidence for the orthogonality dilemma and transferable representation accumulation. (a) Orthogonality-induced degradation under different task similarity levels. (b) Transfer trajectories over repeated training with and without orthogonal constraints. (c) Next-task zero-shot transfer comparison across continual adaptation methods. (d) Forward transfer dynamics across sequential tasks.

Figure 3 : The CoDe-LoRA framework. (Left) Training: Knowledge is disentangled into task-specific experts and a shared branch consolidated via null space projection. (Right) Inference: Inputs are routed to experts via prototype similarity, with a confidence-based fallback to the shared branch for robust prediction.

	Method	Standard CL Benchmark				Large Number of Tasks
	Method	Order-1	Order-2	Order-3	Avg	Order-4	Order-5	Order-6	Avg
Ref.	Per-LoRA (Oracle)	81.4	81.3	81.4	81.4	77.6	77.4	77.5	77.5
Ref.	MTL	80.0	80.0	80.0	80.0	76.5	76.5	76.5	76.5
Seq. FT	SeqFT	18.9	24.9	41.7	28.5	7.4	7.4	7.5	7.4
	SeqLoRA	44.6	32.7	53.7	43.7	0.6	1.9	1.6	1.4
	IncLoRA	66.0	64.9	68.3	66.4	63.3	58.5	61.7	61.2

Table 1 : Performance comparison of different methods using the T5 model on Standard CL Benchmark and Large Number of Tasks. The average accuracy after training on the final task is reported. The top two results (excluding the oracle and MTL references) are bolded and underlined . Subscripts denote standard deviations across 3 random seeds.

Panel of Table 2 — Table 2 : Results across backbones. Std./Long: average accuracy on the Standard CL/Long Sequence benchmarks; OP/BWT: Trace metrics. Co/De-LoRA: single-branch ablations. Top two per column are bolded and underlined .

Figure 5 : Ablation analysis of CoDe-LoRA showing task-wise accuracy and BWT under different ensemble strategies (left) and routing accuracy across varying prototype sample sizes $N$ (right).

Figure 6 : Zero-shot forward transfer on Qwen3-0.6B (Order 1). “N/A” indicates trained tasks.

Method	Train (s)	Infer (s)	Params
O-LoRA	108.8	30.1	9.4M (1.28%)
N-LoRA	98.0	29.9	9.4M (1.28%)
MoLE-CIE	861.0	142.7	32.4M (4.40%)
CoDe-LoRA	98.4	43.1	9.4M (1.28%)

Table 3 : Efficiency on Order 1 (T5-large, one H20 GPU). Train and inference are seconds per task and per evaluated test set; CoDe-LoRA is a single seed, the others are means over three seeds. Params are the stored adapter parameters (CoDe-LoRA stores only the experts) and their share of the backbone.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Notation	Description
$\mathcal{T}^{(t)}$	The $t$ -th task in the sequence
$\mathcal{D}^{(t)}$	Dataset associated with task $\mathcal{T}^{(t)}$
$\theta_{\text{base}}$	Parameters of the frozen backbone
$\theta_{\text{shared}}$	Parameters of consolidation branch
$\theta_{\text{spec}}^{k}$	Parameters of $k$ -th task expert
$W_{\text{acc}}^{(t)}$	Accumulated shared weights up to task $t$

Appendix

Table 4 : Summary of key notations.

Method	$c_{t},\ s_{t}$	Avg Accuracy	Shared Branch
No Scaling	$1,\ 1$	$76.5_{\pm 0.5}$	$60.3_{\pm 1.3}$
Linear Avg	$\frac{t-1}{t},\ \frac{1}{t}$	$77.5_{\pm 0.4}$	$73.6_{\pm 1.0}$
CoDe-LoRA	$\mathbf{\sqrt{\frac{t-1}{t}},\ \frac{1}{\sqrt{t}}}$	$\mathbf{77.8_{\pm 0.4}}$	$\mathbf{75.6_{\pm 0.6}}$

Appendix

Table 5 : Scaling factors on the Standard CL Benchmark (Order-1, T5-Large), measured on the null-space consolidation path where this rule is defined. “Avg Accuracy” is the full model; “Shared Branch” evaluates the consolidated branch alone, which is what the scaling factor controls. Three seeds.

Retained rank	Co-LoRA alone	CoDe-LoRA
1	66.84	79.13
2	72.11	79.13
4	75.86	79.13
8	76.67	79.13

Appendix

Table 6 : SVD retained rank on the Standard CL Benchmark (Order 1, T5-large). Rank 8 is the default.

	Standard (Order 1)		Long (Order 4)
Router	Misrouted	Loss	Misrouted	Loss
Cosine prototype	2.03%	1.43	10.07%	4.74
LDA	0.15%	0.10	1.24%	0.20

Appendix

Table 7 : Routing errors and their cost on T5-large (three seeds). Loss is the AP of the true-task expert minus that of the routed expert.

Figure 7 : Routing accuracy versus support samples per task

N

(T5-large).

Figure 8 : Per-task accuracy of Co, the routed expert and their oracle on the Long benchmark (Order 4).

Scheme family	Outcome
Confidence-threshold fallback	best threshold on development is “never use Co”
Per-expert branch choice	$\approx$ routed expert
Co as an extra expert	no development-qualified candidate
Output selectors (margin, entropy, agreement)	below calibrated De, or kept De
1480-feature selectors	none passed development; 26 rescues vs. 26 breaks on test
Closed-form combinations (PoE, averaging, Dawid–Skene)	do not solve the selection problem

Appendix

Table 8 : Inference-time schemes for combining Co with the routed expert.

Construction	Co alone	Oracle
Trained, SVD merge (main)	76.67 / 66.33	83.79 / 84.61
+ null space, paper scaling	75.61 / 63.78	84.53 / 86.00
+ distillation after SVD	74.77 / 66.34	83.66 / 84.51
+ left-space projection	76.16 / 65.67	83.74 / 84.08
+ two-sided projection	74.88 / 63.91	83.14 / 84.60
Isolated experts, SVD merge	41.87 / 11.06	81.84 / 73.99

Appendix

Table 9 : Constructions of Co (generative, test): Co alone and the oracle that takes the better of Co and the routed expert per example. The routed expert alone scores 79.13 / 76.3–76.5 (Std / Long).

Method	Task	Train (s)	Infer (s)
O-LoRA	DBpedia	141	25
	Amazon	65	27
	Yahoo	186	43
	AG News	44	117
N-LoRA	DBpedia	128	26
	Amazon	60	27

Appendix

Table 10 : Training and inference seconds per task on the Standard benchmark (Order 1, T5-large, one H20 GPU, seed 41).

Method	Train (min)	Infer (min)	Params
O-LoRA	13.3	9.4	35.4M (4.80%)
N-LoRA	12.0	9.4	35.4M (4.80%)
MoLE-CIE	140.3	49.5	60.0M (8.14%)
CoDe-LoRA	12.5	13.4	35.4M (4.80%)

Appendix

Table 11 : End-to-end efficiency on the Long Sequence Benchmark (T=15 tasks). CoDe-LoRA is a single seed, the others are means over three seeds.

No.	Dataset name	Category	Task	Domain	Metric
1	Yelp	CL Benchmark	sentiment analysis	Yelp reviews	accuracy
2	Amazon	CL Benchmark	sentiment analysis	Amazon reviews	accuracy
3	DBpedia	CL Benchmark	topic classification	Wikipedia	accuracy
4	Yahoo	CL Benchmark	topic classification	Yahoo Q&A	accuracy
5	AG News	CL Benchmark	topic classification	news	accuracy
6	MNLI	GLUE	NLI	various	accuracy

Appendix

Table 12 : Datasets and their tasks, categories, domains, and metrics.

Task	Prompts
NLI	What is the logical relationship between the “sentence 1” and the “sentence 2”? Choose one from the option.
QQP	Whether the “first sentence” and the “second sentence” have the same meaning? Choose one from the option.
SC	What is the sentiment of the following paragraph? Choose one from the option.
TC	What is the topic of the following paragraph? Choose one from the option.
BoolQA	According to the following passage, is the question true or false? Choose one from the option.
MultiRC	According to the following passage and question, is the candidate answer true or false? Choose one from the option.

Appendix

Table 13 : Task descriptions and corresponding prompts.

Order	Setting	Task Sequence
1	T5, Llama 2-7B	dbpedia $\rightarrow$ amazon $\rightarrow$ yahoo $\rightarrow$ ag
2	T5, Llama 2-7B	dbpedia $\rightarrow$ amazon $\rightarrow$ ag $\rightarrow$ yahoo
3	T5, Llama 2-7B	yahoo $\rightarrow$ amazon $\rightarrow$ ag $\rightarrow$ dbpedia
4	T5	mnli $\rightarrow$ cb $\rightarrow$ wic $\rightarrow$ copa $\rightarrow$ qqp $\rightarrow$ boolqa $\rightarrow$ rte $\rightarrow$ imdb $\rightarrow$ yelp $\rightarrow$ amazon $\rightarrow$ sst-2 $\rightarrow$ dbpedia $\rightarrow$ ag $\rightarrow$ multirc $\rightarrow$ yahoo
5	T5	multirc $\rightarrow$ boolqa $\rightarrow$ wic $\rightarrow$ mnli $\rightarrow$ cb $\rightarrow$ copa $\rightarrow$ qqp $\rightarrow$ rte $\rightarrow$ imdb $\rightarrow$ sst-2 $\rightarrow$ dbpedia $\rightarrow$ ag $\rightarrow$ yelp $\rightarrow$ amazon $\rightarrow$ yahoo
6	T5	yelp $\rightarrow$ amazon $\rightarrow$ mnli $\rightarrow$ cb $\rightarrow$ copa $\rightarrow$ qqp $\rightarrow$ rte $\rightarrow$ imdb $\rightarrow$ sst-2 $\rightarrow$ dbpedia $\rightarrow$ ag $\rightarrow$ yahoo $\rightarrow$ multirc $\rightarrow$ boolqa $\rightarrow$ wic

Appendix

Table 14 : Task sequences for different model and benchmark settings.

Explore similar work

ChainLoRA: Geometry-Preserving Task Vector Merging for Continual Learning in LLMs
Sep 30, 2026Hang Yin, Haozhe Wang, Yuhua Luo +3Replay-Based Continual Learning Parameter-Efficient Fine-Tuning Methods
Shanghai Jiao Tong University · Shanghai Innovation Institute · Beijing Zhongguancun Academy +3
Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning
Jul 26, 2026Reza Rahimi Azghan, Gautham Krishna Gudur, Giulia Pedrielli +2Replay-Based Continual Learning Catastrophic Forgetting
College of Health Solutions, Arizona State University, Phoenix, Arizona, USA · Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, Texas, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, Arizona, USA +1
ReCoLoRA: Spectrum-Aware Recursive Consolidation for Continual LLM Fine-Tuning
Jul 4, 2026Wentao LuLow-Rank Adaptation Adapters Large Language Model Fine-Tuning

Sep 30, 2026, stat.MLJ/K move · Enter open · S save

ChainLoRA: Geometry-Preserving Task Vector Merging for Continual Learning in LLMs

Hang Yin, Haozhe Wang, Yuhua Luo, Zhangqi Pan+2

Shanghai Jiao Tong University · Shanghai Innovation Institute · Beijing Zhongguancun Academy+3

Replay-Based Continual Learning Parameter-Efficient Fine-Tuning Methods Continual Learning Continual Model Merging

Open paper AskPDF arXiv

Continual parameter-efficient fine-tuning for large language models (LLMs) must balance retention of previously acquired knowledge, adaptation to new tasks, and strict parameter budgets. We present \textbf{ChainLoRA}, a replay-free continual merging framework built on chain-updated task-vector geometry. From a parameter-merging perspective, we formulate a geometric view of forgetting through a measurable interaction between task updates, separating directional overlap from coefficient coupling. Building on this view, ChainLoRA combines chain-updated training with post-stream adaptive SVD merging. During training, initialization and a one-sided orthogonality proxy use only the last carrier, keeping their historical-state footprint and regularization overhead constant as the task stream grows. At merging time, Adaptive SVD extracts a shared carrier and aligns it to the latest task through Procrustes adaptation. Our theoretical analysis shows that Procrustes adaptation facilitates geometric approximate separation of shared and task-specific components. The one-sided proxy further bounds inter-task interference. An effective-rank penalty additionally promotes efficient utilization of the task subspace during continual learning. Experiments show that ChainLoRA achieves state-of-the-art performance among the evaluated replay-free methods on the Large and SuperNI benchmarks, while remaining competitive on Standard CL and attaining almost the closest average scores to the evaluated replay-based method across all three benchmarks.

Abstract

Explore similar work

ChainLoRA: Geometry-Preserving Task Vector Merging for Continual Learning in LLMs

Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning

ReCoLoRA: Spectrum-Aware Recursive Consolidation for Continual LLM Fine-Tuning