Making LoRA Identifiable: Orthogonal Alignment for Continual Learning

Pratik Rakesh Singh; Mohammadi Zaki; Akash Saha; Aneesh Mukkamala; Pankaj Wasnik

cs.LGOct 12, 2026

Making LoRA Identifiable: Orthogonal Alignment for Continual Learning

Authors: Pratik Rakesh Singh, Mohammadi Zaki, Akash Saha, Aneesh Mukkamala, Pankaj Wasnik

Organizations: Sony Research India

Abstract

There is growing interest in applying Parameter-Efficient Fine-Tuning (PEFT) to Continual Learning, as it enables faster training, mitigates catastrophic forgetting, and facilitates better adaptation to new tasks. Prior approaches have explored a range of strategies, including extending new LoRA branches for each incoming task, training under orthogonal loss objectives across all task-specific LoRA modules, and employing gating mechanisms to integrate new and existing LoRA modules. However, these methods incur memory costs that grow linearly with the number of tasks, limiting their scalability. More recent approaches investigate training a single LoRA by leveraging asymmetries in its parameterization; however, they often suffer from insufficient representational alignment across the LoRA parameter space. To address these limitations, we propose Stiefel Optimization and Aligned Rotation (SOAR), a two-stage framework for continual LoRA learning. First, we constrain the A matrix to lie on the Stiefel manifold, thereby reducing coordinate misalignment to the space of orthogonal rotation matrices. Second, we introduce an Orthogonal Procrustes alignment procedure for the B matrix to estimate the optimal rotation, coupled with a per-rank retention merging strategy that quantifies directional agreement between representations of new and previously learned tasks. We conduct extensive experiments across multiple models and benchmarks, demonstrating that SOAR achieves strong continual learning performance without incurring additional memory overhead.

Figures & tables

	Standard CL Benchmark				Large Number of Tasks				SuperNI Benchmark
Method	O1 $\uparrow$	O2 $\uparrow$	O3 $\uparrow$	avg $\uparrow$	O4 $\uparrow$	O5 $\uparrow$	O6 $\uparrow$	avg $\uparrow$	O7 $\uparrow$	O8 $\uparrow$	avg $\uparrow$
SeqLoRA	77.92	78.05	78.10	78.02	61.10	61.33	61.15	61.19	29.44	31.43	30.43
IncLoRA	82.20	82.40	82.92	82.50	65.20	65.50	65.40	65.36	33.44	35.39	34.42
O-LoRA	83.30	83.20	83.35	83.28	67.18	67.08	67.35	67.20	34.85	34.66	34.76
Gain-LoRA	83.27	83.36	83.80	83.48	67.39	67.39	67.41	67.39	36.42	37.38	36.90
CorDA	83.30	83.30	83.15	83.25	67.06	67.12	67.13	67.10	30.01	30.66	30.33

Table 1: (%) Performance comparison on three CL benchmark datasets using Qwen3-8B, across different task order, where

O_{i}

represents

i^{th}

task order.

	Standard CL Benchmark				Large Number of Tasks				SuperNI Benchmark
Method	O1 $\uparrow$	O2 $\uparrow$	O3 $\uparrow$	avg $\uparrow$	O4 $\uparrow$	O5 $\uparrow$	O6 $\uparrow$	avg $\uparrow$	O7 $\uparrow$	O8 $\uparrow$	avg $\uparrow$
QR+lambda	85.25	85.75	86.98	85.99	68.19	68.78	68.507	68.49	47.72	47.96	47.84
Stiefel+lambda	86.10	85.80	86.75	86.21	69.05	69.38	68.96	69.13	48.35	48.43	48.40
QR+SOAR Merging	87.80	87.50	87.70	87.67	70.17	70.01	69.88	70.02	48.91	49.68	49.29
SOAR	89.35	89.95	89.15	89.48	71.23	70.93	70.72	70.96	51.28	51.90	51.59

Table 2: (%) Performance comparison of different initialization and merging on Qwen3-8B.

		Standard CL Benchmark				Large Number of Tasks				SuperNI Benchmark
Model	Method	O1 $\uparrow$	O2 $\uparrow$	O3 $\uparrow$	avg $\uparrow$	O4 $\uparrow$	O5 $\uparrow$	O6 $\uparrow$	avg $\uparrow$	O7 $\uparrow$	O8 $\uparrow$	avg $\uparrow$
Llama-3.2-3B	SLAO	82.12	81.48	83.93	82.51	57.31	62.13	59.42	59.92	46.07	45.52	45.79
Llama-3.2-3B	SOAR	85.88	86.45	87.13	86.48	59.71	63.31	65.93	62.98	47.04	48.87	47.96
Llama-3.1-8B	SLAO	87.08	87.53	87.35	87.32	65.25	68.76	65.80	66.60	47.50	48.18	47.84
Llama-3.1-8B	SOAR	89.35	90.10	89.70	89.72	68.42	69.88	68.49	68.90	49.98	51.83	50.91
Qwen3-8B	SLAO	85.25	85.75	86.98	85.99	68.19	68.78	68.51	68.49	47.72	47.96	47.84

Table 3: Performance comparison of different model sizes and families.

Variant	$\gamma_{i}$	O1	O4	O7	Avg $\mathbf{(\Delta)}$
Random gate	$\sim\mathcal{U}[0,1]$	80.10	62.55	41.59	61.41 (-9.21)
Cosine-only	$\max(c_{i},0)$	86.05	65.33	44.41	65.26 (-5.36)
Residual-only	$\exp(-\delta_{i}/\tau)$	86.65	68.41	49.22	68.09 (-2.53)
SOAR (full)	$\max(c_{i},0)\exp(-\delta_{i}/\tau)$	89.35	71.23	51.28	70.62

Table 4: Additional Ablation on Gating

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

	Standard CL Benchmark				Large Number of Tasks				SuperNI Benchmark
Method	O1 $\uparrow$	O2 $\uparrow$	O3 $\uparrow$	avg $\uparrow$	O4 $\uparrow$	O5 $\uparrow$	O6 $\uparrow$	avg $\uparrow$	O7 $\uparrow$	O8 $\uparrow$	avg $\uparrow$
$r\to 8$	88.82	88.88	88.38	88.69	69.34	69.07	69.70	69.37	49.50	50.09	49.79
$r\to 16$	88.78	88.87	88.60	88.72	70.20	70.13	70.19	70.17	49.58	50.83	50.21
$r\to 32$	89.35	89.95	89.15	89.48	71.23	70.93	70.72	70.96	51.28	51.90	51.59
$r\to 64$	88.60	88.78	89.13	88.83	71.40	70.44	70.80	70.88	51.60	51.42	51.51

Appendix

Table 5: Effect of different ranks on SOAR using Qwen3-8B.

Method	A (SCL)	B (SCL)	C (SCL)	A (LNT)	B (LNT)	C (LNT)	A (SNI)	B (SNI)	C (SNI)
SeqLoRA	-12.40	-12.85	-13.62	-14.41	-14.78	-15.81	-17.57	-18.04	-19.42
IncLoRA	-8.02	-8.74	-9.38	-8.41	-9.12	-10.21	-14.98	-15.46	-16.84
O-LoRA	-7.10	-7.58	-8.16	-5.66	-6.18	-6.91	-13.78	-14.02	-15.18
Gain-LoRA	-7.35	-7.96	-8.42	-4.54	-5.08	-5.84	-11.39	-11.87	-12.73
CoRDA	-7.02	-7.46	-8.37	-7.61	-8.04	-8.92	-15.56	-16.03	-17.44
SLAO	-4.12	-4.38	-5.18	-3.53	-3.91	-4.67	-6.04	-6.28	-6.21

Appendix

Table 6: BWT scores across task orders on different models.

Figure 1: (%) Performance analysis across different $\tau$ values for three task orders.

	Standard CL Benchmark				Large Number of Tasks				SuperNI Benchmark
Method	O1 $\uparrow$	O2 $\uparrow$	O3 $\uparrow$	avg $\uparrow$	O4 $\uparrow$	O5 $\uparrow$	O6 $\uparrow$	avg $\uparrow$	O7 $\uparrow$	O8 $\uparrow$	avg $\uparrow$
SeqLoRA	61.21	61.31	61.50	61.34	49.30	50.10	49.10	49.50	31.00	31.85	31.43
IncLoRA	66.23	66.40	66.44	66.35	53.41	53.10	53.15	53.22	32.64	32.75	32.70
O-LoRA	68.08	68.01	68.16	68.03	56.14	56.25	56.67	56.35	33.07	33.51	33.29
Gain-LoRA	68.45	68.80	68.70	68.65	53.70	53.20	54.10	53.67	33.54	33.70	33.62
CorDA	68.08	68.05	68.10	68.07	54.53	55.20	55.16	55.3	28.00	29.02	28.51

Appendix

Table 7: (%) Performance comparison on three CL benchmark datasets using LLaMA-3.2-3B-Instruct, across different task order, where

O_{i}

represents

i^{th}

task order.

	Standard CL Benchmark				Large Number of Tasks				SuperNI Benchmark
Method	O1 $\uparrow$	O2 $\uparrow$	O3 $\uparrow$	avg $\uparrow$	O4 $\uparrow$	O5 $\uparrow$	O6 $\uparrow$	avg $\uparrow$	O7 $\uparrow$	O8 $\uparrow$	avg $\uparrow$
SeqLoRA	77.3	76.75	77.9	77.32	61.38	60.4	61.76	61.18	29.43	30.3	29.87
IncLoRA	78.34	78.11	77.8	78.08	63.45	64.11	63.76	63.77	31.34	32.44	31.89
O-LoRA	79.82	80.11	79.21	79.71	66.28	65.16	65.67	65.70	34.83	36.11	35.47
Gain-LoRA	80.10	80.55	80.40	80.35	65.47	66.24	65.79	65.83	35.15	36.23	35.69
CorDA	79.87	79.98	79.82	79.89	65.08	64.65	65.67	65.13	31.98	31.05	31.52

Appendix

Table 8: (%) Performance comparison on three CL benchmark datasets using LLaMA-3.1-8B-Instruct, across different task order, where

O_{i}

represents

i^{th}

task order.

Figure 2:

\gamma

variance on similar vs dissimilar task, averaged across layers for rank 32

Figure 3: (%) Performance analysis averaged across the benchmark dataset.

Figure 4: Effect of Procrustes and Per Rank Gating using Qwen3-8B across varying datasets.

Dataset name	Category	Task	Domain	Metric
Amazon Polarity	CL Benchmark	Sentiment analysis	Amazon reviews	Accuracy
DBpedia	CL Benchmark	Topic classification	Wikipedia	Accuracy
Yahoo Answers	CL Benchmark	Topic classification	Yahoo Q&A	Accuracy
AG News	CL Benchmark	Topic classification	News	Accuracy
MNLI	GLUE	Natural language inference	Various	Accuracy
QQP	GLUE	Paraphrase detection	Quora	Accuracy

Appendix

Table 9: Descriptions of datasets used from standard continual learning benchmarks, GLUE, and SuperGLUE.

Method	Time Taken (seconds)
KnoTS	1300.0
LoRA-LEGO	1472.0
GainLoRA	1299.5
IncLoRA	1219.0
SLAO	1254.0
OLoRA	1452.0

Appendix

Table 10: Total compute time taken

$\mathbf{A}$ update rule	O1	O4	O7	Average
Random fixed $\mathbf{A}$	70.80	58.43	34.56	54.59
Freeze after first task	87.45	67.86	42.91	65.92
SOAR	89.35	71.23	51.28	70.62
SOAR + Procrustes $\mathbf{A}$ -merge	89.90	72.80	53.10	71.93

Appendix

Table 11: Ablations on initialisation strategies for

\mathbf{A}

Metric	O1	O4	O7
Mean principal-angle drift of $\mathbf{A}$	$3.50^{\circ}$	$5.04^{\circ}$	$12.59^{\circ}$

Appendix

Table 12: Measured principal-angle drift of

\mathbf{A}

Dataset name	Task	Category	Metric
multi-woz-user-utterance-generation	Dialogue generation	SuperNI	Rouge-L
diplomacy-text-generation	Dialogue generation	SuperNI	Rouge-L
personachat-generate-next	Dialogue generation	SuperNI	Rouge-L
outcome-extraction	Information extraction	SuperNI	Rouge-L
glucose-reverse-cause-event-detection	Information extraction	SuperNI	Rouge-L
evaluation-relation-extraction	Information extraction	SuperNI	Rouge-L

Appendix

Table 13: Descriptions of datasets used from SNLI-derived SuperNI benchmark.

Order	Task Sequence
1	dbpedia $\rightarrow$ ag $\rightarrow$ amazon $\rightarrow$ yahoo
2	dbpedia $\rightarrow$ amazon $\rightarrow$ ag $\rightarrow$ yahoo
3	ag $\rightarrow$ yahoo $\rightarrow$ dbpedia $\rightarrow$ amazon
4	amazon $\rightarrow$ qqp $\rightarrow$ mnli $\rightarrow$ imdb $\rightarrow$ sst-2 $\rightarrow$ wic $\rightarrow$ yahoo $\rightarrow$ boolq
	$\rightarrow$ ag $\rightarrow$ dbpedia $\rightarrow$ rte $\rightarrow$ multirc $\rightarrow$ record
5	mnli $\rightarrow$ imdb $\rightarrow$ rte $\rightarrow$ record $\rightarrow$ boolq $\rightarrow$ qqp $\rightarrow$ sst-2 $\rightarrow$ yahoo

Appendix

Table 14: Eight different task orders used in continual learning experiments.

Orthogonal Procrustes Alignment

See all 11 papers

Jul 26, 2026

Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning

Large language models generalize well to individual tasks but lack an inherent mechanism for learning them sequentially, leading to catastrophic forgetting. To mitigate this, LoRA-based continual learning methods allocate a separate low-rank adapter per task, yet existing approaches either require task identity at inference or sum all adapters indiscriminately, letting irrelevant branches distort the output. Recent gating-based solutions route inputs to the correct adapter but introduce trainable parameters that themselves need protection against forgetting. In this work, we observe that pooled token embeddings from a frozen LLM embedding layer already separate task distributions throughout the learning sequence. A Gaussian mixture model fitted on these embeddings, without any gradient-based training, is sufficient for task-agnostic adapter selection at test time. This eliminates the need for a learned gating module. On the adapter side, constraining each task's parameters to the principal subspace of the pretrained weights via SVD yields a compact latent-space parameterization. Within this subspace, orthogonal regularization directly controls inter-task interference. The resulting system, Latent-LoRA, is replay-free, requires no trainable routing component, and uses substantially fewer parameters per task. Experiments across five model scales and two established continual learning benchmarks show state-of-the-art performance with near-zero forgetting.

Reza Rahimi Azghan, Gautham Krishna Gudur, Giulia Pedrielli +2

College of Health Solutions, Arizona State University, Phoenix, Arizona, USA · Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, Texas, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, Arizona, USA +1

Continual Learning for LLMs Continual LoRA Incremental Learning Parameter-Efficient Fine-Tuning

Ask PDF

May 27, 2026

Janus-LoRA: A Balanced Low-Rank Adaptation for Continual Learning

Low-Rank Adaptation (LoRA) has emerged as a promising paradigm for Continual Learning. It independently updates its low-rank factors (

A

and

B

), creating a composite update to the full weight matrix through their interaction. To prevent catastrophic forgetting, this update should remain orthogonal to the task-specific subspace that contains previously learned knowledge. However, we identify that this composite update systematically violates this orthogonality, reintroducing interference and undermining stability. Furthermore, naively enforcing this orthogonality compromises plasticity, disrupting the delicate stability-plasticity trade-off. To resolve these issues, we propose Janus-LoRA, a framework that restores this balance through two novel components. Specifically, we first introduce Gradient Rectification, a closed-form solution that mathematically decouples LoRA's factor updates, enforcing orthogonality against the historical knowledge subspace identified by an efficient Online Estimation. Next, to enhance plasticity, we introduce a Decoupled Margin Loss that promotes feature-level separation by pushing new feature representations away from old ones, thus creating distinct, low-interference regions for new learning. Comprehensive experiments on challenging benchmarks demonstrate that by harmonizing parameter-level orthogonality with feature-level separation, Janus-LoRA achieves a superior balance and establishes new state-of-the-art performance.

Cheng Chen, Pengpeng Zeng, Yuyu Guo +3

School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China · School of Computer Science and Technology, Tongji University, Shanghai, China · Independent Researcher +1

Stability-Plasticity Dilemma Continual Learning Low-Rank Adaptation Parameter-Efficient Fine-Tuning Continual LoRA

Ask PDF

May 13, 2026

Continual Fine-Tuning of Large Language Models via Program Memory

Parameter-Efficient Fine-Tuning (PEFT), particularly Low-Rank Adaptation (LoRA), has become a standard approach for adapting Large Language Models (LLMs) under limited compute. However, in continual settings where models are updated sequentially with small datasets, conventional LoRA updates struggle to balance rapid adaptation and knowledge retention. Existing methods typically treat the low-rank space as a homogeneous update region, lacking mechanisms to regulate how short-term updates are consolidated over time. We propose a continual LoRA framework with Program memory, inspired by Complementary Learning Systems in neuroscience. Our approach, dubbed ProCL, organizes LoRA adapters into structured program memory slots that are dynamically retrieved through input-conditioned attention. This enables rapid and localized adaptation, encouraging similar inputs to reuse shared adapter regions while reserving unused capacity for future data. The slots are then combined with the underlying adapter, which maintains a distributed representation that gradually accumulates knowledge across tasks to balance plasticity and stability. Our method operates entirely within the LoRA parameterization and incurs no additional inference cost. Experiments on diverse benchmarks demonstrate improved retention and reduced catastrophic forgetting over other continual LoRA strategies.

Hung Le, Svetha Venkatesh

Deakin Applied AI Initiative, Deakin University, Australia

Continual Learning for LLMs Memory-Augmented Language Models Low-Rank Adaptation Continual LoRA Parameter-Efficient Fine-Tuning

Ask PDF

Abstract

Orthogonal Procrustes Alignment

Explore similar work

Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning

Janus-LoRA: A Balanced Low-Rank Adaptation for Continual Learning

Continual Fine-Tuning of Large Language Models via Program Memory