Making LoRA Identifiable: Orthogonal Alignment for Continual Learning

Pratik Rakesh Singh; Mohammadi Zaki; Akash Saha; Aneesh Mukkamala; Pankaj Wasnik

cs.LGOct 12, 2026

Making LoRA Identifiable: Orthogonal Alignment for Continual Learning

Authors: Pratik Rakesh Singh, Mohammadi Zaki, Akash Saha, Aneesh Mukkamala, Pankaj Wasnik

Organizations: Sony Research India

Abstract

There is growing interest in applying Parameter-Efficient Fine-Tuning (PEFT) to Continual Learning, as it enables faster training, mitigates catastrophic forgetting, and facilitates better adaptation to new tasks. Prior approaches have explored a range of strategies, including extending new LoRA branches for each incoming task, training under orthogonal loss objectives across all task-specific LoRA modules, and employing gating mechanisms to integrate new and existing LoRA modules. However, these methods incur memory costs that grow linearly with the number of tasks, limiting their scalability. More recent approaches investigate training a single LoRA by leveraging asymmetries in its parameterization; however, they often suffer from insufficient representational alignment across the LoRA parameter space. To address these limitations, we propose Stiefel Optimization and Aligned Rotation (SOAR), a two-stage framework for continual LoRA learning. First, we constrain the A matrix to lie on the Stiefel manifold, thereby reducing coordinate misalignment to the space of orthogonal rotation matrices. Second, we introduce an Orthogonal Procrustes alignment procedure for the B matrix to estimate the optimal rotation, coupled with a per-rank retention merging strategy that quantifies directional agreement between representations of new and previously learned tasks. We conduct extensive experiments across multiple models and benchmarks, demonstrating that SOAR achieves strong continual learning performance without incurring additional memory overhead.

Figures & tables

	Standard CL Benchmark				Large Number of Tasks				SuperNI Benchmark
Method	O1 $\uparrow$	O2 $\uparrow$	O3 $\uparrow$	avg $\uparrow$	O4 $\uparrow$	O5 $\uparrow$	O6 $\uparrow$	avg $\uparrow$	O7 $\uparrow$	O8 $\uparrow$	avg $\uparrow$
SeqLoRA	77.92	78.05	78.10	78.02	61.10	61.33	61.15	61.19	29.44	31.43	30.43
IncLoRA	82.20	82.40	82.92	82.50	65.20	65.50	65.40	65.36	33.44	35.39	34.42
O-LoRA	83.30	83.20	83.35	83.28	67.18	67.08	67.35	67.20	34.85	34.66	34.76
Gain-LoRA	83.27	83.36	83.80	83.48	67.39	67.39	67.41	67.39	36.42	37.38	36.90
CorDA	83.30	83.30	83.15	83.25	67.06	67.12	67.13	67.10	30.01	30.66	30.33

Table 1: (%) Performance comparison on three CL benchmark datasets using Qwen3-8B, across different task order, where

O_{i}

represents

i^{th}

task order.

	Standard CL Benchmark				Large Number of Tasks				SuperNI Benchmark
Method	O1 $\uparrow$	O2 $\uparrow$	O3 $\uparrow$	avg $\uparrow$	O4 $\uparrow$	O5 $\uparrow$	O6 $\uparrow$	avg $\uparrow$	O7 $\uparrow$	O8 $\uparrow$	avg $\uparrow$
QR+lambda	85.25	85.75	86.98	85.99	68.19	68.78	68.507	68.49	47.72	47.96	47.84
Stiefel+lambda	86.10	85.80	86.75	86.21	69.05	69.38	68.96	69.13	48.35	48.43	48.40
QR+SOAR Merging	87.80	87.50	87.70	87.67	70.17	70.01	69.88	70.02	48.91	49.68	49.29
SOAR	89.35	89.95	89.15	89.48	71.23	70.93	70.72	70.96	51.28	51.90	51.59

Table 2: (%) Performance comparison of different initialization and merging on Qwen3-8B.

		Standard CL Benchmark				Large Number of Tasks				SuperNI Benchmark
Model	Method	O1 $\uparrow$	O2 $\uparrow$	O3 $\uparrow$	avg $\uparrow$	O4 $\uparrow$	O5 $\uparrow$	O6 $\uparrow$	avg $\uparrow$	O7 $\uparrow$	O8 $\uparrow$	avg $\uparrow$
Llama-3.2-3B	SLAO	82.12	81.48	83.93	82.51	57.31	62.13	59.42	59.92	46.07	45.52	45.79
Llama-3.2-3B	SOAR	85.88	86.45	87.13	86.48	59.71	63.31	65.93	62.98	47.04	48.87	47.96
Llama-3.1-8B	SLAO	87.08	87.53	87.35	87.32	65.25	68.76	65.80	66.60	47.50	48.18	47.84
Llama-3.1-8B	SOAR	89.35	90.10	89.70	89.72	68.42	69.88	68.49	68.90	49.98	51.83	50.91
Qwen3-8B	SLAO	85.25	85.75	86.98	85.99	68.19	68.78	68.51	68.49	47.72	47.96	47.84

Table 3: Performance comparison of different model sizes and families.

Variant	$\gamma_{i}$	O1	O4	O7	Avg $\mathbf{(\Delta)}$
Random gate	$\sim\mathcal{U}[0,1]$	80.10	62.55	41.59	61.41 (-9.21)
Cosine-only	$\max(c_{i},0)$	86.05	65.33	44.41	65.26 (-5.36)
Residual-only	$\exp(-\delta_{i}/\tau)$	86.65	68.41	49.22	68.09 (-2.53)
SOAR (full)	$\max(c_{i},0)\exp(-\delta_{i}/\tau)$	89.35	71.23	51.28	70.62

Table 4: Additional Ablation on Gating

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

	Standard CL Benchmark				Large Number of Tasks				SuperNI Benchmark
Method	O1 $\uparrow$	O2 $\uparrow$	O3 $\uparrow$	avg $\uparrow$	O4 $\uparrow$	O5 $\uparrow$	O6 $\uparrow$	avg $\uparrow$	O7 $\uparrow$	O8 $\uparrow$	avg $\uparrow$
$r\to 8$	88.82	88.88	88.38	88.69	69.34	69.07	69.70	69.37	49.50	50.09	49.79
$r\to 16$	88.78	88.87	88.60	88.72	70.20	70.13	70.19	70.17	49.58	50.83	50.21
$r\to 32$	89.35	89.95	89.15	89.48	71.23	70.93	70.72	70.96	51.28	51.90	51.59
$r\to 64$	88.60	88.78	89.13	88.83	71.40	70.44	70.80	70.88	51.60	51.42	51.51

Appendix

Table 5: Effect of different ranks on SOAR using Qwen3-8B.

Method	A (SCL)	B (SCL)	C (SCL)	A (LNT)	B (LNT)	C (LNT)	A (SNI)	B (SNI)	C (SNI)
SeqLoRA	-12.40	-12.85	-13.62	-14.41	-14.78	-15.81	-17.57	-18.04	-19.42
IncLoRA	-8.02	-8.74	-9.38	-8.41	-9.12	-10.21	-14.98	-15.46	-16.84
O-LoRA	-7.10	-7.58	-8.16	-5.66	-6.18	-6.91	-13.78	-14.02	-15.18
Gain-LoRA	-7.35	-7.96	-8.42	-4.54	-5.08	-5.84	-11.39	-11.87	-12.73
CoRDA	-7.02	-7.46	-8.37	-7.61	-8.04	-8.92	-15.56	-16.03	-17.44
SLAO	-4.12	-4.38	-5.18	-3.53	-3.91	-4.67	-6.04	-6.28	-6.21

Appendix

Table 6: BWT scores across task orders on different models.

Figure 1: (%) Performance analysis across different $\tau$ values for three task orders.

	Standard CL Benchmark				Large Number of Tasks				SuperNI Benchmark
Method	O1 $\uparrow$	O2 $\uparrow$	O3 $\uparrow$	avg $\uparrow$	O4 $\uparrow$	O5 $\uparrow$	O6 $\uparrow$	avg $\uparrow$	O7 $\uparrow$	O8 $\uparrow$	avg $\uparrow$
SeqLoRA	61.21	61.31	61.50	61.34	49.30	50.10	49.10	49.50	31.00	31.85	31.43
IncLoRA	66.23	66.40	66.44	66.35	53.41	53.10	53.15	53.22	32.64	32.75	32.70
O-LoRA	68.08	68.01	68.16	68.03	56.14	56.25	56.67	56.35	33.07	33.51	33.29
Gain-LoRA	68.45	68.80	68.70	68.65	53.70	53.20	54.10	53.67	33.54	33.70	33.62
CorDA	68.08	68.05	68.10	68.07	54.53	55.20	55.16	55.3	28.00	29.02	28.51

Appendix

Table 7: (%) Performance comparison on three CL benchmark datasets using LLaMA-3.2-3B-Instruct, across different task order, where

O_{i}

represents

i^{th}

task order.

	Standard CL Benchmark				Large Number of Tasks				SuperNI Benchmark
Method	O1 $\uparrow$	O2 $\uparrow$	O3 $\uparrow$	avg $\uparrow$	O4 $\uparrow$	O5 $\uparrow$	O6 $\uparrow$	avg $\uparrow$	O7 $\uparrow$	O8 $\uparrow$	avg $\uparrow$
SeqLoRA	77.3	76.75	77.9	77.32	61.38	60.4	61.76	61.18	29.43	30.3	29.87
IncLoRA	78.34	78.11	77.8	78.08	63.45	64.11	63.76	63.77	31.34	32.44	31.89
O-LoRA	79.82	80.11	79.21	79.71	66.28	65.16	65.67	65.70	34.83	36.11	35.47
Gain-LoRA	80.10	80.55	80.40	80.35	65.47	66.24	65.79	65.83	35.15	36.23	35.69
CorDA	79.87	79.98	79.82	79.89	65.08	64.65	65.67	65.13	31.98	31.05	31.52

Appendix

Table 8: (%) Performance comparison on three CL benchmark datasets using LLaMA-3.1-8B-Instruct, across different task order, where

O_{i}

represents

i^{th}

task order.

Figure 2:

\gamma

variance on similar vs dissimilar task, averaged across layers for rank 32

Figure 3: (%) Performance analysis averaged across the benchmark dataset.

Figure 4: Effect of Procrustes and Per Rank Gating using Qwen3-8B across varying datasets.

Dataset name	Category	Task	Domain	Metric
Amazon Polarity	CL Benchmark	Sentiment analysis	Amazon reviews	Accuracy
DBpedia	CL Benchmark	Topic classification	Wikipedia	Accuracy
Yahoo Answers	CL Benchmark	Topic classification	Yahoo Q&A	Accuracy
AG News	CL Benchmark	Topic classification	News	Accuracy
MNLI	GLUE	Natural language inference	Various	Accuracy
QQP	GLUE	Paraphrase detection	Quora	Accuracy

Appendix

Table 9: Descriptions of datasets used from standard continual learning benchmarks, GLUE, and SuperGLUE.

Method	Time Taken (seconds)
KnoTS	1300.0
LoRA-LEGO	1472.0
GainLoRA	1299.5
IncLoRA	1219.0
SLAO	1254.0
OLoRA	1452.0

Appendix

Table 10: Total compute time taken

$\mathbf{A}$ update rule	O1	O4	O7	Average
Random fixed $\mathbf{A}$	70.80	58.43	34.56	54.59
Freeze after first task	87.45	67.86	42.91	65.92
SOAR	89.35	71.23	51.28	70.62
SOAR + Procrustes $\mathbf{A}$ -merge	89.90	72.80	53.10	71.93

Appendix

Table 11: Ablations on initialisation strategies for

\mathbf{A}

Metric	O1	O4	O7
Mean principal-angle drift of $\mathbf{A}$	$3.50^{\circ}$	$5.04^{\circ}$	$12.59^{\circ}$

Appendix

Table 12: Measured principal-angle drift of

\mathbf{A}

Dataset name	Task	Category	Metric
multi-woz-user-utterance-generation	Dialogue generation	SuperNI	Rouge-L
diplomacy-text-generation	Dialogue generation	SuperNI	Rouge-L
personachat-generate-next	Dialogue generation	SuperNI	Rouge-L
outcome-extraction	Information extraction	SuperNI	Rouge-L
glucose-reverse-cause-event-detection	Information extraction	SuperNI	Rouge-L
evaluation-relation-extraction	Information extraction	SuperNI	Rouge-L

Appendix

Table 13: Descriptions of datasets used from SNLI-derived SuperNI benchmark.

Order	Task Sequence
1	dbpedia $\rightarrow$ ag $\rightarrow$ amazon $\rightarrow$ yahoo
2	dbpedia $\rightarrow$ amazon $\rightarrow$ ag $\rightarrow$ yahoo
3	ag $\rightarrow$ yahoo $\rightarrow$ dbpedia $\rightarrow$ amazon
4	amazon $\rightarrow$ qqp $\rightarrow$ mnli $\rightarrow$ imdb $\rightarrow$ sst-2 $\rightarrow$ wic $\rightarrow$ yahoo $\rightarrow$ boolq
	$\rightarrow$ ag $\rightarrow$ dbpedia $\rightarrow$ rte $\rightarrow$ multirc $\rightarrow$ record
5	mnli $\rightarrow$ imdb $\rightarrow$ rte $\rightarrow$ record $\rightarrow$ boolq $\rightarrow$ qqp $\rightarrow$ sst-2 $\rightarrow$ yahoo

Appendix

Table 14: Eight different task orders used in continual learning experiments.

Orthogonal Procrustes Alignment

See all 11 papers

Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning
Jul 26, 2026Reza Rahimi Azghan, Gautham Krishna Gudur, Giulia Pedrielli +2Continual Learning for LLMs Continual LoRA
College of Health Solutions, Arizona State University, Phoenix, Arizona, USA · Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, Texas, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, Arizona, USA +1
Janus-LoRA: A Balanced Low-Rank Adaptation for Continual Learning
May 27, 2026Cheng Chen, Pengpeng Zeng, Yuyu Guo +3Stability-Plasticity Dilemma Continual Learning
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China · School of Computer Science and Technology, Tongji University, Shanghai, China · Independent Researcher +1
Continual Fine-Tuning of Large Language Models via Program Memory
May 13, 2026Hung Le, Svetha VenkateshContinual Learning for LLMs Memory-Augmented Language Models
Deakin Applied AI Initiative, Deakin University, Australia

Jul 26, 2026J/K move · Enter open · S save

Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning

Reza Rahimi Azghan, Gautham Krishna Gudur, Giulia Pedrielli, Pavan Turaga+1

College of Health Solutions, Arizona State University, Phoenix, Arizona, USA · Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, Texas, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, Arizona, USA+1

Continual Learning for LLMs Continual LoRA Incremental Learning Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Large language models generalize well to individual tasks but lack an inherent mechanism for learning them sequentially, leading to catastrophic forgetting. To mitigate this, LoRA-based continual learning methods allocate a separate low-rank adapter per task, yet existing approaches either require task identity at inference or sum all adapters indiscriminately, letting irrelevant branches distort the output. Recent gating-based solutions route inputs to the correct adapter but introduce trainable parameters that themselves need protection against forgetting. In this work, we observe that pooled token embeddings from a frozen LLM embedding layer already separate task distributions throughout the learning sequence. A Gaussian mixture model fitted on these embeddings, without any gradient-based training, is sufficient for task-agnostic adapter selection at test time. This eliminates the need for a learned gating module. On the adapter side, constraining each task's parameters to the principal subspace of the pretrained weights via SVD yields a compact latent-space parameterization. Within this subspace, orthogonal regularization directly controls inter-task interference. The resulting system, Latent-LoRA, is replay-free, requires no trainable routing component, and uses substantially fewer parameters per task. Experiments across five model scales and two established continual learning benchmarks show state-of-the-art performance with near-zero forgetting.

Abstract

Orthogonal Procrustes Alignment

Explore similar work

Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning

Janus-LoRA: A Balanced Low-Rank Adaptation for Continual Learning

Continual Fine-Tuning of Large Language Models via Program Memory