ChainLoRA: Geometry-Preserving Task Vector Merging for Continual Learning in LLMs

Hang Yin; Haozhe Wang; Yuhua Luo; Zhangqi Pan; Xiaoxing Wang; Junchi Yan

stat.MLSep 30, 2026

ChainLoRA: Geometry-Preserving Task Vector Merging for Continual Learning in LLMs

Authors: Hang Yin, Haozhe Wang, Yuhua Luo, Zhangqi Pan, Xiaoxing Wang, Junchi Yan

Organizations: Shanghai Jiao Tong University · Shanghai Innovation Institute · Beijing Zhongguancun Academy · StepFun · Huazhong University of Science and Technology · IAAR

Abstract

Continual parameter-efficient fine-tuning for large language models (LLMs) must balance retention of previously acquired knowledge, adaptation to new tasks, and strict parameter budgets. We present \textbf{ChainLoRA}, a replay-free continual merging framework built on chain-updated task-vector geometry. From a parameter-merging perspective, we formulate a geometric view of forgetting through a measurable interaction between task updates, separating directional overlap from coefficient coupling. Building on this view, ChainLoRA combines chain-updated training with post-stream adaptive SVD merging. During training, initialization and a one-sided orthogonality proxy use only the last carrier, keeping their historical-state footprint and regularization overhead constant as the task stream grows. At merging time, Adaptive SVD extracts a shared carrier and aligns it to the latest task through Procrustes adaptation. Our theoretical analysis shows that Procrustes adaptation facilitates geometric approximate separation of shared and task-specific components. The one-sided proxy further bounds inter-task interference. An effective-rank penalty additionally promotes efficient utilization of the task subspace during continual learning. Experiments show that ChainLoRA achieves state-of-the-art performance among the evaluated replay-free methods on the Large and SuperNI benchmarks, while remaining competitive on Standard CL and attaining almost the closest average scores to the evaluated replay-based method across all three benchmarks.

Figures & tables

Figure 1: Potential task interference in short CL (left) and long CL (right).

Figure 2: Overview of ChainLoRA. During the stream, each update is split by thin QR into a row-orthonormal carrier $\mathbf{P}_{i}$ and magnitude $\mathbf{W}_{i}^{(d)}$ ; the next task is initialized from the last carrier, and a one-sided proxy loss encourages disentanglement of common knowledge from that of the task-specific knowledge. After the stream, Adaptive SVD estimates a shared rank- $r$ carrier, rotates it in $\mathrm{SO}(r)$ — the group of $r$ -dimensional rotations — onto the latest-task carrier; residuals are then weighted by their distance from that, as described in Appendix D.3 .

Figure 3: Parameter-space crowding in long-horizon continual learning: short task streams (left) versus long task streams (right).

S_{5}

leads to forgetting.

	Standard CL Benchmark				Large Number of Tasks				SuperNI Benchmark
Method	O1	O2	O3	avg	O4	O5	O6	avg	O1	O2	avg
SeqLoRA	73.3	76.2	78.4	76.0	69.1	66.0	71.1	68.7	18.4	26.8	22.6
IncLoRA	75.3	77.3	78.3	77.0	72.2	71.6	73.8	72.5	22.0	25.6	23.8
O-LoRA	76.1	76.3	79.2	77.2	74.0	72.0	74.6	73.5	23.3	28.4	25.9
InfLoRA	78.4	80.4	79.9	79.6	69.4	67.4	72.5	69.8	16.5	22.1	19.3
LoRM-BA	76.0	76.8	78.3	77.0	71.4	69.0	70.3	70.2	25.6	18.7	22.2

Table 1: Final average task scores (%) for Llama-2-7B-chat (

Oi

=order). Only the replay-free method is compared. Methods are listed in the order described above, while the two merging strategies of LORM are listed separately.

Figure 4: SuperNI SST-2 collapsed.

Variant	Standard CL avg	$\Delta$	Large avg	$\Delta$	SuperNI avg	$\Delta$
Full (Table 1 recipe)	80.1	–	75.5	–	33.8	–
w/o $\mathcal{L}_{g}^{i}$ ( $\lambda_{1}{=}0$ )	79.3	$-0.8$	74.4	$-1.1$	27.6	$-6.2$
w/o $\mathcal{L}_{\mathrm{rank}}^{i}$ ( $\lambda_{2}{=}0$ )	79.0	$-1.1$	75.4	$-0.1$	27.5	$-6.3$
w/o Adaptive SVD Merging	75.1	$-5.0$	73.2	$-2.3$	32.5	$-1.3$
w/o Chain-updated initialization	78.2	$-1.9$	74.8	$-0.7$	30.7	$-3.1$

Table 2: Ablation study of each modules as Table 1 .

	Standard CL Benchmark				Large Number of Tasks				SuperNI Benchmark
Merging	O1	O2	O3	avg	O4	O5	O6	avg	O1	O2	avg
Average	74.2	65.8	55.0	65.0	67.6	66.7	68.4	67.6	28.4	29.2	28.8
SVD (without Procrustes)	41.6	64.3	60.8	55.6	$\dagger$	$\dagger$	54.3	–	10.4	12.2	11.3
Adaptive SVD (ours)	80.5	79.6	80.2	80.1	77.3	75.8	74.6	75.5	29.8	37.8	33.8

Table 3: Merging ablation study.

\dagger

marks results that ended in catastrophic forgetting (AA

=0

	Standard CL Benchmark				Large Number of Tasks				SuperNI Benchmark
Model	O1	O2	O3	avg	O4	O5	O6	avg	O1	O2	avg
Llama-3.2-3B	73.6	70.4	72.1	72.0	73.5	72.9	73.4	73.3	14.5	17.8	16.1
Llama-2-7B-chat	80.5	79.6	80.2	80.1	77.3	75.8	74.6	75.5	29.8	37.8	33.8
Llama-2-13B-chat	75.3	74.5	76.7	75.5	78.8	73.5	77.6	76.6	26.4	21.1	23.7

Table 4: Results across model size 3B/7B/13B.

Figure 5: Random vs. Chain init on Standard CL and Large.

	3B				7B				13B
Task	$a_{i,0}$	$a_{i,i}$	$a_{i,T}$	$\Delta$	$a_{i,0}$	$a_{i,i}$	$a_{i,T}$	$\Delta$	$a_{i,0}$	$a_{i,i}$	$a_{i,T}$	$\Delta$
DBpedia	1.4	86.6	84.6	$-2.0$	60.7	98.8	95.6	$-3.2$	71.4	98.7	97.2	$-1.6$
Amazon	3.5	54.8	52.8	$-2.0$	34.4	62.5	60.0	$-2.4$	44.6	54.0	59.9	$+5.9$
Yahoo	0.0	70.4	67.4	$-3.0$	14.9	74.3	71.2	$-3.1$	34.8	74.9	54.0	$-20.9$
AGNews	17.0	89.3	89.4	$+0.1$	61.9	90.1	90.0	$-0.1$	56.6	90.0	90.1	$+0.0$
AA / mean $\Delta$ on old	73.6			$-2.3$	79.2			$-2.9$	75.3			$-5.5$

Table 5: Standard CL O1 per-task scores (%) at three sizes (DBpedia

\to

Amazon

\to

Yahoo

\to

AGNews);

a_{i,0}

unseen,

a_{i,i}

just after task

i

a_{i,T}

after the stream,

\Delta{=}a_{i,T}{-}a_{i,i}

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

	Standard CL				Large number of tasks				SuperNI
Model	O1	O2	O3	avg	O4	O5	O6	avg	O1	O2	avg
Llama-3.2-3B	$-0.1$	$-7.0$	$-5.7$	$-4.3$	$-0.3$	$-2.9$	$-4.9$	$-2.7$	$-16.7$	$-24.6$	$-20.6$
Llama-2-7B-chat	$-0.6$	$-1.6$	$-1.9$	$-1.4$	$+0.4$	$-4.4$	$-5.7$	$-3.2$	$-10.8$	$-7.1$	$-8.9$
Llama-2-13B-chat	$-5.5$	$-7.1$	$-6.7$	$-6.4$	$-10.6$	$-22.0$	$-13.7$	$-15.4$	$-23.7$	$-30.8$	$-27.3$

Appendix

Table 6: Stored final BWT for the model-size grid in Table 4 . The lower value indicates better performance in hindering forgetting.

Figure 6: Running BWT after each completed task, for the runs in Table 6 .

Figure 7: Running AA after each completed task, for the runs in Table 6 . At step

t

\mathrm{AA}_{t}

averages one score for each task seen so far.

Explore similar work

Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning
Jul 26, 2026Reza Rahimi Azghan, Gautham Krishna Gudur, Giulia Pedrielli +2Replay-Based Continual Learning Catastrophic Forgetting
College of Health Solutions, Arizona State University, Phoenix, Arizona, USA · Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, Texas, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, Arizona, USA +1
ReCoLoRA: Spectrum-Aware Recursive Consolidation for Continual LLM Fine-Tuning
Jul 4, 2026Wentao LuLow-Rank Adaptation Adapters Large Language Model Fine-Tuning
Continual Fine-Tuning of Large Language Models via Program Memory
May 13, 2026Hung Le, Svetha VenkateshTraining-Side Stage-Aware Low-Rank Adaptation Replay-Based Continual Learning
Deakin Applied AI Initiative, Deakin University, Australia

Jul 26, 2026, cs.LGJ/K move · Enter open · S save

Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning

Reza Rahimi Azghan, Gautham Krishna Gudur, Giulia Pedrielli, Pavan Turaga+1

College of Health Solutions, Arizona State University, Phoenix, Arizona, USA · Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, Texas, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, Arizona, USA+1

Replay-Based Continual Learning Catastrophic Forgetting Multi-Lora Continual Learning Token Embeddings

Open paper AskPDF arXiv

Large language models generalize well to individual tasks but lack an inherent mechanism for learning them sequentially, leading to catastrophic forgetting. To mitigate this, LoRA-based continual learning methods allocate a separate low-rank adapter per task, yet existing approaches either require task identity at inference or sum all adapters indiscriminately, letting irrelevant branches distort the output. Recent gating-based solutions route inputs to the correct adapter but introduce trainable parameters that themselves need protection against forgetting. In this work, we observe that pooled token embeddings from a frozen LLM embedding layer already separate task distributions throughout the learning sequence. A Gaussian mixture model fitted on these embeddings, without any gradient-based training, is sufficient for task-agnostic adapter selection at test time. This eliminates the need for a learned gating module. On the adapter side, constraining each task's parameters to the principal subspace of the pretrained weights via SVD yields a compact latent-space parameterization. Within this subspace, orthogonal regularization directly controls inter-task interference. The resulting system, Latent-LoRA, is replay-free, requires no trainable routing component, and uses substantially fewer parameters per task. Experiments across five model scales and two established continual learning benchmarks show state-of-the-art performance with near-zero forgetting.

Abstract

Explore similar work

Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning

ReCoLoRA: Spectrum-Aware Recursive Consolidation for Continual LLM Fine-Tuning

Continual Fine-Tuning of Large Language Models via Program Memory