ChainLoRA: Geometry-Preserving Task Vector Merging for Continual Learning in LLMs

Hang Yin; Haozhe Wang; Yuhua Luo; Zhangqi Pan; Xiaoxing Wang; Junchi Yan

stat.MLSep 30, 2026

ChainLoRA: Geometry-Preserving Task Vector Merging for Continual Learning in LLMs

Authors: Hang Yin, Haozhe Wang, Yuhua Luo, Zhangqi Pan, Xiaoxing Wang, Junchi Yan

Organizations: Shanghai Jiao Tong University · Shanghai Innovation Institute · Beijing Zhongguancun Academy · StepFun · Huazhong University of Science and Technology · IAAR

Abstract

Continual parameter-efficient fine-tuning for large language models (LLMs) must balance retention of previously acquired knowledge, adaptation to new tasks, and strict parameter budgets. We present \textbf{ChainLoRA}, a replay-free continual merging framework built on chain-updated task-vector geometry. From a parameter-merging perspective, we formulate a geometric view of forgetting through a measurable interaction between task updates, separating directional overlap from coefficient coupling. Building on this view, ChainLoRA combines chain-updated training with post-stream adaptive SVD merging. During training, initialization and a one-sided orthogonality proxy use only the last carrier, keeping their historical-state footprint and regularization overhead constant as the task stream grows. At merging time, Adaptive SVD extracts a shared carrier and aligns it to the latest task through Procrustes adaptation. Our theoretical analysis shows that Procrustes adaptation facilitates geometric approximate separation of shared and task-specific components. The one-sided proxy further bounds inter-task interference. An effective-rank penalty additionally promotes efficient utilization of the task subspace during continual learning. Experiments show that ChainLoRA achieves state-of-the-art performance among the evaluated replay-free methods on the Large and SuperNI benchmarks, while remaining competitive on Standard CL and attaining almost the closest average scores to the evaluated replay-based method across all three benchmarks.

Figures & tables

Figure 1: Potential task interference in short CL (left) and long CL (right).

Figure 2: Overview of ChainLoRA. During the stream, each update is split by thin QR into a row-orthonormal carrier $\mathbf{P}_{i}$ and magnitude $\mathbf{W}_{i}^{(d)}$ ; the next task is initialized from the last carrier, and a one-sided proxy loss encourages disentanglement of common knowledge from that of the task-specific knowledge. After the stream, Adaptive SVD estimates a shared rank- $r$ carrier, rotates it in $\mathrm{SO}(r)$ — the group of $r$ -dimensional rotations — onto the latest-task carrier; residuals are then weighted by their distance from that, as described in Appendix D.3 .

Figure 3: Parameter-space crowding in long-horizon continual learning: short task streams (left) versus long task streams (right).

S_{5}

leads to forgetting.

	Standard CL Benchmark				Large Number of Tasks				SuperNI Benchmark
Method	O1	O2	O3	avg	O4	O5	O6	avg	O1	O2	avg
SeqLoRA	73.3	76.2	78.4	76.0	69.1	66.0	71.1	68.7	18.4	26.8	22.6
IncLoRA	75.3	77.3	78.3	77.0	72.2	71.6	73.8	72.5	22.0	25.6	23.8
O-LoRA	76.1	76.3	79.2	77.2	74.0	72.0	74.6	73.5	23.3	28.4	25.9
InfLoRA	78.4	80.4	79.9	79.6	69.4	67.4	72.5	69.8	16.5	22.1	19.3
LoRM-BA	76.0	76.8	78.3	77.0	71.4	69.0	70.3	70.2	25.6	18.7	22.2

Table 1: Final average task scores (%) for Llama-2-7B-chat (

Oi

=order). Only the replay-free method is compared. Methods are listed in the order described above, while the two merging strategies of LORM are listed separately.

Figure 4: SuperNI SST-2 collapsed.

Variant	Standard CL avg	$\Delta$	Large avg	$\Delta$	SuperNI avg	$\Delta$
Full (Table 1 recipe)	80.1	–	75.5	–	33.8	–
w/o $\mathcal{L}_{g}^{i}$ ( $\lambda_{1}{=}0$ )	79.3	$-0.8$	74.4	$-1.1$	27.6	$-6.2$
w/o $\mathcal{L}_{\mathrm{rank}}^{i}$ ( $\lambda_{2}{=}0$ )	79.0	$-1.1$	75.4	$-0.1$	27.5	$-6.3$
w/o Adaptive SVD Merging	75.1	$-5.0$	73.2	$-2.3$	32.5	$-1.3$
w/o Chain-updated initialization	78.2	$-1.9$	74.8	$-0.7$	30.7	$-3.1$

Table 2: Ablation study of each modules as Table 1 .

	Standard CL Benchmark				Large Number of Tasks				SuperNI Benchmark
Merging	O1	O2	O3	avg	O4	O5	O6	avg	O1	O2	avg
Average	74.2	65.8	55.0	65.0	67.6	66.7	68.4	67.6	28.4	29.2	28.8
SVD (without Procrustes)	41.6	64.3	60.8	55.6	$\dagger$	$\dagger$	54.3	–	10.4	12.2	11.3
Adaptive SVD (ours)	80.5	79.6	80.2	80.1	77.3	75.8	74.6	75.5	29.8	37.8	33.8

Table 3: Merging ablation study.

\dagger

marks results that ended in catastrophic forgetting (AA

=0

	Standard CL Benchmark				Large Number of Tasks				SuperNI Benchmark
Model	O1	O2	O3	avg	O4	O5	O6	avg	O1	O2	avg
Llama-3.2-3B	73.6	70.4	72.1	72.0	73.5	72.9	73.4	73.3	14.5	17.8	16.1
Llama-2-7B-chat	80.5	79.6	80.2	80.1	77.3	75.8	74.6	75.5	29.8	37.8	33.8
Llama-2-13B-chat	75.3	74.5	76.7	75.5	78.8	73.5	77.6	76.6	26.4	21.1	23.7

Table 4: Results across model size 3B/7B/13B.

Figure 5: Random vs. Chain init on Standard CL and Large.

	3B				7B				13B
Task	$a_{i,0}$	$a_{i,i}$	$a_{i,T}$	$\Delta$	$a_{i,0}$	$a_{i,i}$	$a_{i,T}$	$\Delta$	$a_{i,0}$	$a_{i,i}$	$a_{i,T}$	$\Delta$
DBpedia	1.4	86.6	84.6	$-2.0$	60.7	98.8	95.6	$-3.2$	71.4	98.7	97.2	$-1.6$
Amazon	3.5	54.8	52.8	$-2.0$	34.4	62.5	60.0	$-2.4$	44.6	54.0	59.9	$+5.9$
Yahoo	0.0	70.4	67.4	$-3.0$	14.9	74.3	71.2	$-3.1$	34.8	74.9	54.0	$-20.9$
AGNews	17.0	89.3	89.4	$+0.1$	61.9	90.1	90.0	$-0.1$	56.6	90.0	90.1	$+0.0$
AA / mean $\Delta$ on old	73.6			$-2.3$	79.2			$-2.9$	75.3			$-5.5$

Table 5: Standard CL O1 per-task scores (%) at three sizes (DBpedia

\to

Amazon

\to

Yahoo

\to

AGNews);

a_{i,0}

unseen,

a_{i,i}

just after task

i

a_{i,T}

after the stream,

\Delta{=}a_{i,T}{-}a_{i,i}

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

	Standard CL				Large number of tasks				SuperNI
Model	O1	O2	O3	avg	O4	O5	O6	avg	O1	O2	avg
Llama-3.2-3B	$-0.1$	$-7.0$	$-5.7$	$-4.3$	$-0.3$	$-2.9$	$-4.9$	$-2.7$	$-16.7$	$-24.6$	$-20.6$
Llama-2-7B-chat	$-0.6$	$-1.6$	$-1.9$	$-1.4$	$+0.4$	$-4.4$	$-5.7$	$-3.2$	$-10.8$	$-7.1$	$-8.9$
Llama-2-13B-chat	$-5.5$	$-7.1$	$-6.7$	$-6.4$	$-10.6$	$-22.0$	$-13.7$	$-15.4$	$-23.7$	$-30.8$	$-27.3$

Appendix

Table 6: Stored final BWT for the model-size grid in Table 4 . The lower value indicates better performance in hindering forgetting.

Figure 6: Running BWT after each completed task, for the runs in Table 6 .

Figure 7: Running AA after each completed task, for the runs in Table 6 . At step

t

\mathrm{AA}_{t}

averages one score for each task seen so far.

Explore similar work

Jul 26, 2026cs.LG

Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning

Large language models generalize well to individual tasks but lack an inherent mechanism for learning them sequentially, leading to catastrophic forgetting. To mitigate this, LoRA-based continual learning methods allocate a separate low-rank adapter per task, yet existing approaches either require task identity at inference or sum all adapters indiscriminately, letting irrelevant branches distort the output. Recent gating-based solutions route inputs to the correct adapter but introduce trainable parameters that themselves need protection against forgetting. In this work, we observe that pooled token embeddings from a frozen LLM embedding layer already separate task distributions throughout the learning sequence. A Gaussian mixture model fitted on these embeddings, without any gradient-based training, is sufficient for task-agnostic adapter selection at test time. This eliminates the need for a learned gating module. On the adapter side, constraining each task's parameters to the principal subspace of the pretrained weights via SVD yields a compact latent-space parameterization. Within this subspace, orthogonal regularization directly controls inter-task interference. The resulting system, Latent-LoRA, is replay-free, requires no trainable routing component, and uses substantially fewer parameters per task. Experiments across five model scales and two established continual learning benchmarks show state-of-the-art performance with near-zero forgetting.

Reza Rahimi Azghan, Gautham Krishna Gudur, Giulia Pedrielli +2

College of Health Solutions, Arizona State University, Phoenix, Arizona, USA · Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, Texas, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, Arizona, USA +1

Replay-Based Continual Learning Catastrophic Forgetting Multi-Lora Continual Learning Token Embeddings

Ask PDF

Jul 4, 2026cs.LG

ReCoLoRA: Spectrum-Aware Recursive Consolidation for Continual LLM Fine-Tuning

Parameter-efficient fine-tuning adapts a large language model to one task cheaply, but across a task sequence LoRA-style methods keep stacking low-rank updates on the same frozen weight, so each new task tends to overwrite the previous ones. We present ReCoLoRA (Recursive Consolidation of Low-Rank Adapters), a spectrum-aware framework for continual fine-tuning: adapters are initialized from a randomized SVD of the pretrained weight, per-layer effective ranks are selected by an elbow criterion, and the principal subspace is adapted before residual capacity is opened. Before each new task, ReCoLoRA re-decomposes the current effective weight, rather than the original one, into a frozen residual, a slowly updated principal component, and a fresh adapter (recursive consolidation), so every task starts from the model that has already absorbed its predecessors. On a six-task continual GLUE sequence over four 7-8B backbones, ReCoLoRA attains the best final average score on three of the four backbones against rank-swept LoRA, PiSSA, AdaLoRA, and DoRA baselines while training fewer parameters; an oracle-routed task-bank variant serves as an upper bound under full task isolation. Code: https://github.com/bhqy666/ReCoLoRA.

Wentao Lu

Low-Rank Adaptation Adapters Large Language Model Fine-Tuning Continual Fine-Tuning Frequency-Aware Regularization

Ask PDF

May 13, 2026cs.LG

Continual Fine-Tuning of Large Language Models via Program Memory

Parameter-Efficient Fine-Tuning (PEFT), particularly Low-Rank Adaptation (LoRA), has become a standard approach for adapting Large Language Models (LLMs) under limited compute. However, in continual settings where models are updated sequentially with small datasets, conventional LoRA updates struggle to balance rapid adaptation and knowledge retention. Existing methods typically treat the low-rank space as a homogeneous update region, lacking mechanisms to regulate how short-term updates are consolidated over time. We propose a continual LoRA framework with \textbf{Pro}gram memory, inspired by \textbf{C}omplementary \textbf{L}earning Systems in neuroscience. Our approach, dubbed \textbf{ProCL}, organizes LoRA adapters into structured program memory slots that are dynamically retrieved through input-conditioned attention. This enables rapid and localized adaptation, encouraging similar inputs to reuse shared adapter regions while reserving unused capacity for future data. The slots are then combined with the underlying adapter, which maintains a distributed representation that gradually accumulates knowledge across tasks to balance plasticity and stability. Our method operates entirely within the LoRA parameterization and incurs no additional inference cost. Experiments on diverse benchmarks demonstrate improved retention and reduced catastrophic forgetting over other continual LoRA strategies.

Hung Le, Svetha Venkatesh

Deakin Applied AI Initiative, Deakin University, Australia

Training-Side Stage-Aware Low-Rank Adaptation Replay-Based Continual Learning Catastrophic Forgetting Model Fine-Tuning Continual Fine-Tuning Peak Memory+1

Ask PDF