PaLoRA: Paced Low-Rank Adaptation for Continual Learning

Yuxuan Li; Fanhu Zeng; Hao Tang

cs.LGOct 3, 2026

PaLoRA: Paced Low-Rank Adaptation for Continual Learning

Authors: Yuxuan Li, Fanhu Zeng, Hao Tang

Organizations: School of Computer Science Peking University · Institute of Automation Chinese Academy of Sciences

Abstract

LoRA-based continual learning methods mitigate catastrophic forgetting through various mechanisms, yet nearly all complement these with small learning rates as a heuristic to restrict gradient scaling magnitude. Such fixed heuristics lack theoretical guidance on how the strength of this restriction should evolve as tasks accumulate. We reveal that even under directional constraints such as nullspace projection, finite-precision updates inevitably leak into the subspace of accumulated prior knowledge along multiple directions. While small learning rates attenuate such leakage, they cannot prevent the accumulated forgetting from intensifying as the effective rank of historical knowledge grows. We show that the optimal magnitude restriction should adaptively increase with this effective rank to balance stability and plasticity, i.e., preservation of previous knowledge and acquisition of new task information. Under an anisotropic leakage model, we derive a pacing law $s^*=\sqrt{R/c}$ that characterizes the optimal scaling of gradient steps, i.e., the magnitude restriction itself, where $R$ is the effective rank of past updates. Based on this insight, we propose PaLoRA, which compresses historical knowledge via adaptive SVD truncation, projects gradients onto the nullspace of prior tasks, and applies rank-aware adaptive pacing. Experiments demonstrate consistent improvements over prior methods, with particularly strong performance in long-horizon settings, achieving substantial gains of 4% accuracy on challenging 50-task ImageNet-A and ImageNet-R benchmarks.

Figures & tables

Figure 1: Adaptive parameter update restrictions progressively shift optimization trajectory toward previous task solutions as more tasks are encountered, better balancing stability and plasticity to approach the joint optimum.

Figure 2: Overview of PaLoRA. (A) Model decomposition: pretrained weights and compressed memory from previous tasks are frozen, while a new low-rank update is learned for the current task. (B) Two-stage process: adaptive-rank compression of accumulated knowledge via SVD, followed by rank-aware gradient pacing with nullspace projection and scaled updates.

Tasks	10		20		25		50
Method	$ACC_{10}$	$\overline{ACC_{10}}$	$ACC_{20}$	$\overline{ACC_{20}}$	$ACC_{25}$	$\overline{ACC_{25}}$	$ACC_{50}$	$\overline{ACC_{50}}$
Pretrain	55.68 $\pm$ 0.00	59.20 $\pm$ 0.00	55.68 $\pm$ 0.00	59.55 $\pm$ 0.00	55.68 $\pm$ 0.00	59.78 $\pm$ 0.00	55.68 $\pm$ 0.00	59.95 $\pm$ 0.00
Sequence	62.11 $\pm$ 2.21	73.39 $\pm$ 0.97	57.30 $\pm$ 0.39	69.56 $\pm$ 0.69	54.48 $\pm$ 2.16	66.98 $\pm$ 0.28	47.48 $\pm$ 0.05	62.94 $\pm$ 0.60
L2P Wang et al. (2022b)	62.28 $\pm$ 0.30	66.98 $\pm$ 0.04	61.24 $\pm$ 0.16	64.73 $\pm$ 0.43	61.03 $\pm$ 0.35	64.37 $\pm$ 0.37	59.44 $\pm$ 0.45	63.16 $\pm$ 0.57
DualPrompt Wang et al. (2022a)	58.91 $\pm$ 0.23	62.66 $\pm$ 0.06	57.78 $\pm$ 0.33	61.91 $\pm$ 0.14	57.53 $\pm$ 0.64	61.66 $\pm$ 0.10	56.28 $\pm$ 0.97	61.72 $\pm$ 0.22
CODA-Prompt Smith et al. (2023)	69.81 $\pm$ 0.06	73.00 $\pm$ 0.15	66.42 $\pm$ 0.13	69.55 $\pm$ 0.35	65.03 $\pm$ 0.23	68.22 $\pm$ 0.18	56.19 $\pm$ 0.16	59.01 $\pm$ 0.27

Table 1: ImageNet-R accuracy (%, mean

\pm

standard deviation over five seeds). PaLoRA achieves the highest final and average accuracy across all splits, with a 4.09-point final-accuracy gain over LoRA - DRS at 50 tasks.

Tasks	10		20		25		50
Method	$ACC_{10}$	$\overline{ACC_{10}}$	$ACC_{20}$	$\overline{ACC_{20}}$	$ACC_{25}$	$\overline{ACC_{25}}$	$ACC_{50}$	$\overline{ACC_{50}}$
Pretrain	83.81 $\pm$ 0.00	88.82 $\pm$ 0.00	83.81 $\pm$ 0.00	89.13 $\pm$ 0.00	83.81 $\pm$ 0.00	89.16 $\pm$ 0.00	83.81 $\pm$ 0.00	89.33 $\pm$ 0.00
Sequence	87.54 $\pm$ 0.94	92.12 $\pm$ 0.78	81.51 $\pm$ 2.60	89.05 $\pm$ 0.48	81.93 $\pm$ 4.13	88.95 $\pm$ 1.11	77.52 $\pm$ 5.94	85.01 $\pm$ 4.23
L2P Wang et al. (2022b)	84.53 $\pm$ 0.71	87.84 $\pm$ 0.39	81.00 $\pm$ 0.51	84.70 $\pm$ 0.94	78.45 $\pm$ 0.37	82.08 $\pm$ 0.08	74.48 $\pm$ 1.35	78.90 $\pm$ 1.78
DualPrompt Wang et al. (2022a)	75.37 $\pm$ 2.35	86.03 $\pm$ 0.66	77.79 $\pm$ 0.04	86.49 $\pm$ 0.52	79.72 $\pm$ 0.85	86.82 $\pm$ 0.12	81.11 $\pm$ 0.77	87.37 $\pm$ 0.05
CODA-Prompt Smith et al. (2023)	84.40 $\pm$ 0.13	88.77 $\pm$ 0.38	81.69 $\pm$ 1.03	85.34 $\pm$ 0.70	79.10 $\pm$ 1.12	82.19 $\pm$ 0.87	72.16 $\pm$ 0.08	82.65 $\pm$ 0.49

Table 2: CIFAR-100 accuracy (%, mean

\pm

standard deviation over five seeds). PaLoRA consistently improves both metrics, reaching

88.96\pm 0.14

final accuracy at 50 tasks.

Tasks	10		20		25		50
Method	$ACC_{10}$	$\overline{ACC_{10}}$	$ACC_{20}$	$\overline{ACC_{20}}$	$ACC_{25}$	$\overline{ACC_{25}}$	$ACC_{50}$	$\overline{ACC_{50}}$
Pretrain	48.85 $\pm$ 0.00	57.39 $\pm$ 0.00	48.85 $\pm$ 0.00	58.47 $\pm$ 0.00	48.85 $\pm$ 0.00	58.56 $\pm$ 0.00	48.85 $\pm$ 0.00	58.97 $\pm$ 0.00
Sequence	34.13 $\pm$ 1.44	53.86 $\pm$ 2.04	23.87 $\pm$ 0.23	46.07 $\pm$ 1.37	25.97 $\pm$ 3.78	43.62 $\pm$ 0.24	15.61 $\pm$ 8.38	37.62 $\pm$ 1.41
L2P Wang et al. (2022b)	36.08 $\pm$ 0.65	41.66 $\pm$ 0.87	33.55 $\pm$ 0.70	38.95 $\pm$ 1.18	33.64 $\pm$ 0.56	39.08 $\pm$ 0.07	32.39 $\pm$ 0.65	37.55 $\pm$ 0.71
DualPrompt Wang et al. (2022a)	42.42 $\pm$ 3.12	53.20 $\pm$ 0.83	48.09 $\pm$ 0.14	56.99 $\pm$ 0.01	48.02 $\pm$ 0.33	57.19 $\pm$ 0.01	47.66 $\pm$ 0.84	57.69 $\pm$ 0.47
CODA-Prompt Smith et al. (2023)	52.17 $\pm$ 0.23	62.64 $\pm$ 0.39	42.59 $\pm$ 0.65	49.77 $\pm$ 1.08	39.30 $\pm$ 0.37	47.66 $\pm$ 0.32	47.53 $\pm$ 0.28	58.05 $\pm$ 0.21

Table 3: ImageNet-A accuracy (%, mean

\pm

standard deviation over five seeds). PaLoRA leads across all splits and exceeds LoRA - DRS by 3.99 points in final accuracy at 50 tasks.

Method	CIFAR-100		ImageNet-R		ImageNet-A
	25 Tasks	50 Tasks	25 Tasks	50 Tasks	25 Tasks	50 Tasks
Sequence	-16.24	-14.72	-36.86	-42.14	-31.62	-53.83
L2P Wang et al. (2022b)	-8.94	-14.07	-9.43	-10.56	-6.80	-11.16
DualPrompt Wang et al. (2022a)	-8.35	-6.46	-8.37	-9.72	-6.36	-8.79
CODA-Prompt Smith et al. (2023)	-9.83	-17.11	-6.02	-9.76	-10.71	-9.46
InfLoRA Liang and Li (2024)	-12.53	-20.45	-15.08	-20.01	-16.47	-18.34

Table 4: Backward transfer (BWT, %; higher is better) on all three datasets under 25- and 50-task splits. PaLoRA yields the least forgetting on CIFAR-100 and ImageNet-A, while LoRA - DRS has slightly better BWT on ImageNet-R. Values are point estimates.

Figure 3: Accuracy trajectories on CIFAR-100 (10 tasks), ImageNet-A (50 tasks), and ImageNet-R (50 tasks), from left to right. PaLoRA maintains strong accuracy, including over long task sequences.

Dataset	Method	10 Tasks		20 Tasks		25 Tasks		50 Tasks
Dataset	Method	$ACC_{10}$	$\overline{ACC_{10}}$	$ACC_{20}$	$\overline{ACC_{20}}$	$ACC_{25}$	$\overline{ACC_{25}}$	$ACC_{50}$	$\overline{ACC_{50}}$
ImageNet-R	SD-LoRA	78.33	82.85	74.28	79.45	75.37	80.82	69.68	76.24
ImageNet-R	PaLoRA	78.82	82.96	78.57	82.74	78.10	82.90	76.40	80.40
CIFAR-100	SD-LoRA	88.05	92.47	84.20	90.48	80.32	87.54	67.70	77.60
CIFAR-100	PaLoRA	90.98	93.99	90.31	93.67	90.25	93.69	88.86	92.53
ImageNet-A	SD-LoRA	57.21	70.68	50.63	65.06	46.94	58.19	31.99	54.78

Table 5: Direct comparison with SD-LoRA (accuracy in %; point estimates). PaLoRA achieves larger final-accuracy gains at 50 tasks than at 10 tasks on all three datasets.

Figure 4: Component ablation on ImageNet-A with 10, 20, and 50 tasks. Projection and pacing improve the trajectories, while triplet regularization provides an additional accuracy gain. Each setting uses a different class split, so early-task curves are not expected to coincide across panels.

Method	$ACC_{10}$	$\overline{ACC_{10}}$	$ACC_{50}$	$\overline{ACC_{50}}$
Sequence	86.88	91.57	81.72	87.99
Sequence + triplet loss	87.93	92.34	80.00	89.25
PaLoRA w/o triplet loss	89.59	93.18	88.23	92.40
PaLoRA	90.98	93.99	88.86	92.53

Table 6: Triplet-loss ablation on CIFAR-100 (accuracy in %; point estimates). PaLoRA retains most of its improvement without the triplet loss, which provides a smaller additional gain.

Figure 5: Pacing-exponent ablation on ImageNet-A with 50 tasks. The exponent

\alpha=0.5

gives the best accuracy while balancing training loss and forgetting, consistent with the predicted square-root denominator scaling.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Accuracy trajectories across CIFAR-100, ImageNet-A, and ImageNet-R under 10-, 20-, 25-, and 50-task splits. PaLoRA maintains strong performance across datasets and task horizons.

Figure 7: SVD compression across datasets with 10, 20, and 50 tasks. At

\tau=0.99

, the effective rank grows more slowly than the uncompressed

R=rt

reference and varies with the task sequence.

Figure 8: Effective-rank choices on ImageNet-A with 10, 20, and 50 tasks. Top: rank growth; bottom: accuracy. Adaptive SVD retains fewer directions than the

k=r

reference while maintaining comparable accuracy.

$C$	$ACC_{50}$	$\overline{ACC_{50}}$
2.25	88.92	92.54
4.00 (default)	88.86	92.53
6.25	88.70	92.40

Appendix

Table 7: Pacing-constant sensitivity on CIFAR-100 with 50 tasks and

\tau=0.99

(accuracy in %; point estimates). Performance varies little over the tested range.

$\tau$	$ACC_{50}$	$\overline{ACC_{50}}$	Avg. Final Rank
0.95	84.16	90.56	39.6
0.99 (default)	88.86	92.53	159.8
0.999	88.74	92.55	390.8

Appendix

Table 8: SVD-threshold sensitivity on CIFAR-100 with 50 tasks and

C=4

(accuracy in %; point estimates). The default threshold balances retained rank and accuracy.

Method	Time per Task (min)
L2P	2.98
DualPrompt	2.95
CODA-Prompt	3.02
BiLoRA	3.45
Sequence	2.67
Sequence + triplet loss	2.80

Appendix

Table 9: Average training time per task on CIFAR-100 with 50 tasks using an RTX 4080 SUPER. PaLoRA adds 0.34 minutes over Sequence and is faster than LoRA - DRS.

Method	$ACC_{50}$	$\overline{ACC_{50}}$
Sequence	81.72	87.99
PaLoRA	88.86	92.53
Sequence + PiSSA	71.85	80.54
PaLoRA + PiSSA	89.30	92.75

Appendix

Table 10: Compatibility with PiSSA on CIFAR-100 with 50 tasks (accuracy in %; point estimates). PaLoRA improves sequential PiSSA adaptation and slightly exceeds its standard-LoRA counterpart.

	ImageNet-R (50 tasks)				ImageNet-A (50 tasks)
Method	DINOv1		iBOT		DINOv1		iBOT
	ACC	$\overline{ACC}$	ACC	$\overline{ACC}$	ACC	$\overline{ACC}$	ACC	$\overline{ACC}$
InfLoRA Liang and Li (2024)	61.52	70.10	58.52	69.32	30.55	42.48	33.31	49.40
BiLoRA Zhu et al. (2025)	62.85	66.73	65.85	70.42	28.57	37.82	34.69	44.50
LoRA - DRS Liu and Chang (2025)	61.58	67.26	65.12	70.94	30.28	40.19	34.89	45.37
PaLoRA (Ours)	66.78	72.80	69.85	75.66	35.94	46.51	43.25	55.18

Appendix

Table 11: Final and average accuracy (%; point estimates) with DINOv1 and iBOT backbones on 50-task ImageNet-R and ImageNet-A. PaLoRA outperforms the compared methods for both backbones.

Continual LoRA

See all 21 papers

May 26, 2026

Energy-Structured Low-Rank Adaptation for Continual Learning

While orthogonal subspace methods try to mitigate task interference in Continual Learning (CL), they often suffer from energy diffusion across the basis, hindering knowledge compaction and exhausting capacity for future tasks. We observe that output feature drift induced by parameter updates is inherently low-rank, and theoretically prove that preserving parameters along the principal directions of this drift minimizes the output reconstruction error. Motivated by this, we propose Energy-Concentrated and Energy-Ordered Low-Rank Adaptation (E

^2

-LoRA). By explicitly ordering and concentrating knowledge into leading ranks, E

^2

-LoRA frees capacity for subsequent tasks. Furthermore, we design a dynamic rank allocation strategy to balance stability and plasticity by jointly optimizing energy retention and model plasticity. Extensive experiments across multiple benchmarks demonstrate that E

^2

-LoRA achieves state-of-the-art performance. Code is available at https://github.com/kiddo127/E2-LoRA.

Longhua Li, Lei Qi, Qi Tian +1

School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Huawei Technologies, Shenzhen, China

Continual Learning Low-Rank Adaptation Adaptive Rank Allocation Parameter-Efficient Fine-Tuning Continual LoRA

Ask PDF

Oct 1, 2026

MuLoRA: Spectrally Balanced Low-Rank Adaptation for Continual Learning

Low-rank adaptation (LoRA) provides a parameter-efficient approach to continual learning, but its nominal rank can conceal a loss of effective adaptation capacity. We identify spectral plasticity collapse: during sequential adaptation, update energy becomes concentrated in a small subset of singular modes, leaving much of the available low-rank space underutilized. This exposes a limitation of interference avoidance alone: protecting historical representations does not ensure that the remaining adaptation capacity is responsive to new tasks or effectively utilized. To address this problem, we propose MuLoRA, which jointly controls capacity allocation and utilization. First, historical whitening identifies input directions with strong current-task response relative to accumulated historical response, yielding a task-adaptive basis that remains fixed during training. Second, approximate polar orthogonalization of momentum updates reduces spectral concentration within theselected space. An orthonormal basis connects these mechanisms by transferring the factor-update spectrum exactly tothe induced weight update. We establish a max--min characterization of exact subspace selection and derive cumulative spectral bounds under controlled cross-step anisotropy. Across five class-incremental benchmarks and eight incremental settings, MuLoRA achieves the highest mean accuracy in 15 of 16 reported metrics.

Junkang Liu

Tianjin University

Continual Learning Low-Rank Adaptation Continual LoRA Class-Incremental Learning

Ask PDF

May 12, 2026

Low-Rank Adapters Initialization via Gradient Surgery for Continual Learning

LoRA is widely adopted for continual fine-tuning of Large Language Models due to its parameter efficiency, modularity across tasks, and compatibility with replay strategies. However, LoRA-based continual learning remains vulnerable to catastrophic forgetting, whose severity depends on how successive task gradients interact: when consecutive task gradients conflict, standard adapter initializations channel updates into subspaces that overwrite previously learned directions. We propose SLICE, a gradient-surgery-based initialization for LoRA adapters in continual learning. SLICE accumulates gradients from both the current task and a replay buffer of prior tasks, reconciles them through a projection operator, and decomposes the result via truncated SVD to initialize the adapter weights. We evaluate SLICE on the TRACE benchmark and sequences of Super-NI tasks, including a set of adversarial Super-NI sequences that we construct by mining task pairs with maximally opposing gradients. Compared to vanilla LoRA, LoRA-GA, and LoRAM, SLICE consistently achieves a better stability-plasticity trade-off, improving Average Performance, Final Performance and Forgetting metrics while preserving General Performance and In Context Performance across both standard and adversarial continual learning sequences.

Joana Pasquali, Ramiro N. Barros, Arthur S. Bianchessi +7

MALTA, Machine Learning Theory and Applications Lab, PUCRS, Porto Alegre, Brazil · Kunumi Institute, Brazil

Continual Learning for LLMs Gradient Interference Low-Rank Adaptation Incremental Learning Catastrophic Forgetting Continual LoRA

Ask PDF

Abstract

Continual LoRA

Explore similar work

Energy-Structured Low-Rank Adaptation for Continual Learning

MuLoRA: Spectrally Balanced Low-Rank Adaptation for Continual Learning

Low-Rank Adapters Initialization via Gradient Surgery for Continual Learning