PaLoRA: Paced Low-Rank Adaptation for Continual Learning

Yuxuan Li; Fanhu Zeng; Hao Tang

cs.LGOct 3, 2026

PaLoRA: Paced Low-Rank Adaptation for Continual Learning

Authors: Yuxuan Li, Fanhu Zeng, Hao Tang

Organizations: School of Computer Science Peking University · Institute of Automation Chinese Academy of Sciences

Abstract

LoRA-based continual learning methods mitigate catastrophic forgetting through various mechanisms, yet nearly all complement these with small learning rates as a heuristic to restrict gradient scaling magnitude. Such fixed heuristics lack theoretical guidance on how the strength of this restriction should evolve as tasks accumulate. We reveal that even under directional constraints such as nullspace projection, finite-precision updates inevitably leak into the subspace of accumulated prior knowledge along multiple directions. While small learning rates attenuate such leakage, they cannot prevent the accumulated forgetting from intensifying as the effective rank of historical knowledge grows. We show that the optimal magnitude restriction should adaptively increase with this effective rank to balance stability and plasticity, i.e., preservation of previous knowledge and acquisition of new task information. Under an anisotropic leakage model, we derive a pacing law $s^*=\sqrt{R/c}$ that characterizes the optimal scaling of gradient steps, i.e., the magnitude restriction itself, where $R$ is the effective rank of past updates. Based on this insight, we propose PaLoRA, which compresses historical knowledge via adaptive SVD truncation, projects gradients onto the nullspace of prior tasks, and applies rank-aware adaptive pacing. Experiments demonstrate consistent improvements over prior methods, with particularly strong performance in long-horizon settings, achieving substantial gains of 4% accuracy on challenging 50-task ImageNet-A and ImageNet-R benchmarks.

Figures & tables

Figure 1: Adaptive parameter update restrictions progressively shift optimization trajectory toward previous task solutions as more tasks are encountered, better balancing stability and plasticity to approach the joint optimum.

Figure 2: Overview of PaLoRA. (A) Model decomposition: pretrained weights and compressed memory from previous tasks are frozen, while a new low-rank update is learned for the current task. (B) Two-stage process: adaptive-rank compression of accumulated knowledge via SVD, followed by rank-aware gradient pacing with nullspace projection and scaled updates.

Tasks	10		20		25		50
Method	$ACC_{10}$	$\overline{ACC_{10}}$	$ACC_{20}$	$\overline{ACC_{20}}$	$ACC_{25}$	$\overline{ACC_{25}}$	$ACC_{50}$	$\overline{ACC_{50}}$
Pretrain	55.68 $\pm$ 0.00	59.20 $\pm$ 0.00	55.68 $\pm$ 0.00	59.55 $\pm$ 0.00	55.68 $\pm$ 0.00	59.78 $\pm$ 0.00	55.68 $\pm$ 0.00	59.95 $\pm$ 0.00
Sequence	62.11 $\pm$ 2.21	73.39 $\pm$ 0.97	57.30 $\pm$ 0.39	69.56 $\pm$ 0.69	54.48 $\pm$ 2.16	66.98 $\pm$ 0.28	47.48 $\pm$ 0.05	62.94 $\pm$ 0.60
L2P Wang et al. (2022b)	62.28 $\pm$ 0.30	66.98 $\pm$ 0.04	61.24 $\pm$ 0.16	64.73 $\pm$ 0.43	61.03 $\pm$ 0.35	64.37 $\pm$ 0.37	59.44 $\pm$ 0.45	63.16 $\pm$ 0.57
DualPrompt Wang et al. (2022a)	58.91 $\pm$ 0.23	62.66 $\pm$ 0.06	57.78 $\pm$ 0.33	61.91 $\pm$ 0.14	57.53 $\pm$ 0.64	61.66 $\pm$ 0.10	56.28 $\pm$ 0.97	61.72 $\pm$ 0.22
CODA-Prompt Smith et al. (2023)	69.81 $\pm$ 0.06	73.00 $\pm$ 0.15	66.42 $\pm$ 0.13	69.55 $\pm$ 0.35	65.03 $\pm$ 0.23	68.22 $\pm$ 0.18	56.19 $\pm$ 0.16	59.01 $\pm$ 0.27

Table 1: ImageNet-R accuracy (%, mean

\pm

standard deviation over five seeds). PaLoRA achieves the highest final and average accuracy across all splits, with a 4.09-point final-accuracy gain over LoRA - DRS at 50 tasks.

Tasks	10		20		25		50
Method	$ACC_{10}$	$\overline{ACC_{10}}$	$ACC_{20}$	$\overline{ACC_{20}}$	$ACC_{25}$	$\overline{ACC_{25}}$	$ACC_{50}$	$\overline{ACC_{50}}$
Pretrain	83.81 $\pm$ 0.00	88.82 $\pm$ 0.00	83.81 $\pm$ 0.00	89.13 $\pm$ 0.00	83.81 $\pm$ 0.00	89.16 $\pm$ 0.00	83.81 $\pm$ 0.00	89.33 $\pm$ 0.00
Sequence	87.54 $\pm$ 0.94	92.12 $\pm$ 0.78	81.51 $\pm$ 2.60	89.05 $\pm$ 0.48	81.93 $\pm$ 4.13	88.95 $\pm$ 1.11	77.52 $\pm$ 5.94	85.01 $\pm$ 4.23
L2P Wang et al. (2022b)	84.53 $\pm$ 0.71	87.84 $\pm$ 0.39	81.00 $\pm$ 0.51	84.70 $\pm$ 0.94	78.45 $\pm$ 0.37	82.08 $\pm$ 0.08	74.48 $\pm$ 1.35	78.90 $\pm$ 1.78
DualPrompt Wang et al. (2022a)	75.37 $\pm$ 2.35	86.03 $\pm$ 0.66	77.79 $\pm$ 0.04	86.49 $\pm$ 0.52	79.72 $\pm$ 0.85	86.82 $\pm$ 0.12	81.11 $\pm$ 0.77	87.37 $\pm$ 0.05
CODA-Prompt Smith et al. (2023)	84.40 $\pm$ 0.13	88.77 $\pm$ 0.38	81.69 $\pm$ 1.03	85.34 $\pm$ 0.70	79.10 $\pm$ 1.12	82.19 $\pm$ 0.87	72.16 $\pm$ 0.08	82.65 $\pm$ 0.49

Table 2: CIFAR-100 accuracy (%, mean

\pm

standard deviation over five seeds). PaLoRA consistently improves both metrics, reaching

88.96\pm 0.14

final accuracy at 50 tasks.

Tasks	10		20		25		50
Method	$ACC_{10}$	$\overline{ACC_{10}}$	$ACC_{20}$	$\overline{ACC_{20}}$	$ACC_{25}$	$\overline{ACC_{25}}$	$ACC_{50}$	$\overline{ACC_{50}}$
Pretrain	48.85 $\pm$ 0.00	57.39 $\pm$ 0.00	48.85 $\pm$ 0.00	58.47 $\pm$ 0.00	48.85 $\pm$ 0.00	58.56 $\pm$ 0.00	48.85 $\pm$ 0.00	58.97 $\pm$ 0.00
Sequence	34.13 $\pm$ 1.44	53.86 $\pm$ 2.04	23.87 $\pm$ 0.23	46.07 $\pm$ 1.37	25.97 $\pm$ 3.78	43.62 $\pm$ 0.24	15.61 $\pm$ 8.38	37.62 $\pm$ 1.41
L2P Wang et al. (2022b)	36.08 $\pm$ 0.65	41.66 $\pm$ 0.87	33.55 $\pm$ 0.70	38.95 $\pm$ 1.18	33.64 $\pm$ 0.56	39.08 $\pm$ 0.07	32.39 $\pm$ 0.65	37.55 $\pm$ 0.71
DualPrompt Wang et al. (2022a)	42.42 $\pm$ 3.12	53.20 $\pm$ 0.83	48.09 $\pm$ 0.14	56.99 $\pm$ 0.01	48.02 $\pm$ 0.33	57.19 $\pm$ 0.01	47.66 $\pm$ 0.84	57.69 $\pm$ 0.47
CODA-Prompt Smith et al. (2023)	52.17 $\pm$ 0.23	62.64 $\pm$ 0.39	42.59 $\pm$ 0.65	49.77 $\pm$ 1.08	39.30 $\pm$ 0.37	47.66 $\pm$ 0.32	47.53 $\pm$ 0.28	58.05 $\pm$ 0.21

Table 3: ImageNet-A accuracy (%, mean

\pm

standard deviation over five seeds). PaLoRA leads across all splits and exceeds LoRA - DRS by 3.99 points in final accuracy at 50 tasks.

Method	CIFAR-100		ImageNet-R		ImageNet-A
	25 Tasks	50 Tasks	25 Tasks	50 Tasks	25 Tasks	50 Tasks
Sequence	-16.24	-14.72	-36.86	-42.14	-31.62	-53.83
L2P Wang et al. (2022b)	-8.94	-14.07	-9.43	-10.56	-6.80	-11.16
DualPrompt Wang et al. (2022a)	-8.35	-6.46	-8.37	-9.72	-6.36	-8.79
CODA-Prompt Smith et al. (2023)	-9.83	-17.11	-6.02	-9.76	-10.71	-9.46
InfLoRA Liang and Li (2024)	-12.53	-20.45	-15.08	-20.01	-16.47	-18.34

Table 4: Backward transfer (BWT, %; higher is better) on all three datasets under 25- and 50-task splits. PaLoRA yields the least forgetting on CIFAR-100 and ImageNet-A, while LoRA - DRS has slightly better BWT on ImageNet-R. Values are point estimates.

Figure 3: Accuracy trajectories on CIFAR-100 (10 tasks), ImageNet-A (50 tasks), and ImageNet-R (50 tasks), from left to right. PaLoRA maintains strong accuracy, including over long task sequences.

Dataset	Method	10 Tasks		20 Tasks		25 Tasks		50 Tasks
Dataset	Method	$ACC_{10}$	$\overline{ACC_{10}}$	$ACC_{20}$	$\overline{ACC_{20}}$	$ACC_{25}$	$\overline{ACC_{25}}$	$ACC_{50}$	$\overline{ACC_{50}}$
ImageNet-R	SD-LoRA	78.33	82.85	74.28	79.45	75.37	80.82	69.68	76.24
ImageNet-R	PaLoRA	78.82	82.96	78.57	82.74	78.10	82.90	76.40	80.40
CIFAR-100	SD-LoRA	88.05	92.47	84.20	90.48	80.32	87.54	67.70	77.60
CIFAR-100	PaLoRA	90.98	93.99	90.31	93.67	90.25	93.69	88.86	92.53
ImageNet-A	SD-LoRA	57.21	70.68	50.63	65.06	46.94	58.19	31.99	54.78

Table 5: Direct comparison with SD-LoRA (accuracy in %; point estimates). PaLoRA achieves larger final-accuracy gains at 50 tasks than at 10 tasks on all three datasets.

Figure 4: Component ablation on ImageNet-A with 10, 20, and 50 tasks. Projection and pacing improve the trajectories, while triplet regularization provides an additional accuracy gain. Each setting uses a different class split, so early-task curves are not expected to coincide across panels.

Method	$ACC_{10}$	$\overline{ACC_{10}}$	$ACC_{50}$	$\overline{ACC_{50}}$
Sequence	86.88	91.57	81.72	87.99
Sequence + triplet loss	87.93	92.34	80.00	89.25
PaLoRA w/o triplet loss	89.59	93.18	88.23	92.40
PaLoRA	90.98	93.99	88.86	92.53

Table 6: Triplet-loss ablation on CIFAR-100 (accuracy in %; point estimates). PaLoRA retains most of its improvement without the triplet loss, which provides a smaller additional gain.

Figure 5: Pacing-exponent ablation on ImageNet-A with 50 tasks. The exponent

\alpha=0.5

gives the best accuracy while balancing training loss and forgetting, consistent with the predicted square-root denominator scaling.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Accuracy trajectories across CIFAR-100, ImageNet-A, and ImageNet-R under 10-, 20-, 25-, and 50-task splits. PaLoRA maintains strong performance across datasets and task horizons.

Figure 7: SVD compression across datasets with 10, 20, and 50 tasks. At

\tau=0.99

, the effective rank grows more slowly than the uncompressed

R=rt

reference and varies with the task sequence.

Figure 8: Effective-rank choices on ImageNet-A with 10, 20, and 50 tasks. Top: rank growth; bottom: accuracy. Adaptive SVD retains fewer directions than the

k=r

reference while maintaining comparable accuracy.

$C$	$ACC_{50}$	$\overline{ACC_{50}}$
2.25	88.92	92.54
4.00 (default)	88.86	92.53
6.25	88.70	92.40

Appendix

Table 7: Pacing-constant sensitivity on CIFAR-100 with 50 tasks and

\tau=0.99

(accuracy in %; point estimates). Performance varies little over the tested range.

$\tau$	$ACC_{50}$	$\overline{ACC_{50}}$	Avg. Final Rank
0.95	84.16	90.56	39.6
0.99 (default)	88.86	92.53	159.8
0.999	88.74	92.55	390.8

Appendix

Table 8: SVD-threshold sensitivity on CIFAR-100 with 50 tasks and

C=4

(accuracy in %; point estimates). The default threshold balances retained rank and accuracy.

Method	Time per Task (min)
L2P	2.98
DualPrompt	2.95
CODA-Prompt	3.02
BiLoRA	3.45
Sequence	2.67
Sequence + triplet loss	2.80

Appendix

Table 9: Average training time per task on CIFAR-100 with 50 tasks using an RTX 4080 SUPER. PaLoRA adds 0.34 minutes over Sequence and is faster than LoRA - DRS.

Method	$ACC_{50}$	$\overline{ACC_{50}}$
Sequence	81.72	87.99
PaLoRA	88.86	92.53
Sequence + PiSSA	71.85	80.54
PaLoRA + PiSSA	89.30	92.75

Appendix

Table 10: Compatibility with PiSSA on CIFAR-100 with 50 tasks (accuracy in %; point estimates). PaLoRA improves sequential PiSSA adaptation and slightly exceeds its standard-LoRA counterpart.

	ImageNet-R (50 tasks)				ImageNet-A (50 tasks)
Method	DINOv1		iBOT		DINOv1		iBOT
	ACC	$\overline{ACC}$	ACC	$\overline{ACC}$	ACC	$\overline{ACC}$	ACC	$\overline{ACC}$
InfLoRA Liang and Li (2024)	61.52	70.10	58.52	69.32	30.55	42.48	33.31	49.40
BiLoRA Zhu et al. (2025)	62.85	66.73	65.85	70.42	28.57	37.82	34.69	44.50
LoRA - DRS Liu and Chang (2025)	61.58	67.26	65.12	70.94	30.28	40.19	34.89	45.37
PaLoRA (Ours)	66.78	72.80	69.85	75.66	35.94	46.51	43.25	55.18

Appendix

Table 11: Final and average accuracy (%; point estimates) with DINOv1 and iBOT backbones on 50-task ImageNet-R and ImageNet-A. PaLoRA outperforms the compared methods for both backbones.

Continual LoRA

See all 22 papers

Energy-Structured Low-Rank Adaptation for Continual Learning
May 26, 2026Longhua Li, Lei Qi, Qi Tian +1Continual Learning Low-Rank Adaptation
School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Huawei Technologies, Shenzhen, China
MuLoRA: Spectrally Balanced Low-Rank Adaptation for Continual Learning
Oct 1, 2026Junkang LiuContinual Learning Low-Rank Adaptation
Tianjin University
Low-Rank Adapters Initialization via Gradient Surgery for Continual Learning
May 12, 2026Joana Pasquali, Ramiro N. Barros, Arthur S. Bianchessi +7Continual Learning for LLMs Gradient Interference
MALTA, Machine Learning Theory and Applications Lab, PUCRS, Porto Alegre, Brazil · Kunumi Institute, Brazil

May 26, 2026J/K move · Enter open · S save

Energy-Structured Low-Rank Adaptation for Continual Learning

Longhua Li, Lei Qi, Qi Tian, Xin Geng

School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Huawei Technologies, Shenzhen, China

Continual Learning Low-Rank Adaptation Adaptive Rank Allocation Parameter-Efficient Fine-Tuning Continual LoRA

Open paper AskPDF arXiv

While orthogonal subspace methods try to mitigate task interference in Continual Learning (CL), they often suffer from energy diffusion across the basis, hindering knowledge compaction and exhausting capacity for future tasks. We observe that output feature drift induced by parameter updates is inherently low-rank, and theoretically prove that preserving parameters along the principal directions of this drift minimizes the output reconstruction error. Motivated by this, we propose Energy-Concentrated and Energy-Ordered Low-Rank Adaptation (E

^2

-LoRA). By explicitly ordering and concentrating knowledge into leading ranks, E

^2

-LoRA frees capacity for subsequent tasks. Furthermore, we design a dynamic rank allocation strategy to balance stability and plasticity by jointly optimizing energy retention and model plasticity. Extensive experiments across multiple benchmarks demonstrate that E

^2

-LoRA achieves state-of-the-art performance. Code is available at https://github.com/kiddo127/E2-LoRA.

Abstract

Continual LoRA

Explore similar work

Energy-Structured Low-Rank Adaptation for Continual Learning

MuLoRA: Spectrally Balanced Low-Rank Adaptation for Continual Learning

Low-Rank Adapters Initialization via Gradient Surgery for Continual Learning