Revisiting Weight Regularization for Low-Rank Continual Learning

Yaoyue Zheng; Yin Zhang; Joost van de Weijer; Gido M van de Ven; Shaoyi Du; Xuetao Zhang; Zhiqiang Tian

cs.LGFeb 19, 2026

Revisiting Weight Regularization for Low-Rank Continual Learning

Authors: Yaoyue Zheng, Yin Zhang, Joost van de Weijer, Gido M van de Ven, Shaoyi Du, Xuetao Zhang, Zhiqiang Tian

Organizations: State Key Laboratory of Human-Machine Hybrid Augmented Intelligence Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Shaanxi, China · Computer Vision Center, Barcelona, Spain · Universitat Aut`onoma de Barcelona, Barcelona, Spain · School of Instrument Science and Engineering, Harbin Institute of Technology, Harbin, China ·

Abstract

Continual Learning (CL) with large-scale pre-trained models (PTMs) has recently gained wide attention, shifting the focus from training from scratch to continually adapting PTMs. This has given rise to a promising paradigm: parameter-efficient continual learning (PECL), where task interference is typically mitigated by assigning a task-specific module during training, such as low-rank adapters. However, weight regularization techniques, such as Elastic Weight Consolidation (EWC)-a key strategy in CL-remain underexplored in this new paradigm. In this paper, we revisit weight regularization in low-rank CL as a new perspective for mitigating task interference in PECL. Unlike existing low-rank CL methods, we mitigate task interference by regularizing a shared low-rank update through EWC, thereby keeping the storage requirement and inference costs constant regardless of the number of tasks. Our proposed method EWC-LoRA leverages a low-rank representation to estimate parameter importance over the full-dimensional space. This design offers a practical, computational- and memory-efficient solution for CL with PTMs, and provides insights that may inform the broader application of regularization techniques within PECL. Extensive experiments on various benchmarks demonstrate the effectiveness of EWC-LoRA, achieving a stability-plasticity trade-off superior to existing low-rank CL approaches. These results indicate that, even under low-rank parameterizations, weight regularization remains an effective mechanism for mitigating task interference. Code is available at: https://github.com/yaoyz96/low-rank-cl.

Figures & tables

Figure 1: Overview of learning task $\mathcal{T}_{t}$ at a specific layer of the ViT model. (a) Prior low-rank CL methods structurally isolate task-specific LoRA parameters by adding a new LoRA branch for each task. (b) The proposed EWC-LoRA employs a shared LoRA module that is learned across all tasks and regularized according to parameter importance measured by a Fisher Information Matrix, which is updated after learning each task.

Strategy	$\overline{A}_{10}$	Avg.	Stability	Plasticity	+ Mem.
w/o $\mathbf{F}$	$82.99_{(0.84)}$	$89.74_{(0.58)}$	$87.56_{(0.09)}$	$\textbf{98.86}_{(0.09)}$	0 GB
Precomputed $\mathbf{F}_{\mathbf{W}}$	$83.87_{(0.21)}$	$89.36_{(0.49)}$	$93.15_{(0.45)}$	$94.74_{(0.56)}$	1 GB
Separate $\mathbf{F}_{\mathbf{A}},\mathbf{F}_{\mathbf{B}}$	$86.41_{(0.69)}$	$91.33_{(0.50)}$	$94.23_{(0.46)}$	$96.47_{(0.21)}$	4 GB
$\mathbf{F}_{\Delta\mathbf{W}}$ (Ours)	$\textbf{87.91}_{(0.57)}$	$\textbf{92.27}_{(0.39)}$	$\textbf{94.45}_{(0.59)}$	$97.99_{(0.50)}$	6 GB

Table 1: Comparison of different Fisher estimation strategies on CIFAR-100. “ + Mem. ” indicates the additional memory required for Fisher estimation and regularization during training.

Tasks	CIFAR-100		DomainNet		ImageNet-R		ImageNet-A
Methods	$\overline{A}_{10}$ ( $\uparrow$ )	Avg. ( $\uparrow$ )	$\overline{A}_{5}$ ( $\uparrow$ )	Avg. ( $\uparrow$ )	$\overline{A}_{10}$ ( $\uparrow$ )	Avg. ( $\uparrow$ )	$\overline{A}_{10}$ ( $\uparrow$ )	Avg. ( $\uparrow$ )
Joint Train	$92.82_{(0.20)}$	$95.41_{(0.05)}$	$76.84_{(0.06)}$	$81.25_{(0.05)}$	$81.69_{(0.28)}$	$86.25_{(0.09)}$	$65.01_{(0.74)}$	$74.10_{(0.44)}$
Finetune	$79.09_{(1.53)}$	$88.17_{(0.45)}$	$65.57_{(0.20)}$	$75.12_{(0.12)}$	$60.42_{(1.64)}$	$73.18_{(0.32)}$	$32.85_{(1.53)}$	$54.55_{(1.44)}$
L2P	$83.18_{(1.20)}$	$87.69_{(1.05)}$	$70.26_{(0.25)}$	$75.83_{(0.98)}$	$71.26_{(0.44)}$	$76.13_{(0.46)}$	$42.94_{(1.27)}$	$51.40_{(1.95)}$
DualPrompt	$81.48_{(0.86)}$	$86.41_{(0.66)}$	$68.26_{(0.90)}$	$73.84_{(0.45)}$	$68.22_{(0.20)}$	$73.81_{(0.39)}$	$45.49_{(0.96)}$	$54.68_{(1.24)}$
CODA-Prompt	$86.31_{(0.12)}$	$90.67_{(0.22)}$	$70.58_{(0.53)}$	$76.68_{(0.44)}$	$74.05_{(0.41)}$	$78.14_{(0.39)}$	$45.36_{(0.78)}$	$57.03_{(0.94)}$

Table 2: Comparison results on CIFAR-100, DomainNet, ImageNet-R, and ImageNet-A (in %). Bold and underline indicate the highest and second-highest scores, respectively.

Panel of Figure 2 — Figure 2: Task-wise performance comparison of different methods across various datasets.

Backbone	Method	Order-1	Order-2	Order-3	Avg.
T5-large	O-LoRA	$75.69$	$74.92$	74.40	$75.01$
T5-large	EWC-LoRA	78.01	76.85	$74.30$	76.39
LLaMA-3.2-1B-Instruct	O-LoRA	$56.96$	$55.74$	$67.32$	$60.01$
	TreeLoRA	$58.54$	$56.96$	$65.42$	$60.30$
	EWC-LoRA	61.17	60.47	67.61	63.08

Table 3: Comparison results (reported as average accuracy (%)) on the standard language CL benchmark with the two pretrained models. Details about the task order are provided in Appendix A.3.1 .

Tasks	CIFAR-100		DomainNet		ImageNet-R		ImageNet-A
Methods	Stability	Plasticity	Stability	Plasticity	Stability	Plasticity	Stability	Plasticity
Vanilla LoRA	$87.56_{(0.09)}$	$98.86_{(0.09)}$	$81.29_{(0.34)}$	$97.94_{(0.16)}$	$78.63_{(1.38)}$	$99.57_{(0.35)}$	$85.56_{(1.82)}$	$97.56_{(0.77)}$
InfLoRA	$94.84_{(0.52)}$	$95.80_{(1.00)}$	$83.80_{(0.37)}$	$97.34_{(0.29)}$	$92.69_{(0.77)}$	$97.33_{(0.46)}$	$88.63_{(3.39)}$	$72.69_{(3.13)}$
SD-LoRA	$91.85_{(0.60)}$	$98.24_{(0.25)}$	$82.45_{(0.19)}$	$98.69_{(0.15)}$	$91.78_{(0.90)}$	$95.61_{(0.32)}$	$88.61_{(1.05)}$	$92.13_{(2.77)}$
EWC-LoRA	$94.45_{(0.59)}$	$97.99_{(0.50)}$	$91.51_{(0.36)}$	$93.83_{(0.23)}$	$95.62_{(0.42)}$	$93.23_{(0.34)}$	$89.52_{(1.14)}$	$92.78_{(1.91)}$

Table 4: Stability (

\uparrow

) and plasticity (

\uparrow

) scores of different low-rank CL methods, reflecting how well each model retains previous knowledge and adapts to new tasks. We report the normalized form of the two metrics, which is independent of the absolute performance on the dataset.

Panel of Figure 3 — Figure 3: (a) Stability–Plasticity curves illustrating the trade-off between retaining previous knowledge and learning new tasks. (b) Performance across a range of regularization strengths $\lambda$ on CIFAR-100 and DomainNet, showing the effect of $\lambda$ on accuracy.

Methods	Memory	CIFAR-100	DomainNet	ImageNet-R	ImageNet-A
Vanilla LoRA	$\sim$ 18 GB	10m22s $\pm$ 7s	32m8s $\pm$ 44s	13m32s $\pm$ 105s	0m55s $\pm$ 21s
InfLoRA	$\sim$ 20 GB	11m9s $\pm$ 4s	42m25s $\pm$ 68s	14m33s $\pm$ 107s	1m16s $\pm$ 22s
SD-LoRA	$\sim$ 37 GB	12m16s $\pm$ 86s	34m18s $\pm$ 160s	22m53s $\pm$ 232s	1m56s $\pm$ 36s
EWC-LoRA	$\sim$ 24 GB	10m31s $\pm$ 3s	33m10s $\pm$ 50s	13m42s $\pm$ 104s	0m55s $\pm$ 21s

Table 5: Comparison of different methods in terms of memory cost and training time. Memory usage is measured on the Quadro RTX 6000 GPU with a batch size of 128. Training time is reported as the average time required to train a single task.

Table 9

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Datasets	Methods
CIFAR-100	optimizer: Adam; schedular: Cosine; batch size: 128; shuffle: False; epochs: 20; rank: 10 lr: 0.0005; classifier lr: 0.005; lr decay: 0.1 lr: 0.008; classifier lr: 0.008; lr decay: 0.1 (SD-LoRA) $\bm{\epsilon:}$ 0.95 (InfLoRA)
DomainNet	optimizer: Adam; schedular: Cosine; batch size: 128; shuffle: False; epochs: 5; rank: 30 lr: 0.0005; classifier lr: 0.005; lr decay: 0.1 lr: 0.02; classifier lr: 0.02; lr decay: 0.0 (SD-LoRA) $\bm{\epsilon:}$ 0.95 (InfLoRA)
ImageNet-R	optimizer: Adam; schedular: Cosine; batch size: 128; shuffle: False; epochs: 50; rank: 10 lr: 0.0005; classifier lr: 0.0005; lr decay: 0.1 lr: 0.01; classifier lr: 0.01; lr decay: 0.0; weight decay: 0.0005 (SD-LoRA) weight decay: 0.005 (EWC-LoRA) $\bm{\epsilon:}$ 0.98 (InfLoRA)
ImageNet-A	optimizer: Adam; schedular: Cosine; batch size: 128; shuffle: False; epochs: 10; rank: 10 lr: 0.0005; classifier lr: 0.005; lr decay: 0.1 $\bm{\epsilon:}$ 0.98 (InfLoRA)

Appendix

Table 8: Hyperparameters for different benchmarks and methods. “ lr ” denotes learning rate. For the parameter

\bm{\epsilon}

, we refer readers to Liang & Li (2024) for details.

	$\gamma=1.0$	$\gamma=0.9$	$\gamma=0.7$	$\gamma=0.5$	$\gamma=0.3$	$\gamma=0$
$\overline{A}_{10}$	$87.64$	$87.47$	$88.07$	$\uline{88.14}$	88.22	$86.63$
Avg.	$91.96$	$92.12$	$\uline{92.42}$	$92.34$	92.46	$92.04$
Stability	$\uline{94.46}$	$94.45$	$94.28$	$94.37$	94.51	$91.81$
Plasticity	$97.66$	$97.99$	$\uline{98.35}$	$98.33$	$98.29$	99.07

Appendix

Table 9: Performance comparison under different values of

\gamma

on CIFAR-100.

Figure 4: Stability-Plasticity trade-off with various task decay factor

\gamma

. A broad range of

\gamma

(0.3–0.9) yields a similar trade-off, indicating that the method is not sensitive to the precise choice of

\gamma

Panel of Figure 5 — Figure 5: Task-wise performance on CIFAR-100 under different $\gamma$ settings.

	Task 1	Task 2	Task 3	Task 4
Order 1	dbpedia	amazon	yahoo	ag
Order 2	dbpedia	amazon	ag	yahoo
Order 3	yahoo	amazon	ag	dbpedia

Appendix

Table 10: Task order of the standard CL benchmark for language tasks.

Panel of Figure 6 — Figure 6: Task-wise performance across the three task orders using the T5-large model.

Panel of Figure 7 — Figure 7: Task-wise performance across the three task orders using the LLaMA-3.2-1B-Instruct model.

Tasks	ImageNet-R (N=5)		ImageNet-R (N=20)
Methods	$\overline{A}_{5}$ ( $\uparrow$ )	Avg. ( $\uparrow$ )	$\overline{A}_{20}$ ( $\uparrow$ )	Avg. ( $\uparrow$ )
Joint Train	$81.69_{(0.30)}$	$85.57_{(0.13)}$	$81.66_{(0.22)}$	$86.41_{(0.10)}$
Finetune	$69.26_{(0.74)}$	$78.88_{(0.31)}$	$47.06_{(2.05)}$	$63.01_{(0.56)}$
InfLoRA	$\textbf{77.37}_{(0.30)}$	$\textbf{82.19}_{(0.24)}$	$69.63_{(0.62)}$	$76.95_{(0.54)}$
SD-LoRA	$74.90_{(1.58)}$	$79.93_{(0.29)}$	$\textbf{72.26}_{(0.37)}$	$\textbf{77.81}_{(0.21)}$
Vanilla LoRA	$70.15_{(1.00)}$	$79.16_{(0.37)}$	$56.17_{(1.50)}$	$69.51_{(0.37)}$

Appendix

Table 11: Comparison results on ImageNet-R across different task lengths (in %).

Panel of Figure 8 — Figure 8: Task-wise performance of LoRA-based methods on CIFAR-100. Each row represents the performance on all previously encountered tasks (x-axis) after learning the current task (y-axis).

Panel of Figure 9 — Figure 9: Task-wise performance of LoRA-based methods on DomainNet. Each row represents the performance on all previously encountered tasks (x-axis) after learning the current task (y-axis).

Figure 10: Trade-off between stability and plasticity.

	CIFAR-100		ImageNet-R
$r$	$\overline{A}_{10}$	Avg.	$\overline{A}_{10}$	Avg.
16	88.31	92.59	72.10	79.18
10	87.91	92.27	72.86	78.95
4	87.79	92.12	72.42	78.78
1	86.77	91.61	70.37	77.27

Appendix

Table 12: Ablation study on LoRA ranks

r

across two benchmarks of varying difficulty.

	CIFAR-100				ImageNet-R
$r$	$\overline{A}_{10}$	Avg.	Stability	Plasticity	$\overline{A}_{10}$	Avg.	Stability	Plasticity
16	83.05	89.71	86.25	100.42	66.05	76.26	74.17	101.41
10	82.99	89.74	87.56	98.86	66.32	76.35	75.69	99.86
4	83.35	90.10	88.70	98.24	67.52	76.33	77.27	97.30
1	83.72	89.94	89.95	97.36	68.43	77.09	81.95	96.75

Appendix

Table 13: Ablation study on LoRA ranks (

r

) in the Vanilla LoRA setting for CIFAR-100 and ImageNet-R. Note that the plasticity is computed based on the upper bound at rank 10. When the rank is 16, the plasticity can exceed 100%.

Strategy	$\overline{A}_{10}$	Avg.	Stability	Plasticity
Uniform $\mathbf{F}=\mathbf{I}$	$83.02$	$88.85$	$92.26$	$94.63$
Dataset-based $\mathbf{F}$	$83.87$	$89.36$	$93.15$	$94.74$

Appendix

Table 14: Regularization using precomputed Fisher on CIFAR-100.

Estimation	$\overline{A}_{10}$	Avg.	Best $\lambda$
Exact	$88.32$	$92.77$	$\lambda=10^{5}$
Exact (n=500)	$88.28$	$92.76$	$\lambda=10^{5}$
Sample	$88.10$	$92.50$	$\lambda=10^{7}$
Empirical	$87.91$	$92.27$	$\lambda=10^{7}$

Appendix

Table 15: Different ways for estimating the Fisher matrix. Final accuracy of each variant using its optimal strength

\lambda

on CIFAR-100.

Panel of Figure 11 — Figure 11: Evaluation of Fisher accuracy as the task index increases on CIFAR-100 with 10 tasks.

Panel of Figure 12 — Figure 12: Evaluation of Fisher accuracy as the task index increases on CIFAR-100 with 20 tasks.

Panel of Figure 13 — Figure 13: Cosine similarity between two optimal Fisher matrices under the rehearsal-based setting.

Incremental Learning

Incremental learning studies how models update from sequential data while retaining earlier knowledge as classes, tasks, or domains change.

See all 141 papers

May 26, 2026

Energy-Structured Low-Rank Adaptation for Continual Learning

While orthogonal subspace methods try to mitigate task interference in Continual Learning (CL), they often suffer from energy diffusion across the basis, hindering knowledge compaction and exhausting capacity for future tasks. We observe that output feature drift induced by parameter updates is inherently low-rank, and theoretically prove that preserving parameters along the principal directions of this drift minimizes the output reconstruction error. Motivated by this, we propose Energy-Concentrated and Energy-Ordered Low-Rank Adaptation (E

^2

-LoRA). By explicitly ordering and concentrating knowledge into leading ranks, E

^2

-LoRA frees capacity for subsequent tasks. Furthermore, we design a dynamic rank allocation strategy to balance stability and plasticity by jointly optimizing energy retention and model plasticity. Extensive experiments across multiple benchmarks demonstrate that E

^2

-LoRA achieves state-of-the-art performance. Code is available at https://github.com/kiddo127/E2-LoRA.

Longhua Li, Lei Qi, Qi Tian +1

School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Huawei Technologies, Shenzhen, China

Continual Learning Low-Rank Adaptation Adaptive Rank Allocation Parameter-Efficient Fine-Tuning Continual LoRA

Ask PDF

Jun 26, 2026

When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning

Low-Rank Adaptation (LoRA) has become the standard tool for parameter-efficient fine-tuning of large pretrained models. When applied sequentially across tasks in Continual Learning (CL), the standard assumption is that each new task requires a dedicated low-rank adapter. In this work, we challenge this assumption empirically and structurally. We show that task-specific LoRA adapters in CL exhibit significant low-rank redundancy: the subspaces spanned by adapters trained on different tasks substantially overlap, and in many cases earlier adapters can faithfully represent later tasks. Building on this observation, we propose LiteLoRA, a plug-and-play gating mechanism that learns at train time whether to recruit a new adapter or reuse existing low-rank representations. Our method reduces the number of active adapters by 20-70% while matching or exceeding state-of-the-art performance on standard CL benchmarks, revealing that structural redundancy is pervasive and that selective learning is sufficient to achieve stability without sacrificing plasticity.

Tanguy Dieudonné, Giulia Lanzillotta, Enis Simsar +2

Department of Computer Science, ETH Zurich, Zurich, Switzerland.

Continual Learning Low-Rank Adaptation Incremental Learning Parameter-Efficient Fine-Tuning Continual LoRA

Ask PDF

Feb 27, 2026

Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning

Continual Learning (CL) requires models to sequentially adapt to new tasks without forgetting old knowledge. Recently, Low-Rank Adaptation (LoRA), a representative Parameter-Efficient Fine-Tuning (PEFT) method, has gained increasing attention in CL. Several LoRA-based CL methods reduce interference across tasks by separating their update spaces, typically building the new space from the estimated null space of past tasks. However, they (i) overlook task-shared directions, which suppresses knowledge transfer, and (ii) fail to capture truly effective task-specific directions since these ``null bases" of old tasks can remain nearly inactive for new task under correlated tasks. To address this, we study LoRA learning capability from a projection energy perspective, and propose Low-rank Decomposition and Adaptation (LoDA). It performs a task-driven decomposition to build general and truly task-specific LoRA subspaces by solving two energy-based objectives, decoupling directions for knowledge sharing and isolation. LoDA fixes LoRA down-projections on two subspaces and learns robust up-projections via a Gradient-Aligned Optimization (GAO) approach. After each task, before integrating the LoRA updates into the backbone, LoDA derives a closed-form recalibration for the general update, approximating a feature-level joint optimum along this task-shared direction. Experiments indicate that LoDA outperforms existing CL methods. Our code is available at https://github.com/HHHLF/LoDA_ICML2026.

Lingfeng He, De Cheng, Huaijie Wang +3

State Key Laboratory of Integrated Services Networks, School of Telecommunications Engineering, Xidian University, Xi’an, China · School of Electronic Engineering, Xidian University, Xi’an, China · AUTHORERR: Missing \icmlaffiliation.

Continual Learning Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Ask PDF

Abstract

Incremental Learning

Explore similar work

Energy-Structured Low-Rank Adaptation for Continual Learning

When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning

Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning