Modulating Merging Strengths via Joint Loss Estimation for LoRA-based Continual Learning

Kun Gu; De Cheng; Zhipeng Xu; Lingfeng He; Di Xu; Nannan Wang

cs.LGNov 3, 2026

Modulating Merging Strengths via Joint Loss Estimation for LoRA-based Continual Learning

Authors: Kun Gu, De Cheng, Zhipeng Xu, Lingfeng He, Di Xu, Nannan Wang

Organizations: Xidian University · Huawei Technologies Co., Ltd.

Abstract

Continual learning (CL) aims to adapt models to new tasks while preserving previously learned knowledge. Recently, Low-Rank Adaptation (LoRA), a representative Parameter-Efficient Fine-Tuning (PEFT) method, has gained increasing attention in CL due to its efficiency and scalability. Several LoRA-based merging methods maintain a single inference model by scaling the newly learned LoRA update with one coefficient and merging it into the previous weights after each task. However, they usually control the whole update with a shared scalar factor, although different update elements may affect previous and current tasks in different ways. To address this issue, we propose Merging Strength Modulation via Joint Loss Estimation (M2LE). It consists of Hessian Information-guided Adaptive Merging (HIM) and Task Simulation Perturbation for Merging (TSP). Instead of searching for a global merge coefficient, HIM directly solves for the target merged update under a joint objective over previous and current tasks. Solving this objective gives an analytic solution that naturally takes the form of element-wise modulation over the current task LoRA update. To improve the stability of the learned update under subsequent merging, we introduce TSP which simulates different tasks interference. Experiments on multiple benchmarks show that M2LE consistently outperforms existing PEFT-based continual learning methods. Code is available at https://github.com/bbdeeer/M2LE_NeurIPS2026.

Figures & tables

Figure 1: Joint loss landscape of the two group merge on 10S-ImageNet-R.

Method	Venue	FR	10S-ImageNet-R		10S-ImageNet-A		10S-CIFAR100		10S-CUB
Method	Venue	FR	$\mathcal{A}_{{Last}}\uparrow$	$\mathcal{A}_{{Avg}}\uparrow$	$\mathcal{A}_{{Last}}\uparrow$	$\mathcal{A}_{{Avg}}\uparrow$	$\mathcal{A}_{{Last}}\uparrow$	$\mathcal{A}_{{Avg}}\uparrow$	$\mathcal{A}_{{Last}}\uparrow$	$\mathcal{A}_{{Avg}}\uparrow$
L2P [ 43 ]	CVPR’22	✗	${72.34}_{\pm 0.17}$	${77.36}_{\pm 0.64}$	${44.04}_{\pm 0.93}$	${51.24}_{\pm 2.26}$	${84.06}_{\pm 0.88}$	${88.26}_{\pm 1.34}$	${67.02}_{\pm 1.90}$	${70.62}_{\pm 1.60}$
CODAPrompt [ 31 ]	CVPR’23	✗	${73.31}_{\pm 0.50}$	${78.47}_{\pm 0.53}$	${52.08}_{\pm 0.12}$	${63.92}_{\pm 0.12}$	${83.21}_{\pm 3.39}$	${87.71}_{\pm 3.17}$	${77.23}_{\pm 1.12}$	${81.90}_{\pm 0.85}$
InfLoRA [ 18 ]	CVPR’24	✗	${74.75}_{\pm 0.64}$	${80.67}_{\pm 0.55}$	${49.20}_{\pm 1.12}$	${60.92}_{\pm 0.61}$	${86.75}_{\pm 0.35}$	${91.72}_{\pm 0.15}$	${70.82}_{\pm 0.23}$	${81.39}_{\pm 0.14}$
PLAN [ 41 ]	ICCV’25	✗	${75.25}_{\pm 0.42}$	${80.41}_{\pm 0.56}$	–	–	${87.54}_{\pm 0.31}$	${92.21}_{\pm 0.35}$	–	–
SEMA [ 40 ]	CVPR’25	✗	${78.00}\phantom{{}_{\pm 0.00}}$	${83.56}\phantom{{}_{\pm 0.00}}$	${53.32}\phantom{{}_{\pm 0.00}}$	${64.53}\phantom{{}_{\pm 0.00}}$	${86.98}\phantom{{}_{\pm 0.00}}$	${91.37}\phantom{{}_{\pm 0.00}}$	–	–

Table 1: Experimental results for 10 incremental tasks on ImageNet-R, ImageNet-A, CIFAR100, and CUB. We report the averaged results over 3 random seeds. Within each group, the highest results are in bold , and the second highest results are underlined . "FR" denotes the use of feature replay.

Method	Venue	FR	5S-ImageNet-R		20S-ImageNet-R		20S-ImageNet-A		10S-DomainNet
Method	Venue	FR	$\mathcal{A}_{{Last}}\uparrow$	$\mathcal{A}_{{Avg}}\uparrow$	$\mathcal{A}_{{Last}}\uparrow$	$\mathcal{A}_{{Avg}}\uparrow$	$\mathcal{A}_{{Last}}\uparrow$	$\mathcal{A}_{{Avg}}\uparrow$	$\mathcal{A}_{{Last}}\uparrow$	$\mathcal{A}_{{Avg}}\uparrow$
L2P [ 43 ]	CVPR’22	✗	${70.83}_{\pm 0.58}$	${78.34}_{\pm 0.47}$	${69.64}_{\pm 0.42}$	${75.28}_{\pm 0.57}$	${40.48}_{\pm 1.78}$	${49.62}_{\pm 1.46}$	${81.17}_{\pm 0.83}$	${87.43}_{\pm 0.95}$
CODAPrompt [ 31 ]	CVPR’23	✗	${74.91}_{\pm 0.33}$	${79.25}_{\pm 0.53}$	${69.96}_{\pm 0.50}$	${75.34}_{\pm 0.85}$	${44.62}_{\pm 1.92}$	${54.86}_{\pm 0.50}$	${80.04}_{\pm 0.79}$	${86.27}_{\pm 0.82}$
InfLoRA [ 18 ]	CVPR’24	✗	${77.52}_{\pm 0.37}$	${82.01}_{\pm 0.12}$	${71.01}_{\pm 0.45}$	${77.28}_{\pm 0.45}$	${46.81}_{\pm 2.30}$	${57.13}_{\pm 0.95}$	${81.45}_{\pm 0.68}$	${87.55}_{\pm 0.57}$
SEMA [ 40 ]	CVPR’25	✗	${79.78}\phantom{{}_{\pm 0.00}}$	${84.75}\phantom{{}_{\pm 0.00}}$	${74.53}\phantom{{}_{\pm 0.00}}$	${81.75}\phantom{{}_{\pm 0.00}}$	–	–	–	–
SD-LoRA [ 45 ]	ICLR’25	✗	${79.15}_{\pm 0.20}$	${83.01}_{\pm 0.42}$	${75.26}_{\pm 0.37}$	${80.22}_{\pm 0.72}$	${49.12}_{\pm 1.78}$	${59.63}_{\pm 1.57}$	${81.01}_{\pm 0.42}$	${86.85}_{\pm 0.40}$

Table 2: Additional results under different task settings, including 5 and 20 tasks on ImageNet-R, 20 tasks on ImageNet-A, and 10 tasks on DomainNet. Within each group, the highest results are in bold , and the second highest results are underlined . “FR” denotes the use of feature replay.

Idx	HIM	TSP	10S-ImageNet-R		10S-ImageNet-A		20S-ImageNet-R		20S-ImageNet-A
Idx	HIM	TSP	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$
1	–	–	73.05 ±0.35	80.40 ±0.60	52.56 ±2.08	63.36 ±1.32	66.14 ±0.20	73.72 ±0.25	46.06 ±2.14	59.43 ±1.77
2	✓	–	80.36 ±0.40	85.78 ±0.44	61.35 ±1.06	70.23 ±2.04	78.04 ±0.35	84.31 ±0.54	52.80 ±1.46	62.64 ±2.14
3	–	✓	74.20 ±0.91	80.67 ±0.47	56.57 ±2.31	66.26 ±1.46	69.17 ±0.10	77.16 ±0.39	49.50 ±0.72	60.45 ±1.52
4	✓	✓	$\mathbf{81.34}_{\bm{\pm}\mathbf{0.16}}$	$\mathbf{86.53}_{\bm{\pm}\mathbf{0.51}}$	$\mathbf{62.45}_{\bm{\pm}\mathbf{1.34}}$	$\mathbf{71.02}_{\bm{\pm}\mathbf{1.82}}$	$\mathbf{78.62}_{\bm{\pm}\mathbf{0.43}}$	$\mathbf{84.62}_{\bm{\pm}\mathbf{0.26}}$	$\mathbf{53.87}_{\bm{\pm}\mathbf{1.45}}$	$\mathbf{63.69}_{\bm{\pm}\mathbf{1.87}}$

Table 3: Ablation studies on ImageNet-R and ImageNet-A under 10 and 20 incremental tasks.

Strategy	10S-ImageNet-R		10S-ImageNet-A
Strategy	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$
Seq-FT	$73.05_{\pm 0.35}$	$80.40_{\pm 0.60}$	$52.56_{\pm 2.08}$	$63.36_{\pm 1.32}$
EMA-style ( $\theta=\frac{t-1}{t}\theta_{t-1}+\frac{1}{t}\theta_{t}$ ) [ 20 ]	$79.14_{\pm 0.10}$	$85.08_{\pm 0.41}$	$59.64_{\pm 1.22}$	$69.52_{\pm 1.82}$
LoRA-P&M [ 28 ]	$78.89_{\pm 0.44}$	$84.99_{\pm 0.45}$	$59.93_{\pm 1.57}$	$69.87_{\pm 2.05}$
HIM-last ( $\mathcal{J}_{old}=\mathcal{L}_{t-1}(\mathbf{W}_{t-1}+\hat{\Delta\mathbf{W}}_{t})-\mathcal{L}_{t-1}(\mathbf{W}_{t-1})$ )	$76.23_{\pm 0.76}$	$83.81_{\pm 0.61}$	$58.17_{\pm 2.38}$	$68.84_{\pm 1.93}$
HIM (Ours)	$\mathbf{80.36}_{\bm{\pm}\mathbf{0.40}}$	$\mathbf{85.78}_{\bm{\pm}\mathbf{0.44}}$	$\mathbf{61.35}_{\bm{\pm}\mathbf{1.06}}$	$\mathbf{70.23}_{\bm{\pm}\mathbf{2.04}}$

Table 4: Comparison of different merge strategies on 10S-ImageNet-R and 10S-ImageNet-A.

Dataset

Metrics

Methods

Task 0

Task 1

Task 2

Task 3

Task 4

Task 5

Task 6

Task 7

Task 8

Task 9

ImageNet-R

Accuracy

\uparrow

w/o TSP

94.96_{\pm 1.23}

91.31_{\pm 1.22}

89.48_{\pm 0.20}

86.91_{\pm 0.67}

84.73_{\pm 0.64}

83.68_{\pm 0.20}

83.00_{\pm 0.61}

82.13_{\pm 0.53}

81.24_{\pm 0.56}

80.36_{\pm 0.40}

w/ TSP

\mathbf{95.14}_{\pm 1.07}

\mathbf{92.14}_{\pm 0.86}

\mathbf{89.93}_{\pm 0.85}

\mathbf{87.58}_{\pm 0.56}

\mathbf{85.86}_{\pm 0.91}

\mathbf{84.49}_{\pm 0.61}

\mathbf{83.67}_{\pm 0.42}

\mathbf{82.98}_{\pm 0.40}

\mathbf{82.14}_{\pm 0.54}

\mathbf{81.34}_{\pm 0.16}

\Delta

+0.18

+0.83

+0.45

+0.67

+1.13

+0.81

+0.67

+0.85

+0.90

+0.98

Forgetting

\downarrow

w/o TSP

–

4.98_{\pm 0.90}

4.87_{\pm 0.60}

6.16_{\pm 0.58}

7.00_{\pm 0.59}

7.34_{\pm 0.20}

7.26_{\pm 0.67}

7.34_{\pm 0.49}

7.43_{\pm 0.44}

7.76_{\pm 0.38}

w/ TSP

–

\mathbf{3.17}_{\pm 0.60}

\mathbf{3.79}_{\pm 0.86}

\mathbf{4.40}_{\pm 0.58}

\mathbf{4.89}_{\pm 0.78}

\mathbf{5.41}_{\pm 0.24}

\mathbf{5.46}_{\pm 0.62}

\mathbf{5.49}_{\pm 0.88}

\mathbf{5.52}_{\pm 0.82}

\mathbf{5.77}_{\pm 0.64}

\Delta

–

-1.81

-1.08

-1.76

-2.11

-1.93

-1.80

-1.85

-1.91

-1.99

Table 6: Task-by-task results (%) on ImageNet-R and ImageNet-A under the 10 task setting. We compare the results with and without TSP. The forgetting metric is defined in Appendix C.1

Figure 2: Analysis of the effect of SAM on 10S-ImageNet-R. Panels (a,b) visualize the task 2 loss landscapes for updates learned without/with SAM, while panels (c,d) show the total old task loss after merging. The dashed line denotes scalar interpolation, and the red star marks the merged solution.

Figure 3: Analysis of scalar merge and HIM without TSP on 10S-ImageNet-R. Panel (a) compares task 0/1 accuracy under different merge choices, and panels (b–d) show the task 0, task 1, and joint loss landscapes. The dashed line denotes scalar interpolation; the green diamond and red star mark the best scalar merge and selected HIM solution, respectively.

Figure 4: Analysis of the effect of TSP on 10S-ImageNet-R. Panels (a–c) show the task 0, task 1, and joint loss landscapes after introducing TSP, corresponding to Fig. 3 (b–d). Panels (d,e) isolate TSP at the task 2 merge step: from the same task 1 model, we learn and merge the task 2 update without/with TSP and visualize the joint loss over old tasks 0 and 1. The dashed line denotes scalar interpolation, and the red star marks the merged solution.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Overall framework of $\mathrm{M}^{2}\mathrm{LE}$ . (B) During training, TSP simulates cross task interference between label-disjoint subsets. (A) After training, HIM adaptively merges the LoRA update using historical and current-task Fisher information.

Panel of Figure 11 — Algorithm 1 Training pipeline of the proposed M erging Strength M odulation via Joint L oss E stimation ( $\text{M}^{2}$ LE).

Metric	Task 0	Task 1	Task 2	Task 3	Task 4	Task 5	Task 6	Task 7	Task 8	Task 9
$S_{t}\uparrow$	–	1.000	0.988	0.986	0.986	0.987	0.988	0.987	0.987	0.986
HIM $\mathcal{A}_{Last}\uparrow$	96.66	92.81	89.41	87.39	84.69	83.84	83.18	82.62	81.68	80.67
Recomputed Fisher $\mathcal{A}_{Last}\uparrow$	96.66	92.81	89.36	87.11	84.76	83.70	83.15	82.70	81.68	80.67

Appendix

Table 10: Task-by-task comparison of accumulated historical Fisher and recomputed Fisher on 10S-ImageNet-R. Results are obtained with a single random seed.

Figure 6: Hyper-parameter analysis on 10S-ImageNet-R. From left to right, $\lambda$ , $\rho$ , and $\beta$ are varied while the other two are fixed. All results are averaged over three random seeds, and error bars denote the standard deviation.

Figure 7: Same as Fig. 6 , but on 10S-ImageNet-A.

Method	ImageNet-R				CIFAR100				CUB
	B0 Inc10		B50 Inc10		B0 Inc10		B50 Inc10		B0 Inc20		B100 Inc20
	$\mathcal{A}_{Last}$	$\mathcal{A}_{Avg}$	$\mathcal{A}_{Last}$	$\mathcal{A}_{Avg}$	$\mathcal{A}_{Last}$	$\mathcal{A}_{Avg}$	$\mathcal{A}_{Last}$	$\mathcal{A}_{Avg}$	$\mathcal{A}_{Last}$	$\mathcal{A}_{Avg}$	$\mathcal{A}_{Last}$	$\mathcal{A}_{Avg}$
CoOp [ 56 ]	37.52	60.73	39.77	54.20	24.24	47.00	24.12	41.23	8.57	27.61	10.14	24.03
ZS-CLIP [ 29 ]	77.17	83.37	77.17	79.57	71.38	81.81	71.38	76.49	63.06	74.38	63.06	67.96
L2P [ 43 ]	66.52	75.97	66.77	72.82	73.03	82.74	73.61	81.14	57.93	70.87	66.12	75.64
DualPrompt [ 42 ]	66.65	76.21	67.58	73.22	72.44	81.63	72.57	80.12	57.46	69.89	64.84	71.40

Appendix

Table 11: Experimental results on ImageNet-R, CIFAR100 and CUB datasets using CLIP as the backbone. All methods are initialized with the CLIP pre-trained on LAION-400M without experience replay for fair comparisons. PROOF † represents our reproduced results of PROOF [ 55 ] without experience replay. The best results are in bold and the second highest results are underlined .

Method	Aircraft				Cars				UCF
	B0 Inc10		B50 Inc10		B0 Inc10		B50 Inc10		B0 Inc20		B100 Inc20
	$\mathcal{A}_{Last}$	$\mathcal{A}_{Avg}$	$\mathcal{A}_{Last}$	$\mathcal{A}_{Avg}$	$\mathcal{A}_{Last}$	$\mathcal{A}_{Avg}$	$\mathcal{A}_{Last}$	$\mathcal{A}_{Avg}$	$\mathcal{A}_{Last}$	$\mathcal{A}_{Avg}$	$\mathcal{A}_{Last}$	$\mathcal{A}_{Avg}$
CoOp [ 56 ]	7.14	14.54	7.77	13.05	21.65	36.46	20.87	37.40	33.46	47.85	24.74	42.02
ZS-CLIP [ 29 ]	17.22	26.66	17.22	21.70	76.37	82.60	76.37	78.32	67.64	75.50	67.64	71.44
L2P [ 43 ]	28.29	47.19	32.13	44.07	61.82	76.63	65.64	76.37	76.43	86.34	76.62	83.95
DualPrompt [ 42 ]	25.83	44.30	33.57	46.07	62.94	76.26	67.55	76.88	75.82	85.21	76.35	84.31

Appendix

Table 12: Experimental results on Aircraft, Cars and UCF datasets using CLIP as the backbone. All methods are initialized with the CLIP pre-trained on LAION-400M without experience replay for fair comparisons. PROOF † represents our reproduced results of PROOF [ 55 ] without experience replay. The best results are in bold and the second highest results are underlined .

Continual LoRA

See all 27 papers

Sep 26, 2026

When the Merge Coefficient Stops Mattering: Proximity Regularized Merging for Continual LoRA Adaptation

Rehearsal-free continual learning with parameter-efficient adapters can be cast as a sequence of task-vector write-in operations: for each new task, a low-rank adapter is learned and merged into a running model. We propose Proximity Regularized Merging (PRM), a minimal modification to sequential LoRA merging that adds a proximal penalty during task-vector training without changing the subsequent write-in rule. PRM acts as a robust task-vector regularizer: in the reported Base->+Prox diagnostics, it improves AAA across multiple write-in rules, backbones, and class-incremental settings, while its fixed-coefficient variant remains competitive with strong coefficient-based baselines. Mechanistically, matched-prefix norm controls and proximal-strength sweeps show that proximal training shrinks the task-vector radius, lowers Fisher-weighted interference, broadens the coefficient plateau, and exposes a stability-plasticity trade-off. Together, these results suggest that the effectiveness of sequential LoRA merging depends not only on how much of a task vector is written in, but also on whether the task vector itself has been trained to be mergeable.

Yixuan Liu, Yuhao Sun, Sen Song +1

Sapient Intelligence · Tsinghua University

Continual Learning Incremental Learning Low-Rank Adaptation

Ask PDF

Feb 27, 2026

Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning

Continual Learning (CL) requires models to sequentially adapt to new tasks without forgetting old knowledge. Recently, Low-Rank Adaptation (LoRA), a representative Parameter-Efficient Fine-Tuning (PEFT) method, has gained increasing attention in CL. Several LoRA-based CL methods reduce interference across tasks by separating their update spaces, typically building the new space from the estimated null space of past tasks. However, they (i) overlook task-shared directions, which suppresses knowledge transfer, and (ii) fail to capture truly effective task-specific directions since these ``null bases" of old tasks can remain nearly inactive for new task under correlated tasks. To address this, we study LoRA learning capability from a projection energy perspective, and propose Low-rank Decomposition and Adaptation (LoDA). It performs a task-driven decomposition to build general and truly task-specific LoRA subspaces by solving two energy-based objectives, decoupling directions for knowledge sharing and isolation. LoDA fixes LoRA down-projections on two subspaces and learns robust up-projections via a Gradient-Aligned Optimization (GAO) approach. After each task, before integrating the LoRA updates into the backbone, LoDA derives a closed-form recalibration for the general update, approximating a feature-level joint optimum along this task-shared direction. Experiments indicate that LoDA outperforms existing CL methods. Our code is available at https://github.com/HHHLF/LoDA_ICML2026.

Lingfeng He, De Cheng, Huaijie Wang +3

State Key Laboratory of Integrated Services Networks, School of Telecommunications Engineering, Xidian University, Xi’an, China · School of Electronic Engineering, Xidian University, Xi’an, China · AUTHORERR: Missing \icmlaffiliation.

Subspace-Based Continual Learning Continual Learning Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Ask PDF

Jun 26, 2026

When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning

Low-Rank Adaptation (LoRA) has become the standard tool for parameter-efficient fine-tuning of large pretrained models. When applied sequentially across tasks in Continual Learning (CL), the standard assumption is that each new task requires a dedicated low-rank adapter. In this work, we challenge this assumption empirically and structurally. We show that task-specific LoRA adapters in CL exhibit significant low-rank redundancy: the subspaces spanned by adapters trained on different tasks substantially overlap, and in many cases earlier adapters can faithfully represent later tasks. Building on this observation, we propose LiteLoRA, a plug-and-play gating mechanism that learns at train time whether to recruit a new adapter or reuse existing low-rank representations. Our method reduces the number of active adapters by 20-70% while matching or exceeding state-of-the-art performance on standard CL benchmarks, revealing that structural redundancy is pervasive and that selective learning is sufficient to achieve stability without sacrificing plasticity.

Tanguy Dieudonné, Giulia Lanzillotta, Enis Simsar +2

Department of Computer Science, ETH Zurich, Zurich, Switzerland.

Continual Learning Low-Rank Adaptation Incremental Learning Parameter-Efficient Fine-Tuning Continual LoRA Adapter-Based Continual Learning

Ask PDF

Abstract

Continual LoRA

Explore similar work

When the Merge Coefficient Stops Mattering: Proximity Regularized Merging for Continual LoRA Adaptation

Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning

When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning