When the Merge Coefficient Stops Mattering: Proximity Regularized Merging for Continual LoRA Adaptation

Yixuan Liu; Yuhao Sun; Sen Song; Jin Li

cs.LGSep 26, 2026

When the Merge Coefficient Stops Mattering: Proximity Regularized Merging for Continual LoRA Adaptation

Authors: Yixuan Liu, Yuhao Sun, Sen Song, Jin Li

Organizations: Sapient Intelligence · Tsinghua University

Abstract

Rehearsal-free continual learning with parameter-efficient adapters can be cast as a sequence of task-vector write-in operations: for each new task, a low-rank adapter is learned and merged into a running model. We propose Proximity Regularized Merging (PRM), a minimal modification to sequential LoRA merging that adds a proximal penalty during task-vector training without changing the subsequent write-in rule. PRM acts as a robust task-vector regularizer: in the reported Base->+Prox diagnostics, it improves AAA across multiple write-in rules, backbones, and class-incremental settings, while its fixed-coefficient variant remains competitive with strong coefficient-based baselines. Mechanistically, matched-prefix norm controls and proximal-strength sweeps show that proximal training shrinks the task-vector radius, lowers Fisher-weighted interference, broadens the coefficient plateau, and exposes a stability-plasticity trade-off. Together, these results suggest that the effectiveness of sequential LoRA merging depends not only on how much of a task vector is written in, but also on whether the task vector itself has been trained to be mergeable.

Figures & tables

Panel of Figure 1 — Figure 1: Paired transfer checks for proximal task-vector shaping. (a) Dataset-level comparison on the CLIP backbone: PRM improves over the corresponding P&M Base across the plotted settings. (b) Merge-rule comparison on CLIP IN-R 10T: the same proximal task-vector-training term improves each reported Base $\rightarrow$ +Prox pair while leaving the write-in rule unchanged. Full values and setup checks are reported in Appendix D and Appendix C .

	ImageNet-R
	T=5		T=10		T=20
Method	Acc $\uparrow$	AAA $\uparrow$	Acc $\uparrow$	AAA $\uparrow$	Acc $\uparrow$	AAA $\uparrow$
Full-FT	64.92 $\pm$ .87 †	75.57 $\pm$ .50 †	60.57 $\pm$ 1.06 †	72.31 $\pm$ 1.09 †	49.95 $\pm$ 1.31 †	65.32 $\pm$ .69 †
L2P	73.04 $\pm$ .71 †	76.94 $\pm$ .61 †	71.26 $\pm$ .44 †	76.13 $\pm$ .46 †	68.97 $\pm$ .51 †	74.16 $\pm$ .32 †
BECAME	81.69 $\pm$ .25	86.25 $\pm$ .37	80.05 $\pm$ .22	85.65 $\pm$ .80	76.60 $\pm$ .45	83.32 $\pm$ .52
CoFiMA	81.99 $\pm$ .25	86.37 $\pm$ .47	80.13 $\pm$ .37	85.80 $\pm$ .60	76.28 $\pm$ .41	83.58 $\pm$ .40

Table 1: Results on ImageNet-R (ViT-B/16 AugReg). PRM -fixed and PRM -Fisher are ours.

Figure 2: Seen-task loss and accuracy along the write-in coefficient on CLIP ViT-B/16 IN-R 10T at task 5. (a) PRM has the lowest loss, highest accuracy, and smallest variation; the fixed

\alpha=0.8

lies near its empirical best. (b) Rescaling the P&M vector to the PRM norm greatly reduces accuracy variation, indicating that scale explains much of the coefficient sensitivity. Bars show Frobenius norms; curves show seen-task accuracy.

Figure 3: Effect of proximal strength on path diagnostics and benchmark performance. The

x

-axis is logarithmic and

\lambda_{\mathrm{prox}}=0

is mapped to

10^{-5}

for visualization. (a) As

\lambda_{\mathrm{prox}}

increases, the fraction of coefficients within 0.25 and 0.5 percentage points of the sweep optimum increases, while

q_{t}

decreases by orders of magnitude. The direction-normalized interference

q_{t}^{\mathrm{dir}}

is not monotonic. (b) Benchmark performance is highest at an intermediate proximal strength and decreases for very large

\lambda_{\mathrm{prox}}

Method	pert	prox	$\alpha$ -rule	AAA	Acc T	Forg	$\overline{W}_{t,0.01}$	$\overline{q}_{t}$ [ $10^{-3}$ ]
Upper block: deployable variants
LoRA baseline	–	–	$\alpha{=}1$	$76.72{\pm}0.81$	$66.54{\pm}1.19$	$26.66{\pm}1.73$	0.38	63.94
P&M-fixed	✓	–	$\alpha{=}0.8$	$82.30{\pm}0.97$	$74.80{\pm}0.40$	$16.06{\pm}0.94$	$0.44$	$45.12$
P&M	✓	–	Fisher	$85.55{\pm}0.90$	$79.96{\pm}0.51$	$3.50{\pm}0.41$	$0.44$	$45.12$
LoRA-Prox	–	✓	$\alpha{=}0.8$	$85.61{\pm}0.48$	$80.27{\pm}0.31$	$7.72{\pm}0.62$	0.84	$4.93$
PRM -fixed	✓	✓	$\alpha{=}0.8$	$86.39{\pm}0.53$	$81.32{\pm}0.25$	$5.62{\pm}0.29$	$0.91$	$3.26$

Table 2: Ablation and NoWrite attribution on ImageNet-R 10T (3 seeds, mean

\pm

std).

\overline{W}_{t,0.01}

is near-optimal plateau width and

\overline{q}_{t}

is Fisher-weighted interference in

10^{-3}

units, averaged over sweep tasks

\{2,5,10\}

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

\alpha_{t}=-\,\frac{\displaystyle\sum_{\ell}\sum_{i=0}^{t}\bigl\langle F_{i}^{(\ell)}\!\odot\!\delta_{t}^{(\ell)},\;W_{<t}^{(\ell)}-\widetilde{W}_{i}^{(\ell)}-\delta_{t}^{(\ell)}\bigr\rangle}{\displaystyle\sum_{\ell}\sum_{i=0}^{t}\bigl\langle F_{i}^{(\ell)}\!\odot\!\delta_{t}^{(\ell)},\;\delta_{t}^{(\ell)}\bigr\rangle},

Appendix

Algorithm 1 PRM for rehearsal-free sequential LoRA merging.

Method	Task-vector training	Write-in rule	Source
Full-FT	joint update of all parameters	no merge (single model)	in-house
LoRA	per-task LoRA, no perturbation	write-in with $\alpha{=}1$	in-house
EWC-LoRA	per-task LoRA + EWC penalty	write-in with $\alpha{=}1$	in-house
L2P, DualPrompt, CODA-Prompt	prompt-pool training (no LoRA merge)	no merge (prompt routing)	lifted from Qiu et al. [2025] where dagger-marked
Inf-LoRA	subspace-projected LoRA training	subspace consolidation	lifted from Qiu et al. [2025] where dagger-marked
SD-LoRA	soft-router LoRA training	router-weighted consolidation	lifted from Qiu et al. [2025] where dagger-marked

Appendix

Table 3: Compared methods and implementation sources. All in-house runs share the protocol of paragraphs A.1–A.2. Dagger entries in result tables are lifted under the matching protocol of Qiu et al. [2025] ; cells without

\pm

aggregate a single available seed. “Same task-vector training” is shorthand for the perturbation-aware LoRA recipe used by P&M; “+ proximal” adds the penalty in Eq. ( 7 ).

	CIFAR-100				DomainNet		CUB-200
	T=10		T=20		T=5		T=10
Method	Acc $\uparrow$	AAA $\uparrow$	Acc $\uparrow$	AAA $\uparrow$	Acc $\uparrow$	AAA $\uparrow$	Acc $\uparrow$	AAA $\uparrow$
Full-FT	69.49 $\pm$ .50 †	80.35 $\pm$ .87 †	51.21 $\pm$ 1.26 †	61.83 $\pm$ 1.20 †	51.46 $\pm$ .47 †	67.08 $\pm$ 1.13 †	51.43 $\pm$ 1.41 †	69.74 $\pm$ .93 †
L2P	83.18 $\pm$ 1.20 †	87.69 $\pm$ 1.05 †	85.66 $\pm$ .16 †	90.60 $\pm$ .23 †	70.26 $\pm$ .25 †	75.83 $\pm$ .98 †	65.18 $\pm$ 2.49 †	76.12 $\pm$ 1.27 †
BECAME	87.25 $\pm$ .49	91.22 $\pm$ 1.76	84.82 $\pm$ .48	89.89 $\pm$ 1.35	83.47 $\pm$ 1.21	87.54 $\pm$ 1.52	69.36 $\pm$ 1.03	76.95 $\pm$ .81
CoFiMA	86.90 $\pm$ .19	90.99 $\pm$ 1.68	84.91 $\pm$ .10	90.23 $\pm$ 1.22	83.71 $\pm$ .95	87.62 $\pm$ 1.17	69.43 $\pm$ 1.15	77.10 $\pm$ 1.55

Appendix

Table 4: Main results on CIFAR-100, DomainNet, and CUB-200 (ViT-B/16 AugReg).

Write-in rule	Backbone	AAA		$\Delta$ AAA	Forg		$\Delta$ Forg
		Base	+Prox		Base	+Prox	(reduction)
P&M ( $\alpha{=}0.8$ )	AugReg	82.30 $\pm$ 0.97	86.39 $\pm$ 0.53	+4.09	16.06 $\pm$ 0.94	5.62 $\pm$ 0.29	+10.44
P&M ( $\alpha{=}0.8$ )	CLIP	60.77 $\pm$ 0.02	84.13 $\pm$ 0.90	+23.36	52.41 $\pm$ 1.19	12.77 $\pm$ 0.69	+39.64
P&M (Fisher $\alpha$ )	AugReg	85.55 $\pm$ 0.90	85.96 $\pm$ 0.48	+0.41	3.50 $\pm$ 0.41	4.28 $\pm$ 0.28	-0.78
P&M (Fisher $\alpha$ )	CLIP	77.53 $\pm$ 0.73	85.55 $\pm$ 0.60	+8.02	7.72 $\pm$ 1.09	5.93 $\pm$ 0.81	+1.79

Appendix

Table 5: Backbone-paired ImageNet-R 10T comparison. Forgetting reduction is Forg(Base)

-

Forg(+Prox), so positive means improvement.

Figure 4: Five-backbone radar on ImageNet-R 10T. Each axis is one backbone (AugReg, CLIP, Sup-21K, DINO, MAE) and the radial value is AAA . The two polylines compare P&M Base against PRM ; PRM improves over P&M on every reported backbone, with the largest lift on the backbones whose unshaped Base is the lowest.

	ImageNet-R
	T=5		T=10		T=20
Method	Acc $\uparrow$	AAA $\uparrow$	Acc $\uparrow$	AAA $\uparrow$	Acc $\uparrow$	AAA $\uparrow$
BECAME	75.20 $\pm$ 1.51	80.18 $\pm$ 1.80	69.68 $\pm$ 0.76	77.58 $\pm$ 0.34	61.35 $\pm$ 1.79	72.24 $\pm$ 2.83
CoFiMA	72.92 $\pm$ 0.85	78.92 $\pm$ 1.62	65.00 $\pm$ 1.04	75.53 $\pm$ 0.39	51.82 $\pm$ 1.35	68.76 $\pm$ 1.14
CoMA	72.47 $\pm$ 0.71	78.78 $\pm$ 1.63	63.54 $\pm$ 1.39	74.96 $\pm$ 0.28	48.43 $\pm$ 0.74	66.63 $\pm$ 1.25
MagMax	62.01 $\pm$ 0.52	67.81 $\pm$ 1.05	58.86 $\pm$ 0.66	65.66 $\pm$ 0.26	55.29 $\pm$ 0.93	64.67 $\pm$ 1.24

Appendix

Table 6: Main results on ImageNet-R with the CLIP-OpenAI ViT-B/16 backbone. PRM-fixed and PRM-Fisher (bold rows) are our methods.

	CIFAR-100				DomainNet		CUB-200
	T=10		T=20		T=5		T=10
Method	Acc $\uparrow$	AAA $\uparrow$	Acc $\uparrow$	AAA $\uparrow$	Acc $\uparrow$	AAA $\uparrow$	Acc $\uparrow$	AAA $\uparrow$
BECAME	69.86 $\pm$ 1.41	78.57 $\pm$ 1.60	58.56 $\pm$ 3.17	68.13 $\pm$ 3.88	80.89 $\pm$ 1.10	83.74 $\pm$ 1.22	43.97 $\pm$ 1.69	54.36 $\pm$ 3.01
CoFiMA	61.99 $\pm$ 1.41	73.91 $\pm$ 1.58	46.26 $\pm$ 1.53	62.65 $\pm$ 1.48	78.98 $\pm$ 0.26	82.99 $\pm$ 0.81	44.10 $\pm$ 1.00	55.51 $\pm$ 2.61
CoMA	58.15 $\pm$ 2.39	72.12 $\pm$ 1.77	36.08 $\pm$ 2.57	56.31 $\pm$ 1.16	78.16 $\pm$ 0.23	82.74 $\pm$ 0.81	39.49 $\pm$ 0.24	53.80 $\pm$ 2.51
MagMax	52.06 $\pm$ 1.73	60.97 $\pm$ 1.05	43.11 $\pm$ 1.56	52.37 $\pm$ 1.06	68.49 $\pm$ 1.53	72.54 $\pm$ 0.60	40.92 $\pm$ 0.45	52.73 $\pm$ 1.57

Appendix

Table 7: Main results on CIFAR-100, DomainNet, and CUB-200 with the CLIP-OpenAI ViT-B/16 backbone.

Merge rule	AAA $\uparrow$			Forgetting $\downarrow$
	Base	+Prox	$\Delta$	Base	+Prox	$\Delta$
P&M (Fisher $\alpha$ )	77.53 $\pm$ 0.73	85.55 $\pm$ 0.60	+8.02	7.72 $\pm$ 1.09	5.93 $\pm$ 0.81	-1.79
P&M ( $\alpha{=}0.8$ fixed)	60.77 $\pm$ 0.02	84.13 $\pm$ 0.90	+23.36	52.41 $\pm$ 1.19	12.77 $\pm$ 0.69	-39.64
CoFiMA	75.53 $\pm$ 0.39	85.77 $\pm$ 0.55	+10.24	26.13 $\pm$ 1.64	9.14 $\pm$ 0.77	-16.99
CoMA	74.96 $\pm$ 0.28	85.75 $\pm$ 0.55	+10.79	28.70 $\pm$ 2.15	9.23 $\pm$ 0.95	-19.47
Model-Avg	77.75 $\pm$ 0.58	85.74 $\pm$ 0.58	+7.99	8.75 $\pm$ 0.64	5.69 $\pm$ 0.77	-3.06

Appendix

Table 8: Beyond P&M generality on CLIP IN-R 10T (3 seeds, mean

\pm

std). Each write-in rule paired with no-prox Base and +Prox (

\lambda_{\mathrm{prox}}=10^{-2}

\Delta\textsc{AAA}

and

\Delta\textsc{Forg}

are computed as +Prox minus Base; positive

\Delta\textsc{AAA}

and negative

\Delta\textsc{Forg}

both indicate improvement. In this setting, every reported rule receives positive

\Delta\textsc{AAA}

and negative

\Delta\textsc{Forg}

. Bold marks improvements.

Merge rule	AAA $\uparrow$			Forgetting $\downarrow$
	Base	+Prox	$\Delta$	Base	+Prox	$\Delta$
P&M (Fisher $\alpha$ )	72.45 $\pm$ 2.96	77.88 $\pm$ 1.27	+5.44	6.26 $\pm$ 0.68	7.42 $\pm$ 1.12	+1.16
P&M ( $\alpha{=}0.8$ fixed)	49.49 $\pm$ 0.80	75.51 $\pm$ 0.99	+26.02	67.48 $\pm$ 1.52	20.71 $\pm$ 1.36	-46.77
CoFiMA	68.76 $\pm$ 1.14	78.14 $\pm$ 1.24	+9.38	33.41 $\pm$ 1.19	14.70 $\pm$ 0.96	-18.71
CoMA	66.63 $\pm$ 1.25	78.04 $\pm$ 1.31	+11.41	40.16 $\pm$ 1.21	14.52 $\pm$ 1.03	-25.64
Model-Avg	72.08 $\pm$ 2.64	77.82 $\pm$ 1.52	+5.74	6.59 $\pm$ 0.84	7.45 $\pm$ 1.12	+0.85

Appendix

Table 9: Beyond P&M generality on CLIP IN-R 20T (3 seeds, mean

\pm

std). Same protocol as Table 8 ; harder split with twice the merge events.

\Delta

columns are +Prox minus Base; positive

\Delta\textsc{AAA}

and negative

\Delta\textsc{Forg}

indicate improvement. Positive

\Delta\textsc{AAA}

persists across all 7 rules; forgetting changes are rule-dependent, with the largest reductions on rules whose Base over-writes (e.g.

\alpha{=}0.8

fixed).

Merge rule	AAA $\uparrow$			Forgetting $\downarrow$
	Base	+Prox	$\Delta$	Base	+Prox	$\Delta$
P&M (Fisher $\alpha$ )	67.85 $\pm$ 3.93	79.13 $\pm$ 0.68	+11.28	8.66 $\pm$ 2.72	9.33 $\pm$ 0.74	+0.67
P&M ( $\alpha{=}0.8$ fixed)	34.81 $\pm$ 2.05	76.51 $\pm$ 0.62	+41.70	82.84 $\pm$ 1.24	19.80 $\pm$ 1.17	-63.04
CoFiMA	62.65 $\pm$ 1.48	78.94 $\pm$ 0.59	+16.29	39.94 $\pm$ 3.20	15.13 $\pm$ 0.66	-24.81
CoMA	56.31 $\pm$ 1.16	79.53 †	+23.22	58.26 $\pm$ 5.10	15.60 †	-42.66
Model-Avg	67.22 $\pm$ 4.37	79.13 $\pm$ 0.93	+11.91	8.91 $\pm$ 2.75	9.26 $\pm$ 0.57	+0.35

Appendix

Table 10: Beyond P&M generality on CLIP CIFAR-100 20T (CLIP backbone).

\dagger

: cell aggregates a single available seed; standard deviation is omitted.

Merge rule	AAA $\uparrow$			Forgetting $\downarrow$
	Base	+Prox	$\Delta$	Base	+Prox	$\Delta$
P&M (Fisher $\alpha$ )	83.72 $\pm$ 1.26	90.34 $\pm$ 0.13	+6.62	11.44 $\pm$ 0.82	6.87 $\pm$ 0.53	-4.58
P&M ( $\alpha{=}0.8$ fixed)	72.30 $\pm$ 0.21	88.69 $\pm$ 1.21	+16.39	37.74 $\pm$ 0.05	9.27 $\pm$ 1.11	-28.46
CoFiMA	82.99 $\pm$ 0.81	90.34 $\pm$ 0.39	+7.35	18.48 $\pm$ 0.54	7.55 $\pm$ 0.67	-10.93
CoMA	82.74 $\pm$ 0.81	90.09 $\pm$ 0.21	+7.35	19.97 $\pm$ 0.53	7.75 $\pm$ 0.73	-12.22
Model-Avg	83.90 $\pm$ 1.13	90.34 $\pm$ 0.38	+6.45	12.02 $\pm$ 0.54	7.08 $\pm$ 0.66	-4.94

Appendix

Table 11: Beyond P&M generality on CLIP DomainNet 5T (CLIP backbone).

Merge rule	AAA $\uparrow$			Forgetting $\downarrow$
	Base	+Prox	$\Delta$	Base	+Prox	$\Delta$
P&M (Fisher $\alpha$ )	54.38 $\pm$ 2.93	61.31 $\pm$ 2.52	+6.94	14.70 $\pm$ 1.50	14.40 $\pm$ 2.26	-0.30
P&M ( $\alpha{=}0.8$ fixed)	46.11 $\pm$ 1.69	58.05 $\pm$ 2.14	+11.94	66.75 $\pm$ 1.13	35.84 $\pm$ 3.39	-30.91
CoFiMA	55.51 $\pm$ 2.61	61.36 $\pm$ 3.10	+5.85	27.14 $\pm$ 1.77	25.27 $\pm$ 2.64	-1.87
CoMA	53.80 $\pm$ 2.51	61.64 $\pm$ 2.61	+7.84	44.65 $\pm$ 0.42	25.58 $\pm$ 2.20	-19.07
Model-Avg	56.04 $\pm$ 2.52	61.82 $\pm$ 2.39	+5.78	16.95 $\pm$ 1.53	15.15 $\pm$ 2.42	-1.80

Appendix

Table 12: Beyond P&M generality on CLIP CUB-200 10T (CLIP backbone).

Setting	Rules improved / total (AAA)	Mean $\Delta$ AAA	Median $\Delta$ AAA	Best $\Delta$ AAA	Mean Forg. reduction
CLIP IN-R 10T	7/7	+12.33	+10.24	+23.36	+15.52
CLIP IN-R 20T	7/7	+10.89	+9.38	+26.02	+15.51
CLIP CIFAR-100 20T	7/7	+20.13	+16.29	+41.70	+23.55
CLIP DomainNet 5T	7/7	+9.51	+7.35	+16.39	+12.80
CLIP CUB-200 10T	7/7	+7.44	+6.94	+11.94	+10.74

Appendix

Table 13: Per-setting transfer summary on CLIP. “Rules improved/total” counts the available merge rules with positive

\Delta\textsc{AAA}

from Base to

+

Prox. Forgetting reduction is Forg(Base)

-

Forg(+Prox), so positive means improvement.

Vector	Direction	Norm	$\\|\delta_{t}\\|_{F}$	$q_{t}$ [ $10^{-3}$ ]	$q_{t}^{\mathrm{dir}}$ [ $10^{-3}$ ]	$R_{t}$ [pp]	$G_{t,\mathrm{fix}}$ [pp]	Acc ${}_{\leq t}{@}\alpha{=}0.8$
P&M raw	P&M	P&M	11.72 $\pm$ 0.50	162.82 $\pm$ 12.22	1.18 $\pm$ 0.01	17.77 $\pm$ 0.53	10.73 $\pm$ 0.46	66.72 $\pm$ 0.61
P&M $\!\to\!$ PRM-norm	P&M	PRM	1.56 $\pm$ 0.02	2.88 $\pm$ 0.11	1.18 $\pm$ 0.01	2.37 $\pm$ 0.44	0.33 $\pm$ 0.11	76.49 $\pm$ 0.09
PRM raw	PRM	PRM	1.56 $\pm$ 0.02	29.91 $\pm$ 0.51	12.29 $\pm$ 0.30	3.07 $\pm$ 0.82	0.27 $\pm$ 0.19	77.25 $\pm$ 0.55
PRM $\!\to\!$ P&M-norm	PRM	P&M	11.72 $\pm$ 0.50	1694.62 $\pm$ 177.89	12.29 $\pm$ 0.30	70.01 $\pm$ 0.49	63.28 $\pm$ 0.74	14.05 $\pm$ 1.31

Appendix

Table 14: Norm-matched task-vector intervention on CLIP IN-R 10T (task 5). Each row fixes the direction source and norm source before evaluating the same common-prefix

\alpha

sweep.

Panel of Figure 5 — Figure 5: Norm-matched intervention on CLIP IN-R 10T (task 5). Rescaling $\delta_{\textsc{P\&M}}$ to PRM ’s norm without altering its direction substantially reduces coefficient sensitivity; inflating $\delta_{\textsc{PRM}{}}$ to P&M’s norm substantially increases coefficient sensitivity.

Panel of Figure 6 — Figure 6: Continuous task-vector rescaling intervention on CLIP IN-R 10T (task 5). We sweep $c\in\{0,\tfrac{1}{4},\tfrac{1}{2},\tfrac{3}{4},1,\tfrac{5}{4},\tfrac{3}{2},2\}$ for both P&M and PRM vectors. The sweep verifies the expected scaling relations $\|\delta(c)\|_{F}=c\|\delta(1)\|_{F}$ and $q_{t}(c)=c^{2}q_{t}(1)$ , while coefficient sensitivity and fixed- $\alpha$ accuracy vary continuously with $c$ .

Backbone	Setting	$\overline{R}_{t}^{\textsc{P\&M}}$ [pp]	$\overline{q}_{t}^{\textsc{P\&M}}$ [ $10^{-3}$ ]	$\Delta\textsc{AAA}$	Forg. reduction
Settings with $\alpha$ -sweep instrumentation (R t available)
AugReg	IR-10T	6.78	43.74	+0.84	-0.78
AugReg	DomainNet 5T	7.92	77.20	+1.81	+3.07
AugReg	IR-5T	7.05	120.97	-0.52	+0.32
AugReg	IR-20T	6.88	15.12	+0.54	-1.28
AugReg	CIFAR-100 10T	8.46	39.92	+0.81	-0.16

Appendix

Table 15: Cross-setting summary.

\overline{R}_{t}

and

\overline{q}_{t}

are baseline P&M values;

\Delta\textsc{AAA}

is PRM -fixed

-

P&M; forgetting reduction is P&M Forg.

-

PRM -Fisher Forg. (positive means improvement). “–” in

\overline{R}_{t}

marks settings without an

\alpha

sweep.

Figure 7: Cross-setting gain pattern. (a) Baseline P&M

\overline{R}_{t}

vs.

\Delta\textsc{AAA}

( PRM -fixed

-

P&M). (b) Baseline P&M

\overline{q}_{t}

vs. forgetting reduction (P&M

-

PRM -Fisher). Markers encode dataset and backbone.

Continual Learning

Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.

See all 305 papers

Jun 17, 2026

Predicting Mergeability of Parameter-Efficient Fine-Tuning Updates

Low-rank adaptation (LoRA) makes it cheap to train many domain- and task-specific language model adapters, but whether two adapters can be merged is usually discovered only after both have been fully trained and evaluated. This late feedback is costly: adapters that are strong in isolation can interfere destructively once their updates are combined. We ask whether this outcome can be anticipated. We formalize adapter mergeability as the degree to which an adapter preserves its single-task utility after merging, and show that it can be forecast from signals measured in the first few percent of training -- chiefly how the low-rank updates and their gradients align across tasks and how much they disturb shared representations. We package these signals into MergeProbe, a lightweight predictor that estimates pairwise and set-level retention and turns the estimate into a concrete decision: merge directly, reweight, prune, or route. On MERGE-PEFT, a five-domain benchmark spanning math, code, science, instruction following, and safety, MergeProbe attains the best average and worst-case retention among strong interference-aware merge baselines while adding far less deployment overhead than full task routing. This turns LoRA merging from a post-hoc engineering step into an anticipatory measurement problem.

Lin Tang, Wei Zhang, Jing Li +3

1Sichuan University, Chengdu, China · University of Electronic Science and Technology of China, Chengdu, China

Low-Rank Adaptation Language Model Merging Model Merging Parameter-Efficient Fine-Tuning

Ask PDF

Sep 30, 2026

ChainLoRA: Geometry-Preserving Task Vector Merging for Continual Learning in LLMs

Continual parameter-efficient fine-tuning for large language models (LLMs) must balance retention of previously acquired knowledge, adaptation to new tasks, and strict parameter budgets. We present ChainLoRA, a replay-free continual merging framework built on chain-updated task-vector geometry. From a parameter-merging perspective, we formulate a geometric view of forgetting through a measurable interaction between task updates, separating directional overlap from coefficient coupling. Building on this view, ChainLoRA combines chain-updated training with post-stream adaptive SVD merging. During training, initialization and a one-sided orthogonality proxy use only the last carrier, keeping their historical-state footprint and regularization overhead constant as the task stream grows. At merging time, Adaptive SVD extracts a shared carrier and aligns it to the latest task through Procrustes adaptation. Our theoretical analysis shows that Procrustes adaptation facilitates geometric approximate separation of shared and task-specific components. The one-sided proxy further bounds inter-task interference. An effective-rank penalty additionally promotes efficient utilization of the task subspace during continual learning. Experiments show that ChainLoRA achieves state-of-the-art performance among the evaluated replay-free methods on the Large and SuperNI benchmarks, while remaining competitive on Standard CL and attaining almost the closest average scores to the evaluated replay-based method across all three benchmarks.

Hang Yin, Haozhe Wang, Yuhua Luo +3

Shanghai Jiao Tong University · Shanghai Innovation Institute · Beijing Zhongguancun Academy +3

Continual Learning for LLMs Low-Rank Adaptation Task Vector Merging Language Model Merging Model Merging Parameter-Efficient Fine-Tuning

Ask PDF

Jul 18, 2026

CT-Merging: Consensus Directions and Task-Level Scaling for LoRA Adapter Merging

LoRA adapters provide an efficient way to specialize a pretrained model for many downstream tasks, but deploying one adapter per task requires adapter storage and task selection at inference time. Model merging addresses this issue by combining independently trained adapters into one multi-task adapter. Recent SVD-based LoRA merging methods mainly focus on constructing shared or task specific directions, while the coefficients assigned to the final directions are often directly from the original task SVD. On a fixed merged basis, inherited coefficients preserve component order with high rank correlation, yet their magnitudes differ substantially from the coefficients induced by the task updates. To address this mismatch, we propose CT-Merging, a LoRA-aware merging algorithm that estimates consensus directions from average task subspace projectors and assigns task-level RMS coefficient scales in the final update. CT-Merging uses repeated support across task SVD subspaces to construct the common basis, while reducing reliance on rank wise SVD magnitudes after direction construction. On the DC-Merge CLIP adapter benchmark, CT-Merging achieves superior average normalized accuracy compared to state-of-the-art merging methods and further improves over DC-Merge by 2.56 points on ViT-B/32 and 1.51 points on ViT-L/14 KnoTS-trained checkpoints.

Keumseo Ryum, Joonhyuk Kang

KAIST

Low-Rank Adaptation Model Merging

Ask PDF

Abstract

Continual Learning

Explore similar work

Predicting Mergeability of Parameter-Efficient Fine-Tuning Updates

ChainLoRA: Geometry-Preserving Task Vector Merging for Continual Learning in LLMs

CT-Merging: Consensus Directions and Task-Level Scaling for LoRA Adapter Merging