A Margin Perspective on LoRA: Robustness to Catastrophic Forgetting and Adapter Merging (MaLoRA)

Ziqing Xu; Hancheng Min; Lachlan Ewen MacDonald; Salma Tarmoun; Enrique Mallada; Weijie Su; Rene Vidal

cs.LGNov 10, 2026

A Margin Perspective on LoRA: Robustness to Catastrophic Forgetting and Adapter Merging (MaLoRA)

Authors: Ziqing Xu, Hancheng Min, Lachlan Ewen MacDonald, Salma Tarmoun, Enrique Mallada, Weijie Su, Rene Vidal

Organizations: Statistics and Data Science, University of Pennsylvania · IDEAS, University of Pennsylvania · INS & SMS, Shanghai Jiao Tong University · ESE, University of Pennsylvania ·

Abstract

Low-rank adaptation (LoRA) is the de facto method for parameter-efficient fine-tuning of large neural networks, achieving strong task adaptation with minimal trainable parameters and low optimization cost. Beyond this efficiency, practitioners have observed two striking properties: robustness to catastrophic forgetting and the ability to merge independently trained adapters into a single model that performs competitively across multiple tasks. These properties are central to continual and multi-task learning, yet remain poorly understood. In this work, we provide a theoretical explanation through a unified margin-based perspective. We analyze LoRA in multiclass linear classification under a near-orthogonal task regime with $\ell_2$ -regularization, and characterize the optimal LoRA adapter across regularization regimes. Our analysis shows that, in an intermediate regime of regularization parameters, the optimal adapter aligns with the max-margin solution on the fine-tuning data. Building on this characterization, we derive two key consequences. First, we obtain closed-form expressions for the margins on pre-training and fine-tuning data, revealing a precise margin trade-off: the regularization parameter controls the balance between retention and adaptation. Second, we analyze adapter merging, proving that merged models achieve positive margin on each task and deriving optimal mixing coefficients that balance margins across tasks and maximize the margin over their union. These results lead to a simple, training-free merging rule, which we term margin-based LoRA merging (MaLoRA). Experiments on modern architectures and real datasets validate our theoretical predictions, showing that MaLoRA matches or outperforms several adapter-merging baselines across a range of vision and language classification tasks.

Figures & tables

Figure 1: Merged-model accuracy across architectures. Each panel shows the accuracy of the merged model evaluated on the combined dataset, across a $50\times 50$ grid of mixing coefficients $(\alpha_{1},\alpha_{2})\in(0,1)^{2}$ . For each architecture, the red star indicates the theoretically predicted optimal coefficients from Theorem 3.3 , while the red triangle marks the empirically optimal coefficients.

	Per-benchmark normalized accuracy						Aggregate metrics
Method	PACS	VLCS	Terra Inc.	MNLI	Amazon	CivilComm.	Avg. norm. acc	Rel. drop
Soup	96.38	96.89	64.31	101.16	100.64	98.27	92.94	7.06
TA	97.52	97.19	64.46	101.27	100.68	97.80	93.15	6.85
TIES	97.52	96.95	63.78	100.87	100.64	98.00	92.96	7.04
DARE-TIES	97.57	97.23	64.23	100.90	100.45	97.92	93.05	6.95
DARE-Linear	96.48	96.77	64.73	101.12	100.50	97.61	92.87	7.13

Table 1: Average normalized accuracy and relative accuracy drop, following Ilharco et al. (2022) . Results are averaged over 10 seeds. Bold indicates the best result in each column.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Superclass	Fine classes (5 per superclass)
aquatic mammals	beaver, dolphin, otter, seal, whale
fish	aquarium fish, flatfish, ray, shark, trout
flowers	orchid, poppy, rose, sunflower, tulip
food containers	bottle, bowl, can, cup, plate
fruit & vegetables	apple, mushroom, orange, pear, sweet pepper
household electrical devices	clock, computer keyboard, lamp, telephone, television

Appendix

Table 2: CIFAR-100 coarse superclasses and their five fine labels.

Figure 2: Distributions of pairwise feature correlation across four pre-trained models. The red dashed line represent the mean value of correlations, and the dashed lines represent mean plus and minus one std.

	CLIP	RESNET	VIT	CONVNEXT
Intra-class mean (std)	0.2713(0.0769)	0.2312(0.0689)	0.2536(0.0652)	0.2731(0.0784)
Inter-class mean (std)	-0.0024(0.0670)	-0.0021(0.0621)	-0.0021(0.0563)	-0.0027(0.0439)

Appendix

Table 3: Mean and standard deviation of intra-class and inter-class feature cosine similarities. Values closer to zero indicate greater orthogonality between features.

Figure 3: Feature correlation analysis across different pretrained models.

Figure 4: Accuracy of the fine-tuned model across varying regularization levels for four different pre-trained models. Each line represents a fine-tuning task, indicated by color. Stars highlight the best-performing regularization value for each task. Vertical dashed lines indicate the theoretically predicted optimal regularization parameter for each setting. Alignment between the star and the dashed line of the same color reflects how well our theory predicts the empirically optimal regularization level.

Model	$\bar{K}=5$	$\bar{K}=10$	$\bar{K}=20$
ResNet-50	Emp: $0.005208$ Thm: $0.007424$	Emp: $0.003092$ Thm: $0.004158$	Emp: $0.002184$ Thm: $0.002121$
ViT-B/16	Emp: $0.001835$ Thm: $0.003304$	Emp: $0.002599$ Thm: $0.002763$	Emp: $0.001090$ Thm: $0.001551$
ConvNeXt	Emp: $0.000647$ Thm: $0.006709$	Emp: $0.002599$ Thm: $0.003114$	Emp: $0.001835$ Thm: $0.002039$
CLIP	Emp: $0.007373$ Thm: $0.001889$	Emp: $0.005208$ Thm: $0.001178$	Emp: $0.001835$ Thm: $0.001030$

Appendix

Table 4: Empirical (Emp) and theoretical (Thm) optimal regularization parameters

\lambda

for different pre-trained models and fine-tuning task sizes (

\bar{K}

Figure 5: Merged-model accuracy across architectures and different number of classes in the fine-tuning tasks. Each panel shows the accuracy of the merged model evaluated on the combined dataset, across a $50\times 50$ grid of mixing coefficients $(\alpha_{1},\alpha_{2})\in(0,1)^{2}$ . For each architecture, the red star indicates the theoretically predicted optimal coefficients from Theorem 3.3 , while the red triangle marks the empirically optimal coefficients.

Method	PACS	VLCS	Terra Inc.	MNLI	Amazon	CivilComm.	Avg
Soup	96.38 $\pm$ 0.77	96.89 $\pm$ 0.47	64.31 $\pm$ 2.53	101.16 $\pm$ 0.19	100.64 $\pm$ 0.54	98.27 $\pm$ 1.66	92.94 $\pm$ 0.53
TA	97.52 $\pm$ 0.97	97.19 $\pm$ 0.96	64.46 $\pm$ 2.65	101.27 $\pm$ 0.24	100.68 $\pm$ 0.64	97.80 $\pm$ 1.69	93.15 $\pm$ 0.58
TIES	97.52 $\pm$ 0.86	96.95 $\pm$ 0.72	63.78 $\pm$ 2.36	100.87 $\pm$ 0.21	100.64 $\pm$ 0.54	98.00 $\pm$ 1.98	92.96 $\pm$ 0.55
DARE-TIES	97.57 $\pm$ 0.85	97.23 $\pm$ 0.96	64.23 $\pm$ 2.27	100.90 $\pm$ 0.23	100.45 $\pm$ 0.65	97.92 $\pm$ 1.94	93.05 $\pm$ 0.55
DARE-Linear	96.48 $\pm$ 0.77	96.77 $\pm$ 0.51	64.73 $\pm$ 2.34	101.12 $\pm$ 0.21	100.50 $\pm$ 0.57	97.61 $\pm$ 1.88	92.87 $\pm$ 0.53
MaLoRA	96.69 $\pm$ 0.83	97.54 $\pm$ 0.62	65.75 $\pm$ 2.31	101.23 $\pm$ 0.22	100.69 $\pm$ 0.56	98.00 $\pm$ 1.99	93.32 $\pm$ 0.55

Appendix

Table 5: Average normalized accuracy (%) following Ilharco et al. [2022] : each task’s merged-model accuracy is divided by the per-domain finetuned model’s accuracy on that task, then averaged across domains within each benchmark. Mean

\pm

std over 10 seeds. Bold = best per column. Values

>

100 indicate the merged model outperforms the per-task finetuned model on that benchmark.

Method	PACS	VLCS	Terra Inc.	MNLI	Amazon	CivilComm.	Avg
Soup	3.62 $\pm$ 0.77	3.11 $\pm$ 0.47	35.69 $\pm$ 2.53	$-1.16$ $\pm$ 0.19	$-0.64$ $\pm$ 0.54	1.73 $\pm$ 1.66	7.06 $\pm$ 0.53
TA	2.48 $\pm$ 0.97	2.81 $\pm$ 0.96	35.54 $\pm$ 2.65	$-1.27$ $\pm$ 0.24	$-0.68$ $\pm$ 0.64	2.20 $\pm$ 1.69	6.85 $\pm$ 0.58
TIES	2.48 $\pm$ 0.86	3.05 $\pm$ 0.72	36.22 $\pm$ 2.36	$-0.87$ $\pm$ 0.21	$-0.64$ $\pm$ 0.54	2.00 $\pm$ 1.98	7.04 $\pm$ 0.55
DARE-TIES	2.43 $\pm$ 0.85	2.77 $\pm$ 0.96	35.77 $\pm$ 2.27	$-0.90$ $\pm$ 0.23	$-0.45$ $\pm$ 0.65	2.08 $\pm$ 1.94	6.95 $\pm$ 0.55
DARE-Linear	3.52 $\pm$ 0.77	3.23 $\pm$ 0.51	35.27 $\pm$ 2.34	$-1.12$ $\pm$ 0.21	$-0.50$ $\pm$ 0.57	2.39 $\pm$ 1.88	7.13 $\pm$ 0.53
MaLoRA	3.31 $\pm$ 0.83	2.46 $\pm$ 0.62	34.25 $\pm$ 2.31	$-1.23$ $\pm$ 0.22	$-0.69$ $\pm$ 0.56	2.00 $\pm$ 1.99	6.68 $\pm$ 0.55

Appendix

Table 6: Average relative accuracy drop (%), the complement of normalized accuracy (

1-\text{merged}/\text{oracle}

, averaged across domains within each benchmark and across benchmarks). Mean

\pm

std over 10 seeds. Bold = smallest per column (least forgetting). Negative values indicate the merged model outperforms the per-task finetuned model on average. Avg is the equally weighted mean across benchmarks; its std is

\sqrt{\sum_{b}\sigma_{b}^{2}}/6

, where

\sigma_{b}

is the per-benchmark std, assuming independent benchmarks.

Dataset	Modality	# domains	# classes	# merged	Domains (pretrain in bold )
PACS	vision	4	7	3	art painting , cartoon, photo, sketch
VLCS	vision	4	5	3	Caltech101, LabelMe, SUN09, VOC2007
Terra Incognita	vision	4	10	3	location_38, location_43, location_46, location_100
MNLI	language	5	3	4	fiction , government, slate, telephone, travel
Amazon Reviews	language	4	2	3	books , electronics, kitchen, movies
CivilComments	language	5	2	4	identity_attack, insult , obscene, sexual_explicit, threat

Appendix

Table 7: Benchmarks used in the main paper. “Domains” lists every domain in the dataset; the pretrain column marks the one folded into base_with_pretrain . “# merged” is the number of LoRA adapters that participate in the dense task-arithmetic merge (i.e. all non-pretrain adapters).

Dataset	Pretrain domain	LoRA rank $r$	Learning rate	Batch size	Weight decay
PACS	art painting	$4$	$5\times 10^{-4}$	$32$	$5\times 10^{-3}$
VLCS	VOC2007	$4$	$5\times 10^{-4}$	$32$	$5\times 10^{-2}$
Terra Incognita	location_100	$4$	$5\times 10^{-4}$	$32$	$5\times 10^{-3}$
MNLI	fiction	$8$	$5\times 10^{-4}$	$32$	$5\times 10^{-3}$
Amazon Reviews	books	$8$	$5\times 10^{-4}$	$128$	$5\times 10^{-3}$
CivilComments	insult	$32$	$5\times 10^{-4}$	$32$	$5\times 10^{-3}$

Appendix

Table 8: Per-dataset training hyperparameters. All adapters use AdamW with LoRA

\alpha\!=\!16

, dropout

0.05

, query and value projections of the final four transformer layers, and

20

training epochs. The pretrain domain is the one folded into the base_with_pretrain model, with the remaining domains fine-tuned on top.

Dataset	# classes	# domains	Same class		Different classes
			Same domain	Different domains	Same domain	Different domains
PACS	7	4	$+0.376\!\pm\!0.155$	$+0.020\!\pm\!0.157$	$+0.139\!\pm\!0.100$	$-0.060\!\pm\!0.090$
VLCS	5	4	$+0.183\!\pm\!0.169$	$+0.029\!\pm\!0.047$	$+0.008\!\pm\!0.042$	$-0.019\!\pm\!0.029$
Terra Incognita	10	4	$+0.312\!\pm\!0.143$	$+0.060\!\pm\!0.085$	$+0.059\!\pm\!0.152$	$-0.033\!\pm\!0.095$
MNLI	3	5	$+0.145\!\pm\!0.132$	$-0.018\!\pm\!0.048$	$+0.143\!\pm\!0.131$	$-0.019\!\pm\!0.048$
Amazon Reviews	2	4	$+0.073\!\pm\!0.024$	$-0.003\!\pm\!0.018$	$+0.039\!\pm\!0.023$	$-0.026\!\pm\!0.020$

Appendix

Table 9: Feature correlations within and across classes and domains. Entries report mean correlations, with

\pm

denoting spread across class–domain cell pairs in each group. A cell is a class–domain combination.

Method	PACS	VLCS	Terra Inc.	MNLI	Amazon	CivilComm.	Avg.	Ratio
Soup	287.6	752.2	904.1	3207.7	253.6	297.4	950.4	$0.98\times$
TA	287.6	752.8	903.8	3206.7	253.5	297.2	950.3	$0.98\times$
DARE-Linear	316.4	784.1	937.1	3230.9	280.4	332.4	980.2	$1.01\times$
DARE-TIES	323.2	787.3	940.8	3242.7	285.2	340.0	986.5	$1.01\times$
TIES	324.9	790.6	939.9	3254.4	290.0	346.2	991.0	$1.02\times$
MaLoRA	295.4	769.6	924.0	3277.4	261.7	307.3	972.6	$1.00\times$

Appendix

Table 10: Mean end-to-end merge-selection runtime (seconds) over ten seeds per benchmark under the matched-budget timing protocol. Avg. is the arithmetic mean across the six benchmarks; the final column divides each method’s Avg. by MaLoRA’s.

Low-Rank Adaptation

Low-rank adaptation (LoRA) fine-tunes pretrained models with compact weight updates, reducing trainable parameters and deployment costs.

See all 573 papers

Predicting Mergeability of Parameter-Efficient Fine-Tuning Updates
Jun 17, 2026Lin Tang, Wei Zhang, Jing Li +3Low-Rank Adaptation Language Model Merging
1Sichuan University, Chengdu, China · University of Electronic Science and Technology of China, Chengdu, China
When the Merge Coefficient Stops Mattering: Proximity Regularized Merging for Continual LoRA Adaptation
Sep 26, 2026Yixuan Liu, Yuhao Sun, Sen Song +1Continual Learning Incremental Learning
Sapient Intelligence · Tsinghua University
When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning
Jun 26, 2026Tanguy Dieudonné, Giulia Lanzillotta, Enis Simsar +2Continual Learning Low-Rank Adaptation
Department of Computer Science, ETH Zurich, Zurich, Switzerland.

Jun 17, 2026J/K move · Enter open · S save

Predicting Mergeability of Parameter-Efficient Fine-Tuning Updates

Lin Tang, Wei Zhang, Jing Li, Hongyu Chen+2

1Sichuan University, Chengdu, China · University of Electronic Science and Technology of China, Chengdu, China

Low-Rank Adaptation Language Model Merging Model Merging Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Low-rank adaptation (LoRA) makes it cheap to train many domain- and task-specific language model adapters, but whether two adapters can be merged is usually discovered only after both have been fully trained and evaluated. This late feedback is costly: adapters that are strong in isolation can interfere destructively once their updates are combined. We ask whether this outcome can be anticipated. We formalize adapter mergeability as the degree to which an adapter preserves its single-task utility after merging, and show that it can be forecast from signals measured in the first few percent of training -- chiefly how the low-rank updates and their gradients align across tasks and how much they disturb shared representations. We package these signals into MergeProbe, a lightweight predictor that estimates pairwise and set-level retention and turns the estimate into a concrete decision: merge directly, reweight, prune, or route. On MERGE-PEFT, a five-domain benchmark spanning math, code, science, instruction following, and safety, MergeProbe attains the best average and worst-case retention among strong interference-aware merge baselines while adding far less deployment overhead than full task routing. This turns LoRA merging from a post-hoc engineering step into an anticipatory measurement problem.

Abstract

Low-Rank Adaptation

Explore similar work

Predicting Mergeability of Parameter-Efficient Fine-Tuning Updates

When the Merge Coefficient Stops Mattering: Proximity Regularized Merging for Continual LoRA Adaptation

When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning