Organizations: Statistics and Data Science, University of Pennsylvania · IDEAS, University of Pennsylvania · INS & SMS, Shanghai Jiao Tong University · ESE, University of Pennsylvania · AMCS, University of Pennsylvania · ECE, Johns Hopkins University · Radiology, University of Pennsylvania · CIS, University of Pennsylvania
Low-rank adaptation (LoRA) is the de facto method for parameter-efficient fine-tuning of large neural networks, achieving strong task adaptation with minimal trainable parameters and low optimization cost. Beyond this efficiency, practitioners have observed two striking properties: robustness to catastrophic forgetting and the ability to merge independently trained adapters into a single model that performs competitively across multiple tasks. These properties are central to continual and multi-task learning, yet remain poorly understood. In this work, we provide a theoretical explanation through a unified margin-based perspective. We analyze LoRA in multiclass linear classification under a near-orthogonal task regime with ℓ2-regularization, and characterize the optimal LoRA adapter across regularization regimes. Our analysis shows that, in an intermediate regime of regularization parameters, the optimal adapter aligns with the max-margin solution on the fine-tuning data. Building on this characterization, we derive two key consequences. First, we obtain closed-form expressions for the margins on pre-training and fine-tuning data, revealing a precise margin trade-off: the regularization parameter controls the balance between retention and adaptation. Second, we analyze adapter merging, proving that merged models achieve positive margin on each task and deriving optimal mixing coefficients that balance margins across tasks and maximize the margin over their union. These results lead to a simple, training-free merging rule, which we term margin-based LoRA merging (MaLoRA). Experiments on modern architectures and real datasets validate our theoretical predictions, showing that MaLoRA matches or outperforms several adapter-merging baselines across a range of vision and language classification tasks.
Figures & tables
Figure 1: Merged-model accuracy across architectures. Each panel shows the accuracy of the merged model evaluated on the combined dataset, across a 50×50 grid of mixing coefficients (α1,α2)∈(0,1)2 . For each architecture, the red star indicates the theoretically predicted optimal coefficients from Theorem 3.3 , while the red triangle marks the empirically optimal coefficients.
Per-benchmark normalized accuracy
Aggregate metrics
Method
PACS
VLCS
Terra Inc.
MNLI
Amazon
CivilComm.
Avg. norm. acc
Rel. drop
Soup
96.38
96.89
64.31
101.16
100.64
98.27
92.94
7.06
TA
97.52
97.19
64.46
101.27
100.68
97.80
93.15
6.85
TIES
97.52
96.95
63.78
100.87
100.64
98.00
92.96
7.04
DARE-TIES
97.57
97.23
64.23
100.90
100.45
97.92
93.05
6.95
DARE-Linear
96.48
96.77
64.73
101.12
100.50
97.61
92.87
7.13
Table 1: Average normalized accuracy and relative accuracy drop, following Ilharco et al. (2022) . Results are averaged over 10 seeds. Bold indicates the best result in each column.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Superclass
Fine classes (5 per superclass)
aquatic mammals
beaver, dolphin, otter, seal, whale
fish
aquarium fish, flatfish, ray, shark, trout
flowers
orchid, poppy, rose, sunflower, tulip
food containers
bottle, bowl, can, cup, plate
fruit & vegetables
apple, mushroom, orange, pear, sweet pepper
household electrical devices
clock, computer keyboard, lamp, telephone, television
Appendix
Table 2: CIFAR-100 coarse superclasses and their five fine labels.
Figure 2: Distributions of pairwise feature correlation across four pre-trained models. The red dashed line represent the mean value of correlations, and the dashed lines represent mean plus and minus one std.
CLIP
RESNET
VIT
CONVNEXT
Intra-class mean (std)
0.2713(0.0769)
0.2312(0.0689)
0.2536(0.0652)
0.2731(0.0784)
Inter-class mean (std)
-0.0024(0.0670)
-0.0021(0.0621)
-0.0021(0.0563)
-0.0027(0.0439)
Appendix
Table 3: Mean and standard deviation of intra-class and inter-class feature cosine similarities. Values closer to zero indicate greater orthogonality between features.
Figure 3: Feature correlation analysis across different pretrained models.
Figure 4: Accuracy of the fine-tuned model across varying regularization levels for four different pre-trained models. Each line represents a fine-tuning task, indicated by color. Stars highlight the best-performing regularization value for each task. Vertical dashed lines indicate the theoretically predicted optimal regularization parameter for each setting. Alignment between the star and the dashed line of the same color reflects how well our theory predicts the empirically optimal regularization level.
Model
Kˉ=5
Kˉ=10
Kˉ=20
ResNet-50
Emp: 0.005208 Thm: 0.007424
Emp: 0.003092 Thm: 0.004158
Emp: 0.002184 Thm: 0.002121
ViT-B/16
Emp: 0.001835 Thm: 0.003304
Emp: 0.002599 Thm: 0.002763
Emp: 0.001090 Thm: 0.001551
ConvNeXt
Emp: 0.000647 Thm: 0.006709
Emp: 0.002599 Thm: 0.003114
Emp: 0.001835 Thm: 0.002039
CLIP
Emp: 0.007373 Thm: 0.001889
Emp: 0.005208 Thm: 0.001178
Emp: 0.001835 Thm: 0.001030
Appendix
Table 4: Empirical (Emp) and theoretical (Thm) optimal regularization parameters λ for different pre-trained models and fine-tuning task sizes ( Kˉ ).
Figure 5: Merged-model accuracy across architectures and different number of classes in the fine-tuning tasks. Each panel shows the accuracy of the merged model evaluated on the combined dataset, across a 50×50 grid of mixing coefficients (α1,α2)∈(0,1)2 . For each architecture, the red star indicates the theoretically predicted optimal coefficients from Theorem 3.3 , while the red triangle marks the empirically optimal coefficients.
Method
PACS
VLCS
Terra Inc.
MNLI
Amazon
CivilComm.
Avg
Soup
96.38 ± 0.77
96.89 ± 0.47
64.31 ± 2.53
101.16 ± 0.19
100.64 ± 0.54
98.27 ± 1.66
92.94 ± 0.53
TA
97.52 ± 0.97
97.19 ± 0.96
64.46 ± 2.65
101.27 ± 0.24
100.68 ± 0.64
97.80 ± 1.69
93.15 ± 0.58
TIES
97.52 ± 0.86
96.95 ± 0.72
63.78 ± 2.36
100.87 ± 0.21
100.64 ± 0.54
98.00 ± 1.98
92.96 ± 0.55
DARE-TIES
97.57 ± 0.85
97.23 ± 0.96
64.23 ± 2.27
100.90 ± 0.23
100.45 ± 0.65
97.92 ± 1.94
93.05 ± 0.55
DARE-Linear
96.48 ± 0.77
96.77 ± 0.51
64.73 ± 2.34
101.12 ± 0.21
100.50 ± 0.57
97.61 ± 1.88
92.87 ± 0.53
MaLoRA
96.69 ± 0.83
97.54 ± 0.62
65.75 ± 2.31
101.23 ± 0.22
100.69 ± 0.56
98.00 ± 1.99
93.32 ± 0.55
Appendix
Table 5: Average normalized accuracy (%) following Ilharco et al. [2022] : each task’s merged-model accuracy is divided by the per-domain finetuned model’s accuracy on that task, then averaged across domains within each benchmark. Mean ± std over 10 seeds. Bold = best per column. Values > 100 indicate the merged model outperforms the per-task finetuned model on that benchmark.
Method
PACS
VLCS
Terra Inc.
MNLI
Amazon
CivilComm.
Avg
Soup
3.62 ± 0.77
3.11 ± 0.47
35.69 ± 2.53
−1.16± 0.19
−0.64± 0.54
1.73 ± 1.66
7.06 ± 0.53
TA
2.48 ± 0.97
2.81 ± 0.96
35.54 ± 2.65
−1.27± 0.24
−0.68± 0.64
2.20 ± 1.69
6.85 ± 0.58
TIES
2.48 ± 0.86
3.05 ± 0.72
36.22 ± 2.36
−0.87± 0.21
−0.64± 0.54
2.00 ± 1.98
7.04 ± 0.55
DARE-TIES
2.43 ± 0.85
2.77 ± 0.96
35.77 ± 2.27
−0.90± 0.23
−0.45± 0.65
2.08 ± 1.94
6.95 ± 0.55
DARE-Linear
3.52 ± 0.77
3.23 ± 0.51
35.27 ± 2.34
−1.12± 0.21
−0.50± 0.57
2.39 ± 1.88
7.13 ± 0.53
MaLoRA
3.31 ± 0.83
2.46 ± 0.62
34.25 ± 2.31
−1.23± 0.22
−0.69± 0.56
2.00 ± 1.99
6.68 ± 0.55
Appendix
Table 6: Average relative accuracy drop (%), the complement of normalized accuracy ( 1−merged/oracle , averaged across domains within each benchmark and across benchmarks). Mean ± std over 10 seeds. Bold = smallest per column (least forgetting). Negative values indicate the merged model outperforms the per-task finetuned model on average. Avg is the equally weighted mean across benchmarks; its std is ∑bσb2/6 , where σb is the per-benchmark std, assuming independent benchmarks.
Table 7: Benchmarks used in the main paper. “Domains” lists every domain in the dataset; the pretrain column marks the one folded into base_with_pretrain . “# merged” is the number of LoRA adapters that participate in the dense task-arithmetic merge (i.e. all non-pretrain adapters).
Dataset
Pretrain domain
LoRA rank r
Learning rate
Batch size
Weight decay
PACS
art painting
4
5×10−4
32
5×10−3
VLCS
VOC2007
4
5×10−4
32
5×10−2
Terra Incognita
location_100
4
5×10−4
32
5×10−3
MNLI
fiction
8
5×10−4
32
5×10−3
Amazon Reviews
books
8
5×10−4
128
5×10−3
CivilComments
insult
32
5×10−4
32
5×10−3
Appendix
Table 8: Per-dataset training hyperparameters. All adapters use AdamW with LoRA α=16 , dropout 0.05 , query and value projections of the final four transformer layers, and 20 training epochs. The pretrain domain is the one folded into the base_with_pretrain model, with the remaining domains fine-tuned on top.
Dataset
# classes
# domains
Same class
Different classes
Same domain
Different domains
Same domain
Different domains
PACS
7
4
+0.376±0.155
+0.020±0.157
+0.139±0.100
−0.060±0.090
VLCS
5
4
+0.183±0.169
+0.029±0.047
+0.008±0.042
−0.019±0.029
Terra Incognita
10
4
+0.312±0.143
+0.060±0.085
+0.059±0.152
−0.033±0.095
MNLI
3
5
+0.145±0.132
−0.018±0.048
+0.143±0.131
−0.019±0.048
Amazon Reviews
2
4
+0.073±0.024
−0.003±0.018
+0.039±0.023
−0.026±0.020
Appendix
Table 9: Feature correlations within and across classes and domains. Entries report mean correlations, with ± denoting spread across class–domain cell pairs in each group. A cell is a class–domain combination.
Method
PACS
VLCS
Terra Inc.
MNLI
Amazon
CivilComm.
Avg.
Ratio
Soup
287.6
752.2
904.1
3207.7
253.6
297.4
950.4
0.98×
TA
287.6
752.8
903.8
3206.7
253.5
297.2
950.3
0.98×
DARE-Linear
316.4
784.1
937.1
3230.9
280.4
332.4
980.2
1.01×
DARE-TIES
323.2
787.3
940.8
3242.7
285.2
340.0
986.5
1.01×
TIES
324.9
790.6
939.9
3254.4
290.0
346.2
991.0
1.02×
MaLoRA
295.4
769.6
924.0
3277.4
261.7
307.3
972.6
1.00×
Appendix
Table 10: Mean end-to-end merge-selection runtime (seconds) over ten seeds per benchmark under the matched-budget timing protocol. Avg. is the arithmetic mean across the six benchmarks; the final column divides each method’s Avg. by MaLoRA’s.
Low-rank adaptation (LoRA) makes it cheap to train many domain- and task-specific language model adapters, but whether two adapters can be merged is usually discovered only after both have been fully trained and evaluated. This late feedback is costly: adapters that are strong in isolation can interfere destructively once their updates are combined. We ask whether this outcome can be anticipated. We formalize adapter mergeability as the degree to which an adapter preserves its single-task utility after merging, and show that it can be forecast from signals measured in the first few percent of training -- chiefly how the low-rank updates and their gradients align across tasks and how much they disturb shared representations. We package these signals into MergeProbe, a lightweight predictor that estimates pairwise and set-level retention and turns the estimate into a concrete decision: merge directly, reweight, prune, or route. On MERGE-PEFT, a five-domain benchmark spanning math, code, science, instruction following, and safety, MergeProbe attains the best average and worst-case retention among strong interference-aware merge baselines while adding far less deployment overhead than full task routing. This turns LoRA merging from a post-hoc engineering step into an anticipatory measurement problem.
Lin Tang, Wei Zhang, Jing Li +3
1Sichuan University, Chengdu, China · University of Electronic Science and Technology of China, Chengdu, China
Rehearsal-free continual learning with parameter-efficient adapters can be cast as a sequence of task-vector write-in operations: for each new task, a low-rank adapter is learned and merged into a running model. We propose Proximity Regularized Merging (PRM), a minimal modification to sequential LoRA merging that adds a proximal penalty during task-vector training without changing the subsequent write-in rule. PRM acts as a robust task-vector regularizer: in the reported Base->+Prox diagnostics, it improves AAA across multiple write-in rules, backbones, and class-incremental settings, while its fixed-coefficient variant remains competitive with strong coefficient-based baselines. Mechanistically, matched-prefix norm controls and proximal-strength sweeps show that proximal training shrinks the task-vector radius, lowers Fisher-weighted interference, broadens the coefficient plateau, and exposes a stability-plasticity trade-off. Together, these results suggest that the effectiveness of sequential LoRA merging depends not only on how much of a task vector is written in, but also on whether the task vector itself has been trained to be mergeable.
Low-Rank Adaptation (LoRA) has become the standard tool for parameter-efficient fine-tuning of large pretrained models. When applied sequentially across tasks in Continual Learning (CL), the standard assumption is that each new task requires a dedicated low-rank adapter. In this work, we challenge this assumption empirically and structurally. We show that task-specific LoRA adapters in CL exhibit significant low-rank redundancy: the subspaces spanned by adapters trained on different tasks substantially overlap, and in many cases earlier adapters can faithfully represent later tasks. Building on this observation, we propose LiteLoRA, a plug-and-play gating mechanism that learns at train time whether to recruit a new adapter or reuse existing low-rank representations. Our method reduces the number of active adapters by 20-70% while matching or exceeding state-of-the-art performance on standard CL benchmarks, revealing that structural redundancy is pervasive and that selective learning is sufficient to achieve stability without sacrificing plasticity.