PGSB: Pretrained-Guided Shared Basis for LoRA Model Merging

Muqing Liu; Chongjie Si; Zhuoya Liu; Yuheng Jia

cs.LGOct 10, 2026

PGSB: Pretrained-Guided Shared Basis for LoRA Model Merging

Authors: Muqing Liu, Chongjie Si, Zhuoya Liu, Yuheng Jia

Organizations: Chien-Shiung Wu College, Southeast University · MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University · School of Computer Science and Engineering, Southeast University · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China

Abstract

Large pretrained models have achieved remarkable success across diverse domains, yet adapting them to multiple downstream tasks remains challenging: full fine-tuning is costly, while maintaining separate task-specific models is storage-inefficient and fails to produce a unified multi-task model. Low-Rank Adaptation (LoRA) offers an efficient alternative for Model Merging by representing each task adaptation as a low-rank update over frozen pretrained weights. However, existing LoRA merging methods often treat task-specific updates as directly comparable objects and merge them in their original low-rank parameter spaces. This overlooks two key factors: which update directions provide reliable shared support across tasks, and how these directions should be re-parameterized with respect to the pretrained weights before merging. To address this limitation, we propose Pretrained-Guided Shared Basis (PGSB), a two-stage framework for LoRA model merging. PGSB first identifies shared directions among task-specific LoRA updates, and then re-parameterizes these directions using their interaction with the pretrained weights, producing a more suitable basis for merging. Extensive experiments demonstrate that PGSB consistently outperforms state-of-the-art baselines across multiple benchmarks. Codes are available at https://github.com/Kitschnada/Pretrained-Basis .

Figures & tables

Figure 1: Overview of PGSB.Given task-specific LoRA updates, PGSB first performs Task-shared Subspace Extraction by decomposing the concatenated updates and their transposed counterparts, producing shared support matrices. PGSB then performs Pretrained-Coupled Coordinate Refinement, where the extracted supports are coupled with the pretrained weight geometry to obtain pretrained-compatible multi-task bases. Finally, task updates are projected into the refined bases, merged by a downstream merge operator

\mathcal{M}

, and reconstructed back to the original parameter space to obtain the final merged model.

Method	Construction object	Complexity
Baseline	Directly merge $\Delta\mathbf{W}_{t}=\mathbf{B}_{t}\mathbf{A}_{t}$	$\mathcal{O}(n^{2}Tr)$
KnOTS	SVD on $[\Delta\mathbf{W}_{1},\dots,\Delta\mathbf{W}_{T}]$	$\mathcal{O}(n^{3}T^{2})$
Core Space	SVD on stacked low-rank factors	$\mathcal{O}(n^{2}Tr+T^{3}r^{3})$
PGSB	Two full-space SVDs	$\mathcal{O}(n^{3}T^{2}+n^{2}Tr+T^{3}r^{3})$
PGSB (efficient)	Two low-rank SVDs	$\mathcal{O}(n^{2}Tr+T^{3}r^{3})$

Table 1: Asymptotic preprocessing complexity for constructing the merging space of one layer before applying a downstream merge operator. We assume square weight matrices of size

n\times n

T

tasks, and LoRA rank

r

with

Tr\ll n

Method	Space	Cars	DTD	EuroSAT	GTSRB	MNIST	RESISC	SUN397	SVHN	Avg
Abs. Accuracy		74.00	58.30	99.00	92.70	99.30	88.40	64.50	96.20	-
TA		83.62	76.00	50.77	44.12	55.39	72.36	97.05	41.31	65.08
TIES	Baseline	82.24	73.26	51.44	37.92	56.28	69.88	96.95	43.89	63.98
	KnOTS	82.58	73.45	48.26	41.83	57.40	70.66	96.58	45.60	64.54
	Core	84.29	79.74	54.10	49.67	64.50	72.72	97.44	44.29	68.34
	PGSB	83.37	76.37	58.40	44.29	82.18	72.08	95.41	66.17	72.28

Table 2: Normalized accuracies of merged models on the visual tasks with ViT-B/32.

Method	Space	Cars	DTD	EuroSAT	GTSRB	MNIST	RESISC	SUN397	SVHN	Avg
Abs. Accuracy		99.77	70.05	98.59	97.20	99.53	95.70	79.60	97.72	-
TA		80.07	79.73	62.10	59.96	82.17	79.53	86.71	64.78	74.38
TIES	Baseline	79.54	78.66	63.11	60.42	84.12	78.94	87.03	69.60	75.18
	KnOTS	80.35	79.57	62.06	62.95	85.43	79.81	87.57	70.99	76.09
	Core	81.91	79.88	63.07	68.95	87.48	81.77	88.50	73.76	78.17
	PGSB	83.15	80.18	65.48	69.37	87.85	82.29	87.99	77.52	79.23

Table 3: Normalized accuracies of merged models on the visual tasks with ViT-L/14.

Method	Space	SNLI	MNLI	SICK	QNLI	RTE	SCITAIL	Avg
Abs. Accuracy	–	92.50	90.31	91.58	94.49	89.86	96.52	–
TA	–	93.58	95.28	88.07	68.76	100.00	96.73	90.40
TIES	Baseline	95.17	96.21	84.20	74.21	100.00	96.73	91.09
	KnOTS	92.67	94.82	91.03	78.93	101.61	97.22	92.72
	Core	91.94	91.12	96.68	79.27	100.81	97.37	92.86
	PGSB	97.19	97.02	87.35	87.54	95.28	97.97	93.73

Table 4: Normalized accuracies of merged models on the NLI tasks for Llama 3 8B.

Space	ARC-c	ARC-e	BoolQ	HellaS.	OBQA	PIQA	SIQA	WinoG.	Avg
Abs. Accuracy	80.38	92.89	89.51	96.28	86.2	90.32	80.6	88.08	–
Baseline	99.68	99.23	97.56	92.8	96.29	96.75	98.1	97.13	97.19
KnOTS	102.34	100.3	98.14	90.11	96.98	97.11	97.02	95.96	97.25
Core	100.42	99.18	95.66	92.47	96.52	95.3	96.26	95.16	96.37
PGSB	100.21	99.55	97.66	93.72	97.68	97.11	97.46	97.22	97.58

Table 5: Normalized accuracies of merged models on the Commonsense Reasoning tasks for Llama 3 8B with TIES merging.

Variant	Cars	DTD	EuroSAT	GTSRB	MNIST	RESISC	SUN397	SVHN	Avg
PGSB	83.37	76.37	58.40	44.29	82.18	72.08	95.41	66.17	72.28
Baseline	82.24	73.26	51.44	37.92	56.28	69.88	96.95	43.89	63.98
TSE-only	84.88	77.37	54.55	46.81	65.46	72.31	96.69	49.64	68.46
No-weighted	81.76	74.63	51.37	36.91	53.58	69.54	98.04	37.06	62.86

Table 6: Ablations on the visual tasks with TIES merging. Each entry reports normalized accuracy on the corresponding dataset. Baseline directly applies TIES to the LoRA updates without any other operations.

Variant	Cars	DTD	EuroSAT	GTSRB	MNIST	RESISC	SUN397	SVHN	Avg
PGSB	83.37	76.37	58.40	44.29	82.18	72.08	95.41	66.17	72.28
KnOTS	82.58	73.45	48.26	41.83	57.40	70.66	96.58	45.60	64.54
KnOTS + PCR	83.00	72.26	56.12	37.34	70.88	70.41	96.66	61.28	68.49
Core Space	84.29	79.74	54.10	49.67	64.50	72.72	97.44	44.29	68.34
Core Space + PCR	83.79	77.00	50.24	47.75	77.04	68.07	96.19	57.93	69.75

Table 7: Extensibility experiments on the visual tasks with TIES merging. Each entry reports normalized accuracy on the corresponding dataset.

Space	CIFAR10	CIFAR100	Pets	Flowers102	Food101	Caltech101	Cats vs Dogs	Avg
Abs. Accuracy	88.86	64.35	86.24	65.98	84.43	79.90	99.62	81.34
Baseline	97.60	78.57	95.63	91.77	97.95	96.72	99.99	94.03
KnOTS	81.87	30.27	85.74	61.90	89.92	85.47	99.94	76.44
Core Space	99.07	90.58	97.84	97.01	98.65	99.50	99.96	97.52
PGSB	99.32	94.65	98.92	97.71	99.04	98.97	100.00	98.38

Table 8: Normalized accuracies of merged models on additional visual tasks after TIES merging. Abs. Accuracy reports the absolute accuracy of the pretrained model, which is used as the normalization reference for each visual task.

Setting	Space	Cars	DTD	EuroSAT	GTSRB	MNIST	RESISC45	SUN397	SVHN	Avg
$r=4$	Abs. Accuracy	68.57	68.83	98.19	96.65	99.49	91.60	71.34	96.05	–
	Baseline	89.59	72.26	60.62	63.59	82.75	80.64	93.68	79.71	77.85
	KnOTS	92.74	75.50	61.60	68.37	82.13	78.06	93.56	76.39	78.54
	Core	92.38	76.66	66.09	66.59	79.32	77.54	92.98	72.31	77.98
	PGSB	92.81	75.81	68.58	65.45	86.83	81.13	93.33	81.66	80.70
$r=64$	Abs. Accuracy	74.63	75.32	98.56	97.87	99.50	93.38	73.48	96.64	–

Table 9: Normalized accuracies of merged models under different LoRA rank configurations on the eight visual tasks using ViT-B/32 with TIES merging.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Merge Space	Merge Time (s)	Peak GPU Memory (MiB)
ViT-B/32	Baseline	0.73	103.5
	KnOTS	5.27	90.6
	Core Space	1.55	20.5
	PGSB (uncached)	5.76	60.1
	PGSB (cached)	0.84	33.6
ViT-L/14	Baseline	2.50	184.0

Appendix

Table 10: Merging time and peak GPU memory consumption of different merging spaces. PGSB (uncached) includes shared basis construction and PCR, while PGSB (cached) reuses precomputed intermediate results.

Merge Space	Search Time (s)
Baseline	806
KnOTS	794
Core Space	1187
PGSB	727

Appendix

Table 11: Total hyperparameter search time on ViT-B/32 using TIES under the same linear-search protocol.

Space	Cars	DTD	EuroSAT	GTSRB	MNIST	RESISC45	SUN397	SVHN	Avg
Abs. Accuracy	94.13	81.81	98.78	97.89	99.53	94.21	76.93	97.27	–
Baseline	98.00	89.73	58.42	90.88	97.83	84.99	94.82	98.38	89.13
KnOTS	98.61	89.79	56.81	92.85	98.34	82.70	94.99	96.80	88.86
Core	99.05	91.35	65.73	95.85	98.34	84.85	95.46	96.61	90.91
PGSB	98.57	95.19	63.33	92.78	98.51	85.93	97.84	97.99	91.27

Appendix

Table 12: Normalized accuracies of merged models on the eight visual tasks using SigLIP-B/16 with TIES merging.

Space	OBQA	PIQA	WinoG.	QNLI	RTE	SciTail	Avg
Abs. Accuracy	86.20	90.32	88.08	94.49	89.86	96.52	–
Baseline	93.27	93.55	99.55	96.97	98.38	98.10	96.64
KnOTS	94.43	94.58	100.63	90.81	97.98	98.29	96.12
Core	100.93	95.54	98.92	88.51	99.19	97.27	96.73
PGSB	96.98	96.08	98.65	95.08	97.98	97.56	97.06

Appendix

Table 13: Normalized accuracies of merged models on heterogeneous language tasks using Llama 3 8B with TIES merging.

Factor	Setting	$\boldsymbol{\Delta_{\mathrm{PCR}}}$
Task similarity	High / Low	+4.40 / +0.84
Number of adapters	$T=2/4/8$	+1.30 / +5.68 / +3.82
LoRA rank	$r=4/16/64/256$	+2.12 / +3.82 / +3.11 / +1.63
Merge operator	TIES / DARE-TIES / TSV / CART	+3.82 / +5.15 / +0.07 / +0.99

Appendix

Table 14: Effect of different factors on the performance improvement introduced by PCR.

\Delta_{\mathrm{PCR}}

denotes the difference in average normalized accuracy between PGSB and TSE-only. Positive values indicate improvements from PCR.

Language Model Merging

Language model merging combines capabilities from multiple models or adapters by composing their weights, updates, or outputs.

See all 109 papers

Crowded in B-Space: Calibrating Shared Directions for LoRA Merging
Apr 18, 2026Yixuan Tang, Yi YangAdapter Tuning Low-Rank Adaptation
The Hong Kong University of Science and Technology
Compress then Merge: From Multiple LoRAs into One Low-Rank Adapter
Jun 2, 2026Zhengbao He, Ruiqi Ding, Zhehao Huang +3Adapter Tuning Low-Rank Adaptation
Institute of Image Processing and Pattern Recognition, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, China · Shanghai Key Laboratory of Flexible Medical Robotics, Tongren Hospital, Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, China
Predicting Mergeability of Parameter-Efficient Fine-Tuning Updates
Jun 17, 2026Lin Tang, Wei Zhang, Jing Li +3Low-Rank Adaptation Language Model Merging
1Sichuan University, Chengdu, China · University of Electronic Science and Technology of China, Chengdu, China

Apr 18, 2026J/K move · Enter open · S save

Crowded in B-Space: Calibrating Shared Directions for LoRA Merging

Yixuan Tang, Yi Yang

The Hong Kong University of Science and Technology

Adapter Tuning Low-Rank Adaptation Task Vector Merging Language Model Merging Model Merging Task Interference

Open paper AskPDF arXiv

Merging separately trained LoRA adapters is a practical alternative to joint multi-task training, but it often hurts performance. Existing methods usually treat the LoRA update

ΔW = BA

as a single object and do not distinguish the two LoRA matrices. We show that the main source of LoRA merge interference comes from the output-side matrix

B

. Across tasks,

B

repeatedly uses a small set of shared directions, while

A

remains much more task-specific. As a result, the merged adapter overemphasizes these shared directions, and task-specific information is lost. We propose Pico (Pre-merge interference calibration in output-space), a data-free method that calibrates

B

before merge by downscaling over-shared directions and then rescaling the merged update. Pico plugs directly into existing merging methods such as Task Arithmetic, TIES, and TSV-M. Across eight different benchmarks from math, coding, finance, and medical domains, Pico improves average accuracy by 3.4-8.3 points over the corresponding base method and achieves the best overall average performance. Pico also enables merged adapters to outperform the LoRA trained with all task data. These results show that LoRA merging works better when the two LoRA matrices are treated separately.

Abstract

Language Model Merging

Explore similar work

Crowded in B-Space: Calibrating Shared Directions for LoRA Merging

Compress then Merge: From Multiple LoRAs into One Low-Rank Adapter

Predicting Mergeability of Parameter-Efficient Fine-Tuning Updates