Preference-Aligned LoRA Merging: Preserving Subspace Coverage and Addressing Directional Anisotropy

Wooseong Jeong; Wonyoung Lee; Kuk-Jin Yoon

cs.CVMar 27, 2026

Preference-Aligned LoRA Merging: Preserving Subspace Coverage and Addressing Directional Anisotropy

Authors: Wooseong Jeong, Wonyoung Lee, Kuk-Jin Yoon

Organizations: KAIST

Abstract

Merging multiple Low-Rank Adaptation (LoRA) modules is promising for constructing general-purpose systems, yet challenging because LoRA update directions span different subspaces and contribute unevenly. When merged naively, such mismatches can weaken the directions most critical to certain task losses while overemphasizing relatively less important ones, ultimately reducing the model's ability to represent all tasks faithfully. We revisit this problem through two perspectives: subspace coverage, which captures how broadly LoRA directions cover diverse representational directions, and anisotropy, which reflects the imbalance of influence across those directions. We propose TARA-Merging (Task-Rank Anisotropy Alignment), which aligns merging weights using a preference-weighted cross-entropy pseudo-loss while preserving task-relevant LoRA subspaces. This ensures broad subspace coverage and mitigates anisotropy via direction-wise reweighting. Across eight vision and six NLI benchmarks, TARA-Merging consistently outperforms vanilla and LoRA-aware baselines, demonstrating strong robustness and generalization, and highlighting the importance of addressing both subspace coverage and anisotropy in LoRA merging.

Figures & tables

Figure 1: Effective rank across layers for an attention projection layer (e.g., query). The gap between $\Delta\bm{X}$ stack and Rank-1 stack reflects merge-induced collapse.

Figure 2: Directional-sensitivity misalignment $\bm{\xi(\rho_{1},\rho_{2})}$ . Larger $\xi$ indicates stronger change in loss-sensitive directions when switching preferences.

Method	Dataset
Method	Cars	DTD	EuroSAT	GTSRB	MNIST	RESISC45	SUN397	SVHN	Avg
	Per‑task absolute accuracies (%)
Finetuned	74.0	58.3	99.0	92.7	99.3	88.4	64.5	96.2	84.1
	Per‑task accuracies of merged models, normalized to finetuned (%)
Vanilla Merging
RegMean [ 40 ]	80.2	71.3	37.9	47.3	43.1	70.5	93.9	43.0	60.9

Table 1: Per-task accuracy on eight image-classification benchmarks. We merge eight ViT-B/32 checkpoints, each fine-tuned with LoRA. The upper panel shows the per-task absolute accuracy of the fine-tuned baselines. The lower panel shows the accuracy of the merged models, normalized by their corresponding fine-tuned baseline (%).

Method	Dataset
Method	MNLI	QNLI	SNLI	RTE	SICK	SCITAIL	Avg
	Per-task absolute accuracies (%)
Finetuned	90.8	95.3	92.1	84.8	91.3	87.3	90.2
	Per-task accuracies of merged models, normalized to finetuned (%)
Vanilla Merging
TA [ 32 ]	67.3	87.3	41.8	95.7	77.9	76.9	74.6

Table 2: Per-task accuracy on six NLI benchmarks. We merge six LLaMA-3 8B checkpoints, each fine-tuned with LoRA. The upper panel shows the absolute per-task accuracy of the fine-tuned baselines. The lower panel shows the merged models’ accuracy, normalized by each task’s corresponding fine-tuned baseline (%).

Method	Hits@1	Hits@3	Hits@5
TA [ 32 ]	43.5	65.2	74.0
TIES [ 99 ]	43.6	65.3	73.9
DARE-TIES [ 107 ]	44.0	66.4	75.1
AdaMerging [ 105 ]	48.1	73.2	83.0
KnOTS-TIES [ 79 ]	46.8	68.1	76.3
KnOTS-DARE-TIES [ 79 ]	45.2	66.9	75.3

Table 3: Joint-Task Evaluation Results. All models share a ViT-B/32 backbone, are fine-tuned with LoRA on separate datasets, and are then merged for joint-task evaluation.

	Seen Tasks							Unseen Tasks			All Tasks
Method	Cars	DTD	GTSRB	RESISC45	SUN397	SVHN	Avg Acc	EuroSAT	MNIST	Avg Acc	Avg Acc
	Per-task accuracies of merged models, normalized to finetuned (%)
TA [ 32 ]	82.0	74.7	42.2	71.1	96.8	42.2	68.2	37.5	48.2	42.9	61.8
TIES [ 99 ]	81.5	73.2	40.3	64.4	95.4	45.7	66.7	37.2	61.8	49.5	62.4
KnOTS-TIES [ 79 ]	82.7	74.4	48.5	72.5	95.3	50.2	70.6	33.2	50.5	41.8	63.4
KnOTS-DARE-TIES [ 79 ]	81.7	72.5	49.4	71.9	94.6	52.5	70.4	31.6	51.2	41.4	63.2

Table 4: Generalization results on two unseen tasks when merging ViT-B/32 models on six tasks.

Panel of Figure 5 — Figure 5: Effective Rank across layers for each module. Higher curves indicate broader subspace coverage. The gap between $\Delta\bm{W}$ stack and Rank-1 stack reflects merge-induced collapse.

Panel of Figure 7 — Figure 7: Condition Number Anisotropy $\bm{\kappa}$ (SVD Basis). Layer-wise condition number $\kappa(\bm{\rho})$ per module under the orthogonal LoRA basis (SVD). Larger $\kappa$ indicates stronger within-preference directional concentration of loss sensitivity. (Here, $\bm{\rho}_{1}$ uniformly weights all tasks, whereas $\bm{\rho}_{2}$ assigns all weight to a single task (one-hot).)

Panel of Figure 8 — Figure 8: Layer-wise Directional-Sensitivity Spectra (RAW vs. SVD). For each module and layer, we form the task-direction sensitivity matrix with entries $J_{i,k}=\langle\nabla f_{i}(W),\,S_{k}\rangle$ and plot the scree curve of its singular values as energy fractions $\sigma_{k}^{2}/\!\sum_{j}\sigma_{j}^{2}$ versus component index $k$ . “RAW” stacks rank-1 LoRA factors directly, while “SVD” uses a shared orthonormal basis. The two spectra are overlaid in each panel. A larger leading mass indicates stronger concentration of sensitivity into a few modes, whereas flatter spectra indicate a more distributed use of LoRA directions.

Figure 9: Directional-sensitivity misalignment $\bm{\xi(\rho_{1},\rho_{2})}$ in the SVD basis.

Method	Dataset
Method	ScienceQA	VizWiz	IconQA	ImageNet	ChartQA	DocVQA	Avg
	Per-task absolute accuracies (%)
Finetuned	86.6	69.7	65.2	96.0	39.3	41.2	66.4
	Per-task accuracies of merged models, normalized to finetuned (%)
TA [ 32 ]	81.6	71.3	57.2	43.6	66.6	79.5	66.6
TIES [ 99 ]	90.9	69.8	66.5	54.8	79.1	81.9	73.8

Table 5: To further address concerns regarding latency coming from gradient-based optimization, we experimented our method with LLaVA [ 50 , 51 ] on merging six VQA experts.

	LLaMA-3-8B		LLaVA-1.5-7B
Method	Time (min)	VRAM (MiB)	Time (min)	VRAM (MiB)
TA [ 32 ]	3.6	16,150	10.6	14,814
TIES [ 99 ]	8.1 (4.5+3.6)	16,150	18.3 (7.7+10.6)	14,814
KnOTS-TIES [ 79 ]	10.8 (7.2+3.6)	16,150	24.3 (13.7+10.6)	14,814
AdaMerging [ 105 ]	8.0	19,918	13.9	19,788
TARA -Variant A	8.2	19,922	14.1	19,788

Table 6: Time and memory cost analysis. We group baselines into gradient-free and gradient-based methods and compare their time and memory cost on LLaMA and LLaVA. For gradient-free methods, we exclude the time spent on grid search for tuning the scaling factor on the validation set. We report only time per iteration for gradient-free methods and all times measured in minutes (min). TIES, KnOTS-TIES, and TARA -Variant B require additional computation to resolve parameter conflicts or perform SVD, so we report both their total time and, in parentheses, the separate contributions from preprocessing and either one validation run (for gradient-free methods) or one optimization run (for gradient-based methods). Memory usage is reported as VRAM in MiB.

Method	Time (min)	VRAM (MiB)
KnOTS-TIES [ 79 ]	2.8	2262
LoRA-LEGO [ 114 ]	4.0	1416
AdaMerging [ 105 ]	4.1	4344
TARA (Variant A)	4.1	4344
TARA (Variant B)	5.1	5922

Table 7: Computational Cost (ViT-B/32). Times denote a single validation pass (learning-free) vs. optimization (learning-based).

Figure 10: Ablation study of the hyperparameter $\alpha$ . Normalized accuracy is reported for two model variants across $\alpha\in\{0.1,0.5,1.0,2.0,5.0\}$ . Both Variant A and Variant B exhibit stable performance over a wide range of $\alpha$ values, indicating that the method is not sensitive to this hyperparameter. Unless otherwise stated, we use $\alpha=1$ for all main experiments.

Figure 11: Two-Task Trade-Offs on CLIP ViT-B/32. Pairs shown from top-left to bottom-right: Cars-MNIST, DTD-RESISC45, EuroSAT-SUN397, GTSRB-SVHN.

Figure 12: Preference Sensitivity under Random Global Preferences. Pairs shown from top-left to bottom-right: Cars-MNIST, DTD-RESISC45, EuroSAT-SUN397, GTSRB-SVHN.

Method	Dataset
Method	Cars	DTD	EuroSAT	GTSRB	MNIST	RESISC45	SUN397	SVHN	Avg
	Per‑task absolute accuracies (%)
Finetuned	76.2	72.5	98.6	98.3	99.2	93.1	73.7	96.7	88.5
	Per-task accuracies of merged models, normalized to finetuned (%)
Vanilla Merging
TA [ 32 ]	85.6	70.8	81.9	89.1	97.9	81.9	89.1	91.9	86.0

Table 9: Using our checkpoints. Per‑task accuracy on eight image‑classification benchmarks. We merge eight ViT‑B/32 checkpoints, each finetuned with LoRA. The upper panel shows the per‑task absolute accuracy of the finetuned baselines; the lower panel reports accuracy of merged models, normalized by their corresponding finetuned baseline (%).

Method	Dataset
Method	Cars	DTD	EuroSAT	GTSRB	MNIST	RESISC45	SUN397	SVHN	Avg
	Per‑task absolute accuracies (%)
Finetuned (KnOTS)	74.0	58.3	99.0	92.7	99.3	88.4	64.5	96.2	84.1
Finetuned (Ours)	76.2	72.5	98.6	98.3	99.2	93.1	73.7	96.7	88.5
	Per‑task accuracies of merged models, normalized to finetuned (%)
TA [ 32 ] (KnOTS)	82.0	73.6	48.8	42.1	53.1	71.5	97.5	41.2	63.7

Table 10: Comparison between merging methods between our checkpoints and KnOTS checkpoints.

	Seen Tasks							Unseen Tasks			All Tasks
Method	Cars	DTD	GTSRB	RESISC45	SUN397	SVHN	Avg Acc	EuroSAT	MNIST	Avg Acc	Avg Acc
	Per-task accuracies of merged models, normalized to finetuned (%)
TA [ 32 ]	85.6	73.0	94.7	82.4	88.4	95.0	86.5	47.3	78.1	62.7	80.6
TIES [ 99 ]	76.5	62.5	71.4	73.3	87.5	75.0	74.4	49.0	64.5	56.8	69.9
KnOTS-TIES [ 79 ]	85.0	69.4	83.9	81.1	89.4	84.8	82.3	56.6	73.4	65.0	78.0
KnOTS-DARE-TIES [ 79 ]	85.7	69.3	85.0	81.4	90.3	86.4	83.0	56.7	74.0	65.4	78.6

Table 11: Using our checkpoints. Generalization results on two unseen tasks when merging ViT-B/32 models trained on six tasks.

Method	Dataset
Method	Cars	DTD	EuroSAT	GTSRB	MNIST	RESISC45	SUN397	SVHN	Avg
	Per-task absolute accuracies (%)
Finetuned	72.1	70.2	98.4	98.1	99.2	93.3	73.4	96.7	87.7
	Per-task accuracies of merged models, normalized to finetuned (%)
Vanilla Merging
TA [ 32 ]	87.4	76.2	74.0	78.1	98.1	83.7	89.7	92.3	84.9

Table 12: Using our checkpoints with LoRA (Rank-4). Per-task accuracy on eight image-classification benchmarks. We merge eight ViT-B/32 checkpoints, each finetuned with LoRA. The upper panel shows the per-task absolute accuracy of the finetuned baselines; the lower panel reports accuracy of merged models, normalized by their corresponding finetuned baseline (%).

Method	r=4	r=16	r=64	r=256	Full
TA [ 32 ]	71.9(82.9)	76.2(86.2)	77.9(86.2)	77.1(85.3)	72.0(80.8)
Adamerging [ 105 ]	76.3(87.2)	78.7(88.5)	80.2(88.7)	79.8(88.6)	79.1(88.2)
TARA	77.5(88.8)	80.5(90.6)	81.9(90.5)	80.9(89.7)	80.5(89.5)

Table 13: Absolute (Normalized) Accuracy across LoRA Ranks.

Method	Adamerging [ 105 ]	CALM [ 100 ]	FW-Merging [ 5 ]	TARA-variant B
Avg. Acc.	78.7	80.0	77.9	80.5
Time (min)	4.1	20.0	13.4	5.1
VRAM (MiB)	4344	8066	7542	5922

Table 14: Comparison with Recent Baselines.

Backbone (# Tasks)	TA [ 32 ]	AdaMer. [ 105 ]	CALM [ 100 ]	FW-Mer. [ 5 ]	TARA
ViT-B/32 (8)	76.2(86.2)	78.7(88.5)	80.0(90.6)	77.9(87.6)	80.5(90.6)
ViT-B/32 (14)	73.6(83.6)	75.0(84.8)	74.7(84.6)	75.4(85.2)	75.4(85.2)
ViT-B/32 (20)	64.6(72.7)	66.6(74.8)	66.9(75.2)	68.2(77.2)	68.7(77.2)
ViT-L/14 (8)	88.0(95.8)	88.9(96.7)	83.0(90.3)	87.0(94.7)	88.9(96.7)
ViT-L/14 (14)	85.0(92.7)	85.2(93.0)	80.5(87.7)	85.7(93.6)	85.9(93.7)
ViT-L/14 (20)	80.7(87.2)	81.3(87.8)	78.5(84.5)	77.4(84.1)	81.5(88.0)

Table 15: Absolute (Normalized) accuracy on vision benchmark.

Low-Rank Adaptation

Low-rank adaptation (LoRA) fine-tunes pretrained models with compact weight updates, reducing trainable parameters and deployment costs.

See all 542 papers

Evolutionary Negative Module Pruning for Better LoRA Merging
Apr 20, 2026Anda Cao, Zhuo Gou, Yi Wang +5Low-Rank Adaptation Language Model Merging
College of Computer Science and Technology, Zhejiang University · School of Software Technology, Zhejiang University · 3State Key Laboratory of Blockchain and Security, Zhejiang University +1
Not All Ranks Are Equal: Budget-Aware LoRA Merging Across Tasks
Sep 3, 2026Avinash Amballa, Yashas Malur Saidutta, Wenbo Li +2Model Merging Low-Rank Adaptation
Samsung Research America
Crowded in B-Space: Calibrating Shared Directions for LoRA Merging
Apr 18, 2026Yixuan Tang, Yi YangAdapter Tuning Low-Rank Adaptation
The Hong Kong University of Science and Technology

Apr 20, 2026J/K move · Enter open · S save

Evolutionary Negative Module Pruning for Better LoRA Merging

Anda Cao, Zhuo Gou, Yi Wang, Kaixuan Chen+4

College of Computer Science and Technology, Zhejiang University · School of Software Technology, Zhejiang University · 3State Key Laboratory of Blockchain and Security, Zhejiang University+1

Low-Rank Adaptation Language Model Merging Model Merging

Open paper AskPDF arXiv

Merging multiple Low-Rank Adaptation (LoRA) experts into a single backbone is a promising approach for efficient multi-task deployment. While existing methods strive to alleviate interference via weight interpolation or subspace alignment, they rest upon the implicit assumption that all LoRA matrices contribute constructively to the merged model. In this paper, we uncover a critical bottleneck in current merging paradigms: the existence of

\textit{negative modules}

-- specific LoRA layers that inherently degrade global performance upon merging. We propose

\textbf{E}

volutionary

\textbf{N}

egative

\textbf{M}

odule

\textbf{P}

runing (

\textbf{ENMP}

), a plug-and-play LoRA pruning method to locate and exclude these detrimental modules prior to merging. By leveraging an evolutionary search strategy, ENMP effectively navigates the discrete, non-differentiable landscape of module selection to identify optimal pruning configurations. Extensive evaluations demonstrate that ENMP consistently boosts the performance of existing merging algorithms, achieving a new state-of-the-art across both language and vision domains. Code is available at https://github.com/CaoAnda/ENMP-LoRAMerging.

Abstract

Low-Rank Adaptation

Explore similar work

Evolutionary Negative Module Pruning for Better LoRA Merging

Not All Ranks Are Equal: Budget-Aware LoRA Merging Across Tasks

Crowded in B-Space: Calibrating Shared Directions for LoRA Merging