Not All Ranks Are Equal: Budget-Aware LoRA Merging Across Tasks

Avinash Amballa; Yashas Malur Saidutta; Wenbo Li; Lazar Valkov; Srinivas Chappidi

cs.LGSep 3, 2026

Not All Ranks Are Equal: Budget-Aware LoRA Merging Across Tasks

Authors: Avinash Amballa, Yashas Malur Saidutta, Wenbo Li, Lazar Valkov, Srinivas Chappidi

Organizations: Samsung Research America

Abstract

Merging low-rank adapters (LoRAs) promises to eliminate the overhead of swapping task-specific weights at inference time. However, existing merging methods assume every layer needs the same rank budget. Further, some methods assume that rank budget needs to be split equally among the tasks too. We show this uniform-budget assumption is a major source of the performance gap between merged and per-task LoRAs. However, rank selection is an NP hard problem. To this end, we introduce Net Utility, a data free metric that first decomposes every task LoRA by its Singular Value Decomposition (SVD) and scores each of those singular directions by its task utility and its interference with other tasks directions. Next, we globally pool these scores to select singular directions with the highest values with a constraint on the total number of directions selected. The proposed Net Utility metric is applied on top of five different merging methods across three different merging spaces. The merging is done over two sets of tasks, vision and language tasks. Net utility based rank allocation outperforms its counterparts without that allocation. On average, over vision tasks it achieves +2.1% improvement in performance, and +2.2% improvement over the language tasks.

Figures & tables

Figure 1: Ranks allocated to each task at different layers on a ViT-B/32 over seven vision tasks. We observe that different layers have different ranks indicating that certain layers are more important for LoRA updates than the others. We also observe that some tasks have less allocation at the lower layers and more at the later layers. The merging method is Task Arithmetic performed over the full space. Our non-uniform allocation improves accuracy by +3.4%

Figure 2: Net Utility based rank selection for Task Arithmetic Merging. (a): Obtain the per-task LoRAs. (b): Decompose all the LoRAs into their SVDs. Compute the Net Utility metric for each of the directions. (c): Pool all the directions and select the top-R directions. (d): Compose the merged LoRA using the selected directions. We naturally end up with variable number of directions being selected across tasks and layers.

			$R=64$			$R=32$			$R=16$
Space	Iso	Method	Unif.	Ours	$\Delta$	Unif.	Ours	$\Delta$	Unif.	Ours	$\Delta$
Full	–	TA	63.5	66.9	$\mathbf{+3.4}$	63.1	65.9	$\mathbf{+2.7}$	62.5	64.9	$\mathbf{+2.3}$
Full	–	TSV	66.4	66.9	$\mathbf{+0.4}$	65.4	65.7	$\mathbf{+0.3}$	63.9	64.7	$\mathbf{+0.8}$
		DARE	63.5	67.3	$\mathbf{+3.8}$	63.1	66.1	$\mathbf{+3.0}$	62.6	65.8	$\mathbf{+3.2}$
		TIES	62.5	65.0	$\mathbf{+2.5}$	62.3	64.8	$\mathbf{+2.4}$	61.8	64.8	$\mathbf{+3.0}$
Core	–	TSV	66.4	66.8	$\mathbf{+0.4}$	65.4	65.7	$\mathbf{+0.3}$	63.9	64.6	$\mathbf{+0.7}$

Table 1: Vision tasks, ViT-B/32, 7 tasks. Normalized accuracy averaged over all 7 tasks.

			$R=64$			$R=32$			$R=16$
Space	Iso	Method	Unif.	Ours	$\Delta$	Unif.	Ours	$\Delta$	Unif.	Ours	$\Delta$
-	–	TA	72.8	75.1	$\mathbf{+2.3}$	72.8	75.0	$\mathbf{+2.3}$	72.6	74.8	$\mathbf{+2.2}$
Full	–	TSV	78.3	81.4	$\mathbf{+3.1}$	77.5	80.6	$\mathbf{+3.1}$	76.8	80.3	$\mathbf{+3.5}$
		DARE	72.3	73.7	$\mathbf{+1.4}$	72.5	73.3	$\mathbf{+0.9}$	72.0	73.4	$\mathbf{+1.3}$
		TIES	71.9	73.6	$\mathbf{+1.7}$	71.3	73.6	$\mathbf{+2.3}$	70.2	73.5	$\mathbf{+3.4}$
Core	–	TSV	78.3	81.4	$\mathbf{+3.1}$	77.5	80.6	$\mathbf{+3.1}$	76.8	80.3	$\mathbf{+3.5}$

Table 2: Language (NLI) tasks, Qwen3-4B, 6 tasks. Normalized accuracy averaged over 6 tasks.

		Vision (ViT-B/32, 7 tasks)			Language (Qwen3-4B, 6 tasks)
Space	Method	Unif.	Ours	$\Delta$	Unif.	Ours	$\Delta$
Full	Iso-C + TA	66.0	67.3	$\mathbf{+1.3}$	81.6	82.6	$\mathbf{+1.0}$
Full	Iso-C + TSV	66.0	67.2	$\mathbf{+1.2}$	81.8	82.3	$\mathbf{+0.5}$

Table 3: Isotropization (Iso-C) at

R{=}16

, full space for both Vision and Language tasks.

	spend the whole budget			admit only $g_{j,l}>0$
$R$	Uniform	Ours	$\Delta$	Ours	used (avg)	unspent	$\Delta$
$112$ (full budget)	$63.75$	$63.75$	$+0.00$	$\mathbf{66.94}$	$61.5$	$\mathbf{45.0\%}$	$\mathbf{+3.19}$
$64$	$63.54$	$\mathbf{66.94}$	$\mathbf{+3.40}$	$66.94$	$61.5$	$3.9\%$	$+3.40$
$32$	$63.15$	$\mathbf{65.87}$	$\mathbf{+2.72}$	$65.87$	$32.0$	$0\%$	$+2.72$
$16$	$62.52$	$\mathbf{64.86}$	$\mathbf{+2.34}$	$64.86$	$16.0$	$0\%$	$+2.34$

Table 4: The budget need not be spent. Net-utility allocation on task arithmetic on vision tasks. used is directions retained per module. At full rank

R=112

only

55\%

carry positive net utility, discarding the rest raises accuracy by

3.19

while leaving over half the budget unspent. At lower ranks, we observe that entire budget is spent.

Figure 3: Fraction of sets that retain the

l

-th singular direction. A prefix rule keeps every direction above a cut and none below it, i.e. the dashed step. On the other hand the net utility produces nothing of the sort.

Figure 4: Total Budget allocation across all 7 vision tasks v/s layers. We observe that later layers require more budget than early layers. We also observe that modules v-proj and out-proj occupy more budget in the later layers.

Figure 5: Net-utility vs performance on Hendrycks Math dataset. We observe that there is a negative correlation to the relative task performance and net utility

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Base model	Qwen3-4B-Instruct
Task type	Sequence classification (3 classes)
LoRA rank $r$	16
LoRA scaling $\alpha$	16
LoRA dropout	0.1
Target modules	$W_{q}$ , $W_{k}$ , $W_{v}$ , $W_{o}$

Appendix

Table 5: Hyperparameters for LoRA adapter training on Qwen3-4B.

Model Merging

Model merging combines trained model weights or adapters to reuse capabilities while reducing the cost of joint training or ensembles.

See all 192 papers

Preference-Aligned LoRA Merging: Preserving Subspace Coverage and Addressing Directional Anisotropy
Mar 27, 2026Wooseong Jeong, Wonyoung Lee, Kuk-Jin YoonLow-Rank Adaptation Language Model Merging
KAIST
Compress then Merge: From Multiple LoRAs into One Low-Rank Adapter
Jun 2, 2026Zhengbao He, Ruiqi Ding, Zhehao Huang +3Adapter Tuning Low-Rank Adaptation
Institute of Image Processing and Pattern Recognition, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, China · Shanghai Key Laboratory of Flexible Medical Robotics, Tongren Hospital, Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, China
Crowded in B-Space: Calibrating Shared Directions for LoRA Merging
Apr 18, 2026Yixuan Tang, Yi YangAdapter Tuning Low-Rank Adaptation
The Hong Kong University of Science and Technology

Mar 27, 2026J/K move · Enter open · S save

Preference-Aligned LoRA Merging: Preserving Subspace Coverage and Addressing Directional Anisotropy

Wooseong Jeong, Wonyoung Lee, Kuk-Jin Yoon

KAIST

Low-Rank Adaptation Language Model Merging Task Vector Merging Representation Geometry

Open paper AskPDF arXiv

Merging multiple Low-Rank Adaptation (LoRA) modules is promising for constructing general-purpose systems, yet challenging because LoRA update directions span different subspaces and contribute unevenly. When merged naively, such mismatches can weaken the directions most critical to certain task losses while overemphasizing relatively less important ones, ultimately reducing the model's ability to represent all tasks faithfully. We revisit this problem through two perspectives: subspace coverage, which captures how broadly LoRA directions cover diverse representational directions, and anisotropy, which reflects the imbalance of influence across those directions. We propose TARA-Merging (Task-Rank Anisotropy Alignment), which aligns merging weights using a preference-weighted cross-entropy pseudo-loss while preserving task-relevant LoRA subspaces. This ensures broad subspace coverage and mitigates anisotropy via direction-wise reweighting. Across eight vision and six NLI benchmarks, TARA-Merging consistently outperforms vanilla and LoRA-aware baselines, demonstrating strong robustness and generalization, and highlighting the importance of addressing both subspace coverage and anisotropy in LoRA merging.

Abstract

Model Merging

Explore similar work

Preference-Aligned LoRA Merging: Preserving Subspace Coverage and Addressing Directional Anisotropy

Compress then Merge: From Multiple LoRAs into One Low-Rank Adapter

Crowded in B-Space: Calibrating Shared Directions for LoRA Merging