Label-Free Cross-Task LoRA Merging with Null-Space Compression

Wonyoung Lee; Wooseong Jeong; Kuk-Jin Yoon

cs.CVMar 27, 2026

Label-Free Cross-Task LoRA Merging with Null-Space Compression

Authors: Wonyoung Lee, Wooseong Jeong, Kuk-Jin Yoon

Organizations: KAIST · *Equal contribution to this work.

Abstract

Model merging combines independently fine-tuned checkpoints without joint multi-task training. In the era of foundation-model, fine-tuning with Low-Rank Adaptation (LoRA) is prevalent, making LoRA merging a promising target. Existing approaches can work in homogeneous settings where all target tasks are classification but often fail when tasks span classification and regression. Approaches using entropy-based surrogates do not apply to regression and are costly for large language models due to long token sequences. We introduce Null-Space Compression (NSC) Merging, a label-free, output-agnostic method that sets merge weights from adapter geometry. Our key observation is that during LoRA finetuning the down-projection factor $A$ in $ΔW = BA$ compresses its null space, and the compression correlates with performance. NSC uses this as an optimization signal for merging that can generalize across classification, regression, and sequence generation. NSC achieves state-of-the-art performance across twenty heterogeneous vision tasks with balanced gains where prior methods overfit subsets of tasks. It also outperforms baselines on six NLI benchmarks and on vision-language evaluations for VQA and image captioning, demonstrating scalability and effectiveness.

Figures & tables

Panel of Figure 1 — Figure 1: Visualization of validation loss, task performance, and the null-space ratio for an output projection layer during LoRA fine-tuning on (a) image classification and (b) depth estimation.

Figure 2: Classification accuracy versus the null-space ratio . Accuracy is averaged within quartiles of the ratio across tasks. Left shows fine-tuned experts, right shows the merged model.

	NYUD-v2 (4)				PASCAL-Context (5)					Taskonomy (11)											Avg
Method	Depth	Semseg	Normal	Edge	Semseg	Parts	Saliency	Normal	Edge	DE	DZ	EO	ET	K2	K3	N	C	R	S2	S2.5
Method	RMSE $\downarrow$	mIoU $\uparrow$	Mean $\downarrow$	L1 $\uparrow$	mIoU $\uparrow$	mIoU $\uparrow$	mIoU $\uparrow$	Mean $\downarrow$	L1 $\uparrow$	L1 $\downarrow$	L1 $\downarrow$	L1 $\downarrow$	L1 $\downarrow$	L1 $\downarrow$	L1 $\downarrow$	L1 $\downarrow$	RMSE $\downarrow$	L1 $\downarrow$	L1 $\downarrow$	L1 $\downarrow$
Finetuned performance
Finetuned	0.657	37.66	25.98	0.051	70.07	54.77	80.00	18.36	0.046	0.016	0.016	0.101	0.171	0.162	0.082	0.217	0.710	0.136	0.170	0.144	–
Merged models, normalized to finetuned (%)
TA [ 30 ]	24.0	1.3	54.9	105.2	4.7	20.6	36.4	62.5	104.3	100.1	99.8	101.7	104.6	106.4	102.2	100.2	103.6	106.2	107.4	98.8	77.2

Table 1: Merging results on NYUD-v2, PASCAL-Context, and Taskonomy using ViT-B. The fine-tuned rows report per-task absolute performance. For each merging baseline, we report the normalized performance (%) relative to the corresponding fine-tuned score. The rightmost Avg column is the mean of the normalized scores across tasks.

Per-task absolute accuracy (%)
Method	MNLI	QNLI	SNLI	RTE	SICK	SciTail	Avg
Finetuned	90.8	94.9	91.8	87.0	90.9	94.8	91.7
Merged models (normalized to finetuned baselines, %)
TA [ 30 ]	92.8	86.8	93.3	93.6	83.8	95.0	90.9
TIES [ 90 ]	94.3	88.8	90.8	89.8	86.6	94.4	90.8
DARE-TIES [ 98 ]	94.3	88.8	90.8	89.8	86.6	94.4	90.8

Table 2: Per-task results on six NLI benchmarks. We merge six LLaMA-3 8B checkpoints, each fine-tuned with LoRA (rank 16). The top panel reports absolute accuracies of the fine-tuned models. The bottom panel reports accuracies of the merged models, normalized to their corresponding fine-tuned baselines (%).

Per-task absolute score
Method	IconQA	VizWiz ${}_{\text{val}}$	ChartQA	DocVQA ${}_{\text{val}}$	COCO	Flickr30k	Avg
Zero-Shot	17.9	55.2	18.2	24.3	109.5	79.2	–
Finetuned	67.8	69.3	39.0	40.7	130.5	91.3	–
Avg. generated tokens	2.1	2.8	4.7	6.7	12.5	14.2	7.17
Merged models (normalized to fine-tuned baselines, %)
TA [ 30 ]	56.8	83.2	73.0	85.2	92.9	97.6	81.4

Table 3: Per-task performance of merged LLaVA-1.5-7B models across six multi-modal benchmarks. Each model fine-tunes the Vicuna-7B language backbone with LoRA [ 28 ] (rank = 16), while fully fine-tuning the multi-modal projector. The upper block reports absolute accuracies of fine-tuned baselines; the lower block presents merged models’ results normalized to their corresponding baselines (%).

Seen

Avg (Seen)

Unseen

Avg (Unseen)

Avg

NYUD-v2

PASCAL-Context

Taskonomy

NYUD-v2

PASCAL-Context

Taskonomy

Method

Semseg

Edge

Parts

Sal

S2.5

Depth

Normal

Semseg

Normal

Edge

mIoU

\uparrow

\uparrow

mIoU

\uparrow

mIoU

\uparrow

\downarrow

\downarrow

\downarrow

\downarrow

\downarrow

\downarrow

RMSE

\downarrow

Mean

\downarrow

mIoU

\uparrow

Mean

\downarrow

\uparrow

\downarrow

\downarrow

\downarrow

RMSE

\downarrow

\downarrow

Finetuned performance (absolute)

Finetuned

37.66

0.051

54.77

80.00

0.0160

0.1713

0.0820

0.2169

0.1357

0.1435

–

0.657

25.98

70.07

18.36

0.046

0.016

0.101

0.162

0.710

0.170

–

Merged models, normalized to finetuned (%)

Table 4: Experiments on generalization to unseen tasks, including additional baselines. Merging results on NYUD-v2, PASCAL-Context, and Taskonomy with a ViT-B backbone. Fine-tuned rows report per-task absolute performance. For each baseline, we report performance normalized (%) to the corresponding fine-tuned score, and we also report averages over seen tasks, unseen tasks, and all tasks.

Targeted Projection Matrix	Number of Activated Transformer Blocks
Targeted Projection Matrix	12	6	3	1
QKVO	83.7	84.4	83.9	83.0
KVO	84.0	84.6	84.0	83.0
VO	84.3	84.7	84.1	82.6
O	84.5	84.9	84.6	82.6

Table 5: Ablation on where the NSC objective is applied: target module and number of blocks. Evaluated on CLIP (ViT-B/32) across eight image classification tasks.

Method	Requirements			Prep.	Opt.	Val.	Total	GPU Mem.
	Prep.	Opt.	Val.	(min)	(min)	(min)	(min)	(GB)
Gradient-Free
TA [ 30 ]	✗	✗	✓	–	–	16.6	16.6	14.4
TIES [ 90 ]	✓	✗	✓	4.5	–	16.6	20.1	14.4
KnOTS-TIES [ 71 ]	✓	✗	✓	6.8	–	16.6	23.4	14.4
Gradient-Based

Table 6: Runtime and memory efficiency. Comparison of preparation, optimization, and validation time (minutes) and memory usage across gradient-free and gradient-based methods. Validation time excludes grid-search overhead for gradient-free methods.

Dataset	Provided Prompt
IconQA , VizWiz	Answer the question using a single word.
ChartQA , DocVQA	Answer the question using a single word or short phrase.
COCO , Flickr30K	Provide a one-sentence caption for the provided image.

Table 7: Vision–Language Tasks Setup.

Panel of Figure 3 — Figure 3: Extended visualization of validation loss, task performance, and the null-space ratio during LoRA fine-tuning on (a) image classification and (b) depth estimation. We additionally show the null-space ratio trajectories of LoRA at query, key, value projection of self-attention module within a single transformer block, further illustrating the null-space compression phenomenon.

Panel of Figure 4 — Figure 4: Null-space ratio of each transformer block during LoRA fine-tuning for (a) image classification and (b) depth estimation. Values are averaged across transformer blocks, showing that null-space compression consistently occurs throughout the model.

Token Position Strategy			Dataset						Performance
First-Token	Last-Token	Full-Sequence	IconQA	VizWiz	ChartQA	DocVQA	COCO	Flickr	Avg	Time (min)
		✓	59.5	82.8	78.0	87.0	91.5	97.3	82.7	83.7
	✓		60.0	82.9	77.7	87.1	91.5	97.2	82.8	65.1
✓			59.7	82.9	78.1	87.1	91.7	96.8	82.7	13.3

Table 8: Ablation on the token position used to compute the NSC objective. We compare First-Token, Last-Token, and Full-Sequence scoring on six VLM benchmarks. We report average normalized performance and optimization time. The chosen design is shaded.

Input Modality		Dataset						Performance
Text	Image	IconQA	VizWiz	ChartQA	DocVQA	COCO	Flickr	Avg	Time (min)
✓		56.6	83.8	74.3	83.9	93.3	97.7	81.6	6.0
	✓	60.2	82.4	78.4	87.0	91.4	97.0	82.7	13.1
✓	✓	59.7	82.9	78.1	87.1	91.7	96.8	82.7	13.3

Table 9: Ablation on the input modality used to compute the NSC objective. We compare text-only, vision-only, and joint vision–language scoring on six VLM benchmarks. We report average normalized performance and optimization time. The chosen design is shaded.

Process Step	AdaMerging	NSC (Ours)
Pre-computation (Gram-Inverse)	✗	0.7
Forward Pass	396.7	407.2
Entropy/NSC Calculation	1.1	4.1
Backpropagation + Optimizer Step	393.4	386.7

Table 10: Detailed compute cost breakdown of AdaMerging and NSC on VLM tasks. All times are in seconds and measured on a single NVIDIA A6000 GPU.

Targeted Projection Matrix	Number of Activated Transformer Blocks
Targeted Projection Matrix	12	6	3	1
QKVO	92.0	92.0	92.0	91.9
KVO	91.6	91.6	91.6	91.4
VO	92.0	92.0	92.0	91.6
O	91.7	91.7	91.8	91.7

Table 11: Ablation on where the NSC objective is applied: target module and number of blocks evaluated on 20 heterogeneous vision tasks.

Benchmark	Method	Avg. Norm. Acc. (mean $\pm$ std)
LLM Tasks	AdaMerging	90.08 $\pm$ 0.44
LLM Tasks	NSC (Ours)	92.25 $\pm$ 0.08

Table 12: Optimization stability of gradient-based merging methods across 5 random seeds.

# Samples per Task	1	10	100	Full Data
Avg. Norm. Acc.	91.88 $\pm$ 0.35	92.19 $\pm$ 0.16	92.25 $\pm$ 0.12	92.25 $\pm$ 0.08

Table 13: Impact of unlabeled sample size on performance of NSC on LLM tasks.

Method	Dataset
Method	Cars	DTD	EuroSAT	GTSRB	MNIST	RESISC45	SUN397	SVHN	Avg
Finetuned	70.7	67.6	98.5	97.6	99.4	91.6	72.0	95.7	86.6
TA [ 30 ]	87.8	78.6	62.9	70.5	92.2	78.8	92.4	85.9	81.1
TIES [ 90 ]	90.6	80.4	78.8	66.1	91.6	81.2	92.5	82.5	82.9
AdaMerging [ 95 ]	90.1	76.8	87.8	86.0	94.7	85.6	90.8	82.8	86.8
NSC (Ours)	89.4	78.7	80.3	74.4	93.4	82.6	92.1	86.0	84.6

Table 14: Performance comparison across visual classification tasks. Absolute accuracies (top), and normalized accuracies of merged models (bottom).

Model Merging

Model merging combines trained model weights or adapters to reuse capabilities while reducing the cost of joint training or ensembles.

See all 192 papers

Evolutionary Negative Module Pruning for Better LoRA Merging
Apr 20, 2026Anda Cao, Zhuo Gou, Yi Wang +5Low-Rank Adaptation Language Model Merging
College of Computer Science and Technology, Zhejiang University · School of Software Technology, Zhejiang University · 3State Key Laboratory of Blockchain and Security, Zhejiang University +1
Compress then Merge: From Multiple LoRAs into One Low-Rank Adapter
Jun 2, 2026Zhengbao He, Ruiqi Ding, Zhehao Huang +3Adapter Tuning Low-Rank Adaptation
Institute of Image Processing and Pattern Recognition, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, China · Shanghai Key Laboratory of Flexible Medical Robotics, Tongren Hospital, Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, China
CT-Merging: Consensus Directions and Task-Level Scaling for LoRA Adapter Merging
Jul 18, 2026Keumseo Ryum, Joonhyuk KangLow-Rank Adaptation Model Merging
KAIST

Apr 20, 2026J/K move · Enter open · S save

Evolutionary Negative Module Pruning for Better LoRA Merging

Anda Cao, Zhuo Gou, Yi Wang, Kaixuan Chen+4

College of Computer Science and Technology, Zhejiang University · School of Software Technology, Zhejiang University · 3State Key Laboratory of Blockchain and Security, Zhejiang University+1

Low-Rank Adaptation Language Model Merging Model Merging

Open paper AskPDF arXiv

Merging multiple Low-Rank Adaptation (LoRA) experts into a single backbone is a promising approach for efficient multi-task deployment. While existing methods strive to alleviate interference via weight interpolation or subspace alignment, they rest upon the implicit assumption that all LoRA matrices contribute constructively to the merged model. In this paper, we uncover a critical bottleneck in current merging paradigms: the existence of

\textit{negative modules}

-- specific LoRA layers that inherently degrade global performance upon merging. We propose

\textbf{E}

volutionary

\textbf{N}

egative

\textbf{M}

odule

\textbf{P}

runing (

\textbf{ENMP}

), a plug-and-play LoRA pruning method to locate and exclude these detrimental modules prior to merging. By leveraging an evolutionary search strategy, ENMP effectively navigates the discrete, non-differentiable landscape of module selection to identify optimal pruning configurations. Extensive evaluations demonstrate that ENMP consistently boosts the performance of existing merging algorithms, achieving a new state-of-the-art across both language and vision domains. Code is available at https://github.com/CaoAnda/ENMP-LoRAMerging.

Abstract

Model Merging

Explore similar work

Evolutionary Negative Module Pruning for Better LoRA Merging

Compress then Merge: From Multiple LoRAs into One Low-Rank Adapter

CT-Merging: Consensus Directions and Task-Level Scaling for LoRA Adapter Merging