DC-Merge: Improving Model Merging with Directional Consistency

Han-Chen Zhang; Zi-Hao Zhou; Mao-Lin Luo; Shimin Di; Min-Ling Zhang; Tong Wei

cs.LGMar 6, 2026

DC-Merge: Improving Model Merging with Directional Consistency

Authors: Han-Chen Zhang, Zi-Hao Zhou, Mao-Lin Luo, Shimin Di, Min-Ling Zhang, Tong Wei

Organizations: School of Computer Science and Engineering, Southeast University, Nanjing 210096, China

Abstract

Model merging aims to integrate multiple task-adapted models into a unified model that preserves the knowledge of each task. In this paper, we identify that the key to this knowledge retention lies in maintaining the directional consistency of singular spaces between merged multi-task vector and individual task vectors. However, this consistency is frequently compromised by two issues: i) an imbalanced energy distribution within task vectors, where a small fraction of singular values dominate the total energy, leading to the neglect of semantically important but weaker components upon merging, and ii) the geometric inconsistency of task vectors in parameter space, which causes direct merging to distort their underlying directional geometry. To address these challenges, we propose DC-Merge, a method for directional-consistent model merging. It first balances the energy distribution of each task vector by smoothing its singular values, ensuring all knowledge components are adequately represented. These energy-balanced vectors are then projected onto a shared orthogonal subspace to align their directional geometries with minimal reconstruction error. Finally, the aligned vectors are aggregated in the shared orthogonal subspace and projected back to the original parameter space. Extensive experiments on vision and vision-language benchmarks show that DC-Merge consistently achieves state-of-the-art performance in both full fine-tuning and LoRA settings. The implementation code is available at https://github.com/Tobeginwith/DC-Merge.

Figures & tables

Panel of Figure 1 — Figure 1: The singular value distribution of task vectors averaged across all layers. We normalize each singular value by the largest one within each dataset to eliminate the magnitude discrepancy among different datasets.

Panel of Figure 2 — Figure 2: (a) Comparison of cross-task transferability before (left) and after (right) balancing the energy distribution. The diagonal elements represent the relative performance of each task with respect to its fine-tuned model, while the off-diagonal elements indicate the relative transfer performance on other tasks, normalized against their zero-shot baseline. (b) Each task vector’s cosine similarity with multi-task vector (lines) and average normalized transfer accuracy of each task vector (bars). We compare original task vectors and their energy-balanced counterparts. The energy-balanced task vectors achieve higher $\mathrm{CosSim}$ with the multi-task vector (we use $\boldsymbol{\Delta}_{\text{Iso-C}}$ [ 45 ] for simplicity) and better cross-task generalization, indicating that balancing the energy distribution across knowledge components enhances multi-task expressiveness.

Panel of Figure 3 — Figure 3: (a) Comparison of $\mathrm{CosSim}$ and $\mathrm{DirSim}$ . $\mathrm{DirSim}$ considers the similarity between every pair of directions equally, whereas $\mathrm{CosSim}$ mainly focuses on the similarity among the dominant directions while ignoring the minor ones. (b) Empirical validation of the importance of preserving directional geometry. Solid line: task performance vs. $\mathrm{DirSim}$ under random directional perturbations; dashed line: task performance vs. $\mathrm{CosSim}$ under energy distribution perturbations. (c) Correlation of task-wise performance with projected $\mathrm{DirSim}$ . The overall correlation is positive (Pearson $R=0.64$ ) and per-method averages follow the same trend. Similar patterns persist under larger task scales as illustrated in Figure 7 . We utilize Normalized Accuracy Improvement (NAI) [ 45 ] to measure task-wise performance in (b) and (c) .

Method	ViT-B-32			ViT-B-16			ViT-L-14
Method	8 tasks	12 tasks	16 tasks	8 tasks	12 tasks	16 tasks	8 tasks	12 tasks	16 tasks
Individual	$87.82$	$88.90$	$87.50$	$89.71$	$90.76$	$89.11$	$92.36$	$93.57$	$92.11$
Task Arithmetic	$52.80_{(61.73)}$	$60.76_{(69.12)}$	$60.04_{(68.94)}$	$57.70_{(65.30)}$	$64.26_{(71.12)}$	$62.40_{(70.16)}$	$68.29_{(74.38)}$	$73.69_{(78.84)}$	$69.98_{(75.58)}$
KnOTS-TIES	$55.93_{(65.03)}$	$63.03_{(71.43)}$	$61.78_{(70.78)}$	$60.80_{(68.59)}$	$66.35_{(73.38)}$	$64.31_{(72.29)}$	$73.61_{(79.92)}$	$75.64_{(80.90)}$	$72.19_{(77.98)}$
WUDI-Merging	$55.25_{(64.38)}$	$62.20_{(70.64)}$	$61.24_{(70.27)}$	$58.95_{(66.63)}$	$65.29_{(72.29)}$	$64.59_{(72.51)}$	$69.78_{(75.91)}$	$74.25_{(79.45)}$	$71.79_{(77.59)}$
TSV-M	$58.91_{(68.25)}$	$65.30_{(73.86)}$	$63.51_{(72.72)}$	$62.97_{(70.87)}$	$68.92_{(76.06)}$	$67.21_{(75.35)}$	$76.52_{(83.00)}$	$79.67_{(85.13)}$	$74.37_{(80.31)}$

Table 1: Average absolute accuracy results on vision model merging benchmarks in LoRA setting; subscript (in parentheses) is the average normalized accuracy. The best results are in bold and the second-best are underlined .

Method	ViT-B-32			ViT-B-16			ViT-L-14
Method	8 tasks	14 tasks	20 tasks	8 tasks	14 tasks	20 tasks	8 tasks	14 tasks	20 tasks
Individual	$92.83$	$90.88$	$91.37$	$94.64$	$92.76$	$93.17$	$95.81$	$94.29$	$94.73$
Weight Averaging	$66.34_{(72.13)}$	$64.34_{(71.12)}$	$61.04_{(67.53)}$	$72.22_{(76.60)}$	$69.46_{(74.82)}$	$65.31_{(70.36)}$	$79.56_{(83.15)}$	$76.73_{(81.10)}$	$71.60_{(75.60)}$
Task Arithmetic	$70.79_{(76.55)}$	$65.32_{(72.09)}$	$60.52_{(66.79)}$	$75.41_{(79.58)}$	$70.52_{(75.89)}$	$65.78_{(70.76)}$	$84.93_{(88.65)}$	$79.41_{(83.95)}$	$74.01_{(78.07)}$
TIES-Merging	$75.09_{(81.08)}$	$68.02_{(74.83)}$	$63.38_{(69.90)}$	$79.74_{(84.34)}$	$73.22_{(78.73)}$	$68.18_{(73.26)}$	$86.88_{(90.69)}$	$79.46_{(84.05)}$	$75.71_{(79.80)}$
Consensus TA	$75.03_{(80.84)}$	$70.39_{(77.36)}$	$65.43_{(71.98)}$	$79.39_{(83.86)}$	$74.39_{(79.92)}$	$69.76_{(74.93)}$	$86.34_{(90.08)}$	$82.22_{(86.94)}$	$79.00_{(83.22)}$

Table 2: Average absolute accuracy results on vision model merging benchmarks in FFT setting; subscript (in parentheses) is the average normalized accuracy. The best results are in bold and the second-best are underlined .

Method	Seen Tasks	Unseen Tasks
Zeroshot	$43.37$	$25.22$
Individual	$69.23$	$-$
Multi Task	$63.62$	$36.06$
Task Arithmetic	$53.93$	$33.31$
DARE-Merging	$53.84$	$33.15$
TIES-Merging	$53.09$	$33.14$

Table 3: Performance on MM-MergeBench [ 70 ] , containing eight seen tasks (LoRA fine-tuned) and four unseen tasks. The best results are in bold and the second-best are underlined . We report average absolute accuracy. See Appendix F.7 for detailed results.

Method	8 tasks	12 tasks	16 tasks
No smoothing	69.27	74.60	74.47
Averaging	73.09 ( +3.82 )	76.42 ( +1.82 )	75.51 ( +1.04 )
Linear smoothing	73.90 ( +4.63 )	77.22 ( +2.62 )	75.80 ( +1.33 )

Table 4: Performance comparison of different smoothing strategies. We report average normalized accuracy using ViT-B-32 .

Method	Tasks	w/o mask	w/ mask
FFT	8 tasks	87.98	93.55 ( +5.57 )
	14 tasks	82.39	90.50 ( +8.11 )
	20 tasks	77.63	88.18 ( +10.55 )
LoRA	8 tasks	73.61	73.90 ( +0.29 )
	12 tasks	75.94	77.22 ( +1.28 )
	16 tasks	74.42	75.80 ( +1.38 )

Table 5: Comparison of performance with and w/o applying masks. We report average normalized accuracy using ViT-B-32 .

Method	8 tasks	12 tasks	16 tasks
Vanilla TA	61.73	69.12	68.94
TA + ES	69.12 ( +7.39 )	74.35 ( +5.23 )	73.01 ( +4.07 )
TA + CSM	68.13 ( +6.40 )	73.92 ( +4.80 )	72.64 ( +3.70 )
TA + CSM + ES	73.82 ( +12.09 )	77.16 ( +8.04 )	75.73 ( +6.79 )
Vanilla TIES	62.09	69.30	70.06
TIES + ES	69.94 ( +7.85 )	74.97 ( +5.67 )	74.74 ( +4.68 )

Table 6: Performance of individually applying energy smoothing (ES) and cover space merging (CSM) as well as combining them to TA or TIES compared with vanilla settings. We report the average normalized accuracy using ViT-B-32 .

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Definition
$\boldsymbol{W}_{0},\{\boldsymbol{W}_{i}\}_{i=1}^{T}$	The weights of pretrained model
$\{\boldsymbol{W}_{i}\}_{i=1}^{T}$	The weights of $T$ tasks fine-tuned model
$\Delta\boldsymbol{W}_{i}$	The task vector of task $i$
$\Delta\boldsymbol{W}_{i}=\sum_{j=1}^{r}\sigma_{i}^{j}\boldsymbol{u}_{i}^{j}\boldsymbol{v}_{i}^{j\top}$	Knowledge decomposition of $\Delta\boldsymbol{W}_{i}$
$\sigma_{i}^{j}\boldsymbol{u}_{i}^{j}\boldsymbol{v}_{i}^{j\top}$	$j$ -th knowledge vector of $\Delta\boldsymbol{W}_{i}$
$\boldsymbol{u}_{i}^{j}\boldsymbol{v}_{i}^{j\top}$	$j$ -th knowledge component of $\Delta\boldsymbol{W}_{i}$

Appendix

Table 7: List of definitions and their corresponding mathematical symbols used in this paper.

Panel of Figure 4 — Figure 4: Optimization trajectory of the cover basis. We start from a poor initialization $\left(\widetilde{\boldsymbol{U}}_{0},\widetilde{\boldsymbol{V}}_{0}\right)$ obtained by SVD of task vectors from TA and iteratively optimize the surrogate objective in Eq. ( 8 ). (a) The optimization trace of alignment score and average normalized accuracy of DC-Merge under linear scale. (b) The same process in semi-log scale for better visibility of early-stage dynamics. Both alignment score and accuracy increase monotonically during training and converge to a stable optimum, demonstrating that higher alignment directly corresponds to stronger task retention. The final accuracy and alignment closely match those of the whitening-based approximation, confirming that whitening yields a near-optimal cover basis.

Figure 5: The average absolute accuracy of DC-Merge with mask size

\{8r,4r,2r,r,r/2,r/4,r/8\}

in FFT (left) and LoRA (right) settings. Performance peaks near

r

for FFT and shows a milder variation for LoRA (slightly improves till

r

then decreases), indicating that moderate masks retain performance while overly small/large masks degrade it. The results are based on ViT-B-32 8-task benchmark.

Figure 6: Illustration of mitigating directional shift when merging in the shared cover space. We take

u_{1}

and

u_{2}

as example. Left: (1) The direction of

u_{1}

and

u_{2}

. (2) Directly merging in the original parameter space yields

u_{1}^{\prime}

and

u_{2}^{\prime}

, causing severe directional deviation. Right: (1) The direction of

u_{1}

and

u_{2}

. (2) Obtain

u_{1}^{\prime}

and

u_{2}^{\prime}

by whitening. (3) Project

u_{1}

and

u_{2}

onto the space spanned by

u_{1}^{\prime}

and

u_{2}^{\prime}

. (4) Aggregate the projections along the directions of

u_{1}^{\prime}

and

u_{2}^{\prime}

respectively as merged knowledge vectors, which significantly alleviates directional shift.

Panel of Figure 7 — Figure 7: (a) Correlation of task-wise performance with $\mathrm{DirSim}$ . The result is based on ViT-B-32 8-task benchmark with our checkpoints, while we utilize checkpoints provided by KnOTS [ 58 ] for Figure 3(c) . (b) Layer-averaged projected $\mathrm{DirSim}$ between merged multi-task vector and original task vectors for varying number of tasks. (c) Task-wise NAI of the merged model for varying number of tasks. (b) and (c) are based on merging LoRA fine-tuned ViT-B-32 models.

Figure 8: Impact of hyperparameter

r

in LoRA and FFT settings. We report average normalized accuracy of DC-Merge on ViT-B-32 8-task benchmark. The empirical results align with our motivation to preserve the directional geometry of each task vector after merging as much as possible.

Method	ViT-B-16			ViT-L-14			LLaVA-v1.5-7B
Method	8 tasks	12 tasks	16 tasks	8 tasks	12 tasks	16 tasks	8 tasks
No smoothing	71.64	76.13	75.48	83.96	85.63	80.86	79.10
Linear smoothing	78.69 ( +7.05 )	80.35 ( +4.22 )	78.82 ( +3.34 )	89.22 ( +5.26 )	89.11 ( +3.48 )	85.38 ( +4.52 )	87.58 ( +8.48 )
Averaging	78.86 ( +7.22 )	80.56 ( +4.43 )	78.91 ( +3.43 )	89.42 ( +5.46 )	89.31 ( +3.68 )	85.71 ( +4.85 )	87.91 ( +8.81 )

Appendix

Table 8: Performance with different smoothing strategies. We report average normalized accuracy.

Panel of Figure 9 — Figure 9: (a) The relationship between energy retention ratios and fractions of singular values averaged across all datasets. The result is based on the ViT-B-32 8-task benchmark. FFT (Truncated) considers reconstructing each task vector using the top- $r$ singular values (and their associated singular vectors) in the full fine-tuning setting without additional smoothing. (b) Comparison of cross-task transferability w/o (left) and w/ (right) further balancing the energy distribution after truncating long-tailed energy parts. Compared with LoRA setting, applying energy smoothing in FFT scenario yields relatively small improvements in generalization.

Figure 10: The relationship between average absolute accuracy of DC-Merge in LoRA/FFT setting and interpolation coefficient

\tau

Method	Datasets
	Cars	DTD	EuroSAT	GTSRB	MNIST	RESISC45	SUN397	SVHN	Average
Task Arithmetic	$82.0$	$73.6$	$48.8$	$42.1$	$53.1$	$71.5$	$97.5$	$41.2$	$63.7$
KnOTS-TIES	$82.7$	$73.7$	$49.3$	$48.9$	$68.9$	$70.9$	$95.5$	$\underline{53.8}$	$68.0$
WUDI-Merging	$82.4$	$73.5$	$48.6$	$46.8$	$54.5$	$72.4$	$96.0$	$46.5$	$65.1$
TSV-M	$\underline{83.9}$	$75.1$	$\underline{52.4}$	$45.5$	$58.3$	$73.1$	$97.6$	$45.3$	$66.4$
Iso-CTS	$83.3$	$\mathbf{84.7}$	$49.0$	$\mathbf{79.6}$	$\underline{69.4}$	$\mathbf{82.3}$	$\mathbf{99.0}$	$53.2$	$\underline{75.1}$

Appendix

Table 9: Performance on ViT-B-32 8-task benchmark using checkpoints provided by KnOTS. We report normalized accuracy.

Method	ViT-B-32	ViT-B-16	ViT-L-14
Zeroshot	$48.3$	$55.5$	$64.7$
Individual	$90.5$	$92.6$	$94.2$
Task Arithmetic	$70.5$	$74.6$	$84.6$
Fisher Merging	$68.3$	$71.7$	$83.7$
RegMean	$71.8$	$76.6$	$82.2$
PCB-Merging	$76.3$	$81.5$	$87.5$

Appendix

Table 10: Performance on 8-task benchmark using checkpoints provided by Task Arithmetic. We report average absolute accuracy.

Figure 11:

\mathrm{DirSim}

of TA w/o and w/ CSM and ES on ViT-B-32 8-task benchmark in LoRA setting. Both two modules in DC-Merge enhance

\mathrm{DirSim}(\Delta\boldsymbol{W}_{i},\Delta\widetilde{\boldsymbol{W}}_{i})

of each task.

Figure 12: Projected

\mathrm{CosSim}

of TA and DC-Merge (Ours) on ViT-B-32 8-task benchmark in LoRA setting.

Method	Seen Tasks									Unseen Tasks
	Cars	DTD	EuroSAT	GTSRB	MNIST	RESISC45	SUN397	SVHN	Average	CIFAR100	Flowers	Pets	STL10	Average
Individual	$76.61$	$67.34$	$98.19$	$98.29$	$99.15$	$93.97$	$72.49$	$96.50$	$87.82$	$-$	$-$	$-$	$-$	$-$
Zeroshot	$59.49$	$44.15$	$44.30$	$32.27$	$47.94$	$60.25$	$63.21$	$31.42$	$47.88$	$64.61$	$66.18$	$87.74$	$96.77$	$78.83$
Task Arithmetic	$59.82$	$44.31$	$46.52$	$34.71$	$63.43$	$65.78$	$62.61$	$45.25$	$52.80$	$43.83$	$\mathbf{62.30}$	$82.12$	$96.14$	$71.10$
KnOTS-TIES	$61.39$	$44.79$	$48.26$	$43.42$	$71.24$	$65.49$	$63.48$	$49.36$	$55.93$	$47.21$	$\underline{62.11}$	$82.97$	$96.03$	$72.08$
WUDI-Merging	$60.89$	$45.64$	$51.82$	$39.90$	$66.96$	$67.40$	$63.32$	$46.12$	$55.25$	$46.81$	$62.05$	$82.46$	$\underline{96.19}$	$71.89$

Appendix

Table 11: Performance of eight seen tasks and four unseen tasks on ViT-B-32 in LoRA setting. We report absolute accuracy.

Method	Seen Tasks									Unseen Tasks
	Cars	DTD	EuroSAT	GTSRB	MNIST	RESISC45	SUN397	SVHN	Average	CIFAR100	Flowers	Pets	STL10	Average
Individual	$89.18$	$77.93$	$98.44$	$99.11$	$99.28$	$96.95$	$80.49$	$97.53$	$92.36$	$-$	$-$	$-$	$-$	$-$
Zeroshot	$77.94$	$55.64$	$64.68$	$50.68$	$76.16$	$71.33$	$68.34$	$58.58$	$65.42$	$76.13$	$79.43$	$93.32$	$99.39$	$87.07$
Task Arithmetic	$79.01$	$57.39$	$66.07$	$55.72$	$80.15$	$74.25$	$68.90$	$64.85$	$68.29$	$57.28$	$65.71$	$83.52$	$\underline{98.95}$	$76.36$
KnOTS-TIES	$81.39$	$60.53$	$71.22$	$65.98$	$89.48$	$79.48$	$69.41$	$71.36$	$73.61$	$57.50$	$72.01$	$90.33$	$98.84$	$79.67$
WUDI-Merging	$79.62$	$58.19$	$66.56$	$57.26$	$85.03$	$76.51$	$68.59$	$66.49$	$69.78$	$60.81$	$69.65$	$87.26$	$\mathbf{99.03}$	$79.19$

Appendix

Table 12: Performance of eight seen tasks and four unseen tasks on ViT-L-14 in LoRA setting. We report absolute accuracy.

Method	Seen Tasks									Unseen Tasks
	SciQA	Image	VQA	REC	OCR	VizWiz	Flickr	IconQA	Average	AVQA	Image-R	S2W	TabMWP	Average
Individual	$83.74$	$96.02$	$67.58$	$43.40$	$65.50$	$64.80$	$57.29$	$75.54$	$69.23$	$-$	$-$	$-$	$-$	$-$
Zeroshot	$61.73$	$40.87$	$62.88$	$36.10$	$41.16$	$41.03$	$49.07$	$14.09$	$43.37$	$51.62$	$28.27$	$5.98$	$15.01$	$25.22$
Multi Task	$76.90$	$74.08$	$67.05$	$35.98$	$65.37$	$66.67$	$56.09$	$66.87$	$63.62$	$76.33$	$41.39$	$8.34$	$18.20$	$36.06$
Task Arithmetic	$71.94$	$57.49$	$67.06$	$38.90$	$62.87$	$44.80$	$49.20$	$39.21$	$53.93$	$74.78$	$37.37$	$7.52$	$13.57$	$33.31$
DARE-Merging	$71.59$	$57.25$	$66.26$	$39.38$	$62.56$	$44.93$	$49.13$	$39.59$	$53.84$	$73.75$	$37.67$	$7.56$	$13.62$	$33.15$

Appendix

Table 13: Task-wise absolute accuracy results on MM-MergeBench [ 70 ] , containing eight seen tasks (LoRA fine-tuned) and four unseen tasks. The best results are in bold and the second-best are underlined .

Method	8 tasks	14 tasks	20 tasks
Zeroshot	$48.26$	$57.21$	$56.10$
Individual	$92.83$	$90.88$	$91.37$
TSV-M	$85.86_{(92.31)}$	$80.06_{(87.88)}$	$77.07_{(84.29)}$
WUDI-Merging	$\underline{86.47_{(93.05)}}$	$78.87_{(86.71)}$	$69.90_{(76.71)}$
Iso-CTS	$86.20_{(91.78)}$	$\underline{81.71_{(89.70)}}$	$\underline{78.05_{(85.48)}}$
DC-Merge	$\mathbf{87.05_{(93.55)}}$	$\mathbf{82.52_{(90.62)}}$	$\mathbf{80.58_{(88.18)}}$

Appendix

Table 14: Performance on ViT-B-32 benchmarks in FFT setting. We report average absolute accuracy and subscript (in parentheses) is the average normalized accuracy.

Figure 13: Performance of DC-Merge with respect to various smoothing levels in linear smoothing. We report average normalized accuracy. When

\rho=1.0

, linear smoothing reduces to simple averaging.

Panel of Figure 14 — Figure 14: Visualization of task representations in the shared cover space. We apply block-diagonal masks to eliminate cross-task directional interference (off-diagonal blocks) when projecting the aggregated task representations back to parameter space, where the mask size implicitly balances directional preservation and task fidelity. (Continued on next page)

Panel of Figure 15 — Figure 15: Visualization of task representations in the shared cover space. We apply block-diagonal masks to eliminate cross-task directional interference (off-diagonal blocks) when projecting the aggregated task representations back to parameter space, where the mask size implicitly balances directional preservation and task fidelity. (Continued on next page)

Panel of Figure 16 — Figure 16: Visualization of task representations in the shared cover space. We apply block-diagonal masks to eliminate cross-task directional interference (off-diagonal blocks) when projecting the aggregated task representations back to parameter space, where the mask size implicitly balances directional preservation and task fidelity.

Figure 17: Layer-wise visualization of projected

\mathrm{DirSim}

with each individual task. The results are based on ViT-B-32 8-task benchmark in LoRA setting. DC-Merge consistently exhibits higher projected

\mathrm{DirSim}

with task vectors across all layers.

Model Merging

Model merging combines trained model weights or adapters to reuse capabilities while reducing the cost of joint training or ensembles.

See all 192 papers

CT-Merging: Consensus Directions and Task-Level Scaling for LoRA Adapter Merging
Jul 18, 2026Keumseo Ryum, Joonhyuk KangLow-Rank Adaptation Model Merging
KAIST
CAMFT: Conflict-Aware Mergeable Fine-Tuning for Large Language Models
Sep 6, 2026Jingang Zhou, Haiyang Guo, Yuan Ma +2LLM Fine-Tuning Language Model Merging
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences
PACT: Preserving Anchored Cores in Task-vectors for Model Merging
Jun 17, 2026Ningyuan Shi, Zhipeng Zhou, Hao Wang +2Multi-Task Learning Task Vector Merging
Shanghai Jiao Tong University · Nanyang Technological University · The Hong Kong University of Science and Technology (Guangzhou)

Jul 18, 2026J/K move · Enter open · S save

CT-Merging: Consensus Directions and Task-Level Scaling for LoRA Adapter Merging

Keumseo Ryum, Joonhyuk Kang

KAIST

Low-Rank Adaptation Model Merging

Open paper AskPDF arXiv

LoRA adapters provide an efficient way to specialize a pretrained model for many downstream tasks, but deploying one adapter per task requires adapter storage and task selection at inference time. Model merging addresses this issue by combining independently trained adapters into one multi-task adapter. Recent SVD-based LoRA merging methods mainly focus on constructing shared or task specific directions, while the coefficients assigned to the final directions are often directly from the original task SVD. On a fixed merged basis, inherited coefficients preserve component order with high rank correlation, yet their magnitudes differ substantially from the coefficients induced by the task updates. To address this mismatch, we propose CT-Merging, a LoRA-aware merging algorithm that estimates consensus directions from average task subspace projectors and assigns task-level RMS coefficient scales in the final update. CT-Merging uses repeated support across task SVD subspaces to construct the common basis, while reducing reliance on rank wise SVD magnitudes after direction construction. On the DC-Merge CLIP adapter benchmark, CT-Merging achieves superior average normalized accuracy compared to state-of-the-art merging methods and further improves over DC-Merge by 2.56 points on ViT-B/32 and 1.51 points on ViT-L/14 KnoTS-trained checkpoints.

Abstract

Model Merging

Explore similar work

CT-Merging: Consensus Directions and Task-Level Scaling for LoRA Adapter Merging

CAMFT: Conflict-Aware Mergeable Fine-Tuning for Large Language Models

PACT: Preserving Anchored Cores in Task-vectors for Model Merging