Escaping Low-Dimensional Overlap: Multi-Task Model Merging via High-Dimensional Sparse Disentanglement

Yihang Zhang; Shengke Sun; Junjie Wen; Feng Zeng

cs.LGAug 26, 2026

Escaping Low-Dimensional Overlap: Multi-Task Model Merging via High-Dimensional Sparse Disentanglement

Authors: Yihang Zhang, Shengke Sun, Junjie Wen, Feng Zeng

Organizations: Central South University · Nanjing University of Science and Technology · Hefei University of Technology

Abstract

Model merging provides an efficient way to construct multi-task generalist models without additional training, but its performance often degrades under severe task interference. Task interference in model merging primarily stems from superposition, where task-specific features become entangled within the parameter space. This entanglement renders conventional decomposition methods insufficient for effectively isolating useful task directions from interfering components. In this paper, we propose a sparse-representation-based merging framework that uses Sparse Autoencoders (SAEs) to project task vectors into a high-dimensional sparse feature space, enabling feature-level disentanglement before fusion. To reduce computational overhead, we further introduce a lightweight Group-Ranked Zeroth-Order Optimizer (GR-ZOO) to identify task-critical layers for selective merging. Experiments on both Qwen2.5-1.5B and Qwen2.5-7B demonstrate that our method consistently outperforms representative baselines, including Task Arithmetic, TIES-Merge, DARE, Fisher-Merge,and several recent training-free merging methods, across mathematical reasoning, code generation, instruction following, and general knowledge tasks. In a highly conflicting four-task setting on Qwen2.5-1.5B, our method further achieves a 2.78% improvement over the strongest baseline.

Figures & tables

Figure 1: Overall architecture of our proposed High-Dimensional Sparse Disentanglement Merging framework. The pipeline first extracts task vectors from expert models and applies GR-ZOO to identify task-critical layers. Non-critical layers are merged through arithmetic addition, while critical layers are projected into a high-dimensional sparse space via SAE-based feature disentanglement, enabling interference-free fusion of overlapping task features.

Methods	Math (GSM8k)	Code (HumanEval+)	IF (IFEval)	Average
Task Arithmetic	83.85	73.80	45.47	67.71
TIES-Merge	83.40	75.00	38.45	65.62
DARE+TIES	83.62	72.60	36.78	64.33
Fisher-Merge	84.00	75.60	41.59	67.06
TSV-Merge	83.32	72.88	45.77	67.32
WUDI-Merging	82.79	73.49	44.74	67.01

Table 1: Main multi-task merging results on Qwen2.5-7B.

Methods	Math	General				Code	Safe	Average
Methods	GSM8k	STEM	Social Sciences	Humanities	Others	HumanEval	BeaverTail	Average
Task Arithmetic	33.74	8.91	9.39	7.48	7.00	0.00	41.60	15.45
TIES-Merge	51.10	19.48	15.70	14.73	11.75	32.93	61.05	29.53
DARE+TIES	52.16	20.44	14.85	13.62	11.14	31.71	49.91	27.69
Fisher-Merge	60.80	22.13	6.47	10.74	5.74	36.59	39.00	25.92
TSV-Merge	52.00	30.10	33.30	29.70	38.80	15.00	29.00	32.24

Table 2: Performance scaling to 4 tasks on Qwen2.5-1.5B. Average denotes the unweighted arithmetic mean over the seven reported evaluation metrics: GSM8K, four General capability domains, HumanEval, and BeaverTail. The same aggregation rule is applied consistently to all methods.

Figure 2: Gradient estimate variance comparison: GR-ZOO effectively smooths the variance compared to standard ZOO.

Figure 3: Performance comparison of different layer selection strategies. GR-ZOO closely approximates the theoretical upper bound (Gradient) while significantly outperforming the Random baseline.

Component	Variant	Metric Avg.
Projection Space	PCA	30.80
	Random Projection	31.01
	SAE	31.56
Layer Selection	First Layers	28.27
	Random Selection	27.64
	Fisher Selection	29.90

Table 3: Average performance of different projection spaces and layer-selection strategies.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Distribution of element-wise parameter differences $\Delta w=w_{\mathrm{Ours}}-w_{\mathrm{TA}}$ across selected parameter indices. The distributions are centered near zero, while their spike-like structures suggest sparse and selective parameter calibration.

Item	Setting
Candidate / selected layers	197 / 16
Fraction of layers receiving SAE processing	8.12%
Number of trained SAEs	16
Cold GR-ZOO time	467.13s / 7.79min
Single-layer SAE training and latent encoding	366.75s / 6.11min
Peak GPU memory	25,153.93 MB

Appendix

Table 4: Computational cost and training configuration of GR-ZOO and SAE. SAE cost is measured for one selected layer.

Metric	Raw	SAE
Pairwise active overlap $\downarrow$	0.3368	0.0983
Active Jaccard $\downarrow$	0.2027	0.0518
Task-specific active ratio $\uparrow$	29.40%	73.84%
Shared active ratio $\downarrow$	70.60%	26.16%
Shared feature ratio $\downarrow$	50.96%	14.52%

Appendix

Table 5: Disentanglement analysis between raw task vectors and SAE latent representations. SAE reduces activation overlap and increases task-specific feature allocation.

$K$	GSM8K	HumanEval	MMLU	Avg.
8	$42.05\pm 0.42$	$37.60\pm 0.70$	$12.08\pm 0.53$	$30.58\pm 0.25$
16	$42.03\pm 0.85$	$36.18\pm 0.93$	$13.68\pm 1.64$	$30.63\pm 0.40$
24	$41.82\pm 1.03$	$37.20\pm 2.44$	$12.63\pm 0.11$	$30.55\pm 1.09$
32	$42.36\pm 0.42$	$36.18\pm 1.27$	$13.31\pm 1.88$	$30.61\pm 0.99$
64	$41.93\pm 1.18$	$37.40\pm 1.27$	$12.75\pm 1.76$	$30.69\pm 0.75$

Appendix

Table 6: Sensitivity to the Top-

K

sparsity level with the similarity threshold fixed at

\tau=10^{-4}

. IFEval-S denotes the strict IFEval score.

$\tau$	GSM8K	HumanEval	MMLU	Avg.
$0$	$42.13\pm 0.23$	$35.57\pm 1.41$	$13.19\pm 1.68$	$30.29\pm 0.61$
$10^{-5}$	$42.15\pm 0.23$	$36.38\pm 1.41$	$12.14\pm 0.70$	$30.22\pm 0.57$
$10^{-4}$	$42.36\pm 0.42$	$36.18\pm 1.27$	$13.31\pm 1.88$	$30.61\pm 0.99$
$10^{-3}$	$42.05\pm 0.29$	$36.59\pm 1.61$	$13.06\pm 1.48$	$30.57\pm 0.92$

Appendix

Table 7: Sensitivity to the similarity threshold

\tau

with the sparsity level fixed at

K=32

. IFEval-S denotes the strict IFEval score.

Method	Math	General				Code	Metric Avg.
Method	GSM8K	STEM	Social Sciences	Humanities	Others	HumanEval	Metric Avg.
Task Arithmetic	53.14	33.63	46.99	33.79	42.91	14.63	37.52
TIES-Merge	51.93	30.35	29.90	27.97	26.77	31.10	33.00
DARE+TIES	56.02	29.26	28.34	25.80	25.29	32.93	32.94
Fisher-Merge	48.45	21.40	10.63	13.82	8.76	37.20	23.38
$L_{1}$ SAE	54.89	35.02	43.68	31.99	36.67	33.54	39.30

Appendix

Table 8: Detailed results for 3-task merging on Qwen2.5-1.5B. Compared with the standard

L_{1}

-regularized SAE, our complete SAE design achieves a higher overall metric average while maintaining a better balance across Math, General, and Code capabilities. Metric Avg. denotes the unweighted mean over the six displayed evaluation metrics.

Method	Math	General				Code	Safety	Metric Avg.
Method	GSM8K	STEM	Social Sciences	Humanities	Others	HumanEval	BeaverTail	Metric Avg.
Task Arithmetic	33.74	8.91	9.39	7.48	7.00	0.00	41.60	15.45
TIES-Merge	51.10	19.48	15.70	14.73	11.75	32.93	61.05	29.53
DARE+TIES	52.16	20.44	14.85	13.62	11.14	31.71	49.91	27.69
Fisher-Merge	60.80	22.13	6.47	10.74	5.74	36.59	39.00	25.92
$L_{1}$ SAE	51.18	27.44	33.44	22.51	26.22	32.93	52.97	35.24

Appendix

Table 9: Detailed results for 4-task merging on Qwen2.5-1.5B. Compared with the standard

L_{1}

-regularized SAE, our complete SAE design improves the overall metric average and better preserves General capabilities under increased cross-task interference. Metric Avg. denotes the unweighted mean over the seven displayed evaluation metrics.

Component	Variant	GSM8K	HumanEval	IFEval	Metric Avg.
Projection Space	PCA	41.09	35.98	15.34	30.80
	Random Projection	40.94	38.41	13.68	31.01
	SAE	42.76	36.59	15.34	31.56
Layer Selection	First Layers	42.84	28.66	13.31	28.27
	Random Selection	40.94	28.66	13.31	27.64
	Fisher Selection	43.06	31.10	15.53	29.90

Appendix

Table 10: Detailed per-task results for the projection-space and layer-selection ablations. For the projection-space comparison, all methods use the same selected layers and fusion procedure. For the layer-selection comparison, all strategies select the same number of layers and use identical SAE and fusion configurations. Metric Avg. denotes the unweighted mean over GSM8K, HumanEval, and IFEval.

Variant	FVU $\downarrow$	Dead Features (%) $\downarrow$	Post-Top- $K$ Active (%) $\downarrow$	Pre-Top- $K$ Active (%)	Norm Std. $\downarrow$	Atom Coherence $\downarrow$
Full SAE	0.1528	0.0488	8.33	20.22	0.0000	0.0176
w/o Top- $K$	0.0212	0.0000	35.76	35.76	0.0000	0.0176
w/o Residual Fitting	0.1539	0.0000	8.33	26.83	0.0000	0.0176
w/o Decoder Normalization	0.1565	0.2441	8.33	18.25	0.0101	0.0176
w/o Orthogonality	0.0110	1.5299	8.33	14.39	0.0000	0.0224
w/o Normalization and Orthogonality	0.0107	5.1921	8.33	13.23	0.0094	0.0224

Appendix

Table 11: Representation-level diagnostics of the individual SAE components. Lower reconstruction error does not necessarily indicate better feature separation, since a dense or redundant representation may obtain lower FVU while exhibiting higher activation density, atom coherence, or dead-feature rates.

Explore similar work

Resolving Interference (RI): Disentangling Models for Improved Model Merging
Mar 13, 2026Pratik Ramesh, George Stoica, Arun Iyer +2Model Merging
Georgia Tech · University of Washington · Microsoft Research +3
Stay Unique, Stay Efficient: Preserving Model Personality in Multi-Task Merging
Dec 1, 2025Kuangpu Guo, Aijing Yu, Jian Liang +4Model Merging
University of Science and Technology of China · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · Institute of Information Engineering, Chinese Academy of Sciences +3
Model Merging in the Essential Subspace
Feb 23, 2026Longhua Li, Lei Qi, Qi Tian +1Model Merging Task Vector Merging
School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Huawei Technologies, Shanghai, China

Mar 13, 2026J/K move · Enter open · S save

Resolving Interference (RI): Disentangling Models for Improved Model Merging

Pratik Ramesh, George Stoica, Arun Iyer, Leshem Choshen+1

Georgia Tech · University of Washington · Microsoft Research+3

Model Merging

Open paper AskPDF arXiv

Model merging has shown that multitask models can be created by directly combining the parameters of different models that are each specialized on tasks of interest. However, models trained independently on distinct tasks often exhibit interference that degrades the merged model's performance. To solve this problem, we formally define the notion of Cross-Task Interference as the drift in the representation of the merged model relative to its constituent models. Reducing cross-task interference is key to improving merging performance. To address this issue, we propose our method, Resolving Interference (RI), a light-weight adaptation framework which disentangles expert models to be functionally orthogonal to the space of other tasks, thereby reducing cross-task interference. RI does this whilst using only unlabeled auxiliary data as input (i.e., no task-data is needed), allowing it to be applied in data-scarce scenarios. RI consistently improves the performance of state-of-the-art merging methods by up to 3.8% and generalization to unseen domains by up to 2.3%. We also find RI to be robust to the source of auxiliary input while being significantly less sensitive to tuning of merging hyperparameters. Our codebase is available at: https://github.com/pramesh39/resolving_interference

Abstract

Explore similar work

Resolving Interference (RI): Disentangling Models for Improved Model Merging

Stay Unique, Stay Efficient: Preserving Model Personality in Multi-Task Merging

Model Merging in the Essential Subspace