Model merging provides an efficient way to construct multi-task generalist models without additional training, but its performance often degrades under severe task interference. Task interference in model merging primarily stems from superposition, where task-specific features become entangled within the parameter space. This entanglement renders conventional decomposition methods insufficient for effectively isolating useful task directions from interfering components. In this paper, we propose a sparse-representation-based merging framework that uses Sparse Autoencoders (SAEs) to project task vectors into a high-dimensional sparse feature space, enabling feature-level disentanglement before fusion. To reduce computational overhead, we further introduce a lightweight Group-Ranked Zeroth-Order Optimizer (GR-ZOO) to identify task-critical layers for selective merging. Experiments on both Qwen2.5-1.5B and Qwen2.5-7B demonstrate that our method consistently outperforms representative baselines, including Task Arithmetic, TIES-Merge, DARE, Fisher-Merge,and several recent training-free merging methods, across mathematical reasoning, code generation, instruction following, and general knowledge tasks. In a highly conflicting four-task setting on Qwen2.5-1.5B, our method further achieves a 2.78% improvement over the strongest baseline.
Figures & tables
Figure 1: Overall architecture of our proposed High-Dimensional Sparse Disentanglement Merging framework. The pipeline first extracts task vectors from expert models and applies GR-ZOO to identify task-critical layers. Non-critical layers are merged through arithmetic addition, while critical layers are projected into a high-dimensional sparse space via SAE-based feature disentanglement, enabling interference-free fusion of overlapping task features.
Methods
Math (GSM8k)
Code (HumanEval+)
IF (IFEval)
Average
Task Arithmetic
83.85
73.80
45.47
67.71
TIES-Merge
83.40
75.00
38.45
65.62
DARE+TIES
83.62
72.60
36.78
64.33
Fisher-Merge
84.00
75.60
41.59
67.06
TSV-Merge
83.32
72.88
45.77
67.32
WUDI-Merging
82.79
73.49
44.74
67.01
Table 1: Main multi-task merging results on Qwen2.5-7B.
Methods
Math
General
Code
Safe
Average
GSM8k
STEM
Social Sciences
Humanities
Others
HumanEval
BeaverTail
Task Arithmetic
33.74
8.91
9.39
7.48
7.00
0.00
41.60
15.45
TIES-Merge
51.10
19.48
15.70
14.73
11.75
32.93
61.05
29.53
DARE+TIES
52.16
20.44
14.85
13.62
11.14
31.71
49.91
27.69
Fisher-Merge
60.80
22.13
6.47
10.74
5.74
36.59
39.00
25.92
TSV-Merge
52.00
30.10
33.30
29.70
38.80
15.00
29.00
32.24
Table 2: Performance scaling to 4 tasks on Qwen2.5-1.5B. Average denotes the unweighted arithmetic mean over the seven reported evaluation metrics: GSM8K, four General capability domains, HumanEval, and BeaverTail. The same aggregation rule is applied consistently to all methods.
Figure 2: Gradient estimate variance comparison: GR-ZOO effectively smooths the variance compared to standard ZOO.
Figure 3: Performance comparison of different layer selection strategies. GR-ZOO closely approximates the theoretical upper bound (Gradient) while significantly outperforming the Random baseline.
Component
Variant
Metric Avg.
Projection Space
PCA
30.80
Random Projection
31.01
SAE
31.56
Layer Selection
First Layers
28.27
Random Selection
27.64
Fisher Selection
29.90
Table 3: Average performance of different projection spaces and layer-selection strategies.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Distribution of element-wise parameter differences Δw=wOurs−wTA across selected parameter indices. The distributions are centered near zero, while their spike-like structures suggest sparse and selective parameter calibration.
Item
Setting
Candidate / selected layers
197 / 16
Fraction of layers receiving SAE processing
8.12%
Number of trained SAEs
16
Cold GR-ZOO time
467.13s / 7.79min
Single-layer SAE training and latent encoding
366.75s / 6.11min
Peak GPU memory
25,153.93 MB
Appendix
Table 4: Computational cost and training configuration of GR-ZOO and SAE. SAE cost is measured for one selected layer.
Metric
Raw
SAE
Pairwise active overlap ↓
0.3368
0.0983
Active Jaccard ↓
0.2027
0.0518
Task-specific active ratio ↑
29.40%
73.84%
Shared active ratio ↓
70.60%
26.16%
Shared feature ratio ↓
50.96%
14.52%
Appendix
Table 5: Disentanglement analysis between raw task vectors and SAE latent representations. SAE reduces activation overlap and increases task-specific feature allocation.
K
GSM8K
HumanEval
MMLU
Avg.
8
42.05±0.42
37.60±0.70
12.08±0.53
30.58±0.25
16
42.03±0.85
36.18±0.93
13.68±1.64
30.63±0.40
24
41.82±1.03
37.20±2.44
12.63±0.11
30.55±1.09
32
42.36±0.42
36.18±1.27
13.31±1.88
30.61±0.99
64
41.93±1.18
37.40±1.27
12.75±1.76
30.69±0.75
Appendix
Table 6: Sensitivity to the Top- K sparsity level with the similarity threshold fixed at τ=10−4 . IFEval-S denotes the strict IFEval score.
τ
GSM8K
HumanEval
MMLU
Avg.
0
42.13±0.23
35.57±1.41
13.19±1.68
30.29±0.61
10−5
42.15±0.23
36.38±1.41
12.14±0.70
30.22±0.57
10−4
42.36±0.42
36.18±1.27
13.31±1.88
30.61±0.99
10−3
42.05±0.29
36.59±1.61
13.06±1.48
30.57±0.92
Appendix
Table 7: Sensitivity to the similarity threshold τ with the sparsity level fixed at K=32 . IFEval-S denotes the strict IFEval score.
Method
Math
General
Code
Metric Avg.
GSM8K
STEM
Social Sciences
Humanities
Others
HumanEval
Task Arithmetic
53.14
33.63
46.99
33.79
42.91
14.63
37.52
TIES-Merge
51.93
30.35
29.90
27.97
26.77
31.10
33.00
DARE+TIES
56.02
29.26
28.34
25.80
25.29
32.93
32.94
Fisher-Merge
48.45
21.40
10.63
13.82
8.76
37.20
23.38
L1 SAE
54.89
35.02
43.68
31.99
36.67
33.54
39.30
Appendix
Table 8: Detailed results for 3-task merging on Qwen2.5-1.5B. Compared with the standard L1 -regularized SAE, our complete SAE design achieves a higher overall metric average while maintaining a better balance across Math, General, and Code capabilities. Metric Avg. denotes the unweighted mean over the six displayed evaluation metrics.
Method
Math
General
Code
Safety
Metric Avg.
GSM8K
STEM
Social Sciences
Humanities
Others
HumanEval
BeaverTail
Task Arithmetic
33.74
8.91
9.39
7.48
7.00
0.00
41.60
15.45
TIES-Merge
51.10
19.48
15.70
14.73
11.75
32.93
61.05
29.53
DARE+TIES
52.16
20.44
14.85
13.62
11.14
31.71
49.91
27.69
Fisher-Merge
60.80
22.13
6.47
10.74
5.74
36.59
39.00
25.92
L1 SAE
51.18
27.44
33.44
22.51
26.22
32.93
52.97
35.24
Appendix
Table 9: Detailed results for 4-task merging on Qwen2.5-1.5B. Compared with the standard L1 -regularized SAE, our complete SAE design improves the overall metric average and better preserves General capabilities under increased cross-task interference. Metric Avg. denotes the unweighted mean over the seven displayed evaluation metrics.
Component
Variant
GSM8K
HumanEval
IFEval
Metric Avg.
Projection Space
PCA
41.09
35.98
15.34
30.80
Random Projection
40.94
38.41
13.68
31.01
SAE
42.76
36.59
15.34
31.56
Layer Selection
First Layers
42.84
28.66
13.31
28.27
Random Selection
40.94
28.66
13.31
27.64
Fisher Selection
43.06
31.10
15.53
29.90
Appendix
Table 10: Detailed per-task results for the projection-space and layer-selection ablations. For the projection-space comparison, all methods use the same selected layers and fusion procedure. For the layer-selection comparison, all strategies select the same number of layers and use identical SAE and fusion configurations. Metric Avg. denotes the unweighted mean over GSM8K, HumanEval, and IFEval.
Variant
FVU ↓
Dead Features (%) ↓
Post-Top- K Active (%) ↓
Pre-Top- K Active (%)
Norm Std. ↓
Atom Coherence ↓
Full SAE
0.1528
0.0488
8.33
20.22
0.0000
0.0176
w/o Top- K
0.0212
0.0000
35.76
35.76
0.0000
0.0176
w/o Residual Fitting
0.1539
0.0000
8.33
26.83
0.0000
0.0176
w/o Decoder Normalization
0.1565
0.2441
8.33
18.25
0.0101
0.0176
w/o Orthogonality
0.0110
1.5299
8.33
14.39
0.0000
0.0224
w/o Normalization and Orthogonality
0.0107
5.1921
8.33
13.23
0.0094
0.0224
Appendix
Table 11: Representation-level diagnostics of the individual SAE components. Lower reconstruction error does not necessarily indicate better feature separation, since a dense or redundant representation may obtain lower FVU while exhibiting higher activation density, atom coherence, or dead-feature rates.
Model merging has shown that multitask models can be created by directly combining the parameters of different models that are each specialized on tasks of interest. However, models trained independently on distinct tasks often exhibit interference that degrades the merged model's performance. To solve this problem, we formally define the notion of Cross-Task Interference as the drift in the representation of the merged model relative to its constituent models. Reducing cross-task interference is key to improving merging performance. To address this issue, we propose our method, Resolving Interference (RI), a light-weight adaptation framework which disentangles expert models to be functionally orthogonal to the space of other tasks, thereby reducing cross-task interference. RI does this whilst using only unlabeled auxiliary data as input (i.e., no task-data is needed), allowing it to be applied in data-scarce scenarios. RI consistently improves the performance of state-of-the-art merging methods by up to 3.8% and generalization to unseen domains by up to 2.3%. We also find RI to be robust to the source of auxiliary input while being significantly less sensitive to tuning of merging hyperparameters. Our codebase is available at: https://github.com/pramesh39/resolving_interference
Pratik Ramesh, George Stoica, Arun Iyer +2
Georgia Tech · University of Washington · Microsoft Research +3
Model merging has emerged as a promising paradigm for enabling multi-task capabilities without additional training. However, traditional basic merging methods often experience performance degradation due to parameter conflicts, even when applied to similar tasks. While recent personalized merging frameworks successfully preserve task-specific information to maintain performance, they typically incur storage overhead. In this paper, we propose Decomposition, Thresholding, and Scaling (DTS), an approximation-based personalized merging framework that pushes task-specific storage efficiency. DTS first applies singular value decomposition to the task-specific information and retains only a small subset of singular values and vectors. It then introduces a novel thresholding strategy that partitions singular vector elements into groups and assigns a scaling factor to each group. To enable generalization to unseen tasks, we further extend DTS with a variant that fuses task-specific information in a data-free manner based on the semantic similarity of task characteristics. Extensive experiments demonstrate that DTS consistently outperforms state-of-the-art baselines while requiring only 1% extra storage per task. Furthermore, experiments on unseen tasks show that the DTS variant achieves significantly better generalization performance. Our code is available at https://github.com/krumpguo/DTS.
Kuangpu Guo, Aijing Yu, Jian Liang +4
University of Science and Technology of China · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · Institute of Information Engineering, Chinese Academy of Sciences +3
Model merging aims to integrate multiple task-specific fine-tuned models derived from a shared pre-trained checkpoint into a single multi-task model without additional training. Despite extensive research, task interference remains a major obstacle that often undermines the performance of merged models. In this paper, we propose ESM (Essential Subspace Merging) , a robust framework for effective model merging. We begin by performing Principal Component Analysis (PCA) on feature shifts induced by parameter updates. The resulting principal directions span an essential subspace that dominantly influences feature representations. Each task's parameter update matrix is projected onto its respective essential subspace for low-rank decomposition before merging. This methodology mitigates inter-task interference while preserving core task-specific functionality. Furthermore, we introduce a multi-level polarized scaling strategy that amplifies parameters containing critical knowledge and suppresses redundant ones, preventing essential knowledge from being overwhelmed during fusion. Extensive experiments across multiple task sets and model scales demonstrate that our method achieves state-of-the-art performance in multi-task model merging. Code is available at https://github.com/kiddo127/ESM.
Longhua Li, Lei Qi, Qi Tian +1
School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Huawei Technologies, Shanghai, China