When Are Nearby Experts Composable? Cross-Task Model Merging in Neural Thickets
Organizations: University of Pennsylvania · Independent Researcher · University of Electronic Science and Technology of China · Chongqing University
Abstract
Random perturbations around a shared checkpoint can produce experts for distinct tasks, but shared origin does not ensure that their gains survive composition. We formulate expert-pair selection for joint base-relative retention, measured by strict joint success and worst-task gain with unconditional abstention accounting. An exact finite-difference decomposition and an arbitrarily small orthogonal counterexample show why proximity and orthogonality alone are insufficient. We instantiate this formulation with quality-constrained spectral selection, a fixed-budget framework combining cross-task screening, layerwise pair geometry, separate validation, and Base fallback. Across six checkpoints, 21 task pairs, and five independent perturbation pools, the procedure improves strict joint success over Top-1 averaging throughout and leads the four-method model-scale comparison on four of six checkpoints. For the five models with at least 1.5B parameters, matched quality-only, global-cosine, and spectral ranking attain 49.1%, 50.5%, and 53.1% joint success. Under this matched protocol, the spectral selector also yields the highest worst-task gain and Weak Pareto rate and the lowest regression rate. Mean worst-task gain reaches +0.61 percentage points, while 17.1% regression and 13.0% abstention show that composability remains conditional. Project page: https://anonymous.4open.science/r/Thickets-Merge/.
Figures & tables
| SJ (%) | WG (pp) | |||||||
|---|---|---|---|---|---|---|---|---|
| Model | Top-1 | TIES | Direct | Ours | Top-1 | TIES | Direct | Ours |
| Qwen2.5-0.5B | 11.4 | 23.8 | 21.9 | 25.7 | -1.02 | -0.28 | -0.12 | -0.06 |
| Qwen2.5-1.5B | 18.1 | 33.3 | 38.1 | 42.9 | -0.71 | +0.15 | +0.23 | +0.42 |
| Qwen2.5-3B | 26.7 | 44.8 | 49.5 | 55.2 | -0.34 | +0.31 | +0.43 | +0.69 |
| Qwen2.5-7B | 33.3 | 52.4 | 61.0 | 59.0 | -0.12 | +0.48 | +0.77 | +0.72 |
| Llama-3.1-8B | 28.6 | 47.6 | 50.5 | 57.1 | -0.22 | +0.36 | +0.44 | +0.65 |
| Method | SJ | WG | Weak Pareto | Mean gain | Regression | Abstain |
|---|---|---|---|---|---|---|
| Top-1 Avg. | 26.9 | -0.33 | 42.1 | +0.36 | 57.9 | 0.0 |
| TIES | 46.3 | +0.38 | 61.3 | +1.09 | 38.7 | 0.0 |
| Direct Search | 49.7 | +0.47 | 64.6 | +1.28 | 35.4 | 0.0 |
| Quality-only, | 49.1 | +0.49 | 66.1 | +1.39 | 20.4 | 13.5 |
| Global Cosine | 50.5 | +0.54 | 67.6 | +1.48 | 19.0 | 13.3 |
| Ours | 53.1 | +0.61 | 69.9 | +1.57 | 17.1 | 13.0 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Component | Setting |
|---|---|
| Candidate population | per independent pool |
| Retained experts | Up to per task; |
| Pair shortlist | Up to distinct expert pairs |
| Coefficient grid | , , , , |
| Merge validation budget | At most proposals per task pair |
| Spectral coefficient | for every primary configuration |
| Case | Model, merge pair, and item | Base | Top-1 | Merge |
| S1 | Llama; GSM8K + ROCStories; GSM8K test 360 | 85 | 85 | 25 |
| S2 | Qwen 3B; MBPP + GSM8K; MBPP task 427, test 680 | Date conversion; outputs below | ||
| Input: 2026-01-02 | 02-2026-01 | 02-2026-01 | 02-01-2026 | |
| S3 | Qwen 3B; ROCStories + GSM8K; story/test 223 | EACDB | CEBDA | EBCAD |
| S4 | Qwen 0.5B; MBPP + GSM8K; GSM8K test 616 | 10 | 12 | 10 |
| S5 | Qwen 3B; ROCStories + GSM8K; story/test 334 | ABDEC | ADEBC | ABDEC |
| Comparison | Both succeed | Ours only succeeds | Comparator only succeeds | Both fail | SJ (pp) | WG (pp) |
|---|---|---|---|---|---|---|
| Ours vs. Quality-only | 246 | 33 | 12 | 234 | +4.0 | +0.12 |
| Ours vs. Global Cosine | 251 | 28 | 14 | 232 | +2.7 | +0.07 |
| Model | Quality-only SJ | Global Cosine SJ | Ours SJ | Ours Cosine (pp) |
|---|---|---|---|---|
| Qwen2.5-1.5B | +2.9 | |||
| Qwen2.5-3B | +2.9 | |||
| Qwen2.5-7B | +2.9 | |||
| Llama-3.1-8B | +4.8 | |||
| OLMo 3-7B | 0.0 | |||
| Total | +2.7 |
| Factor | Setting | SJ (%) | WG (pp) | Regression (%) |
| (Quality-only) | 49.1 | +0.49 | 20.4 | |
| 0.25 | 51.8 | +0.56 | 18.7 | |
| 0.50 | 53.1 | +0.61 | 17.1 | |
| 0.75 | 52.8 | +0.60 | 17.5 | |
| Rank cap | 4 | 52.4 | +0.58 | 17.9 |
| Rank cap | 8 | 53.1 | +0.61 | 17.1 |
Model Merging
Model merging combines trained model weights or adapters to reuse capabilities while reducing the cost of joint training or ensembles.
See all 214 papers