Random perturbations around a shared checkpoint can produce experts for distinct tasks, but shared origin does not ensure that their gains survive composition. We formulate expert-pair selection for joint base-relative retention, measured by strict joint success and worst-task gain with unconditional abstention accounting. An exact finite-difference decomposition and an arbitrarily small orthogonal counterexample show why proximity and orthogonality alone are insufficient. We instantiate this formulation with quality-constrained spectral selection, a fixed-budget framework combining cross-task screening, layerwise pair geometry, separate validation, and Base fallback. Across six checkpoints, 21 task pairs, and five independent perturbation pools, the procedure improves strict joint success over Top-1 averaging throughout and leads the four-method model-scale comparison on four of six checkpoints. For the five models with at least 1.5B parameters, matched quality-only, global-cosine, and spectral ranking attain 49.1%, 50.5%, and 53.1% joint success. Under this matched protocol, the spectral selector also yields the highest worst-task gain and Weak Pareto rate and the lowest regression rate. Mean worst-task gain reaches +0.61 percentage points, while 17.1% regression and 13.0% abstention show that composability remains conditional. Project page: https://anonymous.4open.science/r/Thickets-Merge/.
Figures & tables
Figure 1: From a shared checkpoint to a validated two-expert merge. Screen candidates, rank distinct expert pairs, and validate at most 10×5=50 merges. Screening permits ties with Base; validation requires both gains >0 . Quality entries denote base-relative selection gains. Neighborhood and overlap are schematic; weak Pareto includes strict joint success. The best feasible merge or Base proceeds to held-out testing, without guaranteed improvement.
SJ ↑ (%)
WG ↑ (pp)
Model
Top-1
TIES
Direct
Ours
Top-1
TIES
Direct
Ours
Qwen2.5-0.5B
11.4
23.8
21.9
25.7 ↑1.9
-1.02
-0.28
-0.12
-0.06 ↑0.06
Qwen2.5-1.5B
18.1
33.3
38.1
42.9 ↑4.8
-0.71
+0.15
+0.23
+0.42 ↑0.19
Qwen2.5-3B
26.7
44.8
49.5
55.2 ↑5.7
-0.34
+0.31
+0.43
+0.69 ↑0.26
Qwen2.5-7B
33.3
52.4
61.0
59.0 ↓2.0
-0.12
+0.48
+0.77
+0.72 ↓0.05
Llama-3.1-8B
28.6
47.6
50.5
57.1 ↑6.6
-0.22
+0.36
+0.44
+0.65 ↑0.21
Table 1: Cross-task composability across model scales. Each model covers 21 task pairs and five independent pools (105 outcomes). SJ: strict joint success (%); WG: mean worst-task gain (pp). Bold marks metric-wise row bests. Small arrow values give Ours minus the best of Top-1, TIES, and Direct for each model and metric (pp), computed from displayed entries.
Figure 2: More joint successes do not guarantee positive worst-task gain. Model-wise SJ and WG for four methods. Each row summarizes 21 task pairs and five pools. Our method improves on Top-1 throughout, but Direct Search leads Qwen 7B and TIES leads OLMo 3. No uncertainty intervals are inferred from these model-level aggregates.
Method
SJ ↑
WG ↑
Weak Pareto ↑
Mean gain ↑
Regression ↓
Abstain ↓
Top-1 Avg.
26.9
-0.33
42.1
+0.36
57.9
0.0
TIES
46.3
+0.38
61.3
+1.09
38.7
0.0
Direct Search
49.7
+0.47
64.6
+1.28
35.4
0.0
Quality-only, λ=0
49.1
+0.49
66.1
+1.39
20.4
13.5
Global Cosine
50.5
+0.54
67.6
+1.48
19.0
13.3
Ours
53.1 ↑2.6
+0.61 ↑0.07
69.9 ↑2.3
+1.57 ↑0.09
17.1 ↓1.9
13.0
Table 2: Aggregate comparison for the five ≥ 1.5B models. 525 pair–pool evaluations per method; all metrics include abstentions. Rates are in %; gains are in pp. Bold marks columnwise optima. Small arrow values, omitted for Abstain, give Ours minus the best comparator per metric (pp), from displayed entries; green/red denotes better/worse.
Figure 3: Abstention is part of the deployed outcome. All 525 primary evaluations partition into 279 strict successes, 88 weak-only improvements, 90 regressions, and 68 Base fallbacks. Weak Pareto includes the first two categories. Any conditional rates exclude the 68 fallbacks and use denominator 457; primary comparisons always retain denominator 525.
Figure 4: Joint gains persist across five Qwen2.5-3B pools. Each pool evaluates the same 21 task pairs under the fixed protocol. Points show the individual pools; summaries are mean ± sample SD, not confidence intervals. Pool-level variation addresses candidate-search randomness and does not quantify all test-sample uncertainty.
Figure 5: Corrections and losses in an auxiliary coefficient analysis. Qwen2.5-3B, GSM8K–MBPP, pool 42; all 1,319 GSM8K test items. The selected pair is additionally evaluated at (αGSM,βMBPP)=(0.75,0.50) , outside the five-point primary grid. This run contributes to neither main table. Specialist denotes the GSM8K expert participating in this merge.
Figure 6: Selected qualitative examples of equal-weight merging. S1–S3 correct displayed shared errors; S4 preserves Base correctness. These separately collected 0.5/0.5 examples illustrate behaviors and are distinct from both the primary comparison and the auxiliary coefficient run in Figure 5 .
Table 3: Fixed primary configuration. Settings apply across model families, task pairs, and pools. Gains and thresholds use percentage points (pp).
Case
Model, merge pair, and item
Base
Top-1
Merge
S1
Llama; GSM8K + ROCStories; GSM8K test 360
85
85
25
S2
Qwen 3B; MBPP + GSM8K; MBPP task 427, test 680
Date conversion; outputs below
Input: 2026-01-02
02-2026-01
02-2026-01
02-01-2026
S3
Qwen 3B; ROCStories + GSM8K; story/test 223
EACDB
CEBDA
EBCAD
S4
Qwen 0.5B; MBPP + GSM8K; GSM8K test 616
10
12
10
S5
Qwen 3B; ROCStories + GSM8K; story/test 334
ABDEC
ADEBC
ABDEC
Appendix
Table 4: Case identities and observed answers. Test indices are zero-based within the supplied archive; task and story identifiers are retained as recorded. Qwen denotes Qwen2.5-Instruct; Llama denotes Llama-3.1-8B-Instruct. S2 shows outputs from executing the generated code on the first displayed test, rather than literal model replies. Letter strings preserve the complete story order.
Comparison
Both succeed
Ours only succeeds
Comparator only succeeds
Both fail
Δ SJ (pp)
Δ WG (pp)
Ours vs. Quality-only
246
33
12
234
+4.0
+0.12
Ours vs. Global Cosine
251
28
14
232
+2.7
+0.07
Appendix
Table D.1: Configuration-level paired comparison of strict joint success. Each row partitions all 525 matched model–task-pair–pool configurations. “Ours only” denotes a strict joint success for the spectral selector when the comparator fails; “Comparator only” denotes the reverse. Delta SJ is computed from the underlying counts.
Model
Quality-only SJ
Global Cosine SJ
Ours SJ
Ours − Cosine (pp)
Qwen2.5-1.5B
40/105=38.1%
42/105=40.0%
45/105=42.9%
+2.9
Qwen2.5-3B
53/105=50.5%
55/105=52.4%
58/105=55.2%
+2.9
Qwen2.5-7B
57/105=54.3%
59/105=56.2%
62/105=59.0%
+2.9
Llama-3.1-8B
54/105=51.4%
55/105=52.4%
60/105=57.1%
+4.8
OLMo 3-7B
54/105=51.4%
54/105=51.4%
54/105=51.4%
0.0
Total
258/525=49.1%
265/525=50.5%
279/525=53.1%
+2.7
Appendix
Table D.2: Model-wise strict joint success for the three matched pair-ranking variants. Each model contains 21 task pairs and five perturbation pools, for 105 configurations. The final column reports the count-based difference between Ours and Global Cosine.
Factor
Setting
SJ (%)
WG (pp)
Regression (%)
λ
0 (Quality-only)
49.1
+0.49
20.4
λ
0.25
51.8
+0.56
18.7
λ
0.50
53.1
+0.61
17.1
λ
0.75
52.8
+0.60
17.5
Rank cap
4
52.4
+0.58
17.9
Rank cap
8
53.1
+0.61
17.1
Appendix
Table D.3: One-factor-at-a-time robustness analysis. Unless the indicated factor is changed, all remaining settings are held at the primary configuration. SJ and Regression are percentages; WG is in percentage points.
Multi-task model merging aims to consolidate several task-specific experts into a unified model, yet static merging consistently suffers from parameter interference. While dynamic merging models aim to bridge this gap, many works rely on the costly storage and loading of redundant expert components at inference. In this work, from the perspective of task expert, we view parameter interference as parameter perturbation introduced to each expert during merging process. We show that such parameter perturbations can be modeled as affine transformation, which can be approximated as additive offsets. Motivated by these, we propose Recover Task eXpert (ReTeX), a framework that predicts those offsets, in order to undo parameter interference and recover task-expert performance from a single merged checkpoint. To recover the appropriate expert when task identity is unknown, we introduce a router-free task identifier based on SVD subspace signatures computed offline before inference. At inference, the identifier selects the task whose subspace yields the smallest projection residual for a given input. As a result, ReTeX recovers over 95% of individual-expert performance in both vision and NLP domains, while significantly improving generalization to unseen tasks. Crucially, we also show that the parameter offset prediction leads to emergent adaptive interpolation of expert knowledge for out-of-distribution (OOD) tasks. ReTeX adaptively interpolates seen expert knowledge to handle unseen tasks. Our code is available at https://github.com/BAIKLAB/ReTeX
Jinwook Jung, Taegyu Kim, Kumju Jo +1
Department of Artificial Intelligence, 2Department of Data Science Hanyang University
Pretrained models are typically viewed as generalists, requiring bespoke post-training to achieve task-specific expertise. Gan & Isola (2026) recently proposed a striking alternative: that pretrained models lie in a dense neural thicket of task specialists, which can be surfaced with simple Gaussian weight perturbations. We examine this neural thicket of task-specific experts and identify several key findings that question this perspective. First, we find that the apparent lift of the best-of-N selected perturbation decays quickly with selection-set size m, strongly suggesting overfitting to the selection questions. In fact, for 10 of the 12 model-task pairs we study, a simple null model of pure selection noise predicts more than 75 percent of the top expert's reward lift. Second, the top experts' advantage often vanishes on held-out questions or remains within the run-to-run variability of the unperturbed pretrained model. Third, we find that majority voting over the top-K selected perturbations, termed RandOpt(N, K) by Gan & Isola (2026), is often matched or outperformed by majority voting over K samples from the base model. Overall, our results suggest that random weight perturbations may be ineffective at uncovering task experts around pretrained models, and that task adaptation requires more sophistication than random search. Code and artifacts are available at github.com/asil012/RevisitingNeuralThickets.
Model merging consolidates fine-tuned experts into one multi-task model without retraining. All existing data-free methods approach this problem entirely in weight space. Restricted to arithmetic on parameters, these methods never observe how each expert behaves, a signal that only emerges through forward evaluation. Accessing this behavioral signal requires inputs to evaluate on, which the data-free setting prohibits. We propose Merge++, a post-hoc method that addresses this by inverting the expert checkpoints to synthesize task-representative images, then distilling expert knowledge into the merged model using those images. Merge++ requires no additional data beyond the checkpoints themselves. It applies universally across merging algorithms and operates as a complementary refinement stage independent of the underlying weight-space method. The method consistently improves merging algorithms ranging from simple task arithmetic to state-of-the-art spectral methods, with average gains of +2 to +8 points and up to +25.9 on individual configurations.