Model merging combines separately fine-tuned models by adding their task vectors to a shared backbone. Interference is typically assessed only after constructing and evaluating a candidate merge, making coefficient search expensive. We show that a first-order directional derivative of task A's outputs along task B's vector can predict interference, but only when evaluated at the appropriate model. At the receiver model, this derivative forecasts nearby merges per task and per example, halves the accuracy error of a no-change baseline, and identifies most examples whose predictions flip. For full multi-task merges, a single evaluation is insufficient; two evaluations along the merge path are needed. At the shared pretrained backbone, however, the derivative is nearly orthogonal to realized changes in our vision experiments and only partly predictive in LoRA language-model pilots. Controls link this failure to displacement along the receiver's own task vector, not simply parameter distance; the mismatch appears within the first few dozen fine-tuning steps. Freezing receiver-local responses produces a surrogate affine in the merge coefficients. We use it to construct LR-Merge, a label-free correction that optimizes coefficients through convex teacher matching on unlabeled probes, without backpropagation through merged networks. LR-Merge improves all tested closed-form, isotropic, and sparsified base mergers, with gains generally increasing as more tasks are merged. Using pretrained rather than receiver-local responses instead degrades the base merge. LR-Merge matches exact teacher matching through a merged network, which takes roughly three times longer to converge and performs similarly at an equal compute budget. Once built, the surrogate answers new coefficient and task-subset queries in seconds.
Figures & tables
Figure 1: Interference is receiver-local. zA(θ) : logits on task A ’s probes; JA=∂zA/∂θ ; τB : task B ’s vector. Black: realized drift zA(θA+ατB)−zA(θA) ; blue: forecast at the receiving model, αJA(θA)τB ; orange: forecast at the pretrained model, αJA(θ0)τB , which points in a nearly unrelated direction (formal setup in Section 3 ).
acc. MAE (pp)
α
forecast
no change
flip recall
flip prec.
damage ρ
drift cos
0.2
0.13
0.26
0.82
0.79
0.917
0.946
0.4
0.30
0.60
0.75
0.68
0.824
0.905
Table 1: Pairwise forecast, 30 ordered pairs, checkpoint set 0 , three probe seeds (other sets in Table 9 ). Flips are rare at α=0.2 ( 1.0% of probe–pair cases, and 17 of 30 pairs have none). Rate-matched label-free low-margin and random flaggers reach precision 0.57 and 0.06 ( 0.42 , 0.11 at α=0.4 ). At α=0.4 the per-pair accuracy drop correlates at 0.75 with the forecast and at 0.12 with task-vector cosine.
one JVP at θA
two JVPs on the path (midpoint)
one JVP at θ0
merger
cos
damage ρ
acc MAE
cos
damage ρ
acc MAE
cos
damage ρ
RegMean
0.756
0.668
4.0
0.978
0.973
1.1
-0.209
-0.004
Iso-C
0.793
0.537
3.5
0.989
0.978
2.1
0.160
0.104
Iso-CTS
0.788
0.470
5.3
0.986
0.971
2.8
0.044
0.039
TIES
0.674
0.479
11.8
0.959
0.950
3.2
0.191
0.243
Task arithmetic
0.721
0.509
11.3
0.977
0.964
3.1
0.243
0.207
Table 2: Forecasting complete merges ( s=1 ) produced by five base mergers: six receivers, mean over three checkpoint sets. One JVP is too coarse at this distance, and two along the path are accurate. At s=0.2 one JVP at θA reaches cosine 0.90 – 0.98 for every merger, against −0.14 to 0.17 at θ0 .
anchor
what it tests
mean cos
min cos
dist. from θA
disp. along τA
θA (receiving model)
the forecast as used
0.949 ± 0.008
0.296
0
0
isotropic, matched distance
is distance the cause?
0.945 ± 0.009
0.315
∥τA∥
0.00
isotropic, 1.41× distance
farther still random
0.942 ± 0.009
–
1.41∥τA∥
0.00
span of other task vectors
is the direction structured?
0.665 ± 0.036
−0.320
∥τA∥
0.07
toward θC , matched distance
third model, confound removed
0.461 ± 0.014
−0.007
∥τA∥
0.67
isotropic tilted 45∘ to −τA
direction dose–response
0.352 ± 0.018
–
∥τA∥
0.71
Table 3: Forecast drift cosine by anchor: 30 ordered pairs at α=0.2 , mean ± sd over six checkpoint sets, min = worst pair. Last column: ∣⟨d,τA⟩∣/∥τA∥2 for anchor displacement d from θA . This displacement orders the nine anchors (Spearman −0.98 , p=3×10−5 ), and distance from θA does not ( −0.16 , p=0.70 ).
Figure 2: Cross-task responses re-aim at the start of fine-tuning. (A) Angle between JA(θA(t))τB and JA(θ0)τB along the receiver’s fine-tuning. (B) The θ0 -anchored forecast collapses over the same window, while the forecast at the current checkpoint keeps rising. Receivers SVHN and GTSRB, five sources each, τB fixed at the source’s final checkpoint, α=0.2 , no labels. Lines are means over pairs and three fine-tuning seeds, and the band is the across-seed spread ( t≤500 , seed 0 beyond).
suite (backbone)
checkpoints
base merger
base
+ LR-Merge
Δ
canonical eight (B/32)
ours, 3 sets
RegMean
80.70 ± 0.04
82.73 ± 0.12
+ 2.03 ± 0.08
Iso-C
83.90 ± 0.22
85.36 ± 0.03
+ 1.46 ± 0.19
Iso-CTS
84.23 ± 0.07
86.08 ± 0.06
+ 1.85 ± 0.09
six-task (B/32)
ours, 6 sets
RegMean
87.49 ± 0.22
90.58 ± 0.20
+ 3.10 ± 0.37
Iso-C
88.78 ± 0.35
91.28 ± 0.27
+ 2.50 ± 0.31
Iso-CTS
88.15 ± 0.38
91.49 ± 0.23
+ 3.34 ± 0.49
Table 4: LR-Merge as a correction of base mergers (average accuracy, %). Rows with one checkpoint set are pilots without across-set spread. The correction cost excludes the base merger (Iso-C: 54 s). Iso-C, Iso-CTS and TIES select their global scale by labeled probe accuracy, while RegMean, DOGE and LR-Merge use no labels. Per-set values and the eight-task suite with CIFAR-10 in Appendix D .
correction of the same RegMean base
Δ accuracy
paired vs. LR-Merge
wall-clock (s)
sets
six-task suite
exact teacher matching, converged
+ 3.24 ± 0.23
+0.14±0.24
3447 ± 78
6
exact teacher matching, stopped at LR-Merge’s budget
+ 3.18 ± 0.30
+0.08±0.14
≈900
6
AdaMerging objective on the same base, converged
+ 3.13 ± 0.52
+0.03±0.22
3432 ± 32
6
coordinate descent on the same objective, matched time
+ 2.87 ± 0.27
−0.41±0.28
964
3
LR-Merge , exact responses
+ 3.10 ± 0.37
–
952 ± 15
6
Table 5: Correcting the same RegMean base by different optimizers. Paired differences are (row − LR-Merge) over the listed checkpoint sets.
Figure 3: (A) Accuracy vs unlabeled probe budget on identical pools: RegMean built from the same probes (dashed), LR-Merge on that RegMean, and standalone layer-wise AdaMerging (values in Table 13 ). (B) Cumulative cost of repeated coefficient queries. At twenty queries LR-Merge is 39 – 69× cheaper. Mean ± sd over three checkpoint sets.
arm
cos @ receiver
cos @ θ0
CLIP ViT-B/32 (six sets)
0.949
0.122
CLIP ViT-B/32, canonical eight ( 56 pairs)
0.975
0.180
CLIP ViT-B/16
0.966
0.152
CLIP ViT-L/14
0.924
0.090
RoBERTa-base/LoRA (3 GLUE tasks)
0.77
0.11
Pythia-410M/LoRA (5 tasks)
0.92
0.23–0.68
Table 6: Forecast drift cosine at the receiver vs at θ0 (vision at α=0.2 ; language in teacher-forced logit space, LoRA fine-tunes). Language rows are pilots.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
measurement
value
Section 2 : related approaches
single-task accuracy, fully linearized vs ordinary fine-tunes (six tasks)
87.5 vs 93.8
ranking all 57 task subsets by merged accuracy, one frozen build
Spearman 0.88±0.03
ranking pairwise interference: gradient-mismatch bound, receiver-side term / full mismatch / our forecast
0.66 / 0.34 / 0.75
Section 3 : forecast
flip precision at α=0.2 , 95% CI
0.79 , [0.72,0.91]
Appendix
Table 7: Additional quantitative results, grouped by the main-text section they support. ViT-B/32, six-task suite unless noted.
β
α=0.2
α=0.4
α=0.7
α=1.0
0.3
0.83 / 0.42 / 0.29
0.69 / 0.43 / 0.23
0.51 / 0.45 / 0.15
0.37 / 0.45 / 0.05
0.5
0.91 / 0.66 / 0.21
0.82 / 0.63 / 0.20
0.68 / 0.59 / 0.16
0.56 / 0.55 / 0.10
0.7
0.94 / 0.83 / 0.15
0.88 / 0.79 / 0.15
0.77 / 0.73 / 0.14
0.66 / 0.65 / 0.12
1.0
0.95 / – / 0.11
0.91 / – / 0.12
0.83 / – / 0.11
0.74 / – / 0.11
Appendix
Table 8: Drift cosine for a scaled receiver θ0+βτA : forecast at the actual receiver / at θA / at θ0 , mean over 30 pairs and three probe seeds (seed spread ≤0.02 ). At β=1 the receiver is θA .
checkpoint set
cos @ θA
cos @ θ0
damage ρ
acc. MAE (pp)
0
0.946
0.124
0.916
0.13
1
0.954
0.126
0.925
0.17
2
0.958
0.107
0.927
0.16
3
0.946
0.109
0.916
0.09
4
0.953
0.132
0.930
0.13
5
0.936
0.133
0.916
0.10
Appendix
Table 9: Forecast and anchor gap over six independently fine-tuned checkpoint sets, 30 ordered pairs each at α=0.2 . Last row: the canonical eight tasks, all 56 ordered pairs of one checkpoint set.
responses read at
s0
s1
s2
s3
s4
s5
mean ± sd
standalone selection
current anchor θˉ(r)
80.65
80.18
80.72
79.98
80.83
80.00
80.39 ± 0.38
pretrained θ0
73.77
73.83
73.80
74.00
74.65
73.65
73.95 ± 0.36
paired drop
6.88
6.35
6.92
5.98
6.18
6.35
6.44 ± 0.38
correction of RegMean (base 87.49±0.22 )
current anchor θˉ(r)
90.53
90.68
90.73
90.23
90.80
90.52
90.58 ± 0.20
Appendix
Table 10: Wrong-anchor ablation (six-task, six checkpoint sets): responses read at the current anchor θˉ(r) vs at θ0 , all else fixed, in standalone selection and in the RegMean correction. The correction drop has t(5)=22.2 .
base merger (8-task)
base
+ ours
Δ
sets
cost (s)
RegMean
83.10 ± 0.25
86.19 ± 0.20
+3.09±0.34
6
1685
Iso-C
85.92 ± 0.22
88.28 ± 0.19
+2.36±0.15
6
1700
Iso-CTS
86.18 ± 0.46
88.66 ± 0.22
+2.47±0.52
6
1688
TIES
73.62 ± 0.44
80.62 ± 0.65
+7.00±0.31
6
1693
Appendix
Table 11: Eight-task suite with CIFAR-10 in place of Cars, six checkpoint sets.
T
task arithmetic
RegMean base
RegMean + ours
gain
p
2
90.50 ± 0.39
98.23 ± 0.21
98.28 ± 0.15
+0.05±0.17
0.67
3
86.06 ± 0.48
97.09 ± 0.16
97.41 ± 0.19
+0.32±0.24
0.15
4
81.73 ± 0.25
96.18 ± 0.21
97.03 ± 0.21
+0.85±0.35
0.052
5
75.23 ± 0.12
88.82 ± 0.20
91.05 ± 0.15
+2.23±0.34
0.0077
6
75.80 ± 0.15
87.37 ± 0.25
90.65 ± 0.10
+3.28±0.35
0.0037
7
75.77 ± 0.29
86.53 ± 0.10
89.98 ± 0.15
+3.45±0.25
0.0018
Appendix
Table 12: Average accuracy vs number of merged tasks T on the eight-task suite (tasks added in the order MNIST, SVHN, GTSRB, EuroSAT, DTD, CIFAR-10, RESISC45, SUN397), mean ± sd over three checkpoint sets; task arithmetic at α=0.3 , and p from a paired test of the correction against its base.
probes
RegMean@ b
+ LR-Merge
AdaMerging
gap
LR / Ada (s)
32
86.75 ± 0.22
88.47 ± 0.94
82.86 ± 1.08
+ 5.61
130 / 2378
64
87.18 ± 0.25
89.18 ± 0.61
84.42 ± 0.99
+ 4.76
246 / 3416
128
87.27 ± 0.19
90.18 ± 0.18
85.14 ± 0.75
+ 5.04
480 / 3435
256
87.35 ± 0.26
90.66 ± 0.11
85.41 ± 0.96
+ 5.25
943 / 3437
Appendix
Table 13: Per-budget values behind Figure 3 A, checkpoint sets 0 – 2 . RegMean@ b is built from the same b probes that LR-Merge uses, and AdaMerging is standalone. Building RegMean adds under 5 s.
backbone
params
groups
build (s)
solve (s)
peak GB
cache MB
ViT-B/32
88M
13
201
1.5
6.5
10
ViT-B/16
86M
13
838
1.6
6.8
10
ViT-L/14
304M
25
8704
1.4
19.8
19
Appendix
Table 14: Cost of one response build and one convex solve, six tasks, 256 probes per task, one L40S GPU. ViT-L/14 uses a JVP batch of 8 .
accuracy (6 tasks)
deployed 5 tasks
new task (CIFAR-10)
cost (s)
incremental (new JVPs + re-solve)
89.54 ± 0.34
90.7
83.7 ± 2.4
146 ± 1
+ surrogate re-anchor
90.09 ± 0.33
90.6
87.5 ± 2.0
613
full rebuild (base + correction)
90.65 ± 0.10
89.9
94.5
964
AdaMerging from scratch †
85.2 ± 0.8
–
–
2007
Appendix
Table 15: A sixth task arrives at a deployed five-task corrected merge (three checkpoint sets). † Six-task standalone run of Table 16 , whose per-task CIFAR-10 accuracy is not reported.
method (six-task, from scratch)
avg
sel. cost (s)
Iso-C (closed-form)
88.9 ± 0.3
54
AdaMerging (task-wise)
80.1 ± 0.5
1185
AdaMerging (layer-wise)
85.2 ± 0.8
2007
Exact teacher matching (layer-wise)
85.6
2160
OSP, per-row, all layers (reimpl.; set 0)
70.8
2161
OSP, final MLP only (reimpl.; set 0)
65.9
49
Appendix
Table 16: Standalone selection (six-task, from scratch). LR-Merge matches the optimization-based methods at a fraction of their cost. Closed-form Iso-C is stronger standalone, which is why correction is the intended use. With six re-anchoring rounds LR-Merge reaches 86.17±0.61 .
arm (Pythia-410M/LoRA, five tasks)
held-out NLL
top-1 token acc. (%)
wall (s)
base merge (TA α=0.3 )
2.1154 ± 0.0208
57.10 ± 0.49
–
LR-Merge (ours)
2.0895 ± 0.0225
57.83 ± 0.51
78
exact TM at ours’ budget
2.0935 ± 0.0213
57.89 ± 0.52
≤ 78
exact TM, oracle best step
2.0924 ± 0.0222
57.88 ± 0.53
–
exact TM, final ( 1000 steps)
2.1059 ± 0.0228
57.85 ± 0.48
1227
Appendix
Table 17: Pythia-410M/LoRA correction, mean ± sd over three probe/optimizer draws (one LoRA fine-tune, teacher-forced held-out metrics). Both methods beat the base ( p=0.006 NLL, p=0.0015 token accuracy), and oracle stopping steps differ across draws ( 500/50/100 ).