Consolidating specialized foundation models into a unified generalist is a central goal of post-training. Two approaches are in common use: model merging, which is fast but treated as a heuristic, and multi-teacher on-policy distillation, which matches distributions but demands heavy rollout compute and suffers from cross-task gradient conflict. We show that the two are one objective seen at different orders. Within a shared basin of homologous specialists, the multi-teacher on-policy distillation objective is, to second order, exactly the Fisher-merging objective. Its minimizer is therefore the Fisher merge, a classic model-merging method given by a closed-form combination of the specialists, so merging is optimal, to second order, as a closed-form stage of distillation. Building on this identification, we propose Residual-Governed Branch-and-Merge(RGBM), a two-phase consolidation procedure. Concretely, a full-parameter Fisher merge is intractable on foundation models, so Phase1 restricts the merge to the task-vector subspace. Extending tangent-space analyses of merging, we show that the same coefficients that mix task vectors also mix implicit rewards, which lets a residual program on unlabelled prompts solve for them. The program minimizes the worst per-domain residual and reports its energy-normalized values as the domain residual spectrum. Phase2 lets this spectrum govern what may be added: RGBM passivates domains with low normalized residuals, trains isolated branches only for domains above the spectral gap, and writes back the admitted updates with calibrated shrinkage. On two open-source consolidation suites, the Phase~1 merge alone matches or outperforms fully trained multi-teacher distillation, and RGBM improves on it further with zero extra joint-training steps, whereas every un-gated write-back, including continued joint distillation from the merge, falls below it.
Figures & tables
Figure 1: Merge before you distill (schematic). (a) Weight space: the dashed triangle is the task subspace spanned by the Math, Code, and IF task vectors. A Minimax projection on unlabelled prompts yields the zero-training merge θ0⋆=Wref+Tα⋆ , where T stacks those task vectors, together with the per-domain residuals Ld (red segments). Normalizing these residuals by domain reward energy gives the domain residual spectrum. (b) Capability space: joint distillation from θ0⋆ can leave the basin (dashed path). The normalized spectrum passivates Code and IF; RGBM writes back only the admitted Math branch, shrunk by γ⋆ (the largest factor that still passes the linearity test). See Section 3 for details.
Figure 2: Two-phase RGBM architecture. (1) Phase 1 scores implicit rewards on unlabelled prompts, forms per-domain second-moment matrices Md , and minimizes the worst residual Ld to obtain θ0⋆ . The heatmap is the reward correlation, not Md itself. (2) The bars are the energy-normalized Open-MOPD spectrum L ; the gate cuts at the midpoint of its largest adjacent gap (threshold ϵ ; Eq. 10 ), so only Math is active. (3) Phase 2 writes active updates back with shrinkage γ to obtain θRGBM ; if every domain is passivated, θRGBM=θ0⋆ . See Sections 3.1 and 3.3 for details.
Method
Math
Code
IF
Macro
RouteRL (Oracle Ensemble)
27.24
26.27
53.52
35.68
Multi-Teacher OPD
24.53
21.18
47.12
30.93
Task Arithmetic
24.35
22.81
46.77
31.31
Subspace Fisher Merge
24.11
23.59
47.99
31.90
RGBM Phase 1 (Ours, θ0⋆ )
23.86
24.09
49.60
32.52
Uniform Distillation (from θ0⋆ )
23.97
21.36
47.42
30.91
Table 1: Open-MOPD (SmolLM3-3B; accuracy). Macro: mean of Math, Code, and IF. Bold: best deployable model. Italic: routing oracle. Gated Math: admitted branch only, at the listed γ .
Program
α
Math
Code
IF
Macro
α=1/3
0.333/0.333/0.333
20.81
19.15
44.77
28.24
A scalar L
0.626/0.626/0.626
24.97
23.28
47.64
31.96
A vector L
0.527/0.609/0.821
24.11
23.59
47.99
31.90
C scalar L
0.579/0.579/0.579
24.97
23.49
45.10
31.19
C vector L ( θ0⋆ )
0.597/0.620/0.800
23.86
24.09
49.60
32.52
A scalar L
0.501/0.501/0.501
24.38
23.36
44.86
30.86
Table 2: Open-MOPD coefficient grid. A minimizes the mean residual (uniform pd , Eq. 5 ); C minimizes the worst residual (Eq. 9 ). Scalar shares one coefficient; vector leaves α free. Scores for the four L rows and for α=1/3 are Table 4 ; the four L rows use the same protocol. Coefficients are Math/Code/IF.
Figure 3: Domain score against the raw residual (Open-MOPD). Each panel orders the nine merges of Table 2 by that panel’s score, so the score curve rises by construction. Domain- L is min-max scaled onto that score range. A/C: uniform and minimax. sc/vec: shared scalar and free vector. L / Lb: the program minimizes L or Ld/E[rd2] . lin: α=1/3 . The energy-normalized spectrum is used only for the Phase 2 gate in Section 4.1 .
Figure 4: Joint distillation from θ0⋆ (Open-MOPD, ≤50 steps). (a) Macro under three joint objectives; dashed and dotted lines are RGBM Phase 2 and θ0⋆ . (b) Uniform Distillation: Math stays near the merge before a late drop; Code and IFBench fall. Dashed lines mark RGBM Phase 2 on those axes.
Method
Math
Code
Avg
Domain-Matched Teacher
46.00
61.17
52.50
SFT
44.33
60.80
51.39
OPD
46.38
60.57
52.46
ExOPD
47.65
61.97
53.79
Task Arithmetic
43.05
60.65
50.59
Equal-Weight Average
22.55
55.67
36.75
Table 3: G-OPD (Qwen3-4B). Math / Code: mean of the 4 math / 3 code tasks. Avg: 7-task mean. Italic: teacher oracle.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Linearity diagnostics on the isotropic family ( θ(α)=Wref+α∑dτd ; not the Minimax vector α⋆ ). (a) Macro vs. scalar α . Horizontal lines are RouteRL ( 35.68 ), the 600 -step student ( 30.93 ), and MixSFT ( 25.35 ); the peak at α=0.75 is this scan, not α⋆ . (b) R2 and Icross over 38M tokens.
Domain
Ld(α⋆)
Rank
θ0⋆
Distill. Branch
Outcome-Gated
RL Branch
Math
0.128
1
23.86
+1.10
+1.83
+0.58
Code
0.052
2
24.09
+0.43
+1.31
+0.51
IF
0.015
3
49.60
+0.48
+1.63
+0.57
Appendix
Table 11: Normalized residual spectrum vs. realized in-domain branch gains. Ld(α⋆) vs. peak in-domain accuracy minus θ0⋆ for an isolated branch trained by on-policy distillation, outcome-gated distillation, or RL. A large normalized residual does not imply a large stand-alone gain.
Hyperparameter
Value
Base Model Architecture
SmolLM3-3B ( P≈3.0×109 )
Optimizer
AdamW ( β1=0.9,β2=0.95,ϵ=10−8 )
Peak Learning Rate
1.5×10−6
Learning Rate Schedule
Linear warmup (5 steps) with cosine decay
Global Batch Size
96 prompt sequences per step
Rollout Group Size ( n )
8 samples per prompt
Appendix
Table 12: Hyperparameter configuration for Open-MOPD branch training. G-OPD uses the official protocol of Yang et al. (2026b) .
Multi-teacher on-policy distillation (MOPD) combines independently developed domain teachers into a single student by distilling their predictions on student-generated samples. We study a setting where teachers share a reference model but undergo different post-training procedures, and find that MOPD can struggle to recover some teacher capabilities. Because distillation occurs on student-generated prefixes, the student initialization can strongly affect subsequent recovery. However, initial benchmark performance is not a reliable predictor of a good MOPD initialization. For example, merge initialization can start below SFT warm-up yet finish higher after MOPD. We further find that effective merging depends on both the relative teacher contributions and the overall merge scale, with some strong configurations lying outside the simplex of convex parameter averaging. Thus, selecting a good merge initialization requires evaluating not only its immediate performance but also the learning it enables under MOPD, making one-shot coefficient search difficult. We propose Iterative Merging for MOPD (IM-MOPD), which starts from a uniform merge and progressively adds task-vector increments for under-recovered domains during distillation. In a 5-domain setting, IM-MOPD achieves higher average normalized recovery than MOPD with either uniform merge initialization or SFT warm-up, showing that effective teacher contributions can be determined progressively during training.
On-policy distillation consolidates specialist models into one student at the price of rollouts, teacher passes and gradient steps for every teacher. We show that the consolidation does not need training: with forward passes only, the capabilities of separately tuned specialists can be aggregated into one model that is, on average, more accurate than either specialist alone. When specialists share a base model, distillation has an exact layer-wise solution: each linear layer of the fused model must reproduce every specialist's output on the inputs that specialist receives while reading its own answers, a least-squares problem whose solution needs only the second moments of those inputs. One forward pass over a few hundred rollouts provides them. We call this training-free on-policy distillation. What the rollouts contain matters more than the solver: answers truncated in repetition loops are few among a mathematics specialist's rollouts but carry a large share of their tokens, and removing them lifts the fused model on AIME25 from task arithmetic's level to the specialist's at matched precision. On Olmo-3-7B, a few GPU-hours of forward computation per specialist yield a generalist that keeps most of each specialist's gain on each of seven mathematics and code benchmarks.
Bangji Yang, Jiajun Fan, Zhaoyu Zhu +2
University of Illinois at Urbana-Champaign · Tsinghua University
Combining capabilities of multiple expert models trained starting from the same base checkpoint has become increasingly common in frontier language-model post-training. Recent trends suggest that multi-teacher on-policy distillation (MOPD) outperforms conventional off-policy methods. However, despite the higher inference and environment interaction costs incurred by MOPD, we find that much of its reported accuracy gain is due to certain training design choices and hyperparameter optimization, rather than the algorithm itself. We conduct a controlled self-distillation study across two multi-teacher settings, four models, and eleven benchmarks, comparing off-policy methods, namely supervised fine-tuning (SFT) and soft-label distillation, to hybrid teacher-prefix distillation and MOPD. We found that all four methods achieve nearly identical accuracy. However, MOPD uses 14.8--23.1× SFT's training GPU-hours. We also revisit four recently published comparisons between on-policy and off-policy distillation and find that the reported on-policy gains shrink substantially when SFT baselines are trained on rejection-sampled teacher trajectories and use independently tuned hyperparameters. As a training-free alternative, we also find that simple weight-merging methods can recover expert capabilities with minutes of CPU merging time, although their accuracy degrades as model size decreases and task interference increases. Overall, our results question recent gains reported due to MOPD and suggest careful tuning of more efficient off-policy baselines as a viable alternative.