A Fisher-Geometric View of Multi-Teacher On-Policy Distillation

Liu Kaiyuan; Zhuang Ziyuan; Ma Haoxuan; Gu Yannian; Song Mingfei; Weng Rongxiang; Ye Jieping

cs.LGOct 19, 2026

A Fisher-Geometric View of Multi-Teacher On-Policy Distillation

Authors: Liu Kaiyuan, Zhuang Ziyuan, Ma Haoxuan, Gu Yannian, Song Mingfei, Weng Rongxiang, Ye Jieping

Organizations: Zhejiang University · Meituan LongCat · Nanjing University · Shanghai Jiao Tong University

Abstract

Consolidating specialized foundation models into a unified generalist is a central goal of post-training. Two approaches are in common use: model merging, which is fast but treated as a heuristic, and multi-teacher on-policy distillation, which matches distributions but demands heavy rollout compute and suffers from cross-task gradient conflict. We show that the two are one objective seen at different orders. Within a shared basin of homologous specialists, the multi-teacher on-policy distillation objective is, to second order, exactly the Fisher-merging objective. Its minimizer is therefore the Fisher merge, a classic model-merging method given by a closed-form combination of the specialists, so merging is optimal, to second order, as a closed-form stage of distillation. Building on this identification, we propose Residual-Governed Branch-and-Merge(RGBM), a two-phase consolidation procedure. Concretely, a full-parameter Fisher merge is intractable on foundation models, so Phase1 restricts the merge to the task-vector subspace. Extending tangent-space analyses of merging, we show that the same coefficients that mix task vectors also mix implicit rewards, which lets a residual program on unlabelled prompts solve for them. The program minimizes the worst per-domain residual and reports its energy-normalized values as the domain residual spectrum. Phase2 lets this spectrum govern what may be added: RGBM passivates domains with low normalized residuals, trains isolated branches only for domains above the spectral gap, and writes back the admitted updates with calibrated shrinkage. On two open-source consolidation suites, the Phase~1 merge alone matches or outperforms fully trained multi-teacher distillation, and RGBM improves on it further with zero extra joint-training steps, whereas every un-gated write-back, including continued joint distillation from the merge, falls below it.

Figures & tables

Figure 1: Merge before you distill (schematic). (a) Weight space: the dashed triangle is the task subspace spanned by the Math, Code, and IF task vectors. A Minimax projection on unlabelled prompts yields the zero-training merge $\theta_{0}^{\star}=W_{\mathrm{ref}}+\mathcal{T}\bm{\alpha}^{\star}$ , where $\mathcal{T}$ stacks those task vectors, together with the per-domain residuals $\mathcal{L}_{d}$ (red segments). Normalizing these residuals by domain reward energy gives the domain residual spectrum. (b) Capability space: joint distillation from $\theta_{0}^{\star}$ can leave the basin (dashed path). The normalized spectrum passivates Code and IF; RGBM writes back only the admitted Math branch, shrunk by $\gamma^{\star}$ (the largest factor that still passes the linearity test). See Section 3 for details.

Figure 2: Two-phase RGBM architecture. (1) Phase 1 scores implicit rewards on unlabelled prompts, forms per-domain second-moment matrices $M_{d}$ , and minimizes the worst residual $\mathcal{L}_{d}$ to obtain $\theta_{0}^{\star}$ . The heatmap is the reward correlation, not $M_{d}$ itself. (2) The bars are the energy-normalized Open-MOPD spectrum $\overline{\bm{\mathcal{L}}}$ ; the gate cuts at the midpoint of its largest adjacent gap (threshold $\epsilon$ ; Eq. 10 ), so only Math is active. (3) Phase 2 writes active updates back with shrinkage $\gamma$ to obtain $\theta_{\mathrm{RGBM}}$ ; if every domain is passivated, $\theta_{\mathrm{RGBM}}=\theta_{0}^{\star}$ . See Sections 3.1 and 3.3 for details.

Method	Math	Code	IF	Macro
RouteRL (Oracle Ensemble)	27.24	26.27	53.52	35.68
Multi-Teacher OPD	24.53	21.18	47.12	30.93
Task Arithmetic	24.35	22.81	46.77	31.31
Subspace Fisher Merge	24.11	23.59	47.99	31.90
RGBM Phase 1 (Ours, $\theta_{0}^{\star}$ )	23.86	24.09	49.60	32.52
Uniform Distillation (from $\theta_{0}^{\star}$ )	23.97	21.36	47.42	30.91

Table 1: Open-MOPD (SmolLM3-3B; accuracy). Macro: mean of Math, Code, and IF. Bold: best deployable model. Italic: routing oracle. Gated Math: admitted branch only, at the listed

\gamma

Program	$\bm{\alpha}$	Math	Code	IF	Macro
$\bm{\alpha}=\mathbf{1}/3$	0.333/0.333/0.333	20.81	19.15	44.77	28.24
A scalar $\mathcal{L}$	0.626/0.626/0.626	24.97	23.28	47.64	31.96
A vector $\mathcal{L}$	0.527/0.609/0.821	24.11	23.59	47.99	31.90
C scalar $\mathcal{L}$	0.579/0.579/0.579	24.97	23.49	45.10	31.19
C vector $\mathcal{L}$ ( $\theta_{0}^{\star}$ )	0.597/0.620/0.800	23.86	24.09	49.60	32.52
A scalar $\overline{\mathcal{L}}$	0.501/0.501/0.501	24.38	23.36	44.86	30.86

Table 2: Open-MOPD coefficient grid. A minimizes the mean residual (uniform

p_{d}

, Eq. 5 ); C minimizes the worst residual (Eq. 9 ). Scalar shares one coefficient; vector leaves

\bm{\alpha}

free. Scores for the four

\mathcal{L}

rows and for

\bm{\alpha}=\mathbf{1}/3

are Table 4 ; the four

\overline{\mathcal{L}}

rows use the same protocol. Coefficients are Math/Code/IF.

Figure 3: Domain score against the raw residual (Open-MOPD). Each panel orders the nine merges of Table 2 by that panel’s score, so the score curve rises by construction. Domain-

\mathcal{L}

is min-max scaled onto that score range. A/C: uniform and minimax. sc/vec: shared scalar and free vector. L / Lb: the program minimizes

\mathcal{L}

\mathcal{L}_{d}/\mathbb{E}[r_{d}^{2}]

. lin:

\bm{\alpha}=\mathbf{1}/3

. The energy-normalized spectrum is used only for the Phase 2 gate in Section 4.1 .

Panel of Figure 4 — Figure 4: Joint distillation from $\theta_{0}^{\star}$ (Open-MOPD, $\leq 50$ steps). (a) Macro under three joint objectives; dashed and dotted lines are RGBM Phase 2 and $\theta_{0}^{\star}$ . (b) Uniform Distillation: Math stays near the merge before a late drop; Code and IFBench fall. Dashed lines mark RGBM Phase 2 on those axes.

Method	Math	Code	Avg
Domain-Matched Teacher	46.00	61.17	52.50
SFT	44.33	60.80	51.39
OPD	46.38	60.57	52.46
ExOPD	47.65	61.97	53.79
Task Arithmetic	43.05	60.65	50.59
Equal-Weight Average	22.55	55.67	36.75

Table 3: G-OPD (Qwen3-4B). Math / Code: mean of the 4 math / 3 code tasks. Avg: 7-task mean. Italic: teacher oracle.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Linearity diagnostics on the isotropic family ( $\theta(\alpha)=W_{\mathrm{ref}}+\alpha\sum_{d}\tau_{d}$ ; not the Minimax vector $\bm{\alpha}^{\star}$ ). (a) Macro vs. scalar $\alpha$ . Horizontal lines are RouteRL ( $35.68$ ), the $600$ -step student ( $30.93$ ), and MixSFT ( $25.35$ ); the peak at $\alpha=0.75$ is this scan, not $\bm{\alpha}^{\star}$ . (b) $R^{2}$ and $\mathcal{I}_{\mathrm{cross}}$ over 38M tokens.

Domain	$\overline{\mathcal{L}}_{d}(\bm{\alpha}^{\star})$	Rank	$\theta_{0}^{\star}$	Distill. Branch	Outcome-Gated	RL Branch
Math	$0.128$	1	23.86	$+1.10$	$+1.83$	$+0.58$
Code	$0.052$	2	24.09	$+0.43$	$+1.31$	$+0.51$
IF	$0.015$	3	49.60	$+0.48$	$+1.63$	$+0.57$

Appendix

Table 11: Normalized residual spectrum vs. realized in-domain branch gains.

\overline{\mathcal{L}}_{d}(\bm{\alpha}^{\star})

vs. peak in-domain accuracy minus

\theta_{0}^{\star}

for an isolated branch trained by on-policy distillation, outcome-gated distillation, or RL. A large normalized residual does not imply a large stand-alone gain.

Hyperparameter	Value
Base Model Architecture	SmolLM3-3B ( $P\approx 3.0\times 10^{9}$ )
Optimizer	AdamW ( $\beta_{1}=0.9,\beta_{2}=0.95,\epsilon=10^{-8}$ )
Peak Learning Rate	$1.5\times 10^{-6}$
Learning Rate Schedule	Linear warmup (5 steps) with cosine decay
Global Batch Size	96 prompt sequences per step
Rollout Group Size ( $n$ )	8 samples per prompt

Appendix

Table 12: Hyperparameter configuration for Open-MOPD branch training. G-OPD uses the official protocol of Yang et al. (2026b) .

Language Model Merging

Language model merging combines capabilities from multiple models or adapters by composing their weights, updates, or outputs.

See all 109 papers

No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation
Sep 28, 2026Seonghyeon Kim, Chaeyun Jang, Noah Lee +2Task Arithmetic Language Model Distillation
KAIST AI · Kakao
Training-Free On-Policy Distillation
Oct 15, 2026Bangji Yang, Jiajun Fan, Zhaoyu Zhu +2On-Policy Distillation Knowledge Distillation
University of Illinois at Urbana-Champaign · Tsinghua University
Rethinking Self-Distillation for Multi-Teacher Capability Merging
Oct 3, 2026Roy Xie, Dan Friedman, Feng Nan +7Language Model Distillation Multi-Teacher Knowledge Distillation
Apple · Duke University

Sep 28, 2026J/K move · Enter open · S save

No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation

Seonghyeon Kim, Chaeyun Jang, Noah Lee, Boseop Kim+1

KAIST AI · Kakao

Task Arithmetic Language Model Distillation Multi-Teacher Knowledge Distillation On-Policy Distillation Model Merging

Open paper AskPDF arXiv

Multi-teacher on-policy distillation (MOPD) combines independently developed domain teachers into a single student by distilling their predictions on student-generated samples. We study a setting where teachers share a reference model but undergo different post-training procedures, and find that MOPD can struggle to recover some teacher capabilities. Because distillation occurs on student-generated prefixes, the student initialization can strongly affect subsequent recovery. However, initial benchmark performance is not a reliable predictor of a good MOPD initialization. For example, merge initialization can start below SFT warm-up yet finish higher after MOPD. We further find that effective merging depends on both the relative teacher contributions and the overall merge scale, with some strong configurations lying outside the simplex of convex parameter averaging. Thus, selecting a good merge initialization requires evaluating not only its immediate performance but also the learning it enables under MOPD, making one-shot coefficient search difficult. We propose Iterative Merging for MOPD (IM-MOPD), which starts from a uniform merge and progressively adds task-vector increments for under-recovered domains during distillation. In a 5-domain setting, IM-MOPD achieves higher average normalized recovery than MOPD with either uniform merge initialization or SFT warm-up, showing that effective teacher contributions can be determined progressively during training.

Abstract

Language Model Merging

Explore similar work

No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation

Training-Free On-Policy Distillation

Rethinking Self-Distillation for Multi-Teacher Capability Merging