No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation

Seonghyeon Kim; Chaeyun Jang; Noah Lee; Boseop Kim; Juho Lee

cs.LGSep 28, 2026

No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation

Authors: Seonghyeon Kim, Chaeyun Jang, Noah Lee, Boseop Kim, Juho Lee

Organizations: KAIST AI · Kakao

Abstract

Multi-teacher on-policy distillation (MOPD) combines independently developed domain teachers into a single student by distilling their predictions on student-generated samples. We study a setting where teachers share a reference model but undergo different post-training procedures, and find that MOPD can struggle to recover some teacher capabilities. Because distillation occurs on student-generated prefixes, the student initialization can strongly affect subsequent recovery. However, initial benchmark performance is not a reliable predictor of a good MOPD initialization. For example, merge initialization can start below SFT warm-up yet finish higher after MOPD. We further find that effective merging depends on both the relative teacher contributions and the overall merge scale, with some strong configurations lying outside the simplex of convex parameter averaging. Thus, selecting a good merge initialization requires evaluating not only its immediate performance but also the learning it enables under MOPD, making one-shot coefficient search difficult. We propose Iterative Merging for MOPD (IM-MOPD), which starts from a uniform merge and progressively adds task-vector increments for under-recovered domains during distillation. In a 5-domain setting, IM-MOPD achieves higher average normalized recovery than MOPD with either uniform merge initialization or SFT warm-up, showing that effective teacher contributions can be determined progressively during training.

Figures & tables

Figure 1 : Student initialization matters for MOPD, but stronger initial performance does not necessarily lead to better outcomes. On Qwen3-4B in the 5-domain setting, MOPD initialized from the shared reference model struggles especially with the SFT-trained Tool Use and Medical teachers, while successfully recovering the RL-trained Finance teacher. Merge initialization can start below SFT warm-up yet finish higher after MOPD. Here,

\lambda

denotes the global merge scale, defined as the sum of the merge coefficients. IM-MOPD further improves performance in a single run, without an additional training stage or a separate coefficient search. Open and filled circles denote initial and final normalized scores, respectively (reference: 0; teacher: 1). Experimental details are provided in Appendix B.1 .

Figure 2 : Overview of IM-MOPD. From a general SFT model $\theta_{\mathrm{ref}}$ and $K$ domain teachers, IM-MOPD starts from a uniform merge of task vectors $\tau_{i}=\phi_{i}-\theta_{\mathrm{ref}}$ and alternates MOPD with selective merging: domains below the recovery threshold $\gamma$ receive a fixed increment $\delta\sum_{i\in\mathcal{M}_{t}}\tau_{i}$ before the next round of distillation as a direct weight-space correction for an enhanced initialization. For specific details, see Algorithm 3.2 .

Panel of Figure 3 — Figure 3 : Selecting a fixed merge initialization requires accounting for subsequent learning, ratios, and scale. (a) From the same 5-domain MOPD checkpoint, we continue training with or without a merge intervention. Although merging can reduce performance immediately, the intervened branch achieves higher recovery after 50 additional MOPD updates. (b) 2-domain Medical-IF coefficient grid vs. IM-MOPD. Open/filled circles denote performance before/after MOPD for fixed merge initializations; numbers indicate the Medical coefficient, with the remainder assigned to IF, and the star denotes IM-MOPD. (c) 5-domain MOPD under different merge initialization. Both ratios and scale affect post-MOPD recovery.

Method	MedQA	CaseHOLD	FinQA	IFBench	$\bm{\tau^{2}}$	Norm.
Qwen3-4B
Qwen3-4B-OT3	69.5	61.8	59.1	24.9	5.3	0.0
Domain Teacher	80.7	73.4	75.0	59.4	66.7	100.0
MOPD	69.1	64.7	74.3	56.9	7.9	42.8
SFT Warm-up + MOPD	73.9	68.2	71.1	53.4	31.6	59.0
Uniform Merge + MOPD	72.7	67.6	74.7	53.6	11.4	53.8

Table 1 : MOPD results across 5-domains at Qwen3-4B and Qwen3-1.7B. IM-MOPD achieves the highest average normalized score among the compared student initialization methods at both model sizes, without an additional SFT stage. Domain Teacher reports the performance of the domain specialist for each domain. The best result in each column within each model size is shown in bold .

Figure 4 : Iterative merging improves recovery in domains left behind by fixed initialization. On Qwen3-4B, IM-MOPD starts from Uniform Merge and progressively improves under-recovered domains during MOPD. The largest gains are observed in Medical, Tool Use, and Law, while performance in already well-recovered domains remains broadly stable.

Merge timing	MedQA	CaseHOLD	FinQA	IFBench	$\bm{\tau^{2}}$	Norm.
No merge (Base MOPD)	69.1	64.7	74.3	56.9	7.9	42.8
All at initialization	76.3	70.8	74.0	52.4	57.0	79.0
All after MOPD	68.1	71.0	59.6	38.2	10.5	23.3
Iterative (IM-MOPD)	76.6	70.9	74.0	57.8	69.3	86.9

Table 2 : Merge-timing ablation on Qwen3-4B. We take the final cumulative merge coefficients produced by IM-MOPD and apply them either entirely at initialization or entirely after MOPD. Both controls underperform the original IM-MOPD schedule, showing that when the same final merge coefficients are applied matters. The best result in each column is shown in bold .

Model	$\gamma$	$\delta=.2$	$\delta=.3$	$\delta=.4$
Qwen3-4B	0.6	80.3	86.9	86.1

Table 3 : IM-MOPD sensitivity to

\gamma

and

\delta

. Average normalized recovery (%) across 5 domains.

Panel of Figure 5 — Figure 5 : Teacher continuation is stronger from merge-based student prefixes on Qwen3-4B. In Medical, teacher continuation is consistently stronger from merge-initialized prefixes than from SFT-warm-up prefixes despite similar student-only performance. In Tool Use, IM-MOPD yields higher continuation success from short prefixes despite lower student-only performance at the evaluated checkpoint. The 0% and 100% endpoints denote teacher-only and student-only performance, respectively.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Teacher	Training	Batch	Length	Peak LR	Tokens (M)
Medical	SFT	16	8,192	$10^{-5}$	426
Law	SFT	128	20,480	$10^{-5}$	269
Tool Use	SFT	64	32,768	$5\cdot 10^{-6}$	796
Finance	GRPO	$32\times 8$	16,384	$3\cdot 10^{-6}$	315
IF	GRPO	$64\times 8$	32,768	$3\cdot 10^{-6}$	483

Appendix

Table 4 : Training configurations of the 4B domain teachers.

Teacher	Training	Batch	Length	Peak LR	Tokens (M)
Medical	SFT	16	8,192	$10^{-5}$	426
Law	SFT	64	20,480	$10^{-5}$	269
Tool Use	SFT	64	32,768	$5\cdot 10^{-6}$	796
Finance	GRPO	$32\times 8$	16,384	$3\cdot 10^{-6}$	275
IF	GRPO	$64\times 8$	32,768	$3\cdot 10^{-6}$	424

Appendix

Table 5 : Training configurations of the 1.7B domain teachers.

Student	Training	Batch	Length	Peak LR	Tokens (M)
4B	SFT	32	32,768	$10^{-5}$	$167$
1.7B	SFT	32	32,768	$10^{-5}$	$171$

Appendix

Table 6 : SFT warm-up configurations.

Size	$\gamma$	$\delta$	Update 25	Update 50	Update 75
4B	0.6	0.2	T,M,L,I	T,M	T
4B	0.6	0.3	T,M,L,I	T	T
4B	0.6	0.4	T,M,L,I	T	$\emptyset$
1.7B	0.4	0.1	T,M,L,I	T,M	T
1.7B	0.4	0.2	T,M,F,I	T	T
1.7B	0.4	0.3	T,M,I	T	T,I

Appendix

Table 7 : Realized merge schedules. T/M/L/F/I denote Tool Use, Medical, Law, Finance, and IF.

Initialization	MedQA	CaseHOLD	FinQA	IFEval	$\tau^{2}$
No init	69.5 / 69.1	61.8 / 64.7	59.1 / 74.3	37.5 / 74.9	5.3 / 7.9
SFT warm-up	72.6 / 73.9	65.4 / 68.2	69.1 / 71.1	56.6 / 71.0	30.7 / 31.6
Merge ( $\lambda=1.0$ )	71.7 / 72.7	66.5 / 67.6	69.0 / 74.7	52.5 / 74.7	0.9 / 11.4
Merge ( $\lambda=2.1$ )	73.7 / 77.5	67.9 / 68.9	66.0 / 73.5	52.7 / 66.2	13.2 / 50.9
IM-MOPD	71.7 / 76.6	66.5 / 70.9	69.0 / 74.0	52.5 / 75.0	0.9 / 69.3

Appendix

Table 8 : Five-domain performance for the initializations in Figure 1 . Each entry shows before / after MOPD. IF is evaluated with IFEval prompt-level strict accuracy.

Correction	MedQA	CaseHOLD	FinQA	IFBench	$\tau^{2}$	Norm.
$\delta=0.2$	79.1	70.3	73.9	58.7	36.8	80.3
$\delta=0.3$	76.6	70.9	74.0	57.8	69.3	86.9
$\delta=0.4$	78.9	71.8	73.0	54.2	59.6	86.1

Appendix

Table 9 : 4B correction-size sweep with

\gamma=0.6

$(\gamma,\delta)$	MedQA	CaseHOLD	FinQA	IFBench	$\tau^{2}$	Norm.
$(0.4,0.1)$	53.7	62.9	59.0	34.8	14.0	62.2
$(0.4,0.2)$	52.2	59.5	59.3	34.0	14.0	57.2
$(0.4,0.3)$	53.7	59.4	55.9	42.9	28.1	76.0
$(0.6,0.1)$	54.6	62.1	58.6	35.2	8.8	57.7
$(0.6,0.2)$	55.0	63.9	58.2	38.4	12.3	65.0
$(0.6,0.3)$	56.9	64.7	50.5	41.8	19.3	69.8

Appendix

Table 10 : 1.7B recovery-threshold and correction-size sweep.

	Initialization			After MOPD
Initialization / method	MedQA	IFEval	Norm.	MedQA	IFEval	Norm.
$(0.2,0.8)$	73.4	79.7	76.7	76.0	78.4	86.4
$(0.4,0.6)$	75.6	79.7	86.7	76.7	79.5	91.3
$(0.6,0.4)$	77.4	64.5	75.4	79.4	79.1	102.5
$(0.8,0.2)$	80.0	49.9	68.9	80.3	78.4	105.4
IM-MOPD	78.3	68.8	84.8	80.0	79.3	105.2

Appendix

Table 11 : Two-domain performance before and after MOPD. Simplex coefficients are ordered as Medical and IF. The last row uses uniform initialization and mid-training merging.

Explore similar work

D $^3$ -MOPD: Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation
Aug 25, 2026Zechen Sun, Zhiwei Zhang, Fei Zhao +7Aware Heterogeneous Multi-Teacher Multimodal On-Policy Distillation Teacher-Student Distillation
AllSpark Team
From Gradients to Capabilities: Understanding Multi-Teacher On-Policy Distillation
Oct 1, 2026Siqi Zhu, Suozhi Huang, Kaixuan Zhang +4Aware Heterogeneous Multi-Teacher Multimodal On-Policy Distillation Teacher
University of Illinois Urbana-Champaign · Princeton University · Westlake University
Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillation
Sep 20, 2026Jie Sun, Mao Zheng, Mingyang Song +8Aware Heterogeneous Multi-Teacher Multimodal On-Policy Distillation Teacher
University of Science and Technology of China · Foundation Model Department, Tencent · Tsinghua University +4

Aug 25, 2026, cs.LGJ/K move · Enter open · S save

D $^3$ -MOPD: Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation

Zechen Sun, Zhiwei Zhang, Fei Zhao, Juntao Li+6

AllSpark Team

Aware Heterogeneous Multi-Teacher Multimodal On-Policy Distillation Teacher-Student Distillation Teacher Kullback-Leibler Divergence

Open paper AskPDF arXiv

Multi-teacher on-policy distillation (MOPD) distills several domain-expert teachers into a single student by minimizing per-domain reverse-KL divergence on the student's own rollouts. Existing approaches typically fix the per-domain data mixture before training, overlooking the fact that different domains converge at substantially different rates: some plateau early while others continue to improve throughout the training budget. A fixed mixture therefore wastes compute on fast-converging domains and undertrains slower-converging ones. To address this, we propose D

^3

-MOPD (Dynamic Domain ScheDuling for MOPD), a zero-overhead scheduler that repurposes the per-domain reverse-KL signal already produced during training to adapt the domain mixture online. Running asynchronously outside the training process, an off-process watcher periodically tracks each domain's KL trajectory, estimates remaining headroom and current improvement rate, and accordingly adjusts the domain sampling ratios without altering the core training loop. Our D

^3

-MOPD scales naturally to arbitrary numbers of domains, and the expected benefit grows as more domains introduce more diverse convergence patterns for the scheduler to exploit. On a Qwen3.6-35B-A3B student distilled from four domain-expert teachers, D

^3

-MOPD closes 97% of the average student-to-teacher performance gap, compared with 63% for vanilla MOPD, reaches the same peak performance with an approximately 3

\times

reduction in rollout steps, and surpasses the specialist teachers on three of seven benchmarks.

Abstract

Explore similar work

D3^33-MOPD: Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation

From Gradients to Capabilities: Understanding Multi-Teacher On-Policy Distillation

Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillation

D $^3$ -MOPD: Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation