Combining capabilities of multiple expert models trained starting from the same base checkpoint has become increasingly common in frontier language-model post-training. Recent trends suggest that multi-teacher on-policy distillation (MOPD) outperforms conventional off-policy methods. However, despite the higher inference and environment interaction costs incurred by MOPD, we find that much of its reported accuracy gain is due to certain training design choices and hyperparameter optimization, rather than the algorithm itself. We conduct a controlled self-distillation study across two multi-teacher settings, four models, and eleven benchmarks, comparing off-policy methods, namely supervised fine-tuning (SFT) and soft-label distillation, to hybrid teacher-prefix distillation and MOPD. We found that all four methods achieve \textit{nearly identical} accuracy. However, MOPD uses 14.8--23.1× SFT's training GPU-hours. We also revisit four recently published comparisons between on-policy and off-policy distillation and find that the reported on-policy gains shrink substantially when SFT baselines are trained on rejection-sampled teacher trajectories and use independently tuned hyperparameters. As a training-free alternative, we also find that simple weight-merging methods can recover expert capabilities with minutes of CPU merging time, although their accuracy degrades as model size decreases and task interference increases. Overall, our results question recent gains reported due to MOPD and suggest careful tuning of more efficient off-policy baselines as a viable alternative.
Figures & tables
Figure 1: Well-tuned SFT narrows reported OPD advantages. Four published SFT–OPD comparisons, with SFT and OPD shown as published (pale) and tuned versions (solid hatched): both methods undergo a learning-rate sweep. SFT uses rejection-sampled teacher trajectories. The reported OPD gaps shrink in every comparison; tuning details and full results are in Appendix A .
Figure 2: Self-distillation pipeline. Each teacher is trained independently from the shared base checkpoint. All methods use the same rejection-sampled prompts. Recorded teacher context includes the prompt and prior interaction history. Multi-turn live interaction means executing student actions and receiving fresh environment observations across turns during training.
Single-turn
Multi-turn
Overall
MIA
MATH †
GSM †
AIME24 †
AIME25 †
IFEval †
IFBen †
GPQA ‡
MMLU ‡
IC-SQL †
ALF-s ‡
ALF-u ‡
WShop ‡
ID Rec.
OOD Δ
Base
82.8 ± 0.2
92.4 ± 0.1
27.1 ± 0.7
21.7 ± 0.8
81.9 ± 0.1
25.7 ± 0.3
42.4 ± 0.5
65.6 ± 0.0
61.7 ± 0.1
18.6 ± 0.7
10.4 ± 0.7
3.6 ± 0.2
New A ± n/r
0.0 ± n/r
Teachers
89.6 ± 0.7
93.0 ± 0.1
41.2 ± 1.5
31.7 ± 0.8
88.5 ± 0.2
42.0 ± 0.3
56.1 ± 0.5
69.8 ± 0.1
77.6 ± 0.1
33.6 ± 1.2
32.1 ± 0.7
11.8 ± 0.2
100.0 ± n/r
New A ± n/r
Mix-RL
87.9 ± 1.2
91.3 ± 0.2
45.3 ± 1.7
33.9 ± 0.7
88.0 ± 0.7
42.4 ± 1.3
51.5 ± 1.7
60.3 ± 0.7
76.4 ± 0.9
8.6 ± 1.2
5.5 ± 2.2
5.8 ± 0.5
102.3 ± 5.2
− 0.4 ± 0.6
Merge
89.2 ± 0.2
93.6 ± 0.0
40.4 ± 1.7
32.9 ± 1.4
87.8 ± 0.2
41.0 ± 0.3
47.1 ± 0.6
63.3 ± 0.1
77.9 ± 0.1
7.9 ± 0.7
1.5 ± 0.7
6.5 ± 0.2
98.9 ± 2.4
− 1.1 ± 0.3
SFT
88.9 ± 0.2
92.8 ± 0.1
41.7 ± 1.2
32.2 ± 2.1
88.5 ± 0.7
42.1 ± 1.3
45.6 ± 0.8
63.5 ± 0.2
76.1 ± 0.2
5.5 ± 1.1
4.5 ± 0.7
0.0 ± n/r
98.2 ± 5.6
− 3.0 ± 0.4
Table 1: Capability recovery and retention with Qwen3-8B . Benchmarks are in-domain ( † ) or out-of-domain ( ‡ ) for each multi-teacher setup. ID Rec. is the recovery rate over all in-domain benchmarks in percent ( Equation 6 ); OOD Δ is the mean change from the base model over all OOD benchmarks. Teachers is a column-wise maximum over that composition’s three experts. Entries report mean ± standard deviation across three seeds. Bold marks the best of the four distillation methods per column; Merge is the best merge result from Appendix C .
Qwen3-1.7B
Ministral-3-3B
Qwen3.5-4B
Qwen3-8B
Mean
Method
MIA
A × 3
MIA
A × 3
MIA
A × 3
MIA
A × 3
OOD Δ
Mix-RL
92.3 ± 1.3
101.9 ± 0.8
93.2 ± 4.8
92.7 ± 1.3
103.5 ± 4.1
102.6 ± 1.2
102.3 ± 5.2
100.3 ± 1.2
−1.1
Merge
86.2 ± 0.2
88.3 ± 0.3
99.7 ± 1.7
77.6 ± 0.4
101.8 ± 2.5
86.8 ± 0.5
98.9 ± 2.4
99.0 ± 0.3
−2.7
SFT
94.7 ± 2.9
98.3 ± 0.7
95.5 ± 1.1
95.1 ± 0.3
98.6 ± 3.0
96.4 ± 1.1
98.2 ± 5.6
99.0 ± 1.7
−3.6
Soft-KD
98.2 ± 2.0
96.8 ± 1.7
98.4 ± 1.3
93.3 ± 1.2
97.3 ± 3.4
97.0 ± 1.0
97.3 ± 2.4
97.6 ± 2.1
−4.0
Off-MOPD
97.7 ± 2.3
96.3 ± 2.1
94.9 ± 4.4
95.4 ± 1.3
98.7 ± 3.2
94.8 ± 1.3
96.9 ± 1.7
97.8 ± 1.7
−3.3
Table 2: Recovery and retention across models. Model-specific columns report in-domain recovery (%); OOD Δ averages the reported retention equally over all eight model–setting pairs. Full benchmark results are in Appendix F .
Figure 3: Validation accuracy versus training progress and GPU-hours. Accuracy is averaged across domains. Training progress is the update count divided by the full-run budget; GPU-hours are cumulative and shown on a logarithmic scale. Markers identify the highest retained validation score for each method.
MIA ID Rec.
MIA OOD Δ
Agent × 3 ID Rec.
Agent × 3 OOD Δ
Method
1.7B
8B
1.7B
8B
1.7B
8B
1.7B
8B
SFT
94.7 ± 2.9
92.5 ± 1.6
− 6.3 ± 0.3
− 9.2 ± 0.5
98.3 ± 0.7
99.3 ± 0.5
− 5.4 ± 0.9
− 14.8 ± 0.7
Soft-KD
98.2 ± 2.0
88.6 ± 1.5
− 7.7 ± 0.2
− 8.4 ± 0.1
96.8 ± 1.7
93.0 ± 1.2
− 4.1 ± 0.5
− 11.7 ± 0.7
Off-MOPD
97.7 ± 2.3
80.9 ± 1.0
− 6.9 ± 0.5
− 3.0 ± 0.8
96.3 ± 2.1
78.9 ± 0.9
− 2.6 ± 0.2
− 1.1 ± 0.7
MOPD
96.9 ± 1.8
77.4 ± 1.9
− 6.1 ± 0.8
− 2.3 ± 0.3
96.8 ± 0.6
73.3 ± 0.3
− 2.1 ± 0.5
− 3.3 ± 0.6
Table 3: Teacher size with a fixed Qwen3-1.7B student. Each method distills from either 1.7B or 8B Qwen3 experts. ID Rec. is the percentage of the corresponding teachers’ aggregate in-domain gains over the 1.7B base recovered by the student ( Equation 6 );
Figure 4: Model changes and training sensitivity on Qwen3-8B. (a) Attention-weight changes remain below 0.7% across methods in both settings. (b) Distillation shortens responses on MIA and lengthens them on Agent × 3, with MOPD showing the largest response-length changes. (c) SFT validation scores peak at an intermediate learning rate in both settings and vary by up to 11.3 percentage points across the sweep. (d) Rejection sampling yields the largest recovery gains for off-policy distillation methods.
Teacher
Baseline SFT
OPD
Tuned SFT
RS SFT
Study
Paper
Ours
Paper
Ours
Paper
Ours
Score
Δ
Score
Δ
Open-MOPD
32.4
32.5 ± 0.2
25.7
23.8 ± 0.4
31.2
32.6 ± 0.8
25.1 ± 0.3
+1.3
32.1 ± 0.6
+7.0
SKD
60.0
58.6 ± 1.1
18.7
18.4 ± 1.0
25.3
25.1 ± 0.6
23.1 ± 0.9
+4.7
22.3 ± 1.0
−0.8
TCOD
81.3
84.3 ± 1.5
28.8
65.7 ± 0.9
63.1
78.0 ± 2.3
70.8 ± 2.6
+5.1
75.2 ± 2.6
+4.4
ReOPD
49.9
55.6 ± 0.6
45.0
47.1 ± 1.2
51.0
54.7 ± 0.5
54.0 ± 0.8
+6.9
54.6 ± 0.4
+0.6
Table 4: Reproducing four published SFT–OPD comparisons. Baseline SFT is our initial reproduction; tuned SFT uses the original or unfiltered corpus; RS SFT additionally uses rejection-sampled teacher data. The two Δ columns compare successive SFT recipes. OPD’s advantage over RS SFT is smaller than published in every comparison.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Math
Code
IF
Method
AIME24
AIME25
Mean
LCBv5
LCBv6
Mean
IFEval
IFBench
Mean
Avg.
Teacher, published
23.7
24.8
24.2
22.2
21.3
21.7
74.5
27.7
51.1
32.4
Teacher, reproduced
24.1
27.1
25.6
23.2
22.3
22.7
74.8
23.3
49.0
32.5
SFT (MixSFT), published
15.6
20.3
17.9
16.0
19.2
17.6
66.9
16.0
41.5
25.7
OPD, published
22.0
22.9
22.4
20.8
22.6
21.7
74.5
24.7
49.6
31.2
SFT baseline, reproduced
11.9
17.5
14.7
14.7
18.4
16.5
64.1
15.9
40.0
23.8
Appendix
Table 5: Open-MOPD results by benchmark. Domain means average two benchmarks; Avg. weights the three domains equally. Published rows are taken from Table 2 of Gao et al. (2026) : RouteRL, MixSFT at epoch 4, and Open-MOPD. Teacher rows use routed experts. RS denotes rejection sampling.
Method
Seen (140)
Unseen (134)
Avg.
Teacher, published
85.7
76.9
81.3
Teacher, reproduced
84.2
84.3
84.3
SFT , published
32.1
25.4
28.8
OPD, published
65.7
60.4
63.1
SFT baseline, reproduced
68.6
62.9
65.7
Tuned SFT , without RS
73.1
68.4
70.8
Appendix
Table 6: TCOD results by ALFWorld split. Scores are success rates (%). Avg. gives the equal-weight mean of seen and unseen.
Method
AIME24
AIME25
AMC23
MATH500
Minerva
Olympiad
Avg.
Teacher, published
28.3
23.3
73.1
84.3
38.5
52.0
49.9
Teacher, reproduced
35.4
27.9
80.9
87.9
44.9
56.5
55.6
SFT , published
22.1
22.9
63.8
80.8
36.0
44.7
45.0
OPD, published
28.3
26.3
70.3
85.8
39.8
55.3
51.0
SFT baseline, reproduced
24.6
25.0
71.3
80.2
32.6
49.2
47.1
Tuned SFT , without RS
34.6
29.2
77.5
86.2
42.5
53.9
54.0
Appendix
Table 7: ReOPD results by mathematics benchmark. Avg. weights the six benchmarks equally. Reproduced benchmark values use the original evaluator.
Table 8: Merging hyperparameters. Singleton values denote fixed settings. Per-expert coefficient configurations appear in Tables 9 and 10 .
Recovery (%)
Method
Coefficients (cIF,cMath,cIC)
IF
Math
IC-SQL
Avg.
Qwen3.5-4B all 8 configurations shown, 4 methods
Teacher gain over base (pts)
29.5
11.7
16.1
★
Task arith.
(0.70,1.00,0.70)
103.1
159.2
103.6
122.0
Task arith.
(0.70,0.70,0.70)
86.2
172.0
100.6
119.6
SCE
top- k100%
63.0
176.4
103.6
114.3
Appendix
Table 9: Merge sweeps on MIA . ★ marks the selected merge (the Merge row of Appendix F ) and ρ a depth-ramped coefficient, all three when set on the tuple; bold is each panel’s best average, and gray averages cover the measured axes only.
Recovery (%)
Method
Coefficients (cIC,cALF,cWS)
IC-SQL
ALFWorld
WebShop
Avg.
Qwen3.5-4B 10 of 32 configurations shown, 6 methods
Teacher gain over base (pts)
15.7
72.1
74.4
★
Task arith.
(1.25,0.95,0.80)ρ
73.5
85.2
89.3
82.6
Task arith.
(1.25,0.85,0.80)ρ
60.5
82.2
91.9
78.2
Task arith.
(1.25,0.85,0.80)
37.0
84.2
90.9
70.7
Appendix
Table 10: Merge sweeps on Agent × 3. ★ marks the selected merge (the Merge row of Appendix F ) and ρ a depth-ramped coefficient, all three when set on the tuple; bold is each panel’s best average, and gray averages cover the measured axes only.
Qwen3-1.7B
Ministral-3-3B
Qwen3.5-4B
Qwen3-8B
Method
MIA
A × 3
MIA
A × 3
MIA
A × 3
MIA
A × 3
Selected learning rate ( ×10−6 )
SFT
20
10
2
5
10
5
5
5
Soft-KD
5
6
0.63
10
16
16
6.3
5
Off-MOPD
1
2
0.2
0.95
1
3.2
1
3
MOPD
1.5
1
0.63
0.95
1
3.2
3
3
Appendix
Table 11: Training configuration. Selected learning rates differ across students and setups, so each default grid is re-centered for every student and setup as described in the text.
Domain
Teacher RL training
Validation
Test
Mathematics
DAPO-Math-17k
teacher’s held-out RL split (512)
MATH-500, GSM8K, AIME24, AIME25
Instruction following
IF-RLVR multi-constraint
teacher’s held-out RL split (430)
IFEval, IFBench
InterCode-SQL
124 Spider train DBs
16 held-out Spider train DBs (815)
Spider dev (1,034)
ALFWorld
train
valid_train (200)
valid_seen (140), valid_unseen (134)
WebShop
goals ≥ 1,500
goals 500–1,499 (1,000)
goals 0–499 (500)
Appendix
Table 12: Task splits. Training, validation, and test splits are disjoint and shared by all methods for each student and setup. Validation excludes both teacher and student training data.
Partition
Benchmark
Domain
Metric
n
MIA
Agent × 3
IFEval
instruction following
prompt-strict accuracy
541
ID
OOD
IFBench
instruction following
prompt-strict accuracy
300
ID
OOD
MATH-500
mathematics
exact-match accuracy
500
ID
OOD
GSM8K
mathematics
exact-match accuracy
1,319
ID
OOD
AIME24
mathematics
avg@16 accuracy ( T=0.6 )
30
ID
OOD
Appendix
Table 13: Benchmarks in the unified evaluation. Benchmarks, metrics, and test-set sizes for Tables 1 and F . A benchmark is ID if its domain is covered by the setup’s teachers and OOD otherwise, and every benchmark counts toward that setup’s recovery or retention. Both setups use the same evaluation suite.
Single-turn
Multi-turn
Overall
MIA
MATH †
GSM †
AIME24 †
AIME25 †
IFEval †
IFBen †
GPQA ‡
MMLU ‡
IC-SQL †
ALF-s ‡
ALF-u ‡
WShop ‡
ID Rec.
OOD Δ
Base
73.4 ± 0.9
82.7 ± 0.1
12.9 ± 0.4
11.2 ± 0.8
66.7 ± 0.1
18.3 ± 0.3
30.8 ± 0.5
47.0 ± 0.1
12.4 ± 0.1
9.3 ± 0.7
9.0 ± 0.7
6.6 ± 0.2
New A ± n/r
0.0 ± n/r
Teachers
79.6 ± 0.4
85.7 ± 0.2
21.5 ± 1.4
19.4 ± 1.1
83.4 ± 0.1
42.3 ± 0.3
31.3 ± 0.5
46.8 ± 0.1
61.6 ± 0.1
7.1 ± 0.7
6.7 ± 0.7
4.4 ± 0.2
100.0 ± n/r
New A ± n/r
Mix-RL
79.7 ± 0.3
84.3 ± 0.6
20.0 ± 1.9
20.4 ± 1.9
83.5 ± 0.4
41.6 ± 1.4
29.3 ± 0.9
40.1 ± 0.4
55.1 ± 0.7
5.7 ± 1.4
1.2 ± 1.6
1.8 ± 0.5
92.3 ± 1.3
− 4.7 ± 0.3
Merge
78.3 ± 0.3
83.3 ± 0.2
19.9 ± 0.6
16.5 ± 0.8
84.2 ± 0.1
41.3 ± 0.3
29.3 ± 0.5
40.7 ± 0.1
54.0 ± 0.1
7.1 ± 0.7
3.7 ± 0.7
0.0 ± n/r
86.2 ± 0.2
− 4.5 ± 0.2
SFT
78.9 ± 0.6
85.2 ± 0.5
18.6 ± 1.3
17.8 ± 2.0
84.5 ± 0.2
41.2 ± 0.7
26.3 ± 0.9
41.1 ± 0.2
61.2 ± 0.4
1.2 ± 1.1
0.7 ± 0.7
0.0 ± n/r
94.7 ± 2.9
− 6.3 ± 0.3
Appendix
Table 14: Per-benchmark results for Qwen3-1.7B . Notation follows Table 1 . ID Rec. and OOD Δ pool ALFWorld’s seen (ALF-s) and unseen (ALF-u) splits.
Single-turn
Multi-turn
Overall
MIA
MATH †
GSM †
AIME24 †
AIME25 †
IFEval †
IFBen †
GPQA ‡
MMLU ‡
IC-SQL †
ALF-s ‡
ALF-u ‡
WShop ‡
ID Rec.
OOD Δ
Base
85.4 ± 0.3
91.5 ± 0.0
23.5 ± 0.3
21.9 ± 0.4
56.0 ± 0.2
17.7 ± 0.3
41.4 ± 0.5
60.1 ± 0.0
40.4 ± 0.1
6.4 ± 0.7
9.0 ± 0.7
0.4 ± 0.2
New A ± n/r
0.0 ± n/r
Teachers
86.6 ± 0.2
90.4 ± 0.2
26.2 ± 0.3
23.1 ± 0.7
69.1 ± 0.1
24.3 ± 0.3
39.9 ± 0.5
59.0 ± 0.0
74.1 ± 0.1
23.6 ± 1.2
20.1 ± 0.7
0.4 ± 0.2
100.0 ± n/r
New A ± n/r
Mix-RL
82.6 ± 0.9
89.6 ± 0.5
24.3 ± 1.4
22.5 ± 1.0
70.5 ± 0.4
26.0 ± 1.2
39.2 ± 0.6
46.2 ± 0.9
74.4 ± 0.3
15.0 ± 2.6
13.4 ± 2.0
1.2 ± 0.3
93.2 ± 4.8
− 2.2 ± 0.6
Merge
82.1 ± 0.3
89.6 ± 0.1
21.6 ± 0.6
21.0 ± 0.4
75.7 ± 0.2
25.8 ± 0.4
36.2 ± 0.6
48.3 ± 0.0
77.8 ± 0.1
10.5 ± 0.8
16.2 ± 0.9
0.9 ± 0.2
99.7 ± 1.7
− 2.7 ± 0.1
SFT
83.0 ± 0.4
88.5 ± 0.3
24.7 ± 0.5
24.0 ± 0.8
69.9 ± 0.3
26.1 ± 0.4
38.4 ± 1.0
49.5 ± 1.1
75.0 ± 0.7
9.5 ± 0.8
8.7 ± 0.9
0.2 ± 0.2
95.5 ± 1.1
− 3.1 ± 0.2
Appendix
Table 15: Per-benchmark results for Ministral-3-3B . Notation follows Table 1 . ID Rec. and OOD Δ pool ALFWorld’s seen (ALF-s) and unseen (ALF-u) splits.
Single-turn
Multi-turn
Overall
MIA
MATH †
GSM †
AIME24 †
AIME25 †
IFEval †
IFBen †
GPQA ‡
MMLU ‡
IC-SQL †
ALF-s ‡
ALF-u ‡
WShop ‡
ID Rec.
OOD Δ
Base
88.2 ± 0.2
92.9 ± 0.1
68.1 ± 0.8
58.8 ± 1.5
65.2 ± 0.1
20.7 ± 0.3
49.0 ± 0.5
69.5 ± 0.0
61.5 ± 0.1
20.7 ± 0.7
14.2 ± 0.7
1.6 ± 0.2
New A ± n/r
0.0 ± n/r
Teachers
92.4 ± 0.2
94.4 ± 0.1
68.5 ± 0.9
65.0 ± 1.4
80.6 ± 0.2
39.3 ± 0.3
60.6 ± 0.5
74.5 ± 0.0
77.5 ± 0.1
21.4 ± 0.7
20.1 ± 0.7
2.4 ± 0.2
100.0 ± n/r
New A ± n/r
Mix-RL
92.6 ± 0.4
97.2 ± 0.2
68.9 ± 1.2
64.2 ± 1.5
80.5 ± 0.9
39.9 ± 1.2
55.1 ± 1.7
70.8 ± 0.1
76.6 ± 1.1
17.6 ± 2.5
20.6 ± 1.9
0.3 ± 0.3
103.5 ± 4.1
+ 1.9 ± 0.6
Merge
94.4 ± 0.2
94.6 ± 0.1
68.2 ± 1.3
65.7 ± 1.5
82.8 ± 0.2
35.0 ± 0.3
55.2 ± 0.6
70.3 ± 0.0
78.1 ± 0.1
14.3 ± 0.7
8.2 ± 0.7
0.4 ± 0.2
101.8 ± 2.5
− 0.1 ± 0.2
SFT
93.9 ± 0.3
93.7 ± 0.2
65.2 ± 2.2
64.4 ± 1.3
80.9 ± 0.6
38.1 ± 1.0
52.9 ± 1.1
67.6 ± 0.2
80.6 ± 0.4
12.6 ± 1.5
14.7 ± 1.1
0.7 ± 0.3
98.6 ± 3.0
− 0.7 ± 0.5
Appendix
Table 16: Per-benchmark results for Qwen3.5-4B . Notation follows Table 1 . ID Rec. and OOD Δ pool ALFWorld’s seen (ALF-s) and unseen (ALF-u) splits.
Single-turn
Multi-turn
Overall
MIA
MATH †
GSM †
AIME24 †
AIME25 †
IFEval †
IFBen †
GPQA ‡
MMLU ‡
IC-SQL †
ALF-s ‡
ALF-u ‡
WShop ‡
ID Rec.
OOD Δ
Base
73.4 ± 0.9
82.7 ± 0.1
12.9 ± 0.4
11.2 ± 0.8
66.7 ± 0.1
18.3 ± 0.3
30.8 ± 0.5
47.0 ± 0.1
12.4 ± 0.1
9.3 ± 0.7
9.0 ± 0.7
6.6 ± 0.2
New A ± n/r
0.0 ± n/r
Teachers
89.6 ± 0.7
93.0 ± 0.1
41.2 ± 1.5
31.7 ± 0.8
88.5 ± 0.2
42.0 ± 0.3
56.1 ± 0.5
69.8 ± 0.1
77.6 ± 0.1
33.6 ± 1.2
32.1 ± 0.7
11.8 ± 0.2
100.0 ± n/r
New A ± n/r
SFT
88.3 ± 0.3
91.2 ± 0.3
38.0 ± 1.7
29.9 ± 2.4
88.2 ± 0.4
40.1 ± 0.8
22.9 ± 1.5
30.3 ± 0.8
74.0 ± 0.5
1.9 ± 0.8
2.0 ± 1.1
1.7 ± 0.2
92.5 ± 1.6
− 9.2 ± 0.5
Soft-KD
87.1 ± 0.3
90.3 ± 0.4
34.5 ± 0.7
28.7 ± 1.7
87.7 ± 0.5
37.3 ± 1.0
24.6 ± 0.8
30.6 ± 1.2
76.8 ± 0.1
2.4 ± 0.8
2.7 ± 0.9
2.1 ± 0.5
88.6 ± 1.5
− 8.4 ± 0.1
Off-MOPD
86.0 ± 0.2
90.3 ± 0.3
33.9 ± 1.3
27.4 ± 1.6
85.4 ± 0.3
32.4 ± 0.7
29.0 ± 1.9
42.9 ± 0.4
72.6 ± 0.9
7.1 ± 1.2
8.5 ± 1.1
1.9 ± 0.6
80.9 ± 1.0
− 3.0 ± 0.8
Appendix
Table 17: Per-benchmark results for Qwen3-1.7B with Qwen3-8B teachers. Notation follows Table 1 . ID Rec. and OOD Δ pool ALFWorld’s seen (ALF-s) and unseen (ALF-u) splits. Base repeats Table 14 and Teachers repeats Table 1 , so ID Rec. is normalized by the 8B teachers’ gains over the 1.7B base and is not directly comparable to Table 14 .
Offline-MOPD
MOPD
MIA
s/step
% of step
s/step
% of step
Student rollout
60.89
95.6
276.23
86.5
Teacher scoring
0.60
0.9
8.20
2.6
Student log-probabilities
0.40
0.6
6.60
2.1
Student gradient update
1.80
2.8
28.40
8.9
Full step
63.69
100.0
319.43
100.0
Appendix
Table 18: OPD training-step timing for Qwen3-8B . Batch sizes (Offline-MOPD/MOPD) are 24/96 on MIA and 40/40 on Agent × 3; s/step describes each configuration. Rollout includes student generation and live environment interaction where applicable. Teacher scoring comprises forward passes over student trajectories.
Full run
Best validation checkpoint
Method
LR ( 10−6 )
Budget
Step
GPU-h
Step
Dev (%)
GPU-h
MIA
SFT
5
1920
1920
25.90
1056
65.15
14.57
Soft-KD
6.3
1920
1920
51.90
1440
65.31
38.60
Off-MOPD
1
1140
1140
161.35
1026
65.07
145.21
MOPD
3
540
540
383.32
390
65.21
276.84
Appendix
Table 19: Training configurations and checkpoint costs for Qwen3-8B . Budget is the number of updates in a completed run. OPD cost at update k is C(k)=8kT/3600 GPU-hours, with constant step time T from Table 18 .
Multi-teacher on-policy distillation (MOPD) combines independently developed domain teachers into a single student by distilling their predictions on student-generated samples. We study a setting where teachers share a reference model but undergo different post-training procedures, and find that MOPD can struggle to recover some teacher capabilities. Because distillation occurs on student-generated prefixes, the student initialization can strongly affect subsequent recovery. However, initial benchmark performance is not a reliable predictor of a good MOPD initialization. For example, merge initialization can start below SFT warm-up yet finish higher after MOPD. We further find that effective merging depends on both the relative teacher contributions and the overall merge scale, with some strong configurations lying outside the simplex of convex parameter averaging. Thus, selecting a good merge initialization requires evaluating not only its immediate performance but also the learning it enables under MOPD, making one-shot coefficient search difficult. We propose Iterative Merging for MOPD (IM-MOPD), which starts from a uniform merge and progressively adds task-vector increments for under-recovered domains during distillation. In a 5-domain setting, IM-MOPD achieves higher average normalized recovery than MOPD with either uniform merge initialization or SFT warm-up, showing that effective teacher contributions can be determined progressively during training.
Foundation multimodal large language models are designed to support a broad spectrum of capabilities across diverse domains. Multi-teacher on-policy distillation (MOPD) provides an effective framework for consolidating domain-specific expertise into a single student model. However, MOPD training gradually drives the student away from its initialization model, and general capabilities decline as the displacement grows, resulting in capability interference. A direct remedy is constraining the student toward its initialization, but this suppresses the acquisition of domain expertise as well. We propose Slow-Fast Multi-Teacher On-Policy Distillation (SF-MOPD), which couples a fast model, the current student updated directly by each teacher, with a slow model, an exponential moving average of the student. The slow model absorbs the learning signal gradually, serving as a moving capability reference that fuses the general foundation with confirmed domain expertise. For each teacher, SF-MOPD computes the teacher-induced update in log-probability space and removes only the component that pushes the fast model further away from the slow model, while retaining aligned and orthogonal components. Experiments across multiple model scales demonstrate that SF-MOPD effectively mitigates capability interference, enhances specialized multimodal capabilities, and reduces the average degradation on general-capability benchmarks, consistently outperforming vanilla MOPD.
Xiaofei Yin, Tong Chu, Jiyuan Fu +3
Ant Security Lab, Ant Group, Shanghai, China · Fudan University, Shanghai, China
Multi-teacher on-policy distillation (MOPD) has emerged as a popular post-training paradigm for integrating specialized capabilities in frontier language models. Existing OPD research has primarily focused on optimizing single-task distillation through objective design, distillation scope, and teacher signal construction, whereas MOPD must aggregate multiple capabilities in shared parameters and address the resulting capability seesaw, in which improving one domain suppresses capabilities acquired from another. Inspired by the distinctive update geometry of OPD, we find that parameter updates from different tasks rapidly concentrate in their respective low-dimensional subspaces during MOPD, providing a direct geometric basis for identifying and controlling cross-task interference. We therefore propose PMOPD (Projection-based Multi-Teacher On-Policy Distillation), which constructs subspace memories from the cumulative parameter displacements of different tasks and projects both gradients and optimizer updates to remove components that interfere with protected task directions. We further develop a lightweight conflict probe to characterize task interactions and guide task ordering, together with a cycling strategy that balances subspace estimation and timely task revisitation. Experiments on representative Code, Reason, and Math tasks show that PMOPD improves every evaluated capability over MOPD, raising the average score across the three tasks by 2.54 points on Qwen2.5-7B and 2.09 points on Llama-3.1-8B. These consistent gains establish geometry-aware optimization as an effective and transferable approach to balanced multi-teacher distillation.