Organizations: Australian Institute for Machine Learning, Adelaide University, Adelaide, Australia · StepFun, Shanghai, China · School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Singapore University of Technology and Design, Singapore · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates
Multi-teacher on-policy distillation consolidates expert capabilities by supervising student-generated trajectories, whereas task arithmetic offers a training-free alternative through linear parameter combinations. We view the linear combination of student and expert parameters as a single differentiable model and propose P2-MOPD, which jointly optimizes the mixing coefficients and the student weights under the same on-policy distillation objective. Unlike task arithmetic, which stays within the span of the frozen student and expert parameters for any choice of merging coefficients, P2-MOPD keeps the student trainable, so the experts act as parameter priors without confining it to this span. Across four domains at 1.7B and 4B scales, P2-MOPD achieves faster and more balanced recovery than MOPD from the same initialization, surpassing its final average recovery in half the training steps. At the end of training, our 1.7B model reaches 94.7% average recovery, versus 80.9% for MOPD.
Figures & tables
Figure 1: Average recovery over training at Qwen3-4B (top) and Qwen3-1.7B (bottom). P 2 -MOPD surpasses the final average recovery of MOPD (TA init) (arrow) after half the training steps (star). Mean over three seeds (shaded: ±1 sd); both methods start from the same TA merge (square).
Figure 2: Overview of P 2 -MOPD . Each projection of the student is composed as its own learnable full weight plus a learnable linear combination of frozen expert directions, Wm=Bm+∑kαk,mτk,m ( Equation 4 ), and the composed weights are used as one model for the forward pass. The student generates rollouts on prompts of all domains; the frozen teacher of each prompt’s domain scores the student’s tokens. The sampled tokens and the teacher logits are passed as detached data flow (gray lines), and the resulting MOPD loss is back-propagated to the student weights (student weight channel) and to the coefficients α (coefficient channel). No gradient reaches the experts or the teachers, which come from the same expert bank.
Method
MedQA
CaseHOLD
FinQA
IFBench
Avg. Recovery
Avg. Score
Worst
AULC
Qwen3-4B
Base student ( θ0 )
68.84
62.03
58.76
25.89
0
53.88
–
–
Domain teachers
79.86
73.57
75.41
59.22
100
72.02
–
–
TA merge
73.95
66.00
70.01
33.78
42.98
60.93
23.67
–
MOPD
62.90
66.30
75.04
61.33
46.80
66.39
−53.94
20.43
MOPD (SFT warm-up)
69.76
71.47
74.22
53.11
66.15
67.14
8.32
61.86
Table 1: Final results. Per-domain test scores (%, avg@3: each item is scored by the mean over three samples at temperature 1.0 ) and summary metrics; for P 2 -MOPD , MOPD (TA init) and IM-MOPD all entries are three-seed means and the summary metrics show ± sd. Avg. Recovery: average recovery over the four domains; Avg. Score: mean raw score over the four benchmarks; Worst: worst-domain recovery; AULC: area under the learning curve, i.e. the average recovery averaged over the checkpoints at 25, 50, 75 and 100 steps. The base student and the teachers define recovery 0 and 100 ; the TA merge is the untrained common start. Bold: best trained method.
Figure 3: Balance of recovery across domains. (a) Final worst-domain recovery (bars: three-seed mean; dots: seeds). (b, c) Recovery of the worst domain, MedQA (dashed; mean ±1 sd over three seeds), and the mean recovery of the other three domains (solid). MedQA is the lowest domain for both methods at almost every checkpoint. MOPD (TA init) recovers the other three domains as fast as P 2 -MOPD but leaves MedQA behind.
Qwen3-4B
Qwen3-1.7B
Domain
Final
Δ
Final
Δ
MedQA
89.8
+29.0± 8.3
88.0
+48.2± 14.4
CaseHOLD
91.4
+3.8± 4.2
96.0
+4.5± 3.1
FinQA
97.6
−1.9± 3.5
94.1
+0.8± 15.6
IFBench
100.6
+4.8± 23.2
100.9
+7.6± 12.9
Avg. Recovery
94.9
+8.9± 4.6
94.7
+15.3± 4.5
Table 2: Per-domain recovery (%) of P 2 -MOPD and its difference from MOPD (TA init). Final: recovery of P 2 -MOPD at the final checkpoint (three-seed mean). Δ : mean difference, P 2 -MOPD minus MOPD (TA init), with sd over all twelve seed-paired comparisons per scale (four checkpoints × three seeds).
Figure 4: Task-vector projection of the final displacement from the TA start in α units (three-seed means; seed sd ≤0.007 ). Each line connects MOPD (TA init) to P 2 -MOPD along one task vector. The share of the displacement energy that lies in the task-vector span is 5.2% / 10.7% for MOPD (TA init) and 85.4% / 90.0% for P 2 -MOPD (4B / 1.7B).
Run
αmed
Med KL ↓
MedQA ↑
Avg. ↑
MOPD (TA init)
×
0.115
22.1
78.1
medical prompts only
×
0.098
−6.9
–
P 2 -MOPD
✓
0.032
87.2
95.8
medical prompts only
✓
0.013
90.2
–
random directions
×
0.111
1.4
70.2
learn only αmed
✓
0.028
94.1
95.0
Table 3: Controlled interventions (1.7B, final checkpoint). Each indented row changes one factor of the run above it. αmed : whether a learnable coefficient moves the student along τmed . Med KL: training reverse KL to the medical teacher on student rollouts (mean of steps 91–100). MedQA and Avg.: MedQA recovery and average recovery (%). –: not reported.
Method
Step (s)
Cost
Time to target
Recovery at budget
4B
MOPD (TA init)
180.6
1.00×
1.00×
83.1
P 2 -MOPD
225.6
1.24×
≤0.31×
95.7
1.7B
MOPD (TA init)
153.9
1.00×
1.00×
80.9
P 2 -MOPD
167.4
1.09×
≤0.55×
89.3
Table 4: Cost to reach a fixed quality on 8 GPUs. Step: mean seconds per step. Cost: wall-clock of 100 steps relative to MOPD (TA init). Time to target: wall-clock to exceed the final average recovery of MOPD (TA init), as a fraction of its 100-step wall-clock; checkpoints are 25 steps apart, so these are upper bounds. Recovery at budget: average recovery of the last checkpoint within the wall-clock of 100 steps of MOPD (TA init), which is step 75 for P 2 -MOPD . Recoveries are three-seed means; all granularities are in Table 7 .
Figure 5: Inside versus beyond the task-vector span (module granularity): mixing coefficients only (student weights frozen, confined to the span), student weights only (mixing coefficients frozen; MOPD (TA init)) and both ( P 2 -MOPD ). The last two are three-seed means (shaded: ±1 sd). Dotted: the TA start.
Qwen3-4B
Qwen3-1.7B
Train W
Train α
Avg.
Worst
Avg.
Worst
42.98
23.67
32.41
17.15
✓
52.98
35.62
49.31
27.17
✓
83.13
47.22
80.93
32.03
✓
✓
94.85
89.42
94.74
88.01
Table 5: Inside versus beyond the task-vector span, final checkpoint. Rows from top to bottom: the untrained TA start, mixing coefficients only (student weights frozen, confined to the span), student weights only (mixing coefficients frozen; MOPD (TA init)), and both channels ( P 2 -MOPD ); the last two rows are three-seed means. Avg.: average recovery; Worst: worst-domain recovery.
Qwen3-4B
Qwen3-1.7B
module
layer
model
module
layer
model
# α
1008
144
4
784
112
4
Avg. Rec. ↑
94.9
73.4
96.5
94.7
94.4
97.2
std ↓
4.6
16.6
3.6
4.8
1.5
9.3
Table 6: Coefficient granularity at the final checkpoint. Avg. Rec.: average recovery; std: across-domain std of recovery; # α : number of free coefficients. Module (shaded, the main setting) is a three-seed mean.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Wall (h)
×
Step (s)
Update (s)
4B
MOPD (TA init)
5.26
1.00
180.6
14.7
P 2 -MOPD module
6.51
1.24
225.6
73.8
P 2 -MOPD layer
6.32
1.20
218.2
68.1
P 2 -MOPD model
7.50
1.43
258.9
90.9
1.7B
MOPD (TA init)
4.48
1.00
153.9
11.1
P 2 -MOPD module
4.90
1.09
167.4
37.1
Appendix
Table 7: Measured training cost on 8 GPUs for all coefficient granularities. Wall-clock covers the training job from environment ready to final checkpoint; per-step numbers are 100-step means from the training logs. × : ratio to MOPD (TA init).
Figure 6: Coefficient learning-rate sweep at 1.7B (module granularity); ηα=0 is MOPD (TA init).
ηα
Step 25
Final
AULC
0 , MOPD (TA init)
50.7
80.9
71.4
3×10−3
61.4
91.3
80.5
10−2 , P 2 -MOPD
72.2
94.7
86.7
3×10−2
51.2
69.5
62.6
Appendix
Table 8: Coefficient learning-rate sweep (1.7B). Average recovery at steps 25 and 100, and its area under the learning curve (AULC; mean over the four checkpoints). Rows for ηα=0 and 10−2 are three-seed means.
Med
Law
Fin
IF
4B
0.63
0.37
0.39
0.35–0.36
1.7B
0.65–0.66
0.46
0.36
0.33
Appendix
Table 9: Final mean coefficient per teacher of P 2 -MOPD (module granularity), averaged over modules; ranges are over the three seeds where they differ. All coefficients start at 0.25 .
4B
1.7B
Optimizer steps
100
100
Prompts per step (per domain)
128 (32)
128 (32)
Responses per prompt
1
1
Rollout sampling
T=1.0 , top- p1.0
T=1.0 , top- p1.0
Max prompt / response tokens
15,872 / 16,384
15,872 / 16,384
Distillation support
top- 16 , nucleus 0.99
top- 16 , nucleus 0.99
Appendix
Table 10: Training hyperparameters of P 2 -MOPD and the MOPD baselines.
MedQA
CaseHOLD
FinQA
IFBench
4B
base
68.84
62.03
58.76
25.89
teacher
79.86
73.57
75.41
59.22
span
11.02
11.54
16.65
33.33
1.7B
base
44.10
44.65
43.62
19.89
teacher
58.99
70.52
62.66
46.44
span
14.89
25.87
19.04
26.55
Appendix
Table 11: Base and teacher endpoints (avg@3, %) used for normalized recovery, measured with the same harness. The span steacher−sbase is the denominator of recovery; one raw point corresponds to 100/span recovery points.
Step 25
Step 50
Step 75
Step 100
Scale
Seed
Merged
Pre
Post
Merged
Pre
Post
Merged
Pre
Post
Final
4B
42
Med, Law, IF
58.9
43.2
IF
74.0
74.5
–
83.9
–
88.3
43
Med, Law
62.3
58.0
–
88.3
–
–
90.4
–
86.6
44
Med, Law
59.2
62.2
–
83.1
–
–
85.2
–
89.2
1.7B
42
Fin, IF
42.3
46.4
Fin
49.2
47.7
–
50.4
–
77.4
43
IF
38.7
44.5
Med, Fin, IF
49.8
32.7
Fin
56.3
51.7
67.1
Appendix
Table 12: IM-MOPD merge trajectory. At steps 25, 50 and 75 the recovery-gated rule merges the task vectors of the listed domains into the student; Pre and Post are the average recovery of the checkpoint immediately before and after the merge, and Step 100 is the final checkpoint. –: no merge at that step.
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.
Multi-teacher on-policy distillation (MOPD) combines independently developed domain teachers into a single student by distilling their predictions on student-generated samples. We study a setting where teachers share a reference model but undergo different post-training procedures, and find that MOPD can struggle to recover some teacher capabilities. Because distillation occurs on student-generated prefixes, the student initialization can strongly affect subsequent recovery. However, initial benchmark performance is not a reliable predictor of a good MOPD initialization. For example, merge initialization can start below SFT warm-up yet finish higher after MOPD. We further find that effective merging depends on both the relative teacher contributions and the overall merge scale, with some strong configurations lying outside the simplex of convex parameter averaging. Thus, selecting a good merge initialization requires evaluating not only its immediate performance but also the learning it enables under MOPD, making one-shot coefficient search difficult. We propose Iterative Merging for MOPD (IM-MOPD), which starts from a uniform merge and progressively adds task-vector increments for under-recovered domains during distillation. In a 5-domain setting, IM-MOPD achieves higher average normalized recovery than MOPD with either uniform merge initialization or SFT warm-up, showing that effective teacher contributions can be determined progressively during training.
Multi-teacher on-policy distillation (MOPD) has emerged as a popular post-training paradigm for integrating specialized capabilities in frontier language models. Existing OPD research has primarily focused on optimizing single-task distillation through objective design, distillation scope, and teacher signal construction, whereas MOPD must aggregate multiple capabilities in shared parameters and address the resulting capability seesaw, in which improving one domain suppresses capabilities acquired from another. Inspired by the distinctive update geometry of OPD, we find that parameter updates from different tasks rapidly concentrate in their respective low-dimensional subspaces during MOPD, providing a direct geometric basis for identifying and controlling cross-task interference. We therefore propose PMOPD (Projection-based Multi-Teacher On-Policy Distillation), which constructs subspace memories from the cumulative parameter displacements of different tasks and projects both gradients and optimizer updates to remove components that interfere with protected task directions. We further develop a lightweight conflict probe to characterize task interactions and guide task ordering, together with a cycling strategy that balances subspace estimation and timely task revisitation. Experiments on representative Code, Reason, and Math tasks show that PMOPD improves every evaluated capability over MOPD, raising the average score across the three tasks by 2.54 points on Qwen2.5-7B and 2.09 points on Llama-3.1-8B. These consistent gains establish geometry-aware optimization as an effective and transferable approach to balanced multi-teacher distillation.
Multi-teacher on-policy distillation can yield uneven domain capabilities. We propose Domain-Routed Delta Optimization (DRD-MOPD), composing a frozen base with expert-initialized trainable deltas. Domains update their deltas with independent optimizer states and clipping. After joint training, Post-separate exports the base plus one trained delta per domain, selected by known task domain without retraining. Using AdamW, the unweighted full student reaches 32.83% Macro across six benchmarks after 20 updates, versus 32.21% for a matched shared control. Post-separate raises Macro with Open-MOPD weighting.
Lin Yao
School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Zhongguancun Academy, Beijing, China