Organizations: SIA-Lab of Tsinghua AIR and ByteDance Seed · Institute for AI Industry Research (AIR), Tsinghua University · Department of Computer Science and Technology, Tsinghua University
Multi-teacher on-policy distillation (M-OPD) has emerged as a promising paradigm for consolidating domain-specialized reinforcement learning (RL) experts into a single generalist student via dense, token-level reward supervision. Despite its practical success, the optimization dynamics governing multi-teacher capability integration remain poorly understood, and open, rigorously reproducible recipes are conspicuously lacking. In this work, we establish a controlled M-OPD benchmark on SmolLM3-3B-Base with oracle routing, isolating capability integration from routing ambiguity. Our investigation reveals a pronounced capability integration gap: standard M-OPD captures only 35.6% of the available headroom relative to a domain-routed oracle ensemble, with concise tasks such as instruction following suffering severe degradation and premature stagnation. Crucially, we show that this failure stems not from gradient conflict, but from a severe misallocation of the token-level optimization budget. This pathology is driven by three orthogonal factors: structural sequence-length disparities across domains, dynamic convergence drift due to non-uniform learning rates, and multi-step reward staleness from asynchronous policy updates. To resolve these imbalances, we introduce Open-MOPD, a principled framework incorporating token-share balancing, gap-aware dynamic budget allocation, and student reward refresh. Together, these mechanisms systematically restore cross-domain balance, elevating headroom recovery from 35.6% to 83.4% in a single deployable student. We fully open-source our end-to-end post-training recipe, training trajectories, and evaluation suites on an academically accessible hardware budget.
Figures & tables
Figure 1: The integration gap is real and strongly asymmetric. (a) Per-domain macro-averages of Naive M-OPD and RouteOPD, where the number above each bar indicates the gap in that domain; (b) per-domain validation trajectories of Naive M-OPD, with dashed lines denoting the RouteOPD reference for the corresponding domain; (c) recovered theoretical headroom relative to RouteRL, (current−πmixsft)/(RouteRL−πmixsft) . Instruction following (IF) exhibits the largest absolute gap and the lowest recovered headroom, and is the earliest domain to stop improving.
Stage
Data
Key settings and products
Base model
SmolLM3-3B-Base
3B; 1.7B fails to close long trajectories, while 7B is too costly for repeated ablations (Appendix 8.1 )
I Mixed-domain SFT
Math: OpenR1-Math-93k Code: OCR-50k (sampled from the full set) IF: Instruction-Nemotron
Four epochs; sequence limit 32,768; approximately balanced by response tokens (37/28/35) product πmixsft
mean@64 / mean@10 / mean@1; averaged evenly across the datasets within a domain first, then macro-averaged over the three domains
Table 1: The Open-MOPD pipeline. Starting from a mixed-domain SFT checkpoint ( πmixsft ), domain-specific RL produces three teacher models πϕd . Multi-teacher OPD then trains the student model πθ from the initial checkpoint. All datasets are publicly released for end-to-end reproducibility.
Math (avg@64)
Code (avg@10)
IF
Method
AIME24
AIME25
avg
LCBv5
LCBv6
avg
IFEval
IFB test
avg
Total
Recovery rate
SmolLM3-3B-Base
2.08
1.72
1.90
3.41
6.17
4.79
16.08
13.00
14.54
7.08
—
SFT
πmixsft (e1)
12.24
16.61
14.43
13.29
17.20
15.25
64.70
16.00
40.35
23.34
—
πmixsft (e2)
12.55
17.97
15.26
13.17
16.63
14.90
65.62
16.00
40.81
23.66
—
πmixsft (e3)
15.10
19.53
17.32
14.97
18.57
16.77
66.36
18.00
42.18
25.42
—
Table 2: Integration gap on SmolLM3-3B. Each domain reports its sub-dataset scores and macro-average. We define the total score as the average of the three domain averages. The four SFT rows are checkpoints after successive epochs. Single-domain RL and OPD report results only for their respective domains. Gray-shaded RouteRL and RouteOPD use a separate model for each domain and select that model using the domain label at evaluation time. They therefore cannot be deployed as one model. The recovery rate is RouteRL−SFTRow Total−SFT , using πmixsft (epoch 4) as the SFT baseline. In the Baselines and M-OPD sections, the highest score in each column is boldfaced and the second highest is underlined.
Figure 2: Teacher conflict is measurable but not the bottleneck. (a) Average teacher disagreement ct (log scale) and its distance to the 1-nat criterion. The legend applies to both panels (a) and (b). (b) Fraction of tokens with ct>1 , broken down by domain. (c) Change in total score for four conflict interventions relative to the baseline. Blue points show the conflict mask, where the horizontal axis indicates the filtering threshold when removing top- k% tokens by ct quantile (larger k means lower threshold). Purple points show the consensus method with a fixed threshold of 1 nat; tokens with ct≤1 use the average log-probs of all three teachers, while those with ct>1 keep the routed teacher. Error bars denote validation standard error.
Figure 3: Prompt share and gradient-token share are decoupled. (a) Whole-run averages of prompt share versus token share per domain (log scale; light bars for prompts, solid bars for tokens). IF accounts for ≈20% of prompts but only ≈1% of gradient tokens. (b) Average response length (log scale) accounts for this difference, with ≈10,500 tokens for math/code versus ≈409 tokens for IF. (c) Balancing token share ( 1/3 per domain) solely via oversampling requires a 33.6× prompt multiplier for IF. This severely reduces math and code prompts per batch, making long-chain supervision unsustainable.
Figure 4: The effective budget drifts with distillation progress. (a) Per-token reward magnitude mˉd of each domain (log axis), differing by 4.9 × early on; (b) after normalizing to their respective initial values, the three domains shrink at different rates, which shows that mˉd measures the remaining teacher–student gap; (c) on a trajectory where sdtok is flattened to 1/3 , Equation 7 degenerates to Bd∝mˉd , so the curves show the budget drift caused by the gap alone. Within 25 steps the share of IF falls below the dashed line to about 9%, while code rises to 63.8%. Flattening the token share does not lock the budget in place.
Figure 5: Repeated inner updates change the student policy within a rollout batch. The horizontal axis is the number of inner updates K per rollout batch. (a) Within the same batch, the KL between the rollout policy and the current student policy grows monotonically with K ; (b) the fraction of tokens clipped by PPO rises with K . At K=1 both are 0, and the dense reward is then consistent with the current student.
Figure 6: Overview of Open-MOPD. Prompts are hard-routed by domain label. During rollout, the teacher computes and caches log-probabilities on St once. The student generates responses under its own distribution. Together, both terms form the dense reward for PPO updates. Within each rollout batch, the inner update repeats K times, reusing the teacher term while recomputing the student term via reward refresh. Badges highlight where each mechanism acts: token-share balancing and gap-following allocation determine the domain loss weight wd , while reward refresh rebuilds rt in every inner update.
Mechanism
Measurement
Intervention
token-share balancing
Response-token share sdtok in the current batch
Set wdshare=gd⋆/sdtok to equalise the domains’ token shares in the loss.
gap-following allocation
Running mean reward magnitude md for each domain
Multiply the share weight by the clipped factor (md/mref)α , giving more weight to domains with a larger current reward signal.
reward refresh
Student log-probabilities at each inner update
Recompute the student-dependent reward while reusing the cached teacher log-probabilities.
Table 3: Summary of the three Open-MOPD mechanisms. Each mechanism modifies a different part of the training computation.
Mechanism
Six datasets
Configuration
share
gap
refresh
K
Math
Code
IF
Total
Δ
Naive M-OPD
1
21.26
19.26
43.64
28.05
—
+share
✓
1
20.55
19.57
47.53
29.22
+1.17
+gap
✓
✓
1
21.00
19.31
49.50
29.94
+1.89
Switching to the K=4 throughput setting (256 prompts per update, 4× rollout batch)
Naive M-OPD (same-setting control)
4
21.62
19.72
46.49
29.28
+1.23
Table 4: From Naive M-OPD to the full recipe. Each row changes one thing from the row above, as shown in the first column. The three middle columns show whether token-share balancing, gap-following allocation, and reward refresh are turned on. K is the number of inner updates per rollout batch.
Figure 7: Gap-following allocation dynamically changes the budget trajectory. (a) Weighted token share wdsdtok : static token-share balancing keeps the share fixed at 1/3 (dashed line). Adding gap-following allocation ( α=1 ) dynamically adjusts the budget based on the remaining teacher–student gap. Domains that converge quickly lose budget, while those with larger gaps receive more resources (e.g., code exceeds 50% , whereas IF drops to ∼17% ). (b) IF domain weight (log scale): setting α=1 properly reduces its weight, whereas the inverse formulation ( α=−0.5 ) creates an unstable feedback loop, causing training to collapse at step 74.
Figure 8: Composition of one outer step at K=4 . Averaged over steady training steps. The dense-reward computation accounts for 2.2% of each step. Reward refresh changes only when the student probabilities are read for this computation, without changing total throughput.
Base model
SFT data
Performance
Qwen3-1.7B-Base
OpenR1-Math-93k
AIME24 best 7.08
Truncation evidence: 69.17–80.42%; capacity and trajectory closure insufficient.
SmolLM3-3B-Base
Three-domain mixSFT
AIME24 15.63
Truncation evidence: 15.2%; chosen as the base model of the recipe.
Qwen2.5-7B-Base
OpenR1-Math-93k
AIME24 31.25
Truncation evidence: 12.92%; capable enough, but a full multi-teacher ablation is too costly.
Table 5: Base-model choice for Open-MOPD. The table summarises directly the experimental results used when constructing the recipe. The truncation evidence is reported in a dedicated row below each base model.
Epoch
Step
Qwen3-1.7B-Base
Qwen2.5-7B-Base
1
732
4.58 / 75.42
16.25 / 42.92
2
1464
7.08 / 72.50
20.83 / 35.42
3
2196
6.67 / 80.42
30.83 / 22.08
4
2928
5.83 / 69.17
30.42 / 19.58
5
3660
6.67 / 69.58
31.25 / 12.92
Table 6: Base-model screening under the same OpenR1-Math-93k SFT. Each cell is AIME24 accuracy / generation truncation rate (%). Qwen3-1.7B-Base mainly generates long, unclosed reasoning trajectories at every observed checkpoint; Qwen2.5-7B-Base instead raises accuracy and lowers truncation together as SFT proceeds.
Mixed-domain SFT
RL: math
RL: code
RL: IF
World size
8 × 8
8 × 8
8 × 8
4 × 8
Starting point
SmolLM3-3B-Base
πmixsft
Algorithm
Cross-entropy
GRPO
train batch size
128
128
128
128
mini batch size
—
32
32
32
Learning rate
4×10−5
1×10−6
Table 7: Hyperparameters of stages one and two. SFT is standard supervised fine-tuning; the three RL teachers are all GRPO, each initialised from πmixsft and trained only on the verifiable reward of its own domain. World size is given as number of nodes × GPUs per node, all A100-80GB. “—” means the item does not apply to that stage.
Single-domain OPD (RouteOPD oracle)
Multi-teacher OPD
math
code
IF
Open-MOPD
World size
1 × 8
1 × 8
1 × 8
1 × 8
Teacher
πϕmath
πϕcode
πϕIF
3-teacher hard routing
Student starting point
πmixsft
train batch size
1,024
mini batch size
256 (i.e. K=4 )
Table 8: Hyperparameters of stage three. The left columns are the three single-domain OPD runs (the RouteOPD oracle of the main text) and the right column is multi-teacher OPD. The two share the optimiser, sequence lengths and distillation settings; the token-share balancing / gap-following allocation / reward refresh rows are the values of our mechanisms in the final recipe.
Specializing large language models to vertical domains improves domain-specific behavior but often degrades general capabilities. We study this trade-off in Multi-Teacher On-Policy Distillation (MOPD), where a specialized model learns from domain and general teachers on its own sampled trajectories. Standard MOPD faces two limitations: ordinary on-policy sampling rarely exposes tokens with large positive teacher--student advantages, and advantage sign alone does not establish whether the proposed update direction is reliable. We propose Uncertainty-Calibrated MOPD (UCMOPD), which addresses these limitations through two complementary mechanisms. Golden-Gain Enhancement combines higher-temperature exploration with a standard-temperature anchor and retains trajectories whose positive learning signal matches or exceeds the prompt-specific anchor. Teacher-Endorsement Filtering then uses centered log-likelihood (CLL) to estimate each retained token's plausibility relative to the teacher's uncertainty and probabilistically preserves updates whose directions are supported by that endorsement. Across role-playing and medical-domain specialization, UCMOPD improves the general-capability average over standard MOPD by 4.48% and 7.86%, respectively, while maintaining vertical-domain performance. Component ablations and diagnostic analyses support the intended roles of the two mechanisms: exposing and selecting stronger positive signals at the trajectory level and validating update directions through teacher endorsement at the token level.
Multi-teacher on-policy distillation (MOPD) distills several domain-expert teachers into a single student by minimizing per-domain reverse-KL divergence on the student's own rollouts. Existing approaches typically fix the per-domain data mixture before training, overlooking the fact that different domains converge at substantially different rates: some plateau early while others continue to improve throughout the training budget. A fixed mixture therefore wastes compute on fast-converging domains and undertrains slower-converging ones. To address this, we propose D3-MOPD (Dynamic Domain ScheDuling for MOPD), a zero-overhead scheduler that repurposes the per-domain reverse-KL signal already produced during training to adapt the domain mixture online. Running asynchronously outside the training process, an off-process watcher periodically tracks each domain's KL trajectory, estimates remaining headroom and current improvement rate, and accordingly adjusts the domain sampling ratios without altering the core training loop. Our D3-MOPD scales naturally to arbitrary numbers of domains, and the expected benefit grows as more domains introduce more diverse convergence patterns for the scheduler to exploit. On a Qwen3.6-35B-A3B student distilled from four domain-expert teachers, D3-MOPD closes 97% of the average student-to-teacher performance gap, compared with 63% for vanilla MOPD, reaches the same peak performance with an approximately 3× reduction in rollout steps, and surpasses the specialist teachers on three of seven benchmarks.
Multi-teacher on-policy distillation (MOPD) extends on-policy distillation (OPD) by routing domain-specific data to specialized teachers, integrating their capabilities through dense token-level supervision on student-generated trajectories. However, domain routing only distinguishes the supervision sources for different domains; updates from all domains still act on shared parameters. When teachers differ substantially in response style and domain expertise, supervision that is effective in isolation may still lead to persistent gradient conflicts and capability degradation during joint optimization. To address this challenge, we propose Cross-Teacher Guarding for MOPD (Road MOPD), which introduces non-routed teachers as auxiliary cross-domain critics on the same student trajectories and coordinates multi-teacher supervision through token-level credit assignment and gradient updates. Specifically, cross-teacher-guided credit assignment uses teacher agreement to strengthen token-level corrections jointly supported by the teachers and attenuate conflicting signals; preventive multi-teacher gradient projection uses auxiliary gradients from the same trajectories to constrain potentially conflicting updates and adaptively controls projection strength to retain target-domain learning signals. In experiments, Road MOPD mitigates degradation in the disadvantaged domain, gains over 5 percentage points under high conflict, and improves overall performance in low-conflict and three-domain settings.