Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillation

Huan-ang Gao; Haohan Chi; Yong Yan; Shiyuan Feng; Hanlin Wu; Zheng Jiang; Bingxiang He; Wei-Ying Ma

cs.LGAug 19, 2026

Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillation

Authors: Huan-ang Gao, Haohan Chi, Yong Yan, Shiyuan Feng, Hanlin Wu, Zheng Jiang, Bingxiang He, Wei-Ying Ma, +2 more

Organizations: SIA-Lab of Tsinghua AIR and ByteDance Seed · Institute for AI Industry Research (AIR), Tsinghua University · Department of Computer Science and Technology, Tsinghua University

Abstract

Multi-teacher on-policy distillation (M-OPD) has emerged as a promising paradigm for consolidating domain-specialized reinforcement learning (RL) experts into a single generalist student via dense, token-level reward supervision. Despite its practical success, the optimization dynamics governing multi-teacher capability integration remain poorly understood, and open, rigorously reproducible recipes are conspicuously lacking. In this work, we establish a controlled M-OPD benchmark on SmolLM3-3B-Base with oracle routing, isolating capability integration from routing ambiguity. Our investigation reveals a pronounced capability integration gap: standard M-OPD captures only 35.6% of the available headroom relative to a domain-routed oracle ensemble, with concise tasks such as instruction following suffering severe degradation and premature stagnation. Crucially, we show that this failure stems not from gradient conflict, but from a severe misallocation of the token-level optimization budget. This pathology is driven by three orthogonal factors: structural sequence-length disparities across domains, dynamic convergence drift due to non-uniform learning rates, and multi-step reward staleness from asynchronous policy updates. To resolve these imbalances, we introduce Open-MOPD, a principled framework incorporating token-share balancing, gap-aware dynamic budget allocation, and student reward refresh. Together, these mechanisms systematically restore cross-domain balance, elevating headroom recovery from 35.6% to 83.4% in a single deployable student. We fully open-source our end-to-end post-training recipe, training trajectories, and evaluation suites on an academically accessible hardware budget.

Figures & tables

Figure 1: The integration gap is real and strongly asymmetric. (a) Per-domain macro-averages of Naive M-OPD and RouteOPD, where the number above each bar indicates the gap in that domain; (b) per-domain validation trajectories of Naive M-OPD, with dashed lines denoting the RouteOPD reference for the corresponding domain; (c) recovered theoretical headroom relative to RouteRL,

(\text{current}-\pi_{\mathrm{mixsft}})/(\text{RouteRL}-\pi_{\mathrm{mixsft}})

. Instruction following (IF) exhibits the largest absolute gap and the lowest recovered headroom, and is the earliest domain to stop improving.

Stage	Data	Key settings and products
Base model	SmolLM3-3B-Base	3B; 1.7B fails to close long trajectories, while 7B is too costly for repeated ablations (Appendix 8.1 )
I Mixed-domain SFT	Math: OpenR1-Math-93k Code: OCR-50k (sampled from the full set) IF: Instruction-Nemotron	Four epochs; sequence limit 32,768; approximately balanced by response tokens (37/28/35) product $\pi_{\mathrm{mixsft}}$
II Domain RL teachers	Math: DAPO-Math-17k Code: DeepScaler-24k (LCB-decontaminated) IF: Nemotron-IF-RL-46k	Independent RL per domain, no domain mixing; all forked from $\pi_{\mathrm{mixsft}}$ products $\pi_{\phi_{\mathrm{math}}},\pi_{\phi_{\mathrm{code}}},\pi_{\phi_{\mathrm{IF}}}$
III Multi-teacher OPD	The union of the three domains’ RL prompts above; each prompt is hard-routed to its teacher by domain label	Response limit 16K for math/code and 2K for IF; top- $k{=}16$ ; the student is initialized from $\pi_{\mathrm{mixsft}}$ product $\pi_{\theta}$
Evaluation	Math: AIME24 / AIME25 Code: LiveCodeBench v5/v6 IF: IFEval + IFBench test	mean@64 / mean@10 / mean@1; averaged evenly across the datasets within a domain first, then macro-averaged over the three domains

Table 1: The Open-MOPD pipeline. Starting from a mixed-domain SFT checkpoint (

\pi_{\mathrm{mixsft}}

), domain-specific RL produces three teacher models

\pi_{\phi_{d}}

. Multi-teacher OPD then trains the student model

\pi_{\theta}

from the initial checkpoint. All datasets are publicly released for end-to-end reproducibility.

SFT
	Math (avg@64)			Code (avg@10)			IF
Method	AIME24	AIME25	avg	LCBv5	LCBv6	avg	IFEval	IFB test	avg	Total	Recovery rate
SmolLM3-3B-Base	2.08	1.72	1.90	3.41	6.17	4.79	16.08	13.00	14.54	7.08	—
$\pi_{\mathrm{mixsft}}$ (e1)	12.24	16.61	14.43	13.29	17.20	15.25	64.70	16.00	40.35	23.34	—
$\pi_{\mathrm{mixsft}}$ (e2)	12.55	17.97	15.26	13.17	16.63	14.90	65.62	16.00	40.81	23.66	—
$\pi_{\mathrm{mixsft}}$ (e3)	15.10	19.53	17.32	14.97	18.57	16.77	66.36	18.00	42.18	25.42	—

Table 2: Integration gap on SmolLM3-3B. Each domain reports its sub-dataset scores and macro-average. We define the total score as the average of the three domain averages. The four SFT rows are checkpoints after successive epochs. Single-domain RL and OPD report results only for their respective domains. Gray-shaded RouteRL and RouteOPD use a separate model for each domain and select that model using the domain label at evaluation time. They therefore cannot be deployed as one model. The recovery rate is

\frac{\text{Row Total}-\text{SFT}}{\text{RouteRL}-\text{SFT}}

, using

\pi_{\mathrm{mixsft}}

(epoch 4) as the SFT baseline. In the Baselines and M-OPD sections, the highest score in each column is boldfaced and the second highest is underlined.

Figure 2: Teacher conflict is measurable but not the bottleneck. (a) Average teacher disagreement

c_{t}

(log scale) and its distance to the 1-nat criterion. The legend applies to both panels (a) and (b). (b) Fraction of tokens with

c_{t}>1

, broken down by domain. (c) Change in total score for four conflict interventions relative to the baseline. Blue points show the conflict mask, where the horizontal axis indicates the filtering threshold when removing top-

k\%

tokens by

c_{t}

quantile (larger

k

means lower threshold). Purple points show the consensus method with a fixed threshold of 1 nat; tokens with

c_{t}\leq 1

use the average log-probs of all three teachers, while those with

c_{t}>1

keep the routed teacher. Error bars denote validation standard error.

Figure 3: Prompt share and gradient-token share are decoupled. (a) Whole-run averages of prompt share versus token share per domain (log scale; light bars for prompts, solid bars for tokens). IF accounts for

\approx 20\%

of prompts but only

\approx 1\%

of gradient tokens. (b) Average response length (log scale) accounts for this difference, with

\approx 10,500

tokens for math/code versus

\approx 409

tokens for IF. (c) Balancing token share (

1/3

per domain) solely via oversampling requires a

33.6\times

prompt multiplier for IF. This severely reduces math and code prompts per batch, making long-chain supervision unsustainable.

Figure 4: The effective budget drifts with distillation progress. (a) Per-token reward magnitude

\bar{m}_{d}

of each domain (log axis), differing by 4.9

\times

early on; (b) after normalizing to their respective initial values, the three domains shrink at different rates, which shows that

\bar{m}_{d}

measures the remaining teacher–student gap; (c) on a trajectory where

s_{d}^{\mathrm{tok}}

is flattened to

1/3

, Equation 7 degenerates to

B_{d}\propto\bar{m}_{d}

, so the curves show the budget drift caused by the gap alone. Within 25 steps the share of IF falls below the dashed line to about 9%, while code rises to 63.8%. Flattening the token share does not lock the budget in place.

Figure 5: Repeated inner updates change the student policy within a rollout batch. The horizontal axis is the number of inner updates

K

per rollout batch. (a) Within the same batch, the KL between the rollout policy and the current student policy grows monotonically with

K

; (b) the fraction of tokens clipped by PPO rises with

K

. At

K{=}1

both are 0, and the dense reward is then consistent with the current student.

Figure 6: Overview of Open-MOPD. Prompts are hard-routed by domain label. During rollout, the teacher computes and caches log-probabilities on

\mathcal{S}_{t}

once. The student generates responses under its own distribution. Together, both terms form the dense reward for PPO updates. Within each rollout batch, the inner update repeats

K

times, reusing the teacher term while recomputing the student term via reward refresh. Badges highlight where each mechanism acts: token-share balancing and gap-following allocation determine the domain loss weight

w_{d}

, while reward refresh rebuilds

r_{t}

in every inner update.

Mechanism	Measurement	Intervention
token-share balancing	Response-token share $s_{d}^{\mathrm{tok}}$ in the current batch	Set $w_{d}^{\mathrm{share}}=g_{d}^{\star}/s_{d}^{\mathrm{tok}}$ to equalise the domains’ token shares in the loss.
gap-following allocation	Running mean reward magnitude $m_{d}$ for each domain	Multiply the share weight by the clipped factor $(m_{d}/m_{\mathrm{ref}})^{\alpha}$ , giving more weight to domains with a larger current reward signal.
reward refresh	Student log-probabilities at each inner update	Recompute the student-dependent reward while reusing the cached teacher log-probabilities.

Table 3: Summary of the three Open-MOPD mechanisms. Each mechanism modifies a different part of the training computation.

	Mechanism				Six datasets
Configuration	share	gap	refresh	$K$	Math	Code	IF	Total	$\Delta$
Naive M-OPD				1	21.26	19.26	43.64	28.05	—
+share	✓			1	20.55	19.57	47.53	29.22	$+1.17$
+gap	✓	✓		1	21.00	19.31	49.50	29.94	$+1.89$
Switching to the $K{=}4$ throughput setting (256 prompts per update, $4\times$ rollout batch)
Naive M-OPD (same-setting control)				4	21.62	19.72	46.49	29.28	$+1.23$

Table 4: From Naive M-OPD to the full recipe. Each row changes one thing from the row above, as shown in the first column. The three middle columns show whether token-share balancing, gap-following allocation, and reward refresh are turned on.

K

is the number of inner updates per rollout batch.

Figure 7: Gap-following allocation dynamically changes the budget trajectory. (a) Weighted token share

w_{d}s_{d}^{\mathrm{tok}}

: static token-share balancing keeps the share fixed at

1/3

(dashed line). Adding gap-following allocation (

\alpha{=}1

) dynamically adjusts the budget based on the remaining teacher–student gap. Domains that converge quickly lose budget, while those with larger gaps receive more resources (e.g., code exceeds

50\%

, whereas IF drops to

\sim 17\%

). (b) IF domain weight (log scale): setting

\alpha{=}1

properly reduces its weight, whereas the inverse formulation (

\alpha{=}-0.5

) creates an unstable feedback loop, causing training to collapse at step 74.

Figure 8: Composition of one outer step at

K{=}4

. Averaged over steady training steps. The dense-reward computation accounts for

2.2\%

of each step. Reward refresh changes only when the student probabilities are read for this computation, without changing total throughput.

Base model	SFT data	Performance
Qwen3-1.7B-Base	OpenR1-Math-93k	AIME24 best 7.08
Truncation evidence: 69.17–80.42%; capacity and trajectory closure insufficient.
SmolLM3-3B-Base	Three-domain mixSFT	AIME24 15.63
Truncation evidence: 15.2%; chosen as the base model of the recipe.
Qwen2.5-7B-Base	OpenR1-Math-93k	AIME24 31.25
Truncation evidence: 12.92%; capable enough, but a full multi-teacher ablation is too costly.

Table 5: Base-model choice for Open-MOPD. The table summarises directly the experimental results used when constructing the recipe. The truncation evidence is reported in a dedicated row below each base model.

Epoch	Step	Qwen3-1.7B-Base	Qwen2.5-7B-Base
1	732	4.58 / 75.42	16.25 / 42.92
2	1464	7.08 / 72.50	20.83 / 35.42
3	2196	6.67 / 80.42	30.83 / 22.08
4	2928	5.83 / 69.17	30.42 / 19.58
5	3660	6.67 / 69.58	31.25 / 12.92

Table 6: Base-model screening under the same OpenR1-Math-93k SFT. Each cell is AIME24 accuracy / generation truncation rate (%). Qwen3-1.7B-Base mainly generates long, unclosed reasoning trajectories at every observed checkpoint; Qwen2.5-7B-Base instead raises accuracy and lowers truncation together as SFT proceeds.

	Mixed-domain SFT	RL: math	RL: code	RL: IF
World size	8 $\times$ 8	8 $\times$ 8	8 $\times$ 8	4 $\times$ 8
Starting point	SmolLM3-3B-Base	$\pi_{\mathrm{mixsft}}$
Algorithm	Cross-entropy	GRPO
train batch size	128	128	128	128
mini batch size	—	32	32	32
Learning rate	$4\times 10^{-5}$	$1\times 10^{-6}$

Table 7: Hyperparameters of stages one and two. SFT is standard supervised fine-tuning; the three RL teachers are all GRPO, each initialised from

\pi_{\mathrm{mixsft}}

and trained only on the verifiable reward of its own domain. World size is given as number of nodes

\times

GPUs per node, all A100-80GB. “—” means the item does not apply to that stage.

	Single-domain OPD (RouteOPD oracle)			Multi-teacher OPD
	math	code	IF	Open-MOPD
World size	1 $\times$ 8	1 $\times$ 8	1 $\times$ 8	1 $\times$ 8
Teacher	$\pi_{\phi_{\mathrm{math}}}$	$\pi_{\phi_{\mathrm{code}}}$	$\pi_{\phi_{\mathrm{IF}}}$	3-teacher hard routing
Student starting point	$\pi_{\mathrm{mixsft}}$
train batch size	1,024
mini batch size	256 (i.e. $K{=}4$ )

Table 8: Hyperparameters of stage three. The left columns are the three single-domain OPD runs (the RouteOPD oracle of the main text) and the right column is multi-teacher OPD. The two share the optimiser, sequence lengths and distillation settings; the token-share balancing / gap-following allocation / reward refresh rows are the values of our mechanisms in the final recipe.

Explore similar work

Recovering General Capabilities via Uncertainty-Calibrated Multi-Teacher On-Policy Distillation
Aug 27, 2026Ziyuan Liu, Jiao Ou, Jian Liang +2Uncertainty-Aware Knowledge Distillation Multi-Teacher Knowledge Distillation
Kuaishou Technology Beijing, China
D $^3$ -MOPD: Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation
Aug 25, 2026Zechen Sun, Zhiwei Zhang, Fei Zhao +7Adaptive Sampling Language Model Distillation
AllSpark Team
Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillation
Oct 9, 2026Shi-Yu Tian, Yong Liu, Ximan Liu +10Language Model Distillation Multi-Teacher Knowledge Distillation
Nanjing University

Aug 27, 2026, cs.CLJ/K move · Enter open · S save

Recovering General Capabilities via Uncertainty-Calibrated Multi-Teacher On-Policy Distillation

Ziyuan Liu, Jiao Ou, Jian Liang, Ruiming Tang+1

Kuaishou Technology Beijing, China

Uncertainty-Aware Knowledge Distillation Multi-Teacher Knowledge Distillation On-Policy Distillation Knowledge Distillation

Open paper AskPDF arXiv

Specializing large language models to vertical domains improves domain-specific behavior but often degrades general capabilities. We study this trade-off in Multi-Teacher On-Policy Distillation (MOPD), where a specialized model learns from domain and general teachers on its own sampled trajectories. Standard MOPD faces two limitations: ordinary on-policy sampling rarely exposes tokens with large positive teacher--student advantages, and advantage sign alone does not establish whether the proposed update direction is reliable. We propose Uncertainty-Calibrated MOPD (UCMOPD), which addresses these limitations through two complementary mechanisms. Golden-Gain Enhancement combines higher-temperature exploration with a standard-temperature anchor and retains trajectories whose positive learning signal matches or exceeds the prompt-specific anchor. Teacher-Endorsement Filtering then uses centered log-likelihood (CLL) to estimate each retained token's plausibility relative to the teacher's uncertainty and probabilistically preserves updates whose directions are supported by that endorsement. Across role-playing and medical-domain specialization, UCMOPD improves the general-capability average over standard MOPD by

4.48\%

and

7.86\%

, respectively, while maintaining vertical-domain performance. Component ablations and diagnostic analyses support the intended roles of the two mechanisms: exposing and selecting stronger positive signals at the trajectory level and validating update directions through teacher endorsement at the token level.

Abstract

Explore similar work

Recovering General Capabilities via Uncertainty-Calibrated Multi-Teacher On-Policy Distillation

D3^33-MOPD: Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation

Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillation

D $^3$ -MOPD: Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation