MOPD-Router: Rethinking Teacher Routing in Multi-Teacher On-Policy Distillation

Tianze Xu; Yanzhao Zheng; Zhentao Zhang; Yuanqiang Yu; Chao Ma; Jihuai Zhu; Lelun Wu; Lyumanshan Ye

cs.LGSep 25, 2026

MOPD-Router: Rethinking Teacher Routing in Multi-Teacher On-Policy Distillation

Authors: Tianze Xu, Yanzhao Zheng, Zhentao Zhang, Yuanqiang Yu, Chao Ma, Jihuai Zhu, Lelun Wu, Lyumanshan Ye, +5 more

Organizations: Shanghai Jiao Tong University · GAIR · Alibaba Group · University of Science and Technology of China · Shanghai Innovation Institute

Abstract

Multi-teacher on-policy distillation (MOPD) integrates specialized capabilities into a single student, but existing practice typically hard-routes each prompt to a domain-matched teacher for the entire rollout. This dependence on prompt-level domain labels restricts using unlabeled training mixtures and leaves complementary signals from other teachers unused. We introduce MOPD-Router, a framework that routes supervision over the full teacher pool at each token, without domain labels or training a separate routing model. Its plug-in interface supports different metrics for selecting and weighting teacher-specific OPD signals. Within this interface, we propose ExpertAlign, which scores each teacher by whether its correction to the student at the current token expresses the specialization that teacher acquired during post-training, and compare it against two reference metrics built on teacher confidence (Entropy) and teacher-student discrepancy (Novelty). Experiments on unlabeled and domain-labeled training mixtures under strong-to-weak and same-size distillation scenarios show that ExpertAlign achieves the strongest overall performance in all four settings. On unlabeled data, it improves the overall score by 5.88 (+12.3%) points over Mean aggregation; on domain-labeled data, it outperforms standard MOPD by 3.95 (+7.8%) points without using available domain labels. These results demonstrate token-level routing can exploit cross-domain complementary supervision, and reduce exclusive reliance on prompt-level domain assignment. Code is available at: https://github.com/TURLEing/MOPD-Router.

Figures & tables

Figure 1: Overview of MOPD-Router. (a) Unlike prompt-level hard routing, token-level routing dynamically combines supervision from the full teacher pool. (b) Teachers evaluate the student’s on-policy rollout, while a plug-in metric weights their token-level OPD signals. (c) Three routing metrics: teacher confidence (Entropy), teacher–student difference (Novelty), and specialization–direction alignment (ExpertAlign).

Baseline Models
	Math				Code			IF		Overall
Method	AIME24	AIME25	HMMT Feb.	HMMT Nov.	Human Eval+	MBPP+	LCB-v6	IFEval	IFBench	Avg.
Qwen3-1.7B	12.50	11.67	5.83	4.58	60.40	52.40	12.00	67.65	16.67	27.08
Qwen3-4B	23.33	10.00	10.00	7.08	82.30	67.20	18.29	81.88	24.14	36.02
RL Teacher	58.33	52.08	28.75	35.00	84.14	72.20	24.57	82.80	31.29	52.13
Student: Qwen3-1.7B-Non-Thinking

Table 1: Results (%) on the unlabeled mixed training set. Bold denotes the best distilled result within each student block; shaded rows identify MOPD-Router methods. The final row reports the same-size ExpertAlign result minus Standard MOPD.

Baseline Models
	Math				Code			IF		Overall
Method	AIME24	AIME25	HMMT Feb.	HMMT Nov.	Human Eval+	MBPP+	LCB-v6	IFEval	IFBench	Avg.
Qwen3-1.7B	12.50	11.67	5.83	4.58	60.40	52.40	12.00	67.65	16.67	27.08
Qwen3-4B	23.33	10.00	10.00	7.08	82.30	67.20	18.29	81.88	24.14	36.02
RL Teacher	58.33	52.08	28.75	35.00	84.14	72.20	24.57	82.80	31.29	52.13
Student: Qwen3-1.7B-Non-Thinking

Table 2: Results (%) on the domain-labeled MOPD training set. Formatting follows Table 1 .

	Student: Qwen3-1.7B				Student: Qwen3-4B
Routing metric	Math	Code	IF	Overall	Math	Code	IF	Overall
Unlabeled Training Set
Mean Aggregation	20.31	46.83	44.33	34.49	37.29	59.68	51.34	47.88
ExpertAlign- Uniform	23.12	47.77	42.81	35.71	41.46	61.78	51.31	50.42
ExpertAlign- Cosine	26.04	50.33	46.02	38.58	46.35	62.93	54.81	53.76
Labeled Training Set

Table 3: ExpertAlign ablation with per-domain averages and the Overall score (%). Math, Code, and IF are averaged over the corresponding benchmarks in Tables 1 and 2 . Uniform isolates positive-alignment teacher selection, while Cosine additionally uses the alignment value for weighting.

	Student: Qwen3-1.7B				Student: Qwen3-4B
Method	Math	Code	IF	Overall	Math	Code	IF	Overall
MOPD-Router w/ ExpertAlign	26.98	53.56	46.56	40.19	46.98	64.22	55.31	54.58
w/o Token-Level Routing	22.50	50.25	44.18	36.57	44.48	62.37	54.24	52.61
w/o Cross-Domain Teachers	26.04	51.84	40.20	37.79	45.83	60.50	50.80	51.83
Standard MOPD	25.00	50.02	43.80	37.52	42.92	60.46	51.31	50.63

Table 4: Cross-domain teacher ablation on the labeled training set (%). Domain scores average the corresponding benchmarks in Table 2 . w/o Token-Level Routing assigns fixed teacher weights per response; w/o Cross-Domain Teachers restricts routing to the teacher indicated by the domain label.

Figure 2: Cross-domain routing behavior of ExpertAlign. Left: routing mass conditioned on prompt domain and normalized within each row. Right: routing coefficients on their original scale for a Qwen3-1.7B student rollout. Semantic stage boundaries are manually annotated for exposition.

Method	Avg. resp. length	GPU-hours	E2E ms/token	Rel. E2E cost
Standard MOPD	4,893	692.9	0.32184	$1.00\times$
MOPD-Router
w/ Entropy	3,278	524.2	0.36344	$1.13\times$
w/ Novelty	3,496	539.6	0.35079	$1.09\times$
w/ ExpertAlign	4,278	716.9	0.38086	$1.18\times$

Table 5: End-to-end training cost in the same-size, domain-labeled setting.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

(a) MOPD experiments
Configuration	Value
Training epochs	3
Train batch size	1,024
Mini-batch size	1,024
Learning rate	$1\times 10^{-5}$ , constant
Gradient clipping	1.0

Appendix

Table 6: Training hyperparameters for (a) MOPD experiments and (b) the IF teacher.

Domain label	Count	Proportion
Instruction following	30,545	50.91%
Mathematics	27,476	45.79%
Code	1,979	3.30%

Appendix

Table 7: Distribution of AI-generated teacher-domain labels on the 60K unlabeled training mixture.

AI-generated label	Math	Code	IF	Total
Mathematics	99	1	0	100
Code	24	76	0	100
Instruction following	8	5	87	100

Appendix

Table 8: Comparison between AI-generated domain labels and human-majority labels on the validation sample. Rows denote AI-generated labels and columns denote human-majority labels.

Setting	Standard MOPD	Mean Aggregation	Open-MOPD	ExpertAlign
Unlabeled, 1.7B	$36.33\%$ $[33.4\%,39.3\%]$	$34.49\%$ $[31.7\%,37.4\%]$	$34.02\%$ $[32.1\%,36.0\%]$	$\mathbf{38.58\%}$ $[35.6\%,41.7\%]$
Unlabeled, 4B	$48.76\%$ $[45.5\%,52.2\%]$	$47.88\%$ $[44.6\%,51.3\%]$	$49.41\%$ $[47.7\%,51.2\%]$	$\mathbf{53.76\%}$ $[50.3\%,57.2\%]$
Labeled, 1.7B	$37.52\%$ $[34.4\%,40.8\%]$	$35.78\%$ $[32.9\%,38.8\%]$	$39.08\%$ $[37.4\%,40.8\%]$	$\mathbf{40.19\%}$ $[37.0\%,43.5\%]$
Labeled, 4B	$50.63\%$ $[47.1\%,54.2\%]$	$48.16\%$ $[44.8\%,51.6\%]$	$52.37\%$ $[50.6\%,54.2\%]$	$\mathbf{54.58\%}$ $[51.2\%,58.0\%]$

Appendix

Table 9: Overall scores and paired ExpertAlign differences with 95% bootstrap confidence intervals.

Method	Math	Code	IF	Overall
Standard MOPD	$43.40\pm 0.59$	$60.46\pm 0.08$	$51.72\pm 0.36$	$50.94\pm 0.34$
Open-MOPD	$46.06\pm 0.71$	$60.84\pm 0.58$	$54.25\pm 0.51$	$52.81\pm 0.38$
ExpertAlign	$\mathbf{47.78\pm 0.71}$	$\mathbf{63.58\pm 0.69}$	$\mathbf{55.19\pm 0.63}$	$\mathbf{54.69\pm 0.12}$

Appendix

Table 10: Training-seed robustness on the domain-labeled training set with the Qwen3-4B student. Entries are mean

\pm

sample standard deviation (%) over three training seeds. Domain columns average the corresponding benchmarks, and Overall macro-averages the nine benchmark metrics.

	Predictive entropy $\downarrow$		Routing weight (%)		Top-1 token share (%)
Teacher	1.7B	4B	1.7B	4B	1.7B	4B
Mathematics	0.337	0.294	36.79	36.57	69.02	70.31
Code	0.404	0.366	33.53	33.36	8.97	8.69
Instruction Following	0.421	0.390	29.68	30.07	22.01	21.00

Appendix

Table 11: Teacher confidence and its routing consequences over the full training trajectory of each run. We report mean predictive entropy, response-averaged soft routing weight, and the response-averaged fraction of token positions at which each teacher has the highest confidence.

Routing metric	Math	Code	IF	Overall
Mean Aggregation	20.31	46.83	44.33	34.49
Entropy	21.35	47.95	41.27	34.65
w/ Batch Calibration	21.25	46.42	44.60	34.83

Appendix

Table 12: Effect of batch-wise entropy calibration on the unlabeled training set with the Qwen3-1.7B student. Domain columns average the benchmarks within each capability.

	Student: Qwen3-1.7B				Student: Qwen3-4B
Routing metric	Math	Code	IF	Overall	Math	Code	IF	Overall
Novelty	22.92	49.47	45.13	36.70	40.73	61.70	52.34	50.30
w/o Accessibility	21.56	44.38	43.82	34.12	40.62	61.86	54.81	50.85

Appendix

Table 13: Novelty ablation results (%) on the unlabeled training set. Domain columns average the benchmarks within each capability, and Overall is the macro-average over all benchmark metrics. Bold denotes the better result within each student setting.

Student	Teacher	Top- $k$ overlap	$S$ -mass	$T$ -mass	$\widehat{a}$	Overlap JS
Qwen3-1.7B	Math	0.728	0.990	0.992	0.991	0.0386
	Code	0.736	0.991	0.990	0.990	0.0348
	IF	0.671	0.984	0.978	0.981	0.0606
Qwen3-4B	Math	0.874	0.995	0.997	0.996	0.0158
	Code	0.878	0.995	0.996	0.995	0.0131
	IF	0.762	0.991	0.989	0.990	0.0385

Appendix

Table 14: Per-teacher Novelty statistics averaged over the training trajectory.

S

-mass and

T

-mass denote the probability mass assigned by the student and teacher to their shared top-

k

candidate set.

Explore similar work

Sep 20, 2026cs.CL

Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillation

Multi-teacher on-policy distillation allows a student to learn from complementary specialists on its own trajectories. Domain-routed approaches, however, select one teacher per example and keep it fixed throughout the response. This design both depends on domain labels that mixed training corpora often lack and cannot adapt teacher selection when the expertise required changes within a trajectory. We observe that each specialist deviates more from a shared reference on in-domain prompts than on out-of-domain prompts, on average. Based on this observation, we propose \textbf{TrustMOPD}, which replaces example-level teacher selection with label-free, token-level supervision allocation. At each student-generated prefix, TrustMOPD measures this displacement in next-token preferences, calibrates its magnitude across teachers, and uses the resulting scores as proxies for local reliability to weight teacher-specific distillation losses. Evaluated across mathematics, code, and instruction following, TrustMOPD closes 91.5% and 98.0% of the overall-score gap between the initial student and oracle-routed teachers when trained on \textsc{SingleCap} and \textsc{MultiCap}, respectively, compared with 54.4% and 54.5% for the strongest label-free baseline in each setting. On \textsc{SingleCap}, it approaches label-based MOPD without using domain labels.

Jie Sun, Mao Zheng, Mingyang Song +8

University of Science and Technology of China · Foundation Model Department, Tencent · Tsinghua University +4

Aware Heterogeneous Multi-Teacher Multimodal On-Policy Distillation Teacher AI Trustworthiness Text Corpora

Ask PDF

Aug 25, 2026cs.LG

D $^3$ -MOPD: Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation

Multi-teacher on-policy distillation (MOPD) distills several domain-expert teachers into a single student by minimizing per-domain reverse-KL divergence on the student's own rollouts. Existing approaches typically fix the per-domain data mixture before training, overlooking the fact that different domains converge at substantially different rates: some plateau early while others continue to improve throughout the training budget. A fixed mixture therefore wastes compute on fast-converging domains and undertrains slower-converging ones. To address this, we propose D

^3

-MOPD (Dynamic Domain ScheDuling for MOPD), a zero-overhead scheduler that repurposes the per-domain reverse-KL signal already produced during training to adapt the domain mixture online. Running asynchronously outside the training process, an off-process watcher periodically tracks each domain's KL trajectory, estimates remaining headroom and current improvement rate, and accordingly adjusts the domain sampling ratios without altering the core training loop. Our D

^3

-MOPD scales naturally to arbitrary numbers of domains, and the expected benefit grows as more domains introduce more diverse convergence patterns for the scheduler to exploit. On a Qwen3.6-35B-A3B student distilled from four domain-expert teachers, D

^3

-MOPD closes 97% of the average student-to-teacher performance gap, compared with 63% for vanilla MOPD, reaches the same peak performance with an approximately 3

\times

reduction in rollout steps, and surpasses the specialist teachers on three of seven benchmarks.

Zechen Sun, Zhiwei Zhang, Fei Zhao +7

AllSpark Team

Aware Heterogeneous Multi-Teacher Multimodal On-Policy Distillation Teacher-Student Distillation Teacher Kullback-Leibler Divergence

Ask PDF

Sep 28, 2026cs.LG

Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation

Reinforcement learning can turn one language model into several specialists, each excellent at a single skill such as mathematics, coding or following instructions, but users need one model with all of these skills. Multi-teacher on-policy distillation (MOPD) merges them by letting the specialists teach one student: the student answers each prompt, and the specialist for that prompt's domain gives feedback on every token. This routing decides which specialist teaches, but not how strongly its feedback moves the shared student. In Qwen3.5 models at three sizes, we find that MOPD's student does not beat one taught by the best single specialist and gains little of the mathematics specialist's advantage. The feedback is unbalanced: instruction-following feedback is several times more spread out than mathematics feedback and dominates the student's updates. We propose Domain-Normalized MOPD (DN-MOPD), which keeps the routing and rescales each domain's feedback by its measured spread. On six public benchmarks, DN-MOPD improves the average score over MOPD at every size, across three random seeds and under two answer-length limits, and recovers most of the lost mathematics gain. Controls with fixed domain weights show that the gain comes mainly from turning down instruction-following feedback rather than turning up mathematics alone, and that fixed weights close to those DN-MOPD measures perform comparably. Combining specialists therefore requires deciding not only which one teaches, but also how strongly its feedback counts.

Xin Li, Hao Jiang, Xin Gao +6

Nanyang Technological University · Yale University · University of Manchester

Aware Heterogeneous Multi-Teacher Multimodal On-Policy Distillation Teacher Feedback

Ask PDF

Abstract

Explore similar work

Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillation

D3^33-MOPD: Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation

Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation

D $^3$ -MOPD: Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation