MOPD-Router: Rethinking Teacher Routing in Multi-Teacher On-Policy Distillation

Tianze Xu; Yanzhao Zheng; Zhentao Zhang; Yuanqiang Yu; Chao Ma; Jihuai Zhu; Lelun Wu; Lyumanshan Ye

cs.LGSep 25, 2026

MOPD-Router: Rethinking Teacher Routing in Multi-Teacher On-Policy Distillation

Authors: Tianze Xu, Yanzhao Zheng, Zhentao Zhang, Yuanqiang Yu, Chao Ma, Jihuai Zhu, Lelun Wu, Lyumanshan Ye, +5 more

Organizations: Shanghai Jiao Tong University · GAIR · Alibaba Group · University of Science and Technology of China · Shanghai Innovation Institute

Abstract

Multi-teacher on-policy distillation (MOPD) integrates specialized capabilities into a single student, but existing practice typically hard-routes each prompt to a domain-matched teacher for the entire rollout. This dependence on prompt-level domain labels restricts using unlabeled training mixtures and leaves complementary signals from other teachers unused. We introduce MOPD-Router, a framework that routes supervision over the full teacher pool at each token, without domain labels or training a separate routing model. Its plug-in interface supports different metrics for selecting and weighting teacher-specific OPD signals. Within this interface, we propose ExpertAlign, which scores each teacher by whether its correction to the student at the current token expresses the specialization that teacher acquired during post-training, and compare it against two reference metrics built on teacher confidence (Entropy) and teacher-student discrepancy (Novelty). Experiments on unlabeled and domain-labeled training mixtures under strong-to-weak and same-size distillation scenarios show that ExpertAlign achieves the strongest overall performance in all four settings. On unlabeled data, it improves the overall score by 5.88 (+12.3%) points over Mean aggregation; on domain-labeled data, it outperforms standard MOPD by 3.95 (+7.8%) points without using available domain labels. These results demonstrate token-level routing can exploit cross-domain complementary supervision, and reduce exclusive reliance on prompt-level domain assignment. Code is available at: https://github.com/TURLEing/MOPD-Router.

Figures & tables

Figure 1: Overview of MOPD-Router. (a) Unlike prompt-level hard routing, token-level routing dynamically combines supervision from the full teacher pool. (b) Teachers evaluate the student’s on-policy rollout, while a plug-in metric weights their token-level OPD signals. (c) Three routing metrics: teacher confidence (Entropy), teacher–student difference (Novelty), and specialization–direction alignment (ExpertAlign).

Baseline Models
	Math				Code			IF		Overall
Method	AIME24	AIME25	HMMT Feb.	HMMT Nov.	Human Eval+	MBPP+	LCB-v6	IFEval	IFBench	Avg.
Qwen3-1.7B	12.50	11.67	5.83	4.58	60.40	52.40	12.00	67.65	16.67	27.08
Qwen3-4B	23.33	10.00	10.00	7.08	82.30	67.20	18.29	81.88	24.14	36.02
RL Teacher	58.33	52.08	28.75	35.00	84.14	72.20	24.57	82.80	31.29	52.13
Student: Qwen3-1.7B-Non-Thinking

Table 1: Results (%) on the unlabeled mixed training set. Bold denotes the best distilled result within each student block; shaded rows identify MOPD-Router methods. The final row reports the same-size ExpertAlign result minus Standard MOPD.

Baseline Models
	Math				Code			IF		Overall
Method	AIME24	AIME25	HMMT Feb.	HMMT Nov.	Human Eval+	MBPP+	LCB-v6	IFEval	IFBench	Avg.
Qwen3-1.7B	12.50	11.67	5.83	4.58	60.40	52.40	12.00	67.65	16.67	27.08
Qwen3-4B	23.33	10.00	10.00	7.08	82.30	67.20	18.29	81.88	24.14	36.02
RL Teacher	58.33	52.08	28.75	35.00	84.14	72.20	24.57	82.80	31.29	52.13
Student: Qwen3-1.7B-Non-Thinking

Table 2: Results (%) on the domain-labeled MOPD training set. Formatting follows Table 1 .

	Student: Qwen3-1.7B				Student: Qwen3-4B
Routing metric	Math	Code	IF	Overall	Math	Code	IF	Overall
Unlabeled Training Set
Mean Aggregation	20.31	46.83	44.33	34.49	37.29	59.68	51.34	47.88
ExpertAlign- Uniform	23.12	47.77	42.81	35.71	41.46	61.78	51.31	50.42
ExpertAlign- Cosine	26.04	50.33	46.02	38.58	46.35	62.93	54.81	53.76
Labeled Training Set

Table 3: ExpertAlign ablation with per-domain averages and the Overall score (%). Math, Code, and IF are averaged over the corresponding benchmarks in Tables 1 and 2 . Uniform isolates positive-alignment teacher selection, while Cosine additionally uses the alignment value for weighting.

	Student: Qwen3-1.7B				Student: Qwen3-4B
Method	Math	Code	IF	Overall	Math	Code	IF	Overall
MOPD-Router w/ ExpertAlign	26.98	53.56	46.56	40.19	46.98	64.22	55.31	54.58
w/o Token-Level Routing	22.50	50.25	44.18	36.57	44.48	62.37	54.24	52.61
w/o Cross-Domain Teachers	26.04	51.84	40.20	37.79	45.83	60.50	50.80	51.83
Standard MOPD	25.00	50.02	43.80	37.52	42.92	60.46	51.31	50.63

Table 4: Cross-domain teacher ablation on the labeled training set (%). Domain scores average the corresponding benchmarks in Table 2 . w/o Token-Level Routing assigns fixed teacher weights per response; w/o Cross-Domain Teachers restricts routing to the teacher indicated by the domain label.

Figure 2: Cross-domain routing behavior of ExpertAlign. Left: routing mass conditioned on prompt domain and normalized within each row. Right: routing coefficients on their original scale for a Qwen3-1.7B student rollout. Semantic stage boundaries are manually annotated for exposition.

Method	Avg. resp. length	GPU-hours	E2E ms/token	Rel. E2E cost
Standard MOPD	4,893	692.9	0.32184	$1.00\times$
MOPD-Router
w/ Entropy	3,278	524.2	0.36344	$1.13\times$
w/ Novelty	3,496	539.6	0.35079	$1.09\times$
w/ ExpertAlign	4,278	716.9	0.38086	$1.18\times$

Table 5: End-to-end training cost in the same-size, domain-labeled setting.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

(a) MOPD experiments
Configuration	Value
Training epochs	3
Train batch size	1,024
Mini-batch size	1,024
Learning rate	$1\times 10^{-5}$ , constant
Gradient clipping	1.0

Appendix

Table 6: Training hyperparameters for (a) MOPD experiments and (b) the IF teacher.

Domain label	Count	Proportion
Instruction following	30,545	50.91%
Mathematics	27,476	45.79%
Code	1,979	3.30%

Appendix

Table 7: Distribution of AI-generated teacher-domain labels on the 60K unlabeled training mixture.

AI-generated label	Math	Code	IF	Total
Mathematics	99	1	0	100
Code	24	76	0	100
Instruction following	8	5	87	100

Appendix

Table 8: Comparison between AI-generated domain labels and human-majority labels on the validation sample. Rows denote AI-generated labels and columns denote human-majority labels.

Setting	Standard MOPD	Mean Aggregation	Open-MOPD	ExpertAlign
Unlabeled, 1.7B	$36.33\%$ $[33.4\%,39.3\%]$	$34.49\%$ $[31.7\%,37.4\%]$	$34.02\%$ $[32.1\%,36.0\%]$	$\mathbf{38.58\%}$ $[35.6\%,41.7\%]$
Unlabeled, 4B	$48.76\%$ $[45.5\%,52.2\%]$	$47.88\%$ $[44.6\%,51.3\%]$	$49.41\%$ $[47.7\%,51.2\%]$	$\mathbf{53.76\%}$ $[50.3\%,57.2\%]$
Labeled, 1.7B	$37.52\%$ $[34.4\%,40.8\%]$	$35.78\%$ $[32.9\%,38.8\%]$	$39.08\%$ $[37.4\%,40.8\%]$	$\mathbf{40.19\%}$ $[37.0\%,43.5\%]$
Labeled, 4B	$50.63\%$ $[47.1\%,54.2\%]$	$48.16\%$ $[44.8\%,51.6\%]$	$52.37\%$ $[50.6\%,54.2\%]$	$\mathbf{54.58\%}$ $[51.2\%,58.0\%]$

Appendix

Table 9: Overall scores and paired ExpertAlign differences with 95% bootstrap confidence intervals.

Method	Math	Code	IF	Overall
Standard MOPD	$43.40\pm 0.59$	$60.46\pm 0.08$	$51.72\pm 0.36$	$50.94\pm 0.34$
Open-MOPD	$46.06\pm 0.71$	$60.84\pm 0.58$	$54.25\pm 0.51$	$52.81\pm 0.38$
ExpertAlign	$\mathbf{47.78\pm 0.71}$	$\mathbf{63.58\pm 0.69}$	$\mathbf{55.19\pm 0.63}$	$\mathbf{54.69\pm 0.12}$

Appendix

Table 10: Training-seed robustness on the domain-labeled training set with the Qwen3-4B student. Entries are mean

\pm

sample standard deviation (%) over three training seeds. Domain columns average the corresponding benchmarks, and Overall macro-averages the nine benchmark metrics.

	Predictive entropy $\downarrow$		Routing weight (%)		Top-1 token share (%)
Teacher	1.7B	4B	1.7B	4B	1.7B	4B
Mathematics	0.337	0.294	36.79	36.57	69.02	70.31
Code	0.404	0.366	33.53	33.36	8.97	8.69
Instruction Following	0.421	0.390	29.68	30.07	22.01	21.00

Appendix

Table 11: Teacher confidence and its routing consequences over the full training trajectory of each run. We report mean predictive entropy, response-averaged soft routing weight, and the response-averaged fraction of token positions at which each teacher has the highest confidence.

Routing metric	Math	Code	IF	Overall
Mean Aggregation	20.31	46.83	44.33	34.49
Entropy	21.35	47.95	41.27	34.65
w/ Batch Calibration	21.25	46.42	44.60	34.83

Appendix

Table 12: Effect of batch-wise entropy calibration on the unlabeled training set with the Qwen3-1.7B student. Domain columns average the benchmarks within each capability.

	Student: Qwen3-1.7B				Student: Qwen3-4B
Routing metric	Math	Code	IF	Overall	Math	Code	IF	Overall
Novelty	22.92	49.47	45.13	36.70	40.73	61.70	52.34	50.30
w/o Accessibility	21.56	44.38	43.82	34.12	40.62	61.86	54.81	50.85

Appendix

Table 13: Novelty ablation results (%) on the unlabeled training set. Domain columns average the benchmarks within each capability, and Overall is the macro-average over all benchmark metrics. Bold denotes the better result within each student setting.

Student	Teacher	Top- $k$ overlap	$S$ -mass	$T$ -mass	$\widehat{a}$	Overlap JS
Qwen3-1.7B	Math	0.728	0.990	0.992	0.991	0.0386
	Code	0.736	0.991	0.990	0.990	0.0348
	IF	0.671	0.984	0.978	0.981	0.0606
Qwen3-4B	Math	0.874	0.995	0.997	0.996	0.0158
	Code	0.878	0.995	0.996	0.995	0.0131
	IF	0.762	0.991	0.989	0.990	0.0385

Appendix

Table 14: Per-teacher Novelty statistics averaged over the training trajectory.

S

-mass and

T

-mass denote the probability mass assigned by the student and teacher to their shared top-

k

candidate set.

Explore similar work

Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillation
Sep 20, 2026Jie Sun, Mao Zheng, Mingyang Song +8Aware Heterogeneous Multi-Teacher Multimodal On-Policy Distillation Teacher
University of Science and Technology of China · Foundation Model Department, Tencent · Tsinghua University +4
D $^3$ -MOPD: Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation
Aug 25, 2026Zechen Sun, Zhiwei Zhang, Fei Zhao +7Aware Heterogeneous Multi-Teacher Multimodal On-Policy Distillation Teacher-Student Distillation
AllSpark Team
Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation
Sep 28, 2026Xin Li, Hao Jiang, Xin Gao +6Aware Heterogeneous Multi-Teacher Multimodal On-Policy Distillation Teacher
Nanyang Technological University · Yale University · University of Manchester

Sep 20, 2026, cs.CLJ/K move · Enter open · S save

Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillation

Jie Sun, Mao Zheng, Mingyang Song, Zeyuan Liu+7

University of Science and Technology of China · Foundation Model Department, Tencent · Tsinghua University+4

Aware Heterogeneous Multi-Teacher Multimodal On-Policy Distillation Teacher AI Trustworthiness Text Corpora

Open paper AskPDF arXiv

Multi-teacher on-policy distillation allows a student to learn from complementary specialists on its own trajectories. Domain-routed approaches, however, select one teacher per example and keep it fixed throughout the response. This design both depends on domain labels that mixed training corpora often lack and cannot adapt teacher selection when the expertise required changes within a trajectory. We observe that each specialist deviates more from a shared reference on in-domain prompts than on out-of-domain prompts, on average. Based on this observation, we propose \textbf{TrustMOPD}, which replaces example-level teacher selection with label-free, token-level supervision allocation. At each student-generated prefix, TrustMOPD measures this displacement in next-token preferences, calibrates its magnitude across teachers, and uses the resulting scores as proxies for local reliability to weight teacher-specific distillation losses. Evaluated across mathematics, code, and instruction following, TrustMOPD closes 91.5% and 98.0% of the overall-score gap between the initial student and oracle-routed teachers when trained on \textsc{SingleCap} and \textsc{MultiCap}, respectively, compared with 54.4% and 54.5% for the strongest label-free baseline in each setting. On \textsc{SingleCap}, it approaches label-based MOPD without using domain labels.

Abstract

Explore similar work

Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillation

D3^33-MOPD: Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation

Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation

D $^3$ -MOPD: Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation