Beyond Policy Discrepancy: Capability-Aware Multi-Teacher On-Policy Distillation

Xuanfeng Yang; Le Gai; Jiaming Wang; Xuan Zhang

cs.LGOct 18, 2026

Beyond Policy Discrepancy: Capability-Aware Multi-Teacher On-Policy Distillation

Authors: Xuanfeng Yang, Le Gai, Jiaming Wang, Xuan Zhang

Organizations: Tsinghua University

Abstract

Multi-teacher on-policy distillation (MOPD) integrates domain specialists into a shared student, but determining how strongly each teacher should supervise remains challenging. Existing discrepancy-based allocation can assign strong supervision even when the student already matches or exceeds the teacher's task performance, since behavioral mismatch does not necessarily imply a remaining capability deficit. We propose Capability-Aware Multi-Teacher On-Policy Distillation (CA-MOPD), which combines specialist merging with capability-guided allocation to account for what the student has already learned. CA-MOPD first merges reinforcement-learning specialists into a strong student initialization, then uses verifier-estimated teacher--student capability gaps to adjust domain-level distillation strength. Supervision decreases as capability gaps close and increases when estimated student capability regresses, while dense token-level guidance remains provided by the teachers. Across six benchmarks spanning mathematics, code generation, and instruction following, CA-MOPD achieves an average score of 32.22, outperforming naive MOPD, Open-MOPD, and the merged initialization by 0.86, 0.83, and 0.74 points, respectively. Controlled comparisons further support the benefits of merged initialization and capability-gap guidance under matched training budgets.

Figures & tables

Figure 1: Overview of CA-MOPD. (1) Domain RL specialists sharing a MixSFT base are merged into a strong student initialization. (2) Student responses are verified to track capability relative to a fixed, calibrated teacher reference and the student’s historical best. (3) Remaining capability headroom and regression feedback control domain weights, which combine with token balancing and global scaling to modulate dense teacher supervision.

Method	AIME24	AIME25	LCBv5	LCBv6	IFEval	IFBench	Avg
MixSFT	14.58	17.14	14.49	19.89	68.76	14.67	24.92
RouteRL	23.23	24.27	23.11	22.06	71.90	22.00	31.10
ParamMerge-TA	23.80	23.23	23.83	22.69	73.01	22.33	31.48
Naive MOPD	22.92	24.79	21.92	22.34	74.86	21.33	31.36
Open-MOPD	22.76	25.00	22.99	21.66	75.60	20.33	31.39
CA-MOPD	24.95	25.10	22.75	21.60	75.23	23.67	32.22

Table 1: Overall performance across mathematics, code generation, and instruction following. All values are percentages. Bold indicates the highest score in each column among all single-model methods.

Initialization	Stage	AIME24	AIME25	LCBv5	LCBv6	IFEval	IFBench	Avg
MixSFT	Initial	14.58	17.14	14.49	19.89	68.76	14.67	24.92
MixSFT	+ CA-MOPD	21.93	25.52	23.11	23.31	72.64	22.67	31.53
ParamMerge-TA	Initial	23.80	23.23	23.83	22.69	73.01	22.33	31.48
ParamMerge-TA	+ CA-MOPD	24.95	25.10	22.75	21.60	75.23	23.67	32.22

Table 2: Effect of initialization under a fixed 200-step CA-MOPD budget. We report performance before and after training; bold marks the better post-training result for each metric.

Figure 2: Capability-gap and discrepancy dynamics during 200 steps of ParamMerge-initialized CA-MOPD. Top: fixed calibrated teacher score minus student verifier score, in percentage points; faint lines use raw batch scores and bold lines use the logged EMA. Bottom: the OPD discrepancy proxy, shown as raw values and a trailing 20-step mean. Negative online gaps indicate that the student estimate exceeds the calibrated teacher reference. These are training-time diagnostics on changing prompts, not fixed-benchmark capability trajectories; IF uses the content_ok verifier.

Allocation signal	AIME24	AIME25	LCBv5	LCBv6	IFEval	IFBench	Avg
Capability gap	21.98	25.00	21.74	22.46	74.12	23.00	31.38
Distribution discrepancy	24.06	25.73	20.54	22.17	73.01	21.00	31.09

Table 3: Step-50 comparison of capability-gap and distribution-discrepancy guidance from MixSFT. Both variants retain the capability-based regression guard. Scores are percentages and Avg is the equal-weight six-benchmark mean. Bold marks the higher score in each column.

Domain	Share (%)	Guard steps	Initial	Final	$\Delta$
Mathematics	28.36	172/200	23.52	25.03	$+1.51$
Code	40.13	92/200	23.26	22.18	$-1.08$
Instruction following	31.51	107/200	47.67	49.45	$+1.78$

Table 4: Training allocation and domain-level outcomes for ParamMerge-initialized CA-MOPD. Share is the mean weighted token share over 200 recorded steps; guard steps count actual factor increases above the reconstructed headroom-only reference. Domain scores average the two corresponding benchmarks in Table 1 ; changes are computed before rounding, in percentage points. This is a descriptive comparison, not a component ablation.

Figure 3: Training allocation during ParamMerge-initialized CA-MOPD. Top: logged final capability factors (blue) and reconstructed headroom-only factors on the same trajectory (dashed gray); orange shading marks increases due to the regression guard. Both factors include the same warm-up override. Count denominators indicate the 200-step training budget. Bottom: weighted token shares, with raw values in faint lines and trailing 10-step means in bold; dotted lines indicate equal shares. These are controller diagnostics on changing training batches, not held-out capability curves.

Component	Mean time (s/step)	Share (%)
Student rollout generation	588.22	56.56
Model forward passes	131.19	12.62
Actor optimization	285.03	27.41
Online capability feedback	33.30	3.20

Table 5: Runtime breakdown of CA-MOPD training, averaged over the 200 steps with available timing records. Shares are relative to the total measured step time; the listed components account for

99.79\%

of this total.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Student / teachers	SmolLM3-3B MixSFT / math, code, and IF RL specialists
Initialization	Task Arithmetic, $\alpha_{d}=1$
Hardware	$2\times$ RTX PRO 5000 72GB
Precision	BF16
Training framework	VERL / FSDP with vLLM rollout
Maximum prompt length	2,048 tokens

Appendix

Table 6: Training configuration used for the main CA-MOPD experiment.

Domain	Benchmark	Temp.	Top- $p$	Top- $k$	Max Output Tokens	Max Sequence Length	Metric
Math	AIME24	0.6	0.95	$-1$	30,000	32,768	avg@64
Math	AIME25	0.6	0.95	$-1$	30,000	32,768	avg@64
Code	LCB v5	1.0	0.95	$-1$	30,000	32,768	avg@10
Code	LCB v6	1.0	0.95	$-1$	30,000	32,768	avg@10
IF	IFEval	1.0	0.95	$-1$	30,000	32,768	strict prompt
IF	IFBench	1.0	0.95	$-1$	30,000	32,768	strict prompt

Appendix

Table 7: Benchmark-specific inference and evaluation settings. All evaluations use thinking mode. Top-

k=-1

disables top-

k

filtering.

Language Model Distillation

Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.

See all 508 papers

Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillation
Sep 20, 2026Jie Sun, Mao Zheng, Mingyang Song +8Language Model Distillation Multi-Teacher Knowledge Distillation
University of Science and Technology of China · Foundation Model Department, Tencent · Tsinghua University +4
Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillation
Aug 19, 2026Huan-ang Gao, Haohan Chi, Yong Yan +7Multi-Teacher Knowledge Distillation On-Policy Distillation
SIA-Lab of Tsinghua AIR and ByteDance Seed · Institute for AI Industry Research (AIR), Tsinghua University · Department of Computer Science and Technology, Tsinghua University
No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation
Sep 28, 2026Seonghyeon Kim, Chaeyun Jang, Noah Lee +2Task Arithmetic Language Model Distillation
KAIST AI · Kakao

Sep 20, 2026J/K move · Enter open · S save

Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillation

Jie Sun, Mao Zheng, Mingyang Song, Zeyuan Liu+7

University of Science and Technology of China · Foundation Model Department, Tencent · Tsinghua University+4

Language Model Distillation Multi-Teacher Knowledge Distillation On-Policy Distillation Knowledge Distillation

Open paper AskPDF arXiv

Multi-teacher on-policy distillation allows a student to learn from complementary specialists on its own trajectories. Domain-routed approaches, however, select one teacher per example and keep it fixed throughout the response. This design both depends on domain labels that mixed training corpora often lack and cannot adapt teacher selection when the expertise required changes within a trajectory. We observe that each specialist deviates more from a shared reference on in-domain prompts than on out-of-domain prompts, on average. Based on this observation, we propose TrustMOPD, which replaces example-level teacher selection with label-free, token-level supervision allocation. At each student-generated prefix, TrustMOPD measures this displacement in next-token preferences, calibrates its magnitude across teachers, and uses the resulting scores as proxies for local reliability to weight teacher-specific distillation losses. Evaluated across mathematics, code, and instruction following, TrustMOPD closes 91.5% and 98.0% of the overall-score gap between the initial student and oracle-routed teachers when trained on SingleCap and MultiCap, respectively, compared with 54.4% and 54.5% for the strongest label-free baseline in each setting. On SingleCap, it approaches label-based MOPD without using domain labels. Our code is available at https://github.com/sunjie279/TrustMOPD-.

Abstract

Language Model Distillation

Explore similar work

Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillation

Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillation

No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation