Robust Domain-Mixture Scheduling for Multi-Teacher On-Policy Distillation
Authors: Yankai Chen, Sida Lin, Yifan Li, Hanjun Luo, Hanrong Zhang, Zhuohan Xie, Bowei He, Wei-Chieh Huang, Meng Ding, Wei Xue, Yike Guo, Philip S. Yu, Xue Liu
Organizations: McGill University, Montreal, Canada · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · The Hong Kong University of Science and Technology, Hong Kong, China · The Chinese University of Hong Kong, Hong Kong, China · New York University, New York, USA · University of Illinois Chicago, Chicago, USA · University of Massachusetts Boston, Boston, USA
Multi-teacher on-policy distillation (MOPD) merges the capabilities of multiple domain-expert teachers, such as mathematical reasoning, code generation, and instruction following, into a single compact student. Since all domains share a limited training budget, how to allocate it across domains directly determines the quality of distillation. Existing methods allocate the budget by each domain's learning progress, but offer no guarantee for lagging domains. We therefore introduce distributionally robust optimization (DRO) into MOPD to prioritize the worst-performing domain. However, the per-domain training signals are estimated from a few rollouts and are inherently noisy, so directly applying DRO mistakes noise for the worst case, lets occasional spikes hijack the budget, and protects inherently hard domains rather than truly regressing ones. To address these issues, we propose RDS-MOPD, which keeps the DRO objective and redesigns its three parts: it shifts the allocation only when differences clearly exceed the noise, bounds the influence of any single anomaly, and defines the worst case as the regression from each domain's own best level, all with theoretical guarantees. Experiments show that RDS-MOPD outperforms recent competing methods and ranks first on most mathematical reasoning, code generation, and instruction-following benchmarks.
Figures & tables
Figure 1: Domain weights on the training trajectory. (a) Vanilla DRO-MOPD: a code KL burst (shaded) lifts the code weight from 0.30 to 0.56, which then locks in (hatched); (b) RDS-MOPD keeps re-allocating the budget without lock-in.
Figure 2: Illustration of our RDS-MOPD framework (left) and its robust scheduler (right).
Figure 3: Three modules on the mixture simplex. (a) NRC sizes the KL ball around u by evidence; without it, p flips between vertices. (b) SAE keeps p bounded after a code spike; without it, p is dragged to the code vertex. (c) WDR ’s solution q⋆ lies where the highest iso-line of ⟨q,h⟩ touches the KL ball of p .
Figure 4: RDS-MOPD vs. its variants without the corresponding module (“w/o module”), replayed on the same recorded trajectory. (a) Share of each domain vs. its progress gap in noise standard deviations (shaded: within one std); (b) share shift remaining 40 steps after a KL spike (lower is better); (c) share of the most regressing domain (higher is better).
Figure 5: RDS-MOPD vs. D 3 -MOPD with the code teacher corrupted from step 30 (shaded). (a) Share of code prompts (dashed: RDS-MOPD without corruption); (b) math KL relative to its level at step 30 (lower is better).
Figure 6: Wall-clock time per training step of MOPD methods.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Domain shares during training (seed 0). Filled/open triangles: noise-driven/other large reversals of D 3 -MOPD, marked at the most-changed domain; RDS-MOPD has none.
Figure 8: Sensitivity to ϵ of NRC (a, b) and η,κ of WDR (c, d) ( ▲ : default ϵ=4 , η=1 , κ=2 ): (a) noise-driven reversals per bootstrap replica (mean ± std); (b) mean share change per update; (c) lagging-domain share; (d) max final share (dashed: 1−2λlo , i.e., two domains at λlo ).
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China · Yili Normal University, Yining, China