cs.CLNov 11, 2026

Janus-MOPD: Tracing the Student Learning Dynamics in Multi-Teacher On-Policy Distillation

Authors: Zongji Yu, Ming Zhang, Dongcheng Zhang, Jiawei Zhou, Haotian Pang, Yiliu Sun, Xiang Wang, Youbang Sun, 

Organizations: Shanghai AI Laboratory · Shanghai Jiao Tong University · Tsinghua University · University of Science and Technology of China

Abstract

Multi-domain capabilities are critical for practically applicable Large Reasoning Models (LRMs). Therefore, Multi-teacher On-Policy Distillation (MOPD) has become a crucial part of the post-training process, integrating the capabilities of multiple domain-specific teacher models into a single student model. However, the student usually fails to acquire multi-domain capabilities simultaneously due to undesirable domain heterogeneity. Existing works have address this problem in view of temporal uneven efficiency and spatial hard allocation in external multi-teacher supervision. While in broader multi-domain training, varying degrees of performance degradation is still largely unexplained. In this paper, we shift the focus on how the student learns and integrates multi-domain capabilities internally, and then propose Janus-MOPD, a general learning dynamics framework of geometric representations for LRMs. Specifically, our main focus is on the internal performance of multi-domain capabilities on both temporal and spatial dimensions within the student. Temporally, our study shows that the student exhibits a Expansion-Consolidation-Optimization (ECO) three-phase learning dynamics, together with inherent limitations in learning efficiency and scaling. Spatially, we summarize two types of subspace locking within and across domains by tracing the representation space of the student, while explaining its clear learning directions but inherent cross-domain conflict. Therefore, Janus-MOPD yields a unified learning dynamics formulation, a geometric tracing methodology, and systematic findings of how the student learns and integrates multi-domain capabilities. Overall, we provide a Janus perspective to internally trace the continual evolution of the multi-domain capabilities within the model itself.

Figures & tables

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Slow-Fast Multi-Teacher On-Policy Distillation for Capability Preservation

    Oct 1, 2026Xiaofei Yin, Tong Chu, Jiyuan Fu +3On-Policy DistillationMulti-Teacher Knowledge Distillation

  2. Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillation

    Aug 19, 2026Huan-ang Gao, Haohan Chi, Yong Yan +7Multi-Teacher Knowledge DistillationOn-Policy Distillation