cs.LGNov 9, 2026

Domain-Routed Delta Optimization for Multi-Teacher On-Policy Distillation

Authors: Lin Yao

Organizations: School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Zhongguancun Academy, Beijing, China

Abstract

Multi-teacher on-policy distillation can yield uneven domain capabilities. We propose Domain-Routed Delta Optimization (DRD-MOPD), composing a frozen base with expert-initialized trainable deltas. Domains update their deltas with independent optimizer states and clipping. After joint training, Post-separate exports the base plus one trained delta per domain, selected by known task domain without retraining. Using AdamW, the unweighted full student reaches 32.83% Macro across six benchmarks after 20 updates, versus 32.21% for a matched shared control. Post-separate raises Macro with Open-MOPD weighting.

Figures & tables

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. UP-MOPD: Update Projection in Multi-Teacher On-Policy Distillation

    Oct 6, 2026Taojie Zhu, Jing Jin, Yuan Xia +8Language Model DistillationGradient Projection

  2. P2^2-MOPD: Accelerating Multi-Teacher On-Policy Distillation via Parameter Priors

    Nov 2, 2026Sinuo Wang, Jiacheng Lu, Zeyu Zhang +8Language Model DistillationMulti-Teacher Knowledge Distillation

  3. Composing What Each Teacher Learned: Delta Distillation for Multi-Teacher On-Policy Learning

    Oct 7, 2026Yuanda Xu, Hejian Sang, Zhengze Zhou +4Multi-Teacher Knowledge DistillationOn-Policy Distillation