Domain-Routed Delta Optimization for Multi-Teacher On-Policy Distillation
Organizations: School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Zhongguancun Academy, Beijing, China
Abstract
Multi-teacher on-policy distillation can yield uneven domain capabilities. We propose Domain-Routed Delta Optimization (DRD-MOPD), composing a frozen base with expert-initialized trainable deltas. Domains update their deltas with independent optimizer states and clipping. After joint training, Post-separate exports the base plus one trained delta per domain, selected by known task domain without retraining. Using AdamW, the unweighted full student reaches 32.83% Macro across six benchmarks after 20 updates, versus 32.21% for a matched shared control. Post-separate raises Macro with Open-MOPD weighting.
Figures & tables
| Method | Sample domain | Updated component |
|---|---|---|
| Own | All domains | for each sample |
| Math (avg@64) | Code (avg@10) | IF | |||||||||
| Method / model | AIME24 | AIME25 | Avg | LCB v5 | LCB v6 | Avg | IFEval | IFBench | Avg | Macro | Recovery |
| Initial models | |||||||||||
| MixSFT (epoch 4) ( 3 ) | 15.78 | 19.58 | 17.68 | 14.97 | 19.20 | 17.09 | 67.10 | 19.00 | 43.05 | 25.94 | 0.00 |
| RouteRL (routed teachers) ( 3 ) | 25.89 | 24.90 | 25.39 | 22.69 | 22.29 | 22.49 | 73.75 | 20.33 | 47.04 | 31.64 | 100.00 |
| Distilled baseline | |||||||||||
| Open-MOPD ( 3 ) | 21.98 | 23.39 | 22.68 | 21.20 | 22.29 | 21.74 | 72.09 | 23.00 | 47.54 | 30.66 | 82.72 |
| Mechanism / metric | Shared only | Open-MOPD | Open-MOPD + Ours | Ours | ||
|---|---|---|---|---|---|---|
| Full | Post-separate | Full | Post-separate | |||
| Open-MOPD mechanisms | ||||||
| 2:2:1 prompt sampling (Math:Code:IF) | ||||||
| Response-length / token-share balancing | ||||||
| Teacher–student gap weighting | ||||||
| Student advantage refresh † | ||||||
| Ours components relative to TA initialization Domain Angle to TA init. ( ∘ ) Projection coefficients (Math, Code, IF) TA init. norm Ours norm Outside energy (%) Math 22.26 (1.0015, 0.0037, 0.0001) 0.819420 0.886860 14.3486 Code 11.10 (0.0007, 1.0039, 0.0002) 1.215318 1.243330 3.7047 IF 36.85 (-0.0035, -0.0073, 1.0000) 0.627739 0.784409 35.9447 | Pairwise angles Domain pair TA init. ( ∘ ) Ours ( ∘ ) Math–Code 88.80 87.42 Math–IF 89.99 92.35 Code–IF 89.98 91.05 |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Setting | Value |
|---|---|
| Domains / component order | Math, Code, IF |
| Base / initialization | Frozen MixSFT; original expert-minus-base components, TA start |
| Evaluated checkpoint | 20 optimizer updates (one per rollout) |
| GPUs | 32 (4 nodes, 8 GPUs per node) |
| Global rollout / mini-batch | 768 / 768 prompts |
| Responses per prompt | 1 |
| Benchmark | Prompts | Responses | TA (%) | Full (%) | Split (%) | TA stops | Full stops | Split stops |
|---|---|---|---|---|---|---|---|---|
| AIME24 | 30 | 1,920 | 26.30 | 25.05 | 27.60 | 120 | 202 | 199 |
| AIME25 | 30 | 1,920 | 25.62 | 25.94 | 25.68 | 138 | 137 | 200 |
| LCB v5 | 167 | 1,670 | 24.25 | 23.95 | 24.31 | 245 | 241 | 55 |
| LCB v6 | 175 | 1,750 | 23.60 | 23.54 | 22.80 | 280 | 264 | 63 |
| IFEval | 541 | 541 | 73.20 | 74.49 | 73.20 | 20 | 4 | 6 |
| IFBench test | 300 | 300 | 21.67 | 24.00 | 22.00 | 18 | 7 | 8 |
| Vector / checkpoint | Angle reference | Angle ( ∘ ) | Projection coefficients (Math, Code, IF) | norm | Outside energy (%) |
|---|---|---|---|---|---|
| Composite models relative to the original TA mixture | |||||
| MixSFT ( 3 ) | — | (0.0000, 0.0000, 0.0000) | 0.000000 | — | |
| TA ( 6 ) | 0.00 | (0.9990, 0.9993, 1.0000) | 1.607181 | 0.0576 | |
| Orthogonal merge † | 1.66 | (0.9969, 0.9999, 0.9995) | 1.606737 | 0.0576 | |
| Open-MOPD Final ( 3 ) | 84.40 | (0.0258, 0.0319, 0.3489) | 1.295021 | 97.0211 | |
| Ours | 14.76 | (0.9993, 1.0005, 1.0003) | 1.662509 | 6.4396 | |
| Math (avg@64) | Code (avg@10) | IF | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Method / model | AIME24 | AIME25 | Avg | LCB v5 | LCB v6 | Avg | IFEval | IFBench | Avg | Macro | Recovery |
| MixSFT (epoch 4) ( 3 ) | 15.63 | 20.26 | 17.95 | 15.99 | 19.20 | 17.60 | 66.91 | 16.00 | 41.46 | 25.67 | — |
| Naive M-OPD ( 10 , 3 ) | 20.92 | 21.60 | 21.26 | 17.53 | 20.99 | 19.26 | 68.61 | 18.67 | 43.64 | 28.05 | 35.6 |
| Open-MOPD ( 3 ) | 21.98 | 22.86 | 22.42 | 20.84 | 22.63 | 21.73 | 74.49 | 24.67 | 49.58 | 31.24 | 83.4 |
| ParamMerge-TA ( 6 , 3 ) | 21.93 | 22.76 | 22.34 | 21.74 | 23.14 | 22.44 | 71.53 | 21.67 | 46.60 | 30.46 | 71.7 |
| RouteOPD (routed models) ( 3 ) | 22.34 | 23.96 | 23.15 | 22.28 | 21.14 | 21.71 | 75.60 | 24.00 | 49.80 | 31.55 | 88.0 |
| Variant | AIME24 | AIME25 | LCB v5 | LCB v6 | IFEval | IFBench |
|---|---|---|---|---|---|---|
| Shared only | 25.05 | 25.78 | 22.75 | 22.40 | 73.94 | 23.33 |
| Open-MOPD | 24.79 | 25.73 | 13.59 | 15.26 | 74.86 | 24.00 |
| Open-MOPD + Ours (Full) | 22.81 | 25.31 | 24.13 | 22.97 | 75.79 | 25.33 |
| Open-MOPD + Ours (Post-separate) | 23.96 | 26.77 | 24.97 | 23.71 | 75.97 | 23.00 |
| Ours (Full) | 25.05 | 25.94 | 23.95 | 23.54 | 74.49 | 24.00 |
| Ours (Post-separate) | 27.60 | 25.68 | 24.31 | 22.80 | 73.20 | 22.00 |
Multi-Teacher Knowledge Distillation
Multi-teacher knowledge distillation combines expertise from multiple models to train compact or unified students across tasks and domains.
See all 118 papers