Multi-teacher on-policy distillation (MOPD) extends on-policy distillation (OPD) by routing domain-specific data to specialized teachers, integrating their capabilities through dense token-level supervision on student-generated trajectories. However, domain routing only distinguishes the supervision sources for different domains; updates from all domains still act on shared parameters. When teachers differ substantially in response style and domain expertise, supervision that is effective in isolation may still lead to persistent gradient conflicts and capability degradation during joint optimization. To address this challenge, we propose Cross-Teacher Guarding for MOPD (Road MOPD), which introduces non-routed teachers as auxiliary cross-domain critics on the same student trajectories and coordinates multi-teacher supervision through token-level credit assignment and gradient updates. Specifically, cross-teacher-guided credit assignment uses teacher agreement to strengthen token-level corrections jointly supported by the teachers and attenuate conflicting signals; preventive multi-teacher gradient projection uses auxiliary gradients from the same trajectories to constrain potentially conflicting updates and adaptively controls projection strength to retain target-domain learning signals. In experiments, Road MOPD mitigates degradation in the disadvantaged domain, gains over 5 percentage points under high conflict, and improves overall performance in low-conflict and three-domain settings.
Figures & tables
Figure 1: Vanilla MOPD and Road MOPD . Left: Vanilla MOPD’s domain-routed supervision can produce conflicting updates to shared parameters. Right: Road MOPD uses same-trajectory auxiliary-teacher feedback for (A) agreement-guided token credit assignment and (B) preventive gradient projection under a target-gradient norm-retention constraint.
[0pt][0pt] (a) Math × Medical
Table 1: Main results across paired-domain settings. Domain Avg. averages benchmarks within each domain; Avg. averages the two domains. Bold/underlined scores mark the best/second-best multi-domain methods per student model.
Table 3
Figure 2: Teacher disagreement and local supervision. Left: raw gradient conflict by disagreement quintile (Math: blue; Medical: orange). Right: Math (top) and Medical (bottom) response windows; blue attenuates ( λt<1 ) and orange amplifies ( λt>1 ) token supervision.
Table 5
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Domain
Source
Sampled prompts
Math
DeepMath ( He et al., 2025 ) , difficulty level ≥6 (57K candidates)
10,000
Code
Eurus-RL-Code ( Cui et al., 2025 )
10,000
Medical
MedMCQA training split ( Pal et al., 2022 )
4,000
MedReason ( Wu et al., 2025 )
3,000
Medical-R1-Distill-Data ( FreedomIntelligence, 2025 ; Chen et al., 2024 )
2,000
m23k ( Huang et al., 2025 )
1,000
Appendix
Table 5: Composition of the training prompt pools. Counts denote sampled training prompts. Each paired setting uses the two corresponding domain pools.
Hyperparameter
Value
Training / PPO minibatch size
1,008 / 1,008 prompts
Rollout batches (paired / three-domain)
58 / 116
PPO epochs per rollout batch
1
Responses per prompt
1
Rollout temperature / top- p
1.0 / 1.0
Maximum prompt / response length
2,048 / 16,384 tokens
Appendix
Table 6: Shared optimization and rollout settings of Road MOPD . Each rollout batch produces one optimizer update.
(a) Math × Medical
Math
Medical
Method
AIME24
AIME25
HMMT 25 Feb.
HMMT 25 Nov.
Avg.
MedQA
Med Xpert QA
Med- MCQA
GPQA Med
Avg.
w/o Credit Assign.
59.69
53.54
32.71
37.19
45.78
76.98
15.88
62.08
36.01
47.74
w/o Grad. Protection
61.46
54.48
33.75
39.06
47.19
76.04
15.39
63.28
37.71
48.11
w/o Adaptive Retention
58.13
55.42
33.02
36.56
45.78
77.53
15.59
63.61
36.84
48.40
PCGrad †
62.60
53.54
33.75
37.50
46.85
77.93
14.65
62.54
36.99
48.03
Appendix
Table 7: Per-benchmark breakdown of the ablations and PCGrad comparison. Avg. is the benchmark mean; rounding may cause small differences from the mean of displayed scores. Scores are percentages (higher is better).
Benchmark
Teacher
Student
Math-only
Medical-only
Code-only
Code
HumanEval+
85.98
76.68
81.40
81.10
84.30
MBPP+
79.63
65.08
69.71
72.95
70.44
LCB v6
27.00
17.14
22.00
22.14
28.00
Avg.
64.20
52.97
57.70
58.73
60.91
Math
Appendix
Table 8: Cross-domain evaluation of single-domain training checkpoints. Each Qwen3-4B single-domain model is trained with OPD on 10,000 in-domain prompts for 3 epochs and evaluated across all 11 benchmarks. The best and second-best results among the three checkpoints are shown in bold and underlined, respectively.
Benchmark
Student
Single OPD
Vanilla MOPD
ExOPD
Open- MOPD
CaMOPD
Ours
Math
AIME24
21.46
58.13
60.21
63.33
56.77
61.04
60.73
AIME25
21.77
51.67
55.21
57.71
53.33
54.17
57.40
HMMT25 Feb.
10.00
31.56
31.98
33.85
30.52
31.87
33.33
HMMT25 Nov.
8.23
35.73
37.08
38.23
36.67
38.12
39.48
Avg.
15.37
44.27
46.12
48.28
44.32
46.30
47.73
Appendix
Table 9: Full results for low-conflict Math × Code. Per-benchmark Qwen3-4B scores underlying Table 3 . Scores are percentages (higher is better); Avg. weights benchmarks equally within each domain. Bold and underlining mark the best and second-best multi-domain methods, respectively.
Benchmark
Student
Single OPD
Vanilla MOPD
ExOPD
Open- MOPD
CaMOPD
Ours
Code
HumanEval+
76.68
84.30
86.28
84.76
85.52
84.45
86.59
MBPP+
65.08
70.44
70.63
71.23
70.44
70.90
72.95
LCB
17.14
28.00
26.86
28.57
25.71
27.57
29.29
Avg.
52.97
60.91
61.26
61.52
60.56
60.97
62.94
Math
Appendix
Table 10: Full results for generalization to three domains. Per-benchmark Qwen3-4B scores underlying Table 3 . Scores are percentages (higher is better); Avg. weights benchmarks equally within each domain. Single OPD uses the matching in-domain checkpoint from Table 8 . The best and second-best results among the compared multi-domain methods are shown in bold and underlined, respectively.
Figure 4: Extended Math and Medical examples from the right panels of Figure 2 . Outlines identify the main-text windows within the longer contexts. Blue and orange denote attenuation and amplification on the unchanged scale λt∈[0.5,1.5] . Token indices are zero-based, with half-open intervals. Omitted context and the generation limit are marked.
Figure 5: Three additional Math response excerpts. Panels show the beginning, middle, and end of three distinct sampled responses to AIME2024_0 . Original token coefficients and the [0.5,1.5] color scale are preserved. A dot represents a whitespace-only token; omitted context and the generation cap are marked.
Figure 6: Three additional Medical response excerpts. Panels show the beginning, middle, and end of three distinct sampled responses to MedQA_test_0 , using the same fixed-window rule and unchanged color scale as Figure 5 . These are generated reasoning excerpts, not verified answer explanations.
Figure 7: Training loss and generation behavior across three domain pairings. Columns show individual Road MOPD runs. Top: the logged task-0 surrogate contribution on a symmetric-log scale (linear within ±0.01 ); each panel identifies the recorded domain. Middle: mean response length for each domain. Bottom: percentage of responses across both domains that reach 16,384 tokens. Curves connect all 58 logged updates; negative loss values and fluctuations are preserved.
Figure 8: Minimum norm retention over training. Each panel identifies the primary domain whose gradient is projected; columns use the same runs as Figure 7 . Blue shows the logged ratio under adaptive projection; dashed orange shows the analytical full-projection estimate on the same gradients. The dotted line marks m=0.6 . Ratios are minima over conflicting parameter-tensor units, before merged-gradient clipping. Overlapping curves indicate identical minimum retention. All logged steps are shown without smoothing.