On-policy distillation (OPD) trains a student on the responses it generates. Existing LLM multi-teacher OPD transfers what specialists predict through their output distributions. We introduce Latent-MOPD, to our knowledge the first representation-level multi-teacher OPD method for LLMs. It integrates existing specialists through both their predictions and the hidden states used to compute them, without additional teacher training. To coordinate representation supervision from multiple specialists, we select late-layer targets according to the teacher-student relationship, bridge unequal hidden widths with a shared projection, and group updates by domain. Each teacher's supervision gradually shifts from hidden states to token predictions, with both channels using the same routed specialist. In our main same-family setting, Latent-MOPD outperforms the token-only, representation-only and uniform-averaging baselines on all nine benchmarks across math, code and logic. With the same parameter count as each teacher, the student also surpasses the per-benchmark best teacher on a majority of these benchmarks. With larger, separately developed cross-family teachers, Latent-MOPD outperforms both single-channel baselines on all benchmarks. A same-family all-layer representation-only control remains stable with domain-pure updates but collapses when teacher domains are interleaved within an update. Our results show that a single student can integrate capabilities from several specialists through both their output distributions and internal representations.
Figures & tables
Figure 1: Same-family gains beyond individual teachers, with a cross-family extension. (a) Same-family; (b) cross-family (last 1, Linear). Both panels use shared per-axis scales, with each outer vertex fixed at the same-family Latent-MOPD score. Solid polygons show Latent-MOPD; the dotted ring in (b) repeats the same-family reference. Representation-only denotes OPRD-style (all layers) in (a) and Rep-only (last 1) in (b) . Labels give Latent-MOPD scores; bold with ( > Teacher) marks those above the strongest teacher. Table 1 reports all nine same-family benchmarks.
Figure 2: Three supervision designs on shared student-generated prefixes. (a) Uniform averaging gives each teacher’s token probabilities equal weight. (b) MOPD-style routes token supervision to one frozen teacher. (c) Latent-MOPD adds a representation target from the same teacher, matching gψ(zS) to its hidden state. PG denotes the policy-gradient token update from outputs after the LM heads; bars are schematic distributions. Routed columns illustrate a math prompt. One layer pair is drawn: same-family aligns the last three using the identity, cross-family the last using a shared trainable linear map. Both use crossfade (Figure 4 ).
Figure 3: Representation targets and update organization. (a,b) Native linear CKA between matched block outputs on fixed base-student responses before training. Same-family differences increase near the output (magnified scale); cross-family similarities show a middle-layer trough. Hatching marks the selected depths; bands give one prompt-bootstrap standard deviation (Appendix E ). (c) Same-family all-layer representation-only supervision remains stable with domain-pure updates; interleaved updates collapse and finish far below the base (dotted line). Both use MATH-500 with 8 samples per problem (Appendix F ).
Figure 4: Crossfade across rotating teachers. Domain-pure updates rotate through math, code and logic. Two selected cycles illustrate token weights rising and latent weights fading on each teacher’s own clock (Equation 3 ); bar lengths are schematic. Both regimes use this schedule, with different windows.
Logic
Code
Math
Method
GYM
BBH
MuSR
LCB-e
MBPP
MBPP+
AIME24
AIME25
Minerva
Norm
Teacher (math)
39.1
60.5
50.3
66.8
60.4
51.1
56.5
37.7
33.6
–
Teacher (code)
25.6
41.6
45.1
73.5
62.0
52.6
42.1
30.0
32.2
–
Teacher (logic)
52.8
61.8
50.9
74.3
59.0
49.6
46.2
35.6
33.8
–
Base student
24.6
39.9
46.4
51.1
54.4
45.2
29.2
21.9
29.9
0.00
Uniform averaging
40.7
60.0
50.4
72.8
61.1
50.7
48.1
34.0
34.6 †
0.79
Table 1: Same-family 1.5 B teachers , distilled into the 1.5 B student for 62 steps. Gray rows are teachers; blue rows are Latent-MOPD, with darker blue marking the default last-three-layer setting. Bold and underline mark the best and second-best non-teacher scores; † exceeds the best teacher. Norm : base =0 , best-teacher envelope =1 . Appendix A specifies evaluation, Norm and baseline implementation.
Logic
Code
Math
Method
GYM
BBH
MBPP
MBPP+
AIME24
AIME25
Norm
Teacher (math)
46.1
54.9
78.2
64.3
61.9
52.9
–
Teacher (code)
47.6
74.3
83.7
70.2
64.4
51.0
–
Teacher (logic)
42.0
64.2
80.2
67.2
55.2
41.5
–
Base student
24.6
39.9
54.4
45.2
29.2
21.9
0.00
MOPD-style (token-only)
29.2
45.8
55.4
46.8
36.5
30.2
0.16
Table 2: Cross-family 7 B teachers. Skywork-OR1-Math, AceReason-Nemotron and R1-Distill-7B, each ∼2.3× the student’s hidden width, distilled into the same 1.5 B student. Darker blue highlights the MLP variant with the highest Norm. Score markings and Norm follow Table 1 .
Logic
Code
Math
Method
GYM
BBH
MBPP
MBPP+
AIME24
AIME25
Norm
Teacher (math)
39.1
60.5
60.4
51.1
56.5
37.7
–
Teacher (code)
25.6
41.6
62.0
52.6
42.1
30.0
–
Teacher (logic)
52.8
61.8
59.0
49.6
46.2
35.6
–
Base student
24.6
39.9
54.4
45.2
29.2
21.9
0.00
Parameter merge (start)
46.3
60.6
64.9
55.4
48.1
34.0
0.87
Table 3: Parameter-merge initialization. Uniform merging and 62 -step distillation on the same six benchmarks as Table 2 . Gray rows are teachers; blue marks Latent-MOPD. Bold and underline mark the best and second-best non-teacher scores, respectively.
Figure 5: Training dynamics, general retention and representation alignment. (a) Default same-family (last-three-layer) and cross-family (last-layer) Latent-MOPD on MATH-500, with 8 samples per problem. (b) Same-family general retention under 0 -shot likelihood scoring. (c) Same-family raw representation loss, normalized to each domain’s first logged value and excluding β . Crossfade lasts 10 updates per domain (shaded), followed by token-only supervision. Appendix A specifies scoring and loss measurement.
Figure 6: Domain-conditioned teacher matching after same-family distillation. (a,b) Mean next-token probability cosine on teacher-disagreement positions: input domains are rows and teachers are columns, on one shared color scale; outlines mark row maxima. (c) Change in matching-teacher margin (Latent-MOPD minus MOPD-style) on selected and all positions, with paired 95% instance-bootstrap intervals. Appendix G defines the selection rule, margin and measurement protocol.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Three-teacher supervision: an illustrative schematic. All panels use the same math prompt. (a) The math, code and logic teachers contribute equally to the averaged next-token distribution, with weight 1/3 each. (b) MOPD-style selects the math teacher’s distribution; the code and logic teachers remain inactive. (c) Latent-MOPD retains this token target and adds a hidden-state target from the same math teacher. Hidden-feature tiles are separate from the token distributions; their positions do not denote token or layer correspondence. Values and features are schematic.
Figure 8: Layer selection, crossfade and logic-domain training dynamics. (a) Last-layer and last-three-layer comparisons: representation-only and Latent-MOPD in the same-family (SF) setting, and Latent-MOPD with linear projection in the cross-family (CF) setting. Norm is computed separately within Tables 1 and 2 ; the two regimes have separate axes. (b) Same-family score gains from crossfade over constant weights, in percentage points, computed from the reported Table 1 scores. Both configurations use last-three-layer alignment; the comparison changes the coupled token and representation schedule. (c) Latent-MOPD on GYM-200, with 8 samples per problem: same-family last-three-layer and cross-family last-layer linear configurations; the dashed line marks the base. The validation set is separate from the 925 -problem GYM evaluation in the main tables.
Figure 9: Depth-matched CKA after masking high-activation dimensions. (a) Same-family, with a magnified vertical scale; (b) cross-family, on the full 0 – 1 scale. Each teacher is compared with the base student on the same 192 -prompt diagnostic pool as Figure 3 a,b. Before centering, the top 1% of dimensions by mean absolute activation are zeroed separately in each model and block. Curves show linear CKA at matched decoder-block outputs, before final normalization; shading is one standard deviation over 200 prompt-bootstrap replicates. The masked cross-family curves retain a middle-layer trough, while the same-family minima occur before the final block. Hatching marks the selected training depths: the final three blocks in (a) and the final block in (b).
Figure 10: Interleaved representation-only training under an extended budget. The same-family all-layer control in Figure 3 c, continued to 120 steps. (a) MATH-500; (b) GYM-200. Scores average 8 samples per problem. Solid lines connect all measured checkpoints. Horizontal dashed lines mark the base; vertical dotted lines mark the original 62 -step budget, and diamonds the step- 62 values. Vertical scales differ.
Figure 11: Teacher matching across readout depth in the same-family setting. (a) Latent-MOPD minus MOPD-style matching-teacher margin, by input domain. Shading and endpoint bars show paired 95% pointwise instance-bootstrap intervals. (b) Per-teacher cosine changes on selected math positions at the last four fitted indices and native head; positive values indicate greater similarity under Latent-MOPD. Both panels use the same teacher-disagreement selection rule. Native-head measurements are separate from the fitted readouts.
Multi-t.
Pre-head
On-policy
Parameter merging ( Wortsman et al., 2022 )
✓
—
✗
Multi-teacher feature KD, off-policy ( You et al., 2017 )
✓
✓
✗
Uniform averaging
✓
✗
✓
MOPD ( Ma et al., 2026 )
✓
✗
✓
LastOPD / OPRD ( Yang et al., 2026a ; Yang et al., 2026b )
✗
✓
✓
Latent-MOPD (ours)
✓
✓
✓
Appendix
Table 4: Where multi-teacher capability integration happens. Among the methods compared here, Latent-MOPD combines multiple teachers, hidden-state supervision and student-generated training samples. A dash denotes not applicable.
On-policy distillation (OPD) supervises the student exclusively in the output space by matching next-token distributions. This paradigm suffers from two limitations: (i) a high-variance gradient estimator whose signal-to-noise ratio collapses as the student approaches the teacher, and (ii) an LM-head information bottleneck that discards the teacher's intermediate hidden states. We propose On-Policy Representation Distillation (OPRD), the first method to lift on-policy distillation into the hidden-state space. OPRD aligns student and teacher representations across selected layers on the same on-policy rollouts, providing dense, deterministic, per-layer supervision while bypassing the LM head entirely. Theoretically, OPRD provides a deterministic per-sample gradient, removing the token-level estimation variance that plagues OPD, and exposes structural information that any output-space objective necessarily discards. Empirically, OPRD closes the student-teacher gap on competition mathematics benchmarks (AIME 2024, AIME 2025, and AIMO), where every output-space baseline plateaus below the teacher, while training 1.44x faster and using up to 54% less memory. We further extend OPRD to the cross-architecture setting via OPRD-Bridge. By exploiting the observation that heterogeneous models share a low-rank representational structure, we construct a frozen projector pair that aligns representations across arbitrary depth and width mismatches, shifting the alignment from the output space (which depends on a shared vocabulary) to the representation space. We validate OPRD-Bridge on both cross-architecture (Qwen3-4B -> Qwen3-1.7B-Base) and cross-tokenizer (Phi-4-mini-reasoning -> Qwen3-1.7B-Base) settings, demonstrating successful knowledge transfer even when the vocabulary-based alignment channel is unavailable. Code: https://github.com/ShenzhiYang2000/OPRD.
On-policy distillation (OPD) corrects a student on the responses it writes, but its signal is the teacher's next-token distribution: it tells the student what the teacher says but misses how it thinks. Latent supervision promises the missing part by aligning the student's latent states to the teacher's. Recent methods such as OPRD bring this signal into on-policy distillation. However, we observe two failures of this recipe when distilling Qwen3-4B and Qwen3-8B into Qwen3-1.7B-Base. Early gain, late collapse: latent supervision alone lifts MATH-500 accuracy from 25 to 46 in 10 steps, but subsequent training degrades performance down to 11 with no recovery. Better alignment, worse behavior: although the alignment metric steadily improves throughout this collapse, the most aligned model turns out to be the worst performing. Further analysis suggests a mismatch in how the latent signal is applied: layers paired by depth play different roles in the two models, so continued alignment may pull the student toward teacher states it cannot understand. To address this, we propose LastOPD, which applies the latent signal only at the last-layer state, the common interface both LM heads read, and only during a 10-step crossfade into token-level OPD. This keeps the useful part of the latent signal and hands the student to token-level supervision before the collapse sets in. Extensive experiments show that LastOPD improves MATH-500 over token-only OPD by 5.55 and 4.02 points with the 4B and 8B teachers, leads on most held-out datasets, and reaches the final score of token-only OPD in about half the steps. Code is available at https://github.com/Muyiiiii/LastOPD.
Jie Yang, Zhengyu Fang, Zelin Xu +7
University of Illinois at Chicago · Visa Research · Case Western Reserve University +2
Modern large language models (LLMs) rely on reinforcement learning during post-training to push specific capabilities, yet integrating multiple capabilities into one model remains hard. Existing methods, such as Off-Policy Finetune and Mix-RL, are either inefficient or lose performance. In this work, we propose Multi-teacher On-Policy Distillation (MOPD), a post-training paradigm for combining the capabilities of multiple domain RL teachers: we first run per-domain specialised RL to obtain a set of domain teachers, then distill these teachers into the student on its own rollouts. This eliminates exposure bias and provides a dense optimization signal. On Qwen3-30B-A3B, MOPD outperforms Mix-RL, Cascade RL, Off-Policy Finetune, and Param-Merge baselines, inheriting nearly all of each teacher's capability. MOPD also enables parallel, independent development of domain teachers, removing the cross-domain coupling typical of multi-domain post-training. MOPD has been deployed in the post-training of MiMo-V2-Flash, an industrial-scale frontier model, demonstrating its practical value for capability integration in frontier-scale LLMs.