cs.LGOct 1, 2026
SaveLatent-MOPD: Latent Multi-Teacher On-Policy Distillation
Organizations: Case Western Reserve University · Zillow Group, Inc. · University of Illinois at Chicago · University of Florida
Abstract
On-policy distillation (OPD) trains a student on the responses it generates. Existing LLM multi-teacher OPD transfers what specialists predict through their output distributions. We introduce Latent-MOPD, to our knowledge the first representation-level multi-teacher OPD method for LLMs. It integrates existing specialists through both their predictions and the hidden states used to compute them, without additional teacher training. To coordinate representation supervision from multiple specialists, we select late-layer targets according to the teacher-student relationship, bridge unequal hidden widths with a shared projection, and group updates by domain. Each teacher's supervision gradually shifts from hidden states to token predictions, with both channels using the same routed specialist. In our main same-family setting, Latent-MOPD outperforms the token-only, representation-only and uniform-averaging baselines on all nine benchmarks across math, code and logic. With the same parameter count as each teacher, the student also surpasses the per-benchmark best teacher on a majority of these benchmarks. With larger, separately developed cross-family teachers, Latent-MOPD outperforms both single-channel baselines on all benchmarks. A same-family all-layer representation-only control remains stable with domain-pure updates but collapses when teacher domains are interleaved within an update. Our results show that a single student can integrate capabilities from several specialists through both their output distributions and internal representations.
Figures & tables
Figure 1: Same-family gains beyond individual teachers, with a cross-family extension. (a) Same-family; (b) cross-family (last 1, Linear). Both panels use shared per-axis scales, with each outer vertex fixed at the same-family Latent-MOPD score. Solid polygons show Latent-MOPD; the dotted ring in (b) repeats the same-family reference. Representation-only denotes OPRD-style (all layers) in (a) and Rep-only (last 1) in (b) . Labels give Latent-MOPD scores; bold with ( Teacher) marks those above the strongest teacher. Table 1 reports all nine same-family benchmarks.
Figure 2: Three supervision designs on shared student-generated prefixes. (a) Uniform averaging gives each teacher’s token probabilities equal weight. (b) MOPD-style routes token supervision to one frozen teacher. (c) Latent-MOPD adds a representation target from the same teacher, matching to its hidden state. PG denotes the policy-gradient token update from outputs after the LM heads; bars are schematic distributions. Routed columns illustrate a math prompt. One layer pair is drawn: same-family aligns the last three using the identity, cross-family the last using a shared trainable linear map. Both use crossfade (Figure 4 ).
Figure 3: Representation targets and update organization. (a,b) Native linear CKA between matched block outputs on fixed base-student responses before training. Same-family differences increase near the output (magnified scale); cross-family similarities show a middle-layer trough. Hatching marks the selected depths; bands give one prompt-bootstrap standard deviation (Appendix E ). (c) Same-family all-layer representation-only supervision remains stable with domain-pure updates; interleaved updates collapse and finish far below the base (dotted line). Both use MATH-500 with samples per problem (Appendix F ).
Figure 4: Crossfade across rotating teachers. Domain-pure updates rotate through math, code and logic. Two selected cycles illustrate token weights rising and latent weights fading on each teacher’s own clock (Equation 3 ); bar lengths are schematic. Both regimes use this schedule, with different windows.
| Logic | Code | Math | ||||||||
| Method | GYM | BBH | MuSR | LCB-e | MBPP | MBPP+ | AIME24 | AIME25 | Minerva | Norm |
| Teacher (math) | 39.1 | 60.5 | 50.3 | 66.8 | 60.4 | 51.1 | 56.5 | 37.7 | 33.6 | – |
| Teacher (code) | 25.6 | 41.6 | 45.1 | 73.5 | 62.0 | 52.6 | 42.1 | 30.0 | 32.2 | – |
| Teacher (logic) | 52.8 | 61.8 | 50.9 | 74.3 | 59.0 | 49.6 | 46.2 | 35.6 | 33.8 | – |
| Base student | 24.6 | 39.9 | 46.4 | 51.1 | 54.4 | 45.2 | 29.2 | 21.9 | 29.9 | 0.00 |
| Uniform averaging | 40.7 | 60.0 | 50.4 | 72.8 | 61.1 | 50.7 | 48.1 | 34.0 | 34.6 † | 0.79 |
Table 1: Same-family B teachers , distilled into the B student for steps. Gray rows are teachers; blue rows are Latent-MOPD, with darker blue marking the default last-three-layer setting. Bold and underline mark the best and second-best non-teacher scores; exceeds the best teacher. Norm : base , best-teacher envelope . Appendix A specifies evaluation, Norm and baseline implementation.
| Logic | Code | Math | |||||
| Method | GYM | BBH | MBPP | MBPP+ | AIME24 | AIME25 | Norm |
| Teacher (math) | 46.1 | 54.9 | 78.2 | 64.3 | 61.9 | 52.9 | – |
| Teacher (code) | 47.6 | 74.3 | 83.7 | 70.2 | 64.4 | 51.0 | – |
| Teacher (logic) | 42.0 | 64.2 | 80.2 | 67.2 | 55.2 | 41.5 | – |
| Base student | 24.6 | 39.9 | 54.4 | 45.2 | 29.2 | 21.9 | 0.00 |
| MOPD-style (token-only) | 29.2 | 45.8 | 55.4 | 46.8 | 36.5 | 30.2 | 0.16 |
Table 2: Cross-family B teachers. Skywork-OR1-Math, AceReason-Nemotron and R1-Distill-7B, each the student’s hidden width, distilled into the same B student. Darker blue highlights the MLP variant with the highest Norm. Score markings and Norm follow Table 1 .
| Logic | Code | Math | |||||
|---|---|---|---|---|---|---|---|
| Method | GYM | BBH | MBPP | MBPP+ | AIME24 | AIME25 | Norm |
| Teacher (math) | 39.1 | 60.5 | 60.4 | 51.1 | 56.5 | 37.7 | – |
| Teacher (code) | 25.6 | 41.6 | 62.0 | 52.6 | 42.1 | 30.0 | – |
| Teacher (logic) | 52.8 | 61.8 | 59.0 | 49.6 | 46.2 | 35.6 | – |
| Base student | 24.6 | 39.9 | 54.4 | 45.2 | 29.2 | 21.9 | 0.00 |
| Parameter merge (start) | 46.3 | 60.6 | 64.9 | 55.4 | 48.1 | 34.0 | 0.87 |
Table 3: Parameter-merge initialization. Uniform merging and -step distillation on the same six benchmarks as Table 2 . Gray rows are teachers; blue marks Latent-MOPD. Bold and underline mark the best and second-best non-teacher scores, respectively.
Figure 5: Training dynamics, general retention and representation alignment. (a) Default same-family (last-three-layer) and cross-family (last-layer) Latent-MOPD on MATH-500, with samples per problem. (b) Same-family general retention under -shot likelihood scoring. (c) Same-family raw representation loss, normalized to each domain’s first logged value and excluding . Crossfade lasts updates per domain (shaded), followed by token-only supervision. Appendix A specifies scoring and loss measurement.
Figure 6: Domain-conditioned teacher matching after same-family distillation. (a,b) Mean next-token probability cosine on teacher-disagreement positions: input domains are rows and teachers are columns, on one shared color scale; outlines mark row maxima. (c) Change in matching-teacher margin (Latent-MOPD minus MOPD-style) on selected and all positions, with paired instance-bootstrap intervals. Appendix G defines the selection rule, margin and measurement protocol.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Three-teacher supervision: an illustrative schematic. All panels use the same math prompt. (a) The math, code and logic teachers contribute equally to the averaged next-token distribution, with weight each. (b) MOPD-style selects the math teacher’s distribution; the code and logic teachers remain inactive. (c) Latent-MOPD retains this token target and adds a hidden-state target from the same math teacher. Hidden-feature tiles are separate from the token distributions; their positions do not denote token or layer correspondence. Values and features are schematic.
Figure 8: Layer selection, crossfade and logic-domain training dynamics. (a) Last-layer and last-three-layer comparisons: representation-only and Latent-MOPD in the same-family (SF) setting, and Latent-MOPD with linear projection in the cross-family (CF) setting. Norm is computed separately within Tables 1 and 2 ; the two regimes have separate axes. (b) Same-family score gains from crossfade over constant weights, in percentage points, computed from the reported Table 1 scores. Both configurations use last-three-layer alignment; the comparison changes the coupled token and representation schedule. (c) Latent-MOPD on GYM-200, with samples per problem: same-family last-three-layer and cross-family last-layer linear configurations; the dashed line marks the base. The validation set is separate from the -problem GYM evaluation in the main tables.
Figure 9: Depth-matched CKA after masking high-activation dimensions. (a) Same-family, with a magnified vertical scale; (b) cross-family, on the full – scale. Each teacher is compared with the base student on the same -prompt diagnostic pool as Figure 3 a,b. Before centering, the top of dimensions by mean absolute activation are zeroed separately in each model and block. Curves show linear CKA at matched decoder-block outputs, before final normalization; shading is one standard deviation over prompt-bootstrap replicates. The masked cross-family curves retain a middle-layer trough, while the same-family minima occur before the final block. Hatching marks the selected training depths: the final three blocks in (a) and the final block in (b).
Figure 10: Interleaved representation-only training under an extended budget. The same-family all-layer control in Figure 3 c, continued to steps. (a) MATH-500; (b) GYM-200. Scores average samples per problem. Solid lines connect all measured checkpoints. Horizontal dashed lines mark the base; vertical dotted lines mark the original -step budget, and diamonds the step- values. Vertical scales differ.
Figure 11: Teacher matching across readout depth in the same-family setting. (a) Latent-MOPD minus MOPD-style matching-teacher margin, by input domain. Shading and endpoint bars show paired pointwise instance-bootstrap intervals. (b) Per-teacher cosine changes on selected math positions at the last four fitted indices and native head; positive values indicate greater similarity under Latent-MOPD. Both panels use the same teacher-disagreement selection rule. Native-head measurements are separate from the fitted readouts.
| Multi-t. | Pre-head | On-policy | |
|---|---|---|---|
| Parameter merging ( Wortsman et al., 2022 ) | ✓ | — | ✗ |
| Multi-teacher feature KD, off-policy ( You et al., 2017 ) | ✓ | ✓ | ✗ |
| Uniform averaging | ✓ | ✗ | ✓ |
| MOPD ( Ma et al., 2026 ) | ✓ | ✗ | ✓ |
| LastOPD / OPRD ( Yang et al., 2026a ; Yang et al., 2026b ) | ✗ | ✓ | ✓ |
| Latent-MOPD (ours) | ✓ | ✓ | ✓ |
Table 4: Where multi-teacher capability integration happens. Among the methods compared here, Latent-MOPD combines multiple teachers, hidden-state supervision and student-generated training samples. A dash denotes not applicable.