A key challenge in on-policy distillation (OPD) is extracting effective supervision from teacher knowledge. Supervision from token probabilities may underutilize internal knowledge, while direct hidden-state matching forces the student to shift to the teacher's feature space and complicates cross-architecture transfer. Inspired by contrastive learning's use of relationships between instances, we propose SimOPD to transfer teacher internal knowledge through response similarities. Specifically, SimOPD uses paired memory banks of student-generated responses to extract supervision from the teacher's internal representations as similarity distributions. The student then matches these distributions within its own representation space, retaining representational freedom and removing the need for cross-model projection. Extensive experiments show that SimOPD improves the overall average over the strongest baselines by up to 2.6 and 3.1 percentage points in same- and cross-architecture settings.
Figures & tables
Figure 1: Comparison of different on-policy distillation paradigms.
Figure 2: Overview of SimOPD. For each student-generated response, the teacher and student first compute similarity distributions over paired historical responses using their respective memory banks. The student then minimizes the KL divergence between these distributions, thereby learning teacher-defined response relationships.
Method
Mathematical Reasoning
Code Generation
Overall
AIME24
AIME25
AIMO
Avg.
HumanEval
LeetCode
MBPP
Avg.
Teacher
25.8
17.5
59.0
34.1
87.8
73.3
70.4
77.2
55.6
Initial Student
7.4
5.1
27.1
13.2
57.9
43.2
40.4
47.2
30.2
OPD (sampled-token)
9.4
8.3
37.2
18.3
77.2
40.4
71.2
62.9
40.6
OPD (top- k )
9.0
8.5
35.9
17.8
75.8
40.5
67.1
61.1
39.5
ExOPD
11.7
13.3
42.2
22.4
81.1
41.7
67.7
63.5
43.0
Table 1: Same-architecture results: Qwen3-4B → Qwen3-4B-Base. Math and code use separate training runs. Scores are percentages; Avg. is the mean within each domain, and Overall averages the two domains before rounding. Indented + OPD denotes joint training. Bold and underlined numbers mark the best and second-best distilled results, respectively.
Method
Mathematical Reasoning
Code Generation
Overall
AIME24
AIME25
AIMO
Avg.
HumanEval
LeetCode
MBPP
Avg.
Teacher
25.8
20.8
58.4
35.0
88.4
68.9
70.9
76.1
55.5
Initial Student
0.8
0.8
11.7
4.4
61.4
21.8
52.2
45.1
24.8
OPD (sampled-token)
4.2
3.7
23.5
10.5
64.0
23.3
53.4
46.9
28.7
OPD (top- k )
5.0
4.3
22.9
10.7
61.9
22.0
53.5
45.8
28.3
ExOPD
7.5
4.2
22.8
11.5
62.8
21.7
53.1
45.9
28.7
Table 2: Cross-architecture results: Qwen3-8B → Qwen3-1.7B-Base. Math and code use separate training runs. Scores are percentages; Avg. is the mean within each domain, and Overall averages the two domains before rounding. Indented + OPD denotes joint training. Bold and underlined numbers mark the best and second-best distilled results, respectively.
Method
AIME24
AIME25
AIMO
Avg.
SimOPD
15.6
14.8
43.7
24.7
In-Batch Matching
14.7
12.5
41.3
22.8
Similarity Regression
15.2
11.9
42.5
23.2
Single Memory Bank
13.6
12.7
42.2
22.8
Table 3: Ablation study of SimOPD.
Figure 3: Sensitivity of SimOPD to (a) teacher layer depth, (b) memory bank capacity, (c) similarity temperature, and (d) response positions. Math Avg. denotes the mean Avg@16 (%) across AIME24, AIME25, and AIMO. Each panel varies one factor while keeping the remaining settings fixed.
Method
ARC-C
GPQA
MMLU-Pro
Avg.
Initial Student
31.0
14.1
32.5
25.9
OPD
67.6
21.5
43.5
44.2
PowerOPD
68.2
23.4
51.3
47.6
OPD 2
61.7
21.8
44.9
42.8
SimOPD (ours)
72.3
23.4
50.7
48.8
Table 4: OOD evaluation with Qwen3-4B-Base.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Accuracy
Format errors
Execution failures
OPD
53.40
14.50
32.10
ExOPD
53.10
15.30
31.60
PowerOPD
56.40
2.01
41.59
OPD 2
57.40
1.75
40.85
SimOPD
49.10
23.60
27.30
+ OPD
56.60
13.20
30.20
Appendix
Table 5: MBPP error breakdown under cross-architecture distillation. All values are percentages of evaluated samples.
Method
Time/step (s) ↓
Peak memory (GB/GPU) ↓
ExOPD
70.25
76.45
PowerOPD
66.92
76.39
OPD 2
73.69
79.69
OPRD
83.04
77.56
SimOPD (ours)
87.20
78.77
Appendix
Table 6: Training costs for same-architecture mathematical reasoning on eight A100 80GB GPUs. Memory denotes the maximum per-GPU peak. Lower is better.
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.
On-policy distillation (OPD) supervises the student exclusively in the output space by matching next-token distributions. This paradigm suffers from two limitations: (i) a high-variance gradient estimator whose signal-to-noise ratio collapses as the student approaches the teacher, and (ii) an LM-head information bottleneck that discards the teacher's intermediate hidden states. We propose On-Policy Representation Distillation (OPRD), the first method to lift on-policy distillation into the hidden-state space. OPRD aligns student and teacher representations across selected layers on the same on-policy rollouts, providing dense, deterministic, per-layer supervision while bypassing the LM head entirely. Theoretically, OPRD provides a deterministic per-sample gradient, removing the token-level estimation variance that plagues OPD, and exposes structural information that any output-space objective necessarily discards. Empirically, OPRD closes the student-teacher gap on competition mathematics benchmarks (AIME 2024, AIME 2025, and AIMO), where every output-space baseline plateaus below the teacher, while training 1.44x faster and using up to 54% less memory. We further extend OPRD to the cross-architecture setting via OPRD-Bridge. By exploiting the observation that heterogeneous models share a low-rank representational structure, we construct a frozen projector pair that aligns representations across arbitrary depth and width mismatches, shifting the alignment from the output space (which depends on a shared vocabulary) to the representation space. We validate OPRD-Bridge on both cross-architecture (Qwen3-4B -> Qwen3-1.7B-Base) and cross-tokenizer (Phi-4-mini-reasoning -> Qwen3-1.7B-Base) settings, demonstrating successful knowledge transfer even when the vocabulary-based alignment channel is unavailable. Code: https://github.com/ShenzhiYang2000/OPRD.
On-policy distillation (OPD) consolidates expert capabilities into a single student by matching teacher and student final-layer distributions on student-generated rollouts. We investigate whether depthwise prediction transitions, the changes in predictions between layers, can provide additional task-relevant supervision. Our diagnostic study finds that stronger teacher--student agreement in intermediate transitions is associated with higher answer correctness within questions, with correct responses exhibiting lower mean transition KL across all measured intermediate layer pairs. These observations motivate On-Policy Transition Distillation (OPTD), which augments final-layer distribution matching by aligning teacher and student layer-to-layer log-probability increments on student-generated rollouts. Evaluations across three backbones and eight benchmarks spanning mathematics, coding, and instruction following show that OPTD achieves the best or tied-best overall performance in both dual- and three-domain distillation. In the three-domain setting, OPTD recovers ≈90% of the overall teacher--student capability gap, compared with ≈75% for OPD.
Runyang You, Yongqi Li, Tiezheng Yu +1
The Hong Kong Polytechnic University · Huawei Technologies Ltd.
On-policy distillation (OPD) is a standard tool for transferring teacher behavior to a smaller student, but it implicitly assumes that teacher and student predictions are comparable token by token, an assumption that fails whenever the two models tokenize the same text differently. Under heterogeneous tokenizers, exact shared-token matching silently discards a large fraction of the teacher signal at precisely the positions where vocabularies disagree. We propose Simple Cross-Tokenizer OPD (SimCT), which restores this signal by enlarging the supervision space: alongside shared tokens, SimCT compares teacher and student over short multi-token continuations that both tokenizers can realize, leaving the OPD loss form itself unchanged. We show that these units are the finest jointly tokenizable supervision interface, and that coarser alternatives remove teacher-student distinctions that are useful for on-policy learning. Across three heterogeneous teacher-student pairs on mathematical reasoning and code-generation benchmarks, SimCT shows consistent gains over shared-vocabulary OPD and representative cross-tokenizer baselines, with ablations confirming that the improvements come from recovering supervision discarded by exact shared-token matching. Code is available at https://github.com/sunjie279/SimCT-.
Jie Sun, Mao Zheng, Mingyang Song +6
University of Science and Technology of China · Shanghai Innovation Institute · Large Language Model Department, Tencent +1