On-policy distillation (OPD) trains a student to match the teacher's next-token distributions on the student's own trajectories and has yielded substantial empirical gains. Generalized variants allow the student to surpass the teacher by extrapolating an implicit reward in output space. The language-model head, however, attenuates this change anisotropically: much of the change encoded in the teacher's hidden states reaches the logits at a small fraction of its weight, and the sampled-token log-probability ratios on which output-space extrapolation relies inject noise that the extrapolation amplifies, making training unstable. We observe that reinforcement learning (RL) shifts a model's internal representations relative to its base checkpoint, and that the direction of this shift can be measured at every layer. Motivated by this observation, we propose RIDE (RL-Induced Direction Extrapolation), which extrapolates the RL-induced change directly in representation space: at every layer and token position, RIDE computes the residual between the teacher and its pre-RL checkpoint and regresses the student's hidden states toward targets displaced beyond the teacher along this residual. Conditioned on a sampled trajectory, this regression is equivalent to maximizing a linear directional reward defined by the residual under a quadratic penalty centered at the teacher, which makes explicit how the objective moves the student along the RL-induced direction while limiting its deviation from the teacher. Across four base/RL-teacher pairs spanning different scales, architectures, and pre-training lineages, RIDE approaches or exceeds the RL-trained teacher on every pair and is the only method whose mean does so, and it consistently outperforms output-space extrapolation, which degrades the student whenever the teacher is close to its base. Project page: https://github.com/xixixixixxxx/RIDE.
Figures & tables
Figure 1: The language-model head attenuates RL-induced representation changes unevenly. (a) Residual extrapolation before and after projection through the head. (b) Hidden-state and centered-logit energy shares of the teacher-to-base residual across the singular directions of the head. Whiskers show prompt-bootstrap 95% confidence intervals.
Figure 2: Overview of RIDE. On student-generated prefixes, the frozen base and teacher define layerwise residuals. RIDE extrapolates hidden-state targets along these residuals and trains the student by masked regression toward them. Right: target geometry at a single layer and position.
Figure 3: Avg@16 relative to the RL-trained teacher (points; the horizontal line is the teacher) for every method and base/teacher pair in Table 1 . RIDE is the only method whose mean lies above the line on all four pairs; across-seed standard deviations are in Table 4 .
R1-Distill-1.5B → JustRL-1.5B
Qwen3-4B → Just-Qwen3-4B
Method
AIME24
AIME25
AIMO
Avg.
AIME24
AIME25
AIMO
Avg.
Teacher
50.80
35.60
79.50
55.30
63.13
51.04
82.61
65.59
Student
32.90
21.90
62.20
39.00
61.25
48.75
78.46
62.82
OPD top-1
47.10
33.50
78.10
52.90
52.50
45.83
79.40
59.24
OPD top-16
47.10
34.00
76.50
52.53
52.92
47.08
79.22
59.74
OPRD
49.80
34.60
79.10
54.50
61.46
44.38
80.20
62.01
Table 1: Main results (Avg@16, %) on four base/RL-teacher pairs. Trained-method entries are means over three independent training seeds; the teacher and untouched student are fixed-checkpoint reference evaluations. Each run uses its own on-policy rollouts; within a pair, all runs share prompts, sampling settings, and optimizer schedule. ExOPD and RIDE use λ=1.25 with the pre-RL checkpoint as reference. Bold marks the best distillation method per column.
Figure 4: ExOPD (top) and RIDE (bottom) at six extrapolation coefficients on the R1-Distill-1.5B pair (one run each): validation Avg@16 (%) averaged over AIME24, AIME25, and AIMO; mean response length over the preceding 50 steps; validation format score.
Figure 5: Mechanistic analysis on the R1-Distill-1.5B pair ( λ=1.25 , step 500 , final layer). (a) The 512 weakest head directions hold 79.8% of the residual’s hidden-state energy versus 33.3% under isotropy. (b) Realized student change versus target. (c) Next-token conditional variance of the ExOPD advantage and the RIDE gradient norm across λ .
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Method
λ
AIME24
AIME25
AIMO
Avg@16
Format
Length
RIDE
0.5
42.4
28.2
74.0
48.2
85.3
7430
0.75
49.6
32.5
79.6
53.9
96.0
6789
1 (OPRD)
46.5
35.0
81.3
54.3
97.2
5726
1.15
48.8
36.0
80.9
55.2
96.4
5663
1.25
49.8
35.6
80.9
55.4
96.6
5569
1.35
49.8
34.8
81.4
55.3
94.2
5570
Appendix
Table 2: Extrapolation-coefficient sweep on the R1-Distill-1.5B pair (one run per coefficient). Per-benchmark and Avg@16 columns are validation Avg@16 (%) at the final checkpoint; “Format” is the validation format score (%); “Length” is the mean response length (tokens) over the 50 steps preceding the final checkpoint. Bold marks the best final Avg@16 within each method.
Figure 6: Final Avg@16 (%) per benchmark as a function of the extrapolation coefficient on the R1-Distill-1.5B pair (one run per coefficient). The dotted line marks λ=1 , which is OPRD for RIDE and sampled-token OPD for ExOPD.
Figure 7: Extrapolation-coefficient sweep viewed per run. (a) Final Avg@16 against mean response length at the final checkpoint; markers are labeled by λ , connected in increasing λ , and sized by validation format score. (b) Mean (marker) and min–max range (bar) of validation Avg@16 over the last 200 training steps (steps 300 – 500 ) for the six coefficients tracked in Fig. 4 .
Target construction
Avg@16
Teacher matching (OPRD, λ=1 )
54.50
Random direction, same norm
55.04
Reversed direction ( λ=0.75 )
53.90
Mismatched origin ( hT−hB′ )
54.75
Trajectory-mismatched residual
55.12
RIDE (RL-induced residual)
56.38
Appendix
Table 3: Direction controls at fixed displacement magnitude; Avg@16 (%) averaged over AIME24, AIME25, and AIMO on the R1-Distill-1.5B pair; λ=1.25 unless stated.
Base/RL-teacher pair
Teacher
OPRD
RIDE
R1-Distill-1.5B
55.30
54.50±1.04
56.38±0.97
Qwen3-4B
65.59
62.01±0.96
66.07±0.88
Llama-3.2-3B
13.01
10.94±0.90
13.33±1.13
Phi-4-mini
17.96
17.33±1.06
18.30±1.05
Appendix
Table 4: Mean ± SD of macro Avg@16 (%) over three independent training runs (seeds 14 , 42 , and 2027 ). Means are taken from Table 1 ; SDs are across training seeds. Teacher scores are fixed-checkpoint references.
Rollout
Prompt dataset
DAPO-Math-17K
Prompts per step / responses per prompt
8 / 2
Sampling temperature
1.00
Maximum response length
16,384 tokens
Optimization
Steps
500
Appendix
Table 5: Training and evaluation configuration shared by all methods and all four base/teacher pairs.
Method
Frozen passes
Reference
To trainer
Target computed
OPD top-1 / top-16
1 (teacher)
no
T×1 / T×16 log-probs
on trainer
OPRD
1 (teacher)
no
T×L×d hidden states
on teacher worker
ExOPD
2 (teacher, base)
yes
T×1 log-ratios
on trainer
RIDE
2 (teacher, base)
yes
T×L×d hidden states
on teacher worker
Appendix
Table 6: Per-rollout cost of each method. “Frozen passes” counts forward passes of frozen models on the student’s rollout; “Reference” is whether the pre-RL checkpoint must be loaded; “To trainer” is the supervision tensor transmitted from the teacher worker per rollout ( V : vocabulary size; L , d : supervised layers and hidden width).
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.
On-policy distillation (OPD) provides dense, per-token supervision for language model post-training, but its effectiveness is bottlenecked by teacher quality: external teachers suffer from distribution mismatch, while self-distillation with privileged conditioning is limited by in-context learning capacity. We propose RISE (Recursive Improvement via Self-Extrapolating Policy Distillation), which constructs a synthetic teacher directly from the model's own RLVR training trajectory. By extrapolating the displacement between the current checkpoint and a trailing anchor---in parameter space or output logit space---RISE converts a sparse outcome-induced parameter update into a dense token-level target, without any external model or privileged conditioning. RISE combines RLVR and OPD in a complementary loop: outcome rewards ground the extrapolation toward correct reasoning, while the extrapolated teacher refines token-level decisions. Moreover, since the teacher is refreshed every iteration as the student improves, distillation becomes a recursive improvement mechanism rather than a one-shot compression step. Experiments spanning mathematical reasoning, multi-domain STEM, code generation, and multi-turn agentic tasks show that RISE outperforms RLVR-only training and on-policy self-distillation across all settings.
On-policy distillation (OPD) improves the reasoning capabilities of small language models through token-level teacher supervision on student-generated trajectories. Yet can teachers that excel at solving problems independently also guide student reasoning effectively? Prior work shows that when student prefixes follow reasoning paths that differ from the teacher's own or contain errors, teachers can be less accurate when continuing from these prefixes than when solving problems independently. To this end, we propose Prep-OPD, which uses reinforcement learning (RL) before distillation to train the teacher to adapt to the student's existing reasoning state and correct course when errors arise. Training optimizes teacher continuations from fixed student prefixes using final-answer correctness as the reward. The prepared teacher then trains the student through trajectory guidance and token-level supervision. We evaluate Prep-OPD on eight mathematical reasoning benchmarks, using Qwen3-4B-Instruct-2507 as the teacher and Qwen3-0.6B and Qwen3-1.7B as students. With the 4B teacher and 1.7B student, Prep-OPD improves average accuracy over standard OPD and the strongest baseline, Relay-OPD, by 8.28 and 2.30 percentage points, respectively. Controlled experiments further show that teacher RL conditioned on student-generated prefixes yields higher student accuracy than problem-start teacher RL with and without handoff on Qwen3-1.7B. Reusing the same prepared teacher also improves Qwen3-0.6B.
Xiaoyu Ma, Haoyue Liu, Zhichao Wang +2
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen · Shenzhen Future Network of Intelligence Institute (FNiI-Shenzhen) · Guangdong Provincial Key Laboratory of Future Networks of Intelligence, CUHK(SZ)
Policy-shift distillation reuses the difference between a teacher's log probabilities before and after reinforcement learning (RL) as supervision on student-generated trajectories. However, this endpoint policy shift cannot reveal whether local action preferences evolved consistently during teacher training. The same endpoint shift can result from early and late shifts that reinforce or partially cancel each other. We investigate whether agreement between early and late teacher policy shifts can guide selective transfer. We introduce Waypoint-guided On-Policy Distillation (Waypoint-OPD), which uses an intermediate teacher checkpoint from the same RL run to compare early and late shifts in relative action preferences at student-visited prefixes. We gate the endpoint shift according to the directional agreement between the early and late shifts, suppressing it when they point in opposing directions. Compared with endpoint-only transfer, our method requires only one additional frozen-teacher scoring pass. Using RL-trained Qwen3-4B teachers, we evaluate our method with Qwen3-8B, Qwen3-14B, and Qwen3-30B-A3B as student models in mathematical and code reasoning. In all six domain--student settings, Waypoint-OPD achieves the highest domain-average score among the compared distillation methods. Ablations with the Qwen3-8B student show that removing or shuffling gates reduces mathematical reasoning performance. These results support the use of directional agreement between early and late teacher policy shifts to weight the endpoint shift during student training.