On-policy distillation (OPD) trains a student on its own rollouts under a frozen teacher's per-token supervision. Does this dense signal reduce the training FLOPs needed to reach a target accuracy compared with reinforcement learning? We systematically compare vanilla reverse-KL OPD with Group Relative Policy Optimization (GRPO) across task difficulty, student and teacher size, domain, and optimization choices, measuring held-out accuracy against training FLOPs. With an external post-trained teacher, OPD generally loses to GRPO in accuracy per training FLOP at matched settings, even when the teacher is further trained with GRPO, and often suffers response-length inflation. Yet a different teacher changes the picture: when the teacher is the student's own GRPO checkpoint, OPD recovers that checkpoint's accuracy on math with 50× fewer distillation FLOPs than GRPO needs to reach the same accuracy. Following the same teacher setting, we consistently observe this high efficiency with a larger student, on code, and in a second model family. On a single domain, however, the GRPO checkpoint is already the desired model, so distilling it adds to total compute. Across domains, separate GRPO checkpoints hold different capabilities, and no single checkpoint serves every domain. Multi-teacher OPD (DeepSeek-AI, 2026) consolidates per-domain GRPO experts into a single student that matches each expert's accuracy, with the distillation stage costing only 5.2% (on 2 domains) and 8.0% (on 4 domains) of the specialist training FLOPs. Counting specialist training and distillation together, this pipeline reaches joint GRPO's performance on the 2-domain mix with 42.0% fewer total FLOPs.
Figures & tables
Figure 1: OPD compute efficiency, at a glance. (a) GRPO achieves higher performance at fewer FLOPs than OPD across external teacher sizes (up to 32B) on the OOD math evaluation (AIME24/25 and AMC23); the same ordering holds across the other settings. (b) On the OOD math eval with a Qwen3-4B-Base student, OPD with a self-GRPO teacher (a GRPO checkpoint of the student) is the only setup that matches GRPO performance with 50.0× fewer FLOPs than GRPO, and 135.6× fewer FLOPs than OPD with an external teacher Qwen3-8B . (c) On a multi-domain data mix with math and code, running multi-teacher OPD with self-GRPO teachers (per domain) matches joint GRPO at 42.0true% fewer total FLOPs. A single external 8 B teacher substantially underperforms despite its stronger overall performance; the trained student demonstrates severe degenerate behavior like unstoppable repetition, and Chinese characters given an English prompt.
Figure 2: GRPO vs OPD across difficulty levels. We stratify DAPO-Math-17K by mean@32 solve rate of the student ( Qwen3-1.7B-Base ) and teacher ( Qwen3-4B ). This yields eight subsets, each corresponding to a performance level by student and teacher; each subplot shows in-domain mean@16 against cumulative training FLOPs. GRPO outperforms every OPD variant in every cell, including the cell most favourable to OPD (top-right: student barely solves, teacher performs very well). OOD results in Figure 8 .
Figure 3: Impact of running GRPO on teachers. Qwen3-4B-Base student on DAPO-Math-17K; x is cumulative training FLOPs, y is Math OOD macro mean@16 ; dotted horizontals mark each teacher’s own accuracy. We sweep teachers at 4 B, 8 B, and 14 B across three types: GRPO-trained base (left), external thinking (centre), and GRPO on external thinking (right). Left: all three GRPO-trained base teachers quickly converge the student to similar performance; only the same-size 4 B teacher is fully recovered, despite larger teacher having a higher performance ceiling; Centre: student performances climb much more slowly to the same level with GRPO-trained base; all students stay well below any teacher throughout training. Right: OPD with a further-GRPO-trained thinking teacher is more unstable than with a thinking teacher.
Figure 4: Initial top- 16 overlap correlates with the minimal teacher–student gap .
Figure 5: Multi-teacher OPD beats joint GRPO on domain-averaged performance. Qwen3-4B-Base student. Left: 2 -domain average (math + code). Right: 4 -domain average (math, code, instruction following, knowledge). Joint GRPO variants train on all domains simultaneously; multi-teacher OPD merges per-domain GRPO experts into a single student. In two plots, the cost of Self-GRPO OPD with multiple teachers starts from the total FLOPs used to obtain all self-GRPO teachers. A single external 8 B-thinking teacher (purple) can improve over initial student on 2-domain setting, but collapse on 4-domain setting. Per-domain breakdowns are in Appendix J .
Appendix figures & tables38 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: No paper in our audit reports a FLOPs-based OPD compute win over GRPO. We filter 406 OPD papers posted after October 2025. Each bar is the number of papers that pass the criterion above it, and the italic text below each bar names what the next filter removes. 170 papers use a teacher that is distinct from the student and at least as large; 70 of these compare against GRPO as a baseline; 3 of those claim that OPD is cheaper than GRPO; and none of the three measures compute in FLOPs.
Thinking mode
Domain
Temperature
top- p
top- k
Response maximum (tokens)
Off
Math
0.6
0.95
–
7168
On
Math
0.6
0.95
20
30720
On
Code (LCBv6)
0.6
0.95
20
31744
On
MMLU-Pro
0.6
0.95
20
7168
On
IFBench
0.6
0.95
20
30720
Appendix
Table 1: Evaluation generation hyperparameters by thinking mode and domain.
Setting
Value
Framework
verl 0.7.0.dev (custom fork; see § D.9 for the native replication)
Rollout inference
SGLang 0.5.2
Rollout group size G
8 rollouts per prompt
Batch size (prompts)
64
Response-length maximum
7168 tokens (unless noted otherwise)
Rollout sampling
temperature =1.0 , top- $p=$1.0 , top- k disabled
Appendix
Table 2: Shared training hyperparameters. Both GRPO and OPD use identical values for every setting in this table.
closed-form top- k vs sampled-token, no-think mode
§ D.4
Thinking mode
OpenThoughts-3-SFT init, no-think and think teachers
§ D.5
Student initialization
Qwen3-4B-Base , OpenThoughts-3-SFT variants
§ D.6
Appendix
Table 5: Deviations from the shared recipe by ablation axis. Each row names the appendix section documenting the change.
Hyperparameter
Value
Dataset
OpenThoughts-3, 1.2M-example single-turn subset
Prompts trained on
128×3000 = 384000
Loss support
response tokens only (all assistant messages)
Global batch size
128
Max sequence length
16384
Total epochs
1 (single pass)
Appendix
Table 6: SFT hyperparameters. We follow the SFT recipe from the Thinking Machines Lab OPD blog post ( Thinking Machines Lab, 2025 ) as closely as verl allows.
Table 7: SFT learning-rate sweep for the cold-start initialization. mean@16 (%) is reported. Best per row in bold.
Figure 7: On the full DAPO-Math-17k training set, GRPO reaches higher accuracy than OPD with less compute. Qwen3-1.7B-Base student, no-think, learning rate 2\times10−6 , batch size 64 , 8 rollouts per prompt, 200 steps; OPD uses Qwen3-4B , Qwen3-8B , or Qwen3-32B as the teacher. Points are evaluations every 20 steps, plotted against cumulative training FLOPs. Left: mean@16 on a held-out DAPO validation split. Right: OOD macro mean@16 over AIME24, AIME25, and AMC23. GRPO finishes at about 15EFLOP . Every OPD run stays below GRPO while spending 1.4 – 4.7× as much compute.
Figure 8: On OOD math, OPD overtakes GRPO only on the hardest training subsets. Companion to Figure 2 , with OOD macro mean@16 (AIME24, AIME25, AMC23) in place of in-domain accuracy. Rows bin the training problems by the student’s mean@32 ( =0 , >0 ), and columns bin them by the teacher’s mean@32 ( =0 , (0,0.3] , (0.3,0.6] , >0.6$$ ). Qwen3-1.7B-Base student; Qwen3-4B , Qwen3-8B , and Qwen3-32B teachers; no-think, learning rate 2\times10−6 , batch size 64 , 8 rollouts per prompt, 200 steps. GRPO wins the five cells marked “GRPO wins”, and OPD with the 32 B teacher has the lowest peak among the OPD runs in each of them. In the other three cells (the teacher =0 column and the cell with student =0 and teacher ∈(0,0.3] ), every OPD run finishes above GRPO on OOD math, even though GRPO wins the same cells in-domain.
Figure 9: GRPO also outperforms OPD across the difficulty grid with the post-trained Qwen3-1.7B student. Same stratification as Figure 2 , with Qwen3-1.7B (no-think) as the student and Qwen3-4B and Qwen3-32B (no-think) as teachers. Rows bin the training problems by the student’s mean@32 ( =0 , (0,0.3] , (0.3,0.6] , (0.6,1) ), and column groups bin them by the teacher’s mean@32 ( =0 , (0,0.3] , >0.3$$ ). Each cell shows in-domain validation mean@16 (left) and OOD macro mean@16 (right) against cumulative training FLOPs; grey cells have too few training problems. The top row trains on the full DAPO-Math-17k, and horizontal lines mark the teachers’ OOD accuracy ( Qwen3-32B : 38.6true% ; Qwen3-4B : 34.1true% ). Each cell trains on 442 problems from its subset, and “pool n ” gives the size of the full subset. GRPO outperforms both OPD runs in every cell, both in-domain and OOD.
Figure 10: No learning rate lets OPD reach the best GRPO accuracy. Qwen3-1.7B-Base student, no-think, full DAPO-Math-17k; OOD macro mean@16 against cumulative training FLOPs for learning rates 2\times10−6 , 5\times10−6 , 1\times10−5 , 2\times10−5 , and 5\times10−5 . One panel per method: GRPO (top left) and OPD with the Qwen3-4B , Qwen3-8B , and Qwen3-32B teachers. The star in each OPD panel marks the best GRPO accuracy (learning rate 2\times10−6 , about 19true% ). GRPO degrades as the learning rate grows and collapses at 5\times10−5 . OPD is more stable across learning rates, but its best configuration ( Qwen3-4B teacher at 5\times10−6 , about 18.4true% ) still ends below the star.
Figure 11: At a fixed prompt budget, no batch size lets OPD catch up with GRPO. Every configuration trains on 12800 prompts: batch size 64 for 200 steps (1 × ), 128 for 100 steps (2 × ), or 256 for 50 steps (4 × ). Qwen3-1.7B-Base student, no-think, DAPO-Math-17k, learning rate 2\times10−6 ; DAPO validation mean@16 against cumulative training FLOPs, one panel per method. The smallest batch is the best or tied-best setting for GRPO and for OPD with the 4 B and 32 B teachers, while the 8 B teacher shows no consistent ordering. No OPD run reaches the best GRPO accuracy (star) at any batch size.
Figure 12: The reverse-KL estimator barely changes OPD accuracy. The two OPD losses differ only in the reverse-KL estimator. Solid lines use the closed-form reverse KL over the student’s top- k=16 tokens; dashed lines use the sampled-token k1 estimator at the token the student generated. Qwen3-1.7B-Base student, no-think, DAPO-Math-17k, learning rate 2\times10−6 , with GRPO shown for reference. Rows are training subsets that the student never solves, binned by the teacher’s mean@32 ( (0,30\text{true}\mathrm{%}] , (30\text{true}\mathrm{%},60\text{true}\mathrm{%}] , >60\text{true}\mathrm{%}$$ ). Columns are the Qwen3-4B , Qwen3-8B , and Qwen3-32B teachers. Left block: mean@16 on a filtered DAPO validation set of 128 problems. Right block: OOD macro mean@16 . The x -axis is the training step. The two estimators trace nearly the same curves in every cell.
Figure 13: Thinking mode hurts OPD much more than GRPO. Qwen3-1.7B-Base student initialized from OpenThoughts-3 SFT, DAPO-Math-17k; OOD macro mean@16 against cumulative training FLOPs. Left: student and teachers in no-think mode. Right: student and teachers in thinking mode. The vertical dashed line marks the compute at which GRPO finishes. Switching from no-think to thinking lowers the final accuracy of GRPO by 2.1 points ( 18.9true% to 16.8true% ) and of OPD with the 8 B teacher by 2.1 points ( 17.5true% to 15.4true% ). OPD with the 4 B and 32 B teachers loses 8.2 and 7.7 points, ending at 9.1true% and 9.0true% .
Figure 14: An SFT cold start does not change the ordering of GRPO and OPD ( Qwen3-1.7B-Base , no-think). OOD macro mean@16 (AIME24, AIME25, AMC23) against the training step, one panel per method. Each panel compares the base initialization (blue) with an OpenThoughts-3 SFT initialization (red; SFT learning rate 1\times10−5 , step 3000 ). All runs use no-think mode, learning rate 2\times10−6 , DAPO-Math-17k, batch size 64 , 200 steps, and a 7168 -token response limit. The SFT initialization starts higher (about 8.5true% versus 5.5true% ), but the difference disappears by step 20 . GRPO reaches about 19true% under both initializations, and every OPD run stays below the best GRPO accuracy (star).
Figure 15: With thinking enabled, GRPO still leads OPD after an SFT cold start ( Qwen3-1.7B-Base ). OOD macro mean@16 (AIME24, AIME25, AMC23) against cumulative training FLOPs, with OPD using Qwen3-4B or Qwen3-8B as the teacher. Both panels use thinking mode, DAPO-Math-17k, batch size 64 , 8 rollouts per prompt, a 7168 -token training limit, and a 30720 -token evaluation limit. The vertical dashed line marks the compute at which GRPO finishes. Left: SFT initialization at the best SFT learning rate ( 1\times10−4 ). Right: SFT initialization at a lower SFT learning rate ( 1\times10−5 ). GRPO reaches 24.5true% (left) and 18.7true% (right). On the left, OPD with the 4 B teacher comes within 0.3 points of GRPO after about 1.7× the compute. On the right, no OPD run reaches GRPO.
Figure 16: From a strong SFT initialization, OPD falls below the starting student while GRPO improves it ( Qwen3-4B-Base ). The student is Qwen3-4B-Base after OpenThoughts-3 SFT (SFT learning rate 1\times10−4 ), which starts at 51.0true% OOD macro. OPD uses Qwen3-32B or QwQ-32B as the teacher; QwQ-32B generated the SFT corpus, so its outputs match the student’s starting distribution. Panels show the OOD macro mean@16 and its components (AIME24, AIME25, AMC23) against cumulative training FLOPs. GRPO improves to 57.7true% within about 40EFLOP . Both OPD runs drop to about 36true% OOD macro and remain 8 – 15 points below the starting student through 120EFLOP . The QwQ-32B teacher only delays the drop.
Figure 17: External thinking teachers, 4B and 8B students. Two panels, one per student size. Curves are training trajectories on DAPO-Math-17K for GRPO and for OPD against external Qwen3- { 4B , 8B , 32B } thinking teachers; x is cumulative training FLOPs, y is OOD macro mean@16 (AIME24 + AIME25 + AMC23). GRPO curves: solid up to optimizer step 200 , dotted for steps 200 – 400 (the OPD runs trained for at most 200 steps, so the solid segments are the matched-step comparison). No OPD run reaches its teacher’s accuracy within the compute budget, and a larger external teacher does not close the gap.
Figure 18: Switching the training set from DAPO-Math-17k to DeepMath-103k does not change the ordering. Qwen3-1.7B-Base student after OpenThoughts-3 SFT, thinking mode; OPD uses the Qwen3-4B teacher. OOD macro mean@16 against cumulative training FLOPs; solid lines train on DeepMath-103k and dashed lines on DAPO-Math-17k. On both training sets, OPD first rises, then drops to 2.5true% – 4.3true% , and recovers only to 7.5true% – 9.1true% by about 17EFLOP . GRPO reaches 19.0true% (DeepMath) and 17.4true% (DAPO) within 8EFLOP .
Figure 19: Code domain training: in-domain collapse, out-of-domain math transfer. Qwen3-4B-Base student trained on LiveCodeBench-v1–v5 (top row) and strict CodeForces (bottom row) with GRPO or external OPD- 32 B, lr 2\times10−6 , 16384 -token response maximum; x -axis is cumulative training compute (EFLOPs). Left column: LCBv6 mean@16 (in-domain code). OPD ( Qwen3-32B ) loses to GRPO at every evaluated compute budget on both training sets. Right column: math OOD macro mean@16 on AIME24 + AIME25 + AMC23. Despite training only on code, OPD- 32 B reaches ≈31\text{true}\mathrm{%}$$ – 34true% math OOD, beating the code-trained GRPO by roughly 10 points on both training sets — an additional evidence for OPD’s cross-domain transfer (Appendix F ).
Figure 20: The same failure appears in a different framework (tinker, Qwen3.5-9B-Base ). GRPO and OPD on DeepMath from a shared rank- 128 SFT checkpoint, with Qwen3.5-9B as the OPD teacher, for 40 optimizer steps. Top left: training accuracy ( 256 rollouts per step). Top right: AIME24 validation accuracy ( 30 problems, 8 samples each). Bottom left: mean rollout length; the dashed line is the 16384 -token limit. Bottom right: fraction of evaluation responses truncated at the limit. The dotted vertical line marks where training resumed from the step- 20 checkpoint. GRPO holds AIME24 accuracy near 0.54 . OPD falls to 0.013 as its rollouts grow to the limit, and 98.75true% of its evaluation responses are truncated. The ordering and the length saturation match the verl results in Figures 21 and 22 .
Figure 21: Replicate GRPO > OPD ordering on different frameworks. The compute ordering survives the change of framework: GRPO is above every OPD run per FLOP in both the old/custom and latest/native stacks. On latest/native stack, we follow their default choice to use sampled-token reverse KL estimator. Other than the difference we specified, the hyperparamters and configurations match between left and right plots.
Figure 22: With an external thinking teacher, OPD rollouts fill the length limit within ten steps. Qwen3-4B-Base student on DAPO-Math-17k, trained with GRPO, with OPD from the external Qwen3-4B thinking model (the same size as the student), and with OPD from the self-GRPO teacher. Left: mean response length per optimizer step; the dashed line is the 7168 -token training limit. Right: fraction of rollouts truncated at the limit. OPD with the external teacher reaches the limit by step 9 and truncates every rollout for 182 of 200 steps. GRPO and OPD with the self-GRPO teacher end at similar lengths ( 3813 and 3741 tokens) and never saturate the limit.
Figure 23: External thinking teachers make OPD students two to three times as long as the teacher. Qwen3-4B-Base student; mean response length on the OOD math evaluation ( 16 samples per problem) over training steps. Teachers at 4 B, 8 B, and 14 B are of three types: GRPO-trained base models (left), external thinking models (centre), and external thinking models further trained with GRPO (right). Dotted horizontal lines mark each teacher’s own response length on the same prompts, and the grey dashed curve is GRPO. With both kinds of thinking teacher, the student’s responses grow to 2 – 3× the teacher’s length, close to the 30720 -token evaluation limit. With GRPO-trained base teachers the gap is much smaller: the student matches the 4 B teacher’s length and is 1.4× and 1.9× as long as the 8 B and 14 B teachers.
metric
GRPO
OPD, self-GRPO teacher
OPD, external teacher
repetition loop (naive detection)
16.6%
15.2%
63.6%
median response length (tokens)
6,504
6,542
30,720 (= maximum)
↪ if detected loops removed
4,573
4,937
21,031
mean response length (tokens)
11,233
11,605
30,229
↪ if detected loops removed
7,763
8,353
19,627
≥95% of eval maximum
19.0%
21.0%
97.8%
Appendix
Table 8: Rollout pathologies at step 200 on math-only training ( 500 rollouts per run, Qwen3 tokenizer). Values in bold red are ≥2× the GRPO baseline on the same metric. “Repetition loop (naive detection)” flags a rollout when the 100 -char window response[-200:-100] appears ≥3 times elsewhere in the response — a conservative probe that catches short-period token-level loops but misses longer-period, paraphrased, and semantic repetition. “ ↪ if detected loops removed” cuts each detected-loop response at the first occurrence of its probe pattern and re-measures length; because the probe is conservative, this is an upper bound on unique-content length, and still contains an unknown amount of undetected higher-order repetition. Even against this upper bound the external-teacher response is ∼4× the GRPO de-looped median.
Task
Length (tokens)
Math
7,721
Code
1,003
IFBench
829
MMLU-Pro
804
Appendix
Table 9: Single-task GRPO reference length per task ( Qwen3-4B-Base ): mean evaluation response length, in tokens, at the checkpoint used as the multi-teacher OPD teacher.
Figure 24: Multi-teacher OPD with self-GRPO teachers keeps the specialists’ response length, while an external teacher drives responses to the generation limit. Evaluation response length of Qwen3-4B-Base students trained on (a) the 2 -domain mix (math, code) and (b) the 4 -domain mix (math, code, IFBench, MMLU-Pro). Lengths are shown as a ratio to the single-task GRPO specialist for each domain (Table 9 ). The dashed line marks a ratio of 1 , and the y -axis is logarithmic. Hatched bars reach the generation limit. Multi-teacher OPD stays within 0.93 – 1.19× of the specialists on every domain except IFBench ( 1.65 – 2.25× ). Joint GRPO produces 0.87 – 5.31× the specialist length, and OPD with the external Qwen3-8B teacher produces 3.98 – 37.05× .
Figure 25: The self-GRPO teacher result holds for an 8B student (companion to Figure 3 ). Qwen3-8B-Base student on DAPO-Math-17k; mean@16 against training FLOPs. Dotted lines mark each teacher’s accuracy, and the y -axis is broken to show the external teachers. OPD with the self-GRPO teacher reaches the teacher’s 40.1true% OOD macro within about 3EFLOP (group size G=8 ) or 1EFLOP ( G=1 ), against 27.9EFLOP for GRPO. With the external Qwen3-8B and Qwen3-32B teachers, OPD ends at 37true% – 38true% after 65 – 100EFLOP , far below those teachers’ own 76.9true% and 81.3true% .
Figure 26: On math, G=1 makes OPD with an external teacher more compute-efficient, but the self-GRPO teacher is still far cheaper. OOD macro mean@16 on DAPO-Math-17k for Qwen3-4B-Base (left) and Qwen3-8B-Base (right) students. Colour marks the teacher (blue: self-GRPO; green: external Qwen3-8B thinking); marker shape marks the rollout setting. With the external teacher, G=1 beats G=8 at every compute budget and reaches 47.7true% (4B) and 50.6true% (8B), above GRPO. The self-GRPO teacher at G=1 reaches its teacher’s accuracy within about 1EFLOP , an order of magnitude less compute.
Figure 27: The self-GRPO teacher also works from an instruction-tuned SFT model ( Olmo-3-7B-Instruct-SFT ). OOD macro mean@16 and per-benchmark accuracy against training FLOPs on DAPO-Math-17k. GRPO trains for 200 steps at G=8 ; OPD distills the step- 200 GRPO checkpoint into the starting model at G=1 . OPD matches the teacher’s 40.0true% OOD macro with 56× fewer FLOPs ( 44 – 69× per benchmark) and peaks at 51.2true% .
student base weight WB
W1
κspec
ρΣ
δΣ
∥ΔW∥
Qwen3-4B-Base
Qwen3-4B
3571
0.155
1.6\times10−2
162
Qwen3-4B-Base
GRPO-trained Qwen3-4B-Base
2.81
0.063
5.3\times10−5
1.33
Qwen3-4B-Base
mid checkpoint of OPD ( Qwen3-4B )
3.16
0.033
5.7\times10−5
2.69
Qwen3-4B-Base
last checkpoint of OPD ( Qwen3-4B )
3.42
0.025
6.1\times10−5
3.76
Qwen3-8B-Base
Qwen3-8B
2077
0.155
1.3\times10−2
202
Qwen3-8B-Base
GRPO-trained Qwen3-8B-Base
1.19
0.125
7.7\times10−5
1.48
Appendix
Table 10: Spectrum-based scalars of Zhu et al. (2026a) between a base checkpoint WB and a second checkpoint W1 , with ΔW=W1−WB . All scalars are computed in the base’s own SVD frames and aggregated over all 2-D weight matrices. Within each scale, the first two rows compare a candidate teacher against the base: the external thinking teacher and the student’s own GRPO-trained checkpoint. The remaining rows compare the OPD student against the same base (4B: external Qwen3-4B teacher, steps 100 and 200; 8B: self-GRPO teacher, step 100). External thinking teachers sit ∼3 orders of magnitude farther from the base in κspec than GRPO checkpoints. OPD students stay in the GRPO range even when they are trained against an external teacher.
Figure 28: On code, G=1 does not rescue OPD with an external teacher. Qwen3-4B-Base student trained on LiveCodeBench v1–v5; LCBv6 mean@16 against training FLOPs (log scale). Solid lines use G=8 and dashed lines G=1 ; the external teacher is Qwen3-8B (thinking). The self-GRPO teacher reaches 47.3true% within 0.25EFLOP , above GRPO ( 46.4true% ). With the external teacher, G=1 improves on G=8 at the default batch size but not with an 8× batch, and both G=1 runs stop at 32true% – 36true% , more than 10 points below GRPO.
Domain
Training data
Prompt maximum
Response cap
Batch size
Steps
Math
DAPO-Math-17K
1024
7168
64
200
Code
LiveCodeBench v1–v5
1024
16384
32
80
IFBench
IFBench-RLVR
1024
10240
64
200
MMLU-Pro
MMLU-Pro 5 -shot
4096
3072
64
200
Appendix
Table 11: Per-domain deviations for the four individual GRPO specialists. All other hyperparameters follow Tables 2 and 3 .
Figure 29: Per-domain breakdown of the 4 -domain mix. Qwen3-4B-Base student trained on math, code, IFBench, and MMLU-Pro; mean@16 on each domain against the training FLOPs spent on that domain. Dashed lines mark the self-GRPO teacher (green) and the external Qwen3-8B teacher (purple; math and code values lie above the plotted range). All OPD runs use G=1 . Self-GRPO OPD, with one or multiple teachers, reaches or comes within one point of the specialists on every domain within about 1EFLOP . Joint GRPO trails the specialists on math and code but exceeds them on IFBench. OPD with an external teacher stays well below the specialists on code and IFBench.
Figure 30: Per-domain breakdown of the 2 -domain math and code mix. Same methods and conventions as Figure 29 . Self-GRPO OPD passes the specialists ( 33.9true% on math, 46.4true% on code) within 1EFLOP on math and 0.2EFLOP on code. Joint GRPO with instance-level mixing needs about 15EFLOP to reach the math specialist and stays near 41true% on code. OPD with an external teacher stays below 32true% on math and 30true% on code.
Figure 31: Group-size ablation for multi-teacher OPD on the 4 -domain mix. Qwen3-4B-Base student; the individual GRPO specialists serve as teachers, and their accuracy is the dashed line. FLOPs exclude the cost of training the teachers. At G=1 , multi-teacher OPD reaches or comes within 0.5 points of the specialists on all four domains within 0.4EFLOP . At G=8 , it needs more compute and stays below the specialists on IFBench ( 45.1true% vs. 50.0true% ) and MMLU-Pro ( 59.2true% vs. 61.3true% ).
Figure 32: Group-size ablation for multi-teacher OPD on the 2 -domain math and code mix. Same setup as Figure 31 , with an added G=4 run. Smaller groups reach the specialists’ accuracy with less compute: on math, G=1 , G=4 , and G=8 need about 0.2 , 0.8 , and 1.6EFLOP , against 10EFLOP for the GRPO specialist; on code, about 0.17 , 0.3 , and 0.9EFLOP .
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.
On-policy distillation (OPD) accelerates post-training by providing dense token-level supervision from a frozen teacher on the student's own rollouts. Vanilla OPD applies this supervision uniformly across prompts, without checking whether the teacher is reliable for each prompt. Because reverse KL is mode-seeking, a confidently wrong teacher can induce a strong yet misleading update. Distributional proxies, such as entropy or teacher-student likelihood agreement, measure uncertainty or agreement but do not directly verify outcome correctness. We introduce Teacher-Gated On-Policy Distillation (TGOPD), built on the principle that teacher reliability should be verified at the prompt level before dense supervision is admitted. TGOPD estimates reliability from a small set of verifier-scored teacher probes and routes each prompt exclusively to dense OPD when the reliability check passes or to verifier-grounded GRPO otherwise. Across 4B and 35B students in mathematics, code, and instruction following, TGOPD outperforms Vanilla OPD in all six single-domain settings and achieves higher seven-benchmark averages at both scales under multi-domain training. By using otherwise-idle teacher capacity for reliability estimation, TGOPD also reduces teacher-side compute waste in asynchronous OPD, increasing teacher-node GPU utilization from 9.8% to 78.9% in the measured 4B single-domain run.
On-policy distillation (OPD) trains a student on its own generated responses using dense, token-level supervision from a stronger teacher. Vanilla OPD treats all teacher signals equally, assuming that the teacher's supervision is equally important for every token. However, teacher signals at different tokens may have very different effects on the student's performance: some correct important reasoning errors, while others have little effect on the final answer. Motivated by this observation, we introduce Dr. OPD (OPD Done Right), which defines the optimal weighted OPD to maximize the student's performance. We formulate Dr. OPD as a bilevel optimization problem in which the student learns from weighted teacher supervision, while the weights are selected to maximize the expected reward of the resulting student. To solve Dr. OPD, we develop an efficient iterative solver that updates the token weights and student policy alternatively. At each round, it updates weights in closed form and then takes one gradient step on the resulting weighted OPD objective. Under regularity conditions, we show that this weighted update achieves a higher expected reward than a vanilla OPD update. Empirically, across strong-to-weak and same-size distillation on math and code, Dr. OPD consistently outperforms all evaluated baselines. In particular, in the strong-to-weak distillation setting, Dr. OPD improves average math performance by 9.7 points over vanilla OPD, and enables the smaller student to surpass its larger teacher.
On-policy distillation (OPD) is increasingly studied as a cheaper alternative to reinforcement learning, yet it remains a form of knowledge distillation (KD) that relies on teacher supervision. We revisit OPD from the KD perspective by asking where the teacher's advantage lies along student-generated trajectories. Controlled token flips at teacher-student disagreements directly measure the accuracy gained from teacher corrections. Single-token flips yield a modest average gain on selected failed trajectories, whereas repeated flips raise accuracy from 28.4% to 74.5%, close to the teacher's 76.9%. Gains continue to accumulate with more flips, but mass- and KL-based selection outperform random flips by 15-18 percentage points at counts of similar magnitude. Selective intervention remains useful after OPD, but greater argmax agreement does not consistently improve accuracy across model pairs. We advocate training schedules that learn more valuable teacher corrections while maintaining distributional stability. Controlled flips and fork-resolution analysis evaluate OPD beyond accuracy by measuring which teacher preferences the student acquires and what further correction can recover.