On-policy distillation (OPD) provides token-level teacher feedback, yet common sampled-token updates assign credit locally and omit the effects of earlier actions on subsequent feedback. We introduce \method{} (Group-Relative On-Policy Distillation), which decouples feedback computation from credit assignment by aggregating token feedback into calibrated sequence-level credit. We establish a theoretical basis for this approach by showing that, under ideal on-policy assumptions, broadcasting raw summed feedback to every response token yields an unbiased estimate of the negative gradient of the sequence-level reverse-KL objective. To support effective learning in practice, \method{} discounts later feedback that may become less reliable under distribution shift and uses same-prompt rollout groups to calibrate teacher guidance. It preserves relative response preferences while retaining a controlled fraction of the group mean, which we term collective teacher pressure. Across five Qwen3 teacher--student settings and six mathematical reasoning benchmarks, \method{} outperforms vanilla OPD in macro Avg@K in every setting, including gains of 10.9 and 11.5 percentage points in two settings with matched thinking patterns. It also maintains stable training under mismatched thinking patterns, where vanilla OPD collapses. Ablations show that retaining an appropriate amount of collective teacher pressure can improve learning, while excessive collective teacher pressure or an unrestricted feedback horizon can destabilize training.
Figures & tables
Figure 1: Sequence credit and training stability across five teacher–student pairings. (a) Illustrative local feedback favors incorrect B , whereas trajectory credit favors correct A ; outcomes do not enter the objective. (b) JH(H) measures how strongly prefix scores favor correct responses (higher is better). On fixed responses, it peaks around 1,500–3,000 tokens; the horizontal scale is cube-root. (c,d) Training directly on the discounted trajectory score causes verifier reward to fall as response length approaches the generation cap.
Figure 2: GR-OPD overview. (1) The student samples responses to the same prompt and receives token feedback from teacher–student log-probability differences. (2) Discounting with γ aggregates feedback into a trajectory score while down-weighting later positions. (3) Calibration preserves relative response preferences and retains a fraction λ of the group mean, termed collective teacher pressure . The two components are added and normalized by a common group standard deviation. (4) Each response’s advantage is broadcast to its tokens for a PPO-style clipped update.
Table 1: Per-suite Avg@K (%) across five teacher–student pairings. Bold marks the higher score within each pairing; Macro is the unweighted mean over the six suites before rounding. “Collapse” marks a failed training trajectory.
Figure 3: Training dynamics across two matched Instruct pairings and one cross-pattern pairing. Purple curves denote vanilla OPD, and GR-OPD colors denote λ . (a) Calibrating collective teacher pressure at γ=.99 : λ is the retained fraction of the shared component. (b) Controlling the feedback horizon: solid and dashed GR-OPD curves denote γ=.99 and γ=1 . Faint lines show per-step values, bold lines show ten-step moving averages, and endpoint dots mark the last recorded step.
Teacher → student
λ
Normalized pressure magnitude proxy
Retained pressure magnitude proxy
I4nothink→I1.7nothink
.5
3.21
1.60
I4nothink→I1.7nothink
.25
3.16
0.79
I4nothink→I1.7nothink
.2
3.17
0.63
I32nothink→I1.7nothink
.25
3.61
0.90
I32nothink→I1.7nothink
.2
3.56
0.71
I4think→B1.7
0
7.23
0
Table 2: Collective teacher pressure before and after calibration ( γ=.99 ). Cross-pattern runs show larger normalized pressure than matched runs; λ controls the amount retained alongside relative response preferences. Both columns report batch-level magnitude proxies (Appendix C.7 ).
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Prompts per rollout batch
64
Responses per prompt, G
8
Configured per-device training batch size
1
Gradient-accumulation steps
8
Student data-parallel replicas
8
Responses per global optimization minibatch
64
Appendix
Table 3: GR-OPD training configuration. Sampling settings refer to student rollouts during training.
Model used in this paper
Official checkpoint
Public revision
Qwen3-1.7B-Base
Qwen/Qwen3-1.7B-Base
ea980cb
Qwen3-4B-Base
Qwen/Qwen3-4B-Base
906bfd4
Qwen3-1.7B-Instruct
Qwen/Qwen3-1.7B
70d244c
Qwen3-4B-Instruct
Qwen/Qwen3-4B
1cfa9a7
Qwen3-32B-Instruct
Qwen/Qwen3-32B
9216db5
Appendix
Table 4: Official checkpoints for the models and tokenizers. Repository links point to fixed public revisions on Hugging Face.
Benchmark
Questions
K
AIME 2024
30
32
AIME 2025
30
32
AMC 2023
40
32
MATH-500
500
8
Minerva Math
272
8
OlympiadBench (math subset)
170
8
Appendix
Table 5: Evaluation question counts and samples per question. The six suites contain 1,042 questions in total.
Model
Mode
Role
Macro Avg@K
Mean tokens
Qwen3-1.7B-Base
default
Student
23.099
4,166.4
Qwen3-4B-Base
default
Teacher
31.244
3,245.5
Qwen3-1.7B-Instruct
nothink
Student
32.995
1,662.1
Qwen3-4B-Instruct
nothink
Teacher
42.970
1,937.5
Qwen3-4B-Instruct
think
Teacher
74.574
10,138.4
Qwen3-32B-Instruct
think
Teacher
78.954
9,221.9
Appendix
Table 6: Pre-OPD reference capability. Macro Avg@K and mean response length are averaged equally over the six suites. Mean response length is measured in tokens.
Figure 4: Training dynamics for γ=1 and γ=.99 on I32nothink→I1.7nothink . (A) Mean response length in tokens (logarithmic axis); (B) mean task-verifier reward; (C) Pearson correlation between trajectory score and response length; and (D) Spearman correlation between trajectory score and verifier reward. Correlations pool responses within each rollout batch. Curves use centered moving averages as specified above.