On-policy distillation (OPD) trains a student on its own responses using dense token-level guidance from a stronger teacher. In long-context tasks, however, token-level teacher support can favor locally plausible responses that omit evidence distributed across the input or violate global task constraints. Task-specific verifiers, in contrast, evaluate task completion at the response level and may return graded rewards that reflect partial success. We diagnose this mismatch on fixed responses from two representative long-context evidence-aggregation tasks. Across longer input ranges, trajectory-level OPD scores become progressively less aligned with verifier rewards, indicating teacher-verifier disagreement. Motivated by this observation, we introduce Group-Calibrated On-Policy Distillation (GC-OPD). GC-OPD separately normalizes verifier rewards and trajectory-level OPD scores within each rollout group and uses their difference as a signed teacher-verifier disagreement residual. Relative-advantage-based credit assignment (RACA) distributes this trajectory-level residual across tokens according to their relative OPD advantages while preserving the original OPD signal. Across five long-context benchmarks, post-training with GC-OPD raises the five-benchmark averages of the official Qwen3-4B and Qwen3-8B checkpoints from 29.08 to 40.47 and from 35.12 to 44.65, respectively. Vanilla OPD reaches 39.31 and 43.56 under the same setup. Controlled ablations show that the signed residual is more effective than either an additional OPD-derived term or direct group-normalized verifier reward addition, while RACA further improves over uniform token allocation. Together, these results demonstrate that group-relative residual calibration can incorporate verifier outcomes without discarding dense token-level guidance. Code is available at https://github.com/SolereZhang/GC-OPD.
Figures & tables
Figure 1: Teacher–verifier disagreement in two evidence-aggregation tasks: Multi-Table Extraction (MTE) and High-Recall Retrieval (HRR). Pairwise disagreement rate captures ranking conflicts, while OPD preference gap captures the direction and magnitude of OPD preference along the verifier ordering. Both indicate weaker alignment over longer input ranges. See Section 3.2 for details.
Figure 2: Overview of GC-OPD. GC-OPD computes group-normalized verifier rewards and trajectory-level OPD scores, forms their signed residual, and uses RACA to distribute the residual across response tokens while retaining the original OPD advantage.
Task Family
# Samples
Native Verifier
Precise Long-Range Retrieval
4,693
Exact Match (B)
Evidence-Grounded Reasoning
3,204
Accuracy (B)
Multi-Table Extraction
729
IoU (G)
High-Recall Retrieval
540
Set F1 (G)
Fragment Matching and Induction
148
Subset EM (G)
Sequence Reconstruction
69
Pairwise Acc. (G)
Table 1: Composition and reward interfaces of the 9,527-prompt GoLongRL subset after the 32K filter. “B” denotes a binary reward in {0,1} ; “G” denotes a graded score in [0,1] .
Figure 3: Held-out validation used to select the residual coefficient β . The fixed set contains 231 GoLongRL examples after 32K-token filtering and has no overlap with training. White squares show the step-100 reward. Circles show the mean over steps 60, 70, 80, 90, and 100. Both summaries select β=0.10 at both model scales.
Model
Method
Benchmark
Avg.
DocMath
Frames
MRCR
CorpusQA
LBv1QA
Qwen3-4B
Raw
29.08
43.63
26.58
22.02
6.99
46.20
OPD
39.31
49.37
30.95
26.90
32.22
57.10
ExOPD †
38.22
47.75
29.61
25.95
31.91
55.90
Uni-OPD †
38.53
51.88
29.98
27.93
27.36
55.50
PowerOPD †
38.88
49.63
30.10
28.36
32.52
53.80
Table 2: Qwen3-4B and Qwen3-8B results on five long-context benchmarks (%). “Avg.” is the unweighted five-task mean. † denotes an implementation of the method’s principal training-signal mechanism under the shared setup.
Variant
Avg.
DocMath
Frames
MRCR
CorpusQA
LBv1QA
Δ
Vanilla OPD
43.56
55.13
34.59
30.44
39.82
57.80
–
Additional OPD
43.60
54.63
34.59
30.03
39.51
59.20
+0.04
Direct reward
44.19
55.13
34.59
30.40
40.73
60.10
+0.63
GC-OPD
44.65
55.50
34.59
31.10
43.77
58.30
+1.10
Table 3: Response-level signal ablation on Qwen3-8B. All augmented variants use RACA with coefficient β=0.10 and differ only in whether the added signal is At(i) , R~(i) , or ρ(i)=R~(i)−s~(i) . “Avg.” is the unweighted five-task mean, and “ Δ ” is computed from the unrounded average relative to vanilla OPD.
Token allocation
Avg.
DocMath
Frames
MRCR
CorpusQA
LBv1QA
Δ
Vanilla OPD
43.56
55.13
34.59
30.44
39.82
57.80
–
Absolute OPD
43.93
54.63
35.56
29.53
41.34
58.60
+0.38
Uniform
44.28
54.12
37.01
30.64
40.12
59.50
+0.72
RACA
44.65
55.50
34.59
31.10
43.77
58.30
+1.10
Table 4: Token-credit ablation on Qwen3-8B. All residual variants use the same signed residual with β=0.10 and differ only in token allocation. “Avg.” is the unweighted five-task mean, and “ Δ ” is computed from the unrounded average relative to vanilla OPD.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Task-conditioned teacher–verifier disagreement in four GoLongRL task families. Bars report prompt-macro pairwise disagreement rates and top-1 mismatch rates over rollout groups with at least two distinct verifier rewards. Overall pools all nine task families.
Prompt length
Samples
≤2 K
2,222
2–8K
1,958
8–16K
2,155
16–24K
1,914
24–32K
1,278
Total
9,527
Appendix
Table 5: Prompt-length distribution of the exact 32K-filtered training file.
Setting
Value
Training horizon
100 steps
Prompt batch size
32
Responses per prompt
8
Maximum prompt length
32,768
Training response cap
10,240
Learning rate
10−6
Appendix
Table 6: Shared training configuration and GC-OPD settings.
Figure 5: Illustrative fixed-rollout visualization for the 17,265-token case. It reproduces the selected context evidence and all eight independently rerolled student responses; the native label is B.
Correct B response , ρB=+3.464102
Incorrect D response , ρD=−1.154701
Token
At
ut
ct
ΔAt
At
Token
At
ut
ct
ΔAt
At
[
-4.88599
-1.85923
0.26959
0.09339
-4.79260
[
-4.88599
-1.98073
0.24248
-0.02800
-4.91399
Answer
-0.02188
0.83034
1.39285
0.48250
0.46062
Answer
-0.02188
0.68817
1.33112
-0.15370
-0.17558
]
-0.64095
0.48803
1.23928
0.42930
-0.21165
]
-0.64095
0.34849
1.17250
-0.13539
-0.77634
B
-1.91057
-0.21400
0.89341
0.30949
-1.60108
D
-0.66057
0.33773
1.16728
-0.13479
-0.79535
EOS
-0.15839
0.75486
1.36047
0.47128
0.31290
EOS
-0.17101
0.60634
1.29421
-0.14944
-0.32046
Appendix
Table 8: Representative token-level RACA recalculations for the correct-B and incorrect-D responses in Figure 5 under the final GC-OPD setting. Values are rounded; ranking checks use unrounded values.
On-policy distillation (OPD) provides token-level teacher feedback, yet common sampled-token updates assign credit locally and omit the effects of earlier actions on subsequent feedback. We introduce \method{} (Group-Relative On-Policy Distillation), which decouples feedback computation from credit assignment by aggregating token feedback into calibrated sequence-level credit. We establish a theoretical basis for this approach by showing that, under ideal on-policy assumptions, broadcasting raw summed feedback to every response token yields an unbiased estimate of the negative gradient of the sequence-level reverse-KL objective. To support effective learning in practice, \method{} discounts later feedback that may become less reliable under distribution shift and uses same-prompt rollout groups to calibrate teacher guidance. It preserves relative response preferences while retaining a controlled fraction of the group mean, which we term collective teacher pressure. Across five Qwen3 teacher--student settings and six mathematical reasoning benchmarks, \method{} outperforms vanilla OPD in macro Avg@K in every setting, including gains of 10.9 and 11.5 percentage points in two settings with matched thinking patterns. It also maintains stable training under mismatched thinking patterns, where vanilla OPD collapses. Ablations show that retaining an appropriate amount of collective teacher pressure can improve learning, while excessive collective teacher pressure or an unrestricted feedback horizon can destabilize training.
Yushu Wang, Renkai Yang, Jingcao Xu +4
Shanghai Jiao Tong University, Shanghai, China · Alibaba Group, Hangzhou, China
On-policy distillation is a powerful way to transfer reasoning ability from a strong teacher to a smaller student: the student samples trajectories from its own policy, and the teacher provides dense token-level supervision on the states the student actually visits. However, this supervision is not always reliable: a teacher can assign high likelihood to plausible but incorrect solutions, or low likelihood to correct student solutions that follow different reasoning paths. Unconditionally distilling the teacher can therefore reinforce bad modes or erase useful student behavior. To address these limitations, we introduce RG-OPD: Reward-Gated On-Policy Distillation that uses verifier feedback to decide when teacher logits should be trusted. RG-OPD bridges sparse verifier rewards and dense teacher logits, preserving token-level supervision while filtering misleading teacher signals. Across reasoning and coding benchmarks, RG-OPD produces stronger distilled students, outperforming both vanilla reverse-KL distillation and the recent TSD-KD baseline. At 1K generation length, RG-OPD improves over reverse-KL by 2.9 points and over TSD-KD by 4.9 points; in the long-generation setting, it improves over the untuned student by 8.2 points. Our code is available at https://github.com/UoC-tail/RG-OPD.
Mohammad Sadegh Akhondzadeh, Vijay Lingam, Atula Tejaswi +3
On-policy distillation (OPD) accelerates post-training by providing dense token-level supervision from a frozen teacher on the student's own rollouts. Vanilla OPD applies this supervision uniformly across prompts, without checking whether the teacher is reliable for each prompt. Because reverse KL is mode-seeking, a confidently wrong teacher can induce a strong yet misleading update. Distributional proxies, such as entropy or teacher-student likelihood agreement, measure uncertainty or agreement but do not directly verify outcome correctness. We introduce Teacher-Gated On-Policy Distillation (TGOPD), built on the principle that teacher reliability should be verified at the prompt level before dense supervision is admitted. TGOPD estimates reliability from a small set of verifier-scored teacher probes and routes each prompt exclusively to dense OPD when the reliability check passes or to verifier-grounded GRPO otherwise. Across 4B and 35B students in mathematics, code, and instruction following, TGOPD outperforms Vanilla OPD in all six single-domain settings and achieves higher seven-benchmark averages at both scales under multi-domain training. By using otherwise-idle teacher capacity for reliability estimation, TGOPD also reduces teacher-side compute waste in asynchronous OPD, increasing teacher-node GPU utilization from 9.8% to 78.9% in the measured 4B single-domain run.