Beyond Teacher Likelihood: Group-Calibrated On-Policy Distillation for Long-Context Reasoning

Zhu Zhang; Jixun Wang; Xiaoang Xu; Xiaorong Wang; Zihan Zhou; Zhiyuan Wang; Shuo Wang; Chaojun Xiao

cs.LGAug 19, 2026

Beyond Teacher Likelihood: Group-Calibrated On-Policy Distillation for Long-Context Reasoning

Authors: Zhu Zhang, Jixun Wang, Xiaoang Xu, Xiaorong Wang, Zihan Zhou, Zhiyuan Wang, Shuo Wang, Chaojun Xiao,

Organizations: Tsinghua University · OpenBMB · Beijing University of Posts and Telecommunications

Abstract

On-policy distillation (OPD) trains a student on its own responses using dense token-level guidance from a stronger teacher. In long-context tasks, however, token-level teacher support can favor locally plausible responses that omit evidence distributed across the input or violate global task constraints. Task-specific verifiers, in contrast, evaluate task completion at the response level and may return graded rewards that reflect partial success. We diagnose this mismatch on fixed responses from two representative long-context evidence-aggregation tasks. Across longer input ranges, trajectory-level OPD scores become progressively less aligned with verifier rewards, indicating teacher-verifier disagreement. Motivated by this observation, we introduce Group-Calibrated On-Policy Distillation (GC-OPD). GC-OPD separately normalizes verifier rewards and trajectory-level OPD scores within each rollout group and uses their difference as a signed teacher-verifier disagreement residual. Relative-advantage-based credit assignment (RACA) distributes this trajectory-level residual across tokens according to their relative OPD advantages while preserving the original OPD signal. Across five long-context benchmarks, post-training with GC-OPD raises the five-benchmark averages of the official Qwen3-4B and Qwen3-8B checkpoints from 29.08 to 40.47 and from 35.12 to 44.65, respectively. Vanilla OPD reaches 39.31 and 43.56 under the same setup. Controlled ablations show that the signed residual is more effective than either an additional OPD-derived term or direct group-normalized verifier reward addition, while RACA further improves over uniform token allocation. Together, these results demonstrate that group-relative residual calibration can incorporate verifier outcomes without discarding dense token-level guidance. Code is available at https://github.com/SolereZhang/GC-OPD.

Figures & tables

Figure 1: Teacher–verifier disagreement in two evidence-aggregation tasks: Multi-Table Extraction (MTE) and High-Recall Retrieval (HRR). Pairwise disagreement rate captures ranking conflicts, while OPD preference gap captures the direction and magnitude of OPD preference along the verifier ordering. Both indicate weaker alignment over longer input ranges. See Section 3.2 for details.

Figure 2: Overview of GC-OPD. GC-OPD computes group-normalized verifier rewards and trajectory-level OPD scores, forms their signed residual, and uses RACA to distribute the residual across response tokens while retaining the original OPD advantage.

Task Family	# Samples	Native Verifier
Precise Long-Range Retrieval	4,693	Exact Match (B)
Evidence-Grounded Reasoning	3,204	Accuracy (B)
Multi-Table Extraction	729	IoU (G)
High-Recall Retrieval	540	Set F1 (G)
Fragment Matching and Induction	148	Subset EM (G)
Sequence Reconstruction	69	Pairwise Acc. (G)

Table 1: Composition and reward interfaces of the 9,527-prompt GoLongRL subset after the 32K filter. “B” denotes a binary reward in

\{0,1\}

; “G” denotes a graded score in

[0,1]

Figure 3: Held-out validation used to select the residual coefficient

\beta

. The fixed set contains 231 GoLongRL examples after 32K-token filtering and has no overlap with training. White squares show the step-100 reward. Circles show the mean over steps 60, 70, 80, 90, and 100. Both summaries select

\beta=0.10

at both model scales.

Model	Method	Benchmark
Model	Method	Avg.	DocMath	Frames	MRCR	CorpusQA	LBv1QA
Qwen3-4B	Raw	29.08	43.63	26.58	22.02	6.99	46.20
	OPD	39.31	49.37	30.95	26.90	32.22	57.10
	ExOPD †	38.22	47.75	29.61	25.95	31.91	55.90
	Uni-OPD †	38.53	51.88	29.98	27.93	27.36	55.50
	PowerOPD †	38.88	49.63	30.10	28.36	32.52	53.80

Table 2: Qwen3-4B and Qwen3-8B results on five long-context benchmarks (%). “Avg.” is the unweighted five-task mean.

\dagger

denotes an implementation of the method’s principal training-signal mechanism under the shared setup.

Variant	Avg.	DocMath	Frames	MRCR	CorpusQA	LBv1QA	$\boldsymbol{\Delta}$
Vanilla OPD	43.56	55.13	34.59	30.44	39.82	57.80	–
Additional OPD	43.60	54.63	34.59	30.03	39.51	59.20	$+0.04$
Direct reward	44.19	55.13	34.59	30.40	40.73	60.10	$+0.63$
GC-OPD	44.65	55.50	34.59	31.10	43.77	58.30	$\mathbf{+1.10}$

Table 3: Response-level signal ablation on Qwen3-8B. All augmented variants use RACA with coefficient

\beta=0.10

and differ only in whether the added signal is

A^{(i)}_{t}

\tilde{R}^{(i)}

, or

\rho^{(i)}=\tilde{R}^{(i)}-\tilde{s}^{(i)}

. “Avg.” is the unweighted five-task mean, and “

\Delta

” is computed from the unrounded average relative to vanilla OPD.

Token allocation	Avg.	DocMath	Frames	MRCR	CorpusQA	LBv1QA	$\boldsymbol{\Delta}$
Vanilla OPD	43.56	55.13	34.59	30.44	39.82	57.80	–
Absolute OPD	43.93	54.63	35.56	29.53	41.34	58.60	$+0.38$
Uniform	44.28	54.12	37.01	30.64	40.12	59.50	$+0.72$
RACA	44.65	55.50	34.59	31.10	43.77	58.30	$\mathbf{+1.10}$

Table 4: Token-credit ablation on Qwen3-8B. All residual variants use the same signed residual with

\beta=0.10

and differ only in token allocation. “Avg.” is the unweighted five-task mean, and “

\Delta

” is computed from the unrounded average relative to vanilla OPD.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Task-conditioned teacher–verifier disagreement in four GoLongRL task families. Bars report prompt-macro pairwise disagreement rates and top-1 mismatch rates over rollout groups with at least two distinct verifier rewards. Overall pools all nine task families.

Prompt length	Samples
$\leq 2$ K	2,222
2–8K	1,958
8–16K	2,155
16–24K	1,914
24–32K	1,278
Total	9,527

Appendix

Table 5: Prompt-length distribution of the exact 32K-filtered training file.

Setting	Value
Training horizon	100 steps
Prompt batch size	32
Responses per prompt	8
Maximum prompt length	32,768
Training response cap	10,240
Learning rate	$10^{-6}$

Appendix

Table 6: Shared training configuration and GC-OPD settings.

Figure 5: Illustrative fixed-rollout visualization for the 17,265-token case. It reproduces the selected context evidence and all eight independently rerolled student responses; the native label is B.

Correct B response , $\rho_{\mathrm{B}}=+3.464102$						Incorrect D response , $\rho_{\mathrm{D}}=-1.154701$
Token	$A_{t}$	$u_{t}$	$c_{t}$	$\Delta A_{t}$	$\widehat{A}_{t}$	Token	$A_{t}$	$u_{t}$	$c_{t}$	$\Delta A_{t}$	$\widehat{A}_{t}$
[	-4.88599	-1.85923	0.26959	0.09339	-4.79260	[	-4.88599	-1.98073	0.24248	-0.02800	-4.91399
Answer	-0.02188	0.83034	1.39285	0.48250	0.46062	Answer	-0.02188	0.68817	1.33112	-0.15370	-0.17558
]	-0.64095	0.48803	1.23928	0.42930	-0.21165	]	-0.64095	0.34849	1.17250	-0.13539	-0.77634
B	-1.91057	-0.21400	0.89341	0.30949	-1.60108	D	-0.66057	0.33773	1.16728	-0.13479	-0.79535
EOS	-0.15839	0.75486	1.36047	0.47128	0.31290	EOS	-0.17101	0.60634	1.29421	-0.14944	-0.32046

Appendix

Table 8: Representative token-level RACA recalculations for the correct-B and incorrect-D responses in Figure 5 under the final GC-OPD setting. Values are rounded; ranking checks use unrounded values.

On-Policy Distillation

On-policy distillation trains a model on its own outputs using teacher guidance, aligning supervision with the states it encounters at deployment.

See all 507 papers

GR-OPD: Group-Relative Sequence Credit for On-Policy Distillation
Oct 16, 2026Yushu Wang, Renkai Yang, Jingcao Xu +4On-Policy Distillation Knowledge Distillation
Shanghai Jiao Tong University, Shanghai, China · Alibaba Group, Hangzhou, China
Reward-Gated On-Policy Distillation
Jul 4, 2026Mohammad Sadegh Akhondzadeh, Vijay Lingam, Atula Tejaswi +3Language Model Distillation On-Policy Distillation
University of Cologne · AWS AI · UT Austin +1
Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation
Sep 2, 2026Zhiwei Zhang, Zechen Sun, Fei Zhao +6On-Policy Distillation RL Post-Training
AllSpark Team

Oct 16, 2026J/K move · Enter open · S save

GR-OPD: Group-Relative Sequence Credit for On-Policy Distillation

Yushu Wang, Renkai Yang, Jingcao Xu, Zili Geng+3

Shanghai Jiao Tong University, Shanghai, China · Alibaba Group, Hangzhou, China

On-Policy Distillation Knowledge Distillation

Open paper AskPDF arXiv

On-policy distillation (OPD) provides token-level teacher feedback, yet common sampled-token updates assign credit locally and omit the effects of earlier actions on subsequent feedback. We introduce \method{} (Group-Relative On-Policy Distillation), which decouples feedback computation from credit assignment by aggregating token feedback into calibrated sequence-level credit. We establish a theoretical basis for this approach by showing that, under ideal on-policy assumptions, broadcasting raw summed feedback to every response token yields an unbiased estimate of the negative gradient of the sequence-level reverse-KL objective. To support effective learning in practice, \method{} discounts later feedback that may become less reliable under distribution shift and uses same-prompt rollout groups to calibrate teacher guidance. It preserves relative response preferences while retaining a controlled fraction of the group mean, which we term collective teacher pressure. Across five Qwen3 teacher--student settings and six mathematical reasoning benchmarks, \method{} outperforms vanilla OPD in macro Avg@K in every setting, including gains of 10.9 and 11.5 percentage points in two settings with matched thinking patterns. It also maintains stable training under mismatched thinking patterns, where vanilla OPD collapses. Ablations show that retaining an appropriate amount of collective teacher pressure can improve learning, while excessive collective teacher pressure or an unrestricted feedback horizon can destabilize training.

Abstract

On-Policy Distillation

Explore similar work

GR-OPD: Group-Relative Sequence Credit for On-Policy Distillation

Reward-Gated On-Policy Distillation

Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation