DualOPSD: Closing the Feedback Loop in On-Policy Self-Distillation

Yutong Chen; Guangfu Guo; Zhichao Xu; Kunpeng Liu

cs.LGAug 26, 2026

DualOPSD: Closing the Feedback Loop in On-Policy Self-Distillation

Authors: Yutong Chen, Guangfu Guo, Zhichao Xu, Kunpeng Liu

Organizations: Department of Computer Science, Clemson University, 105 Sikes Hall, Clemson, SC 29634, USA · Department of Computer Science, University of Utah

Abstract

On-policy self-distillation (OPSD) combines student generated trajectories with dense supervision from a privileged teacher of the same model. Its pointwise clipping protects the student from locally incompatible teacher directions, but the fixed teacher receives no feedback about the supervision the student rejects, which limits effective learning from the teacher. We introduce DualOPSD, which alternates updates in a coupled teacher-student system. The student retains OPSD's clipped forward KL objective. The privileged teacher adapts to the updated student after each student step. A residual target based on the frozen initial model preserves the effect of privileged context during teacher adaptation. Across Qwen3-8B/4B/1.7B and three math benchmarks, DualOPSD improves avg@12 accuracy over OPSD by 9.83 percentage points. Further diagnostic analyses show lower bidirectional KL, less persistent mismatch, and retained sensitivity to privileged reference. Together, these results demonstrate that DualOPSD can be a more effective form of privileged self-distillation. Code is available at: https://github.com/on1262/DualOPSD.

Figures & tables

Figure 1: Overview of DualOPSD. (a) OPSD updates only the student by minimizing a pointwise clipped forward KL divergence to a fixed privileged teacher. (b) In DualOPSD, the student retains OPSD’s clipped forward KL update. The teacher adapts toward the updated student while adding a scaled residual, which is used to preserve the privileged effect during adaptation. Flames denote where gradients are propagated.

Figure 2: Distributional mismatch and temporal recurrence on Qwen3-4B. OPSD has a larger distributional gap and more recurrent clipped forward KL contributions. Our methods (DualOPSD and DualOPSD w/o Residual) substantially change both patterns. Left: full

D_{\mathrm{KL}}(p_{T}\|p_{S})

over training. Middle: full

D_{\mathrm{KL}}(p_{S}\|p_{T})

over training. Right:

R_{10}

(from step 20 to 100) for dominant mismatch directions. Error bars in all panels are sample standard deviations over 3 training seeds.

Model	Method	AIME24	AIME25	HMMT25	Mean
1.7B	Base	$11.94\pm 1.94$	$9.54\pm 1.12$	$5.74\pm 0.42$	$9.07\pm 0.82$
	SFT	$8.80\pm 1.12$	$7.87\pm 0.64$	$1.67\pm 0.56$	$6.11\pm 0.65$
	GRPO	$12.50\pm 2.82$	$\underline{9.72\pm 0.96}$	$4.81\pm 0.32$	$9.01\pm 0.83$
	OPSD ∗	$\underline{14.17\pm 1.39}$	$9.72\pm 0.73$	$\underline{6.48\pm 1.25}$	$\underline{10.12\pm 0.62}$
	DualOPSD w/o Residual	$11.57\pm 1.16$	$8.52\pm 1.05$	$4.72\pm 0.73$	$8.27\pm 0.93$
	DualOPSD	$\mathbf{15.00\pm 2.65}$	$\mathbf{12.22\pm 1.73}$	$\mathbf{7.59\pm 2.05}$	$\mathbf{11.60\pm 1.82}$

Table 1: Avg@12 accuracy on three mathematics benchmarks (%). Entries report means

\pm

sample standard deviations across three seeds. The final column reports the mean accuracy across the three benchmarks. The 1.7B, 4B, and 8B residual variants use

\beta=0.8

\beta=0.7

, and

\beta=0.5

. ∗ denotes that OPSD is reported at the more accurate of its 100-step and matched-compute checkpoints. The per-seed measurements underlying Table 1 are reported in Appendix E .

Model	Method	PFLOPs	GPU h	$\Delta$ Acc.	$\Delta$ tokens	Trunc. (%) $\downarrow$	$E\uparrow$
Overall	OPSD	$161.78$	1.00	$+4.59$	$+3.27\mathrm{K}$	$6.45$	1.27
	OPSD, matched compute	$436.52$	3.19	$+2.89$	$+4.66\mathrm{K}$	$13.14$	0.59
	DualOPSD w/o Residual	$318.59$	1.40	$+11.26$	$+3.13\mathrm{K}$	2.42	2.33
	DualOPSD	$435.81$	1.78	+14.42	$+5.51\mathrm{K}$	$6.68$	2.57

Table 2: Overall cost and generation behavior for distillation methods, averaged across the 1.7B, 4B, and 8B scales.

\Delta

Acc. is the change in the three-benchmark avg@12 mean relative to the Base, in percentage points.

\Delta

tokens is the change in mean observed completion tokens relative to Base. Length efficiency

E

is defined in Equation 15 .

Figure 3: Qwen3-4B accuracy over training for OPSD and DualOPSD. Each curve shows the mean avg@12 across three benchmarks at 5 checkpoints from step 20 to 100.

Figure 4: Teacher accuracy and privileged-reference sensitivity on Qwen3-4B. Left: teacher and student accuracy across five checkpoints on 500 diagnostic problems. Middle: normalized Jensen–Shannon (JS) retention. The horizontal line at one denotes the initial sensitivity level. Right: fraction of positive forward KL contribution mass removed by the student cap, averaged over the five checkpoints. OPSD appears only in the right panel.

Qwen3-1.7B
Metric	High Clip	Low Clip	No Clip
Removed KL	Upper	Lower	None
Acc. (%)	$7.59\pm 0.48$	$9.88\pm 0.81$	$\mathbf{11.60\pm 1.82}$
Len. (K)	$2.13\pm 0.16$	$4.40\pm 0.67$	$5.22\pm 0.96$
Trunc. (%)	$1.23\pm 0.33$	$6.42\pm 0.63$	$8.77\pm 3.23$
Qwen3-4B

Table 3: Teacher-side clipping ablation on the DualOPSD teacher objective at each Qwen3 scale. Acc. is the mean avg@12 across the three benchmarks, Len. is the mean length of non-truncated responses in K tokens, and Trunc. is the truncation rate. Entries report means

\pm

sample standard deviations across three training seeds.

Model	OPSD	DualOPSD	Reduction (%)
1.7B	$0.823\pm 0.022$	$\mathbf{0.467\pm 0.049}$	43.2
4B	$0.810\pm 0.006$	$\mathbf{0.266\pm 0.069}$	67.2
8B	$0.830\pm 0.007$	$\mathbf{0.390\pm 0.039}$	53.0

Table 4: Teacher–student formatting distance at step 100 on 500 diagnostic problems. Lower values indicate closer agreement.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Signal	Setting	OPSD	DualOPSD w/o Residual	DualOPSD
Clipped forward KL	$k=1$	0.720 $\pm$ 0.005	0.445 $\pm$ 0.008	0.314 $\pm$ 0.031
	$k=5$	0.698 $\pm$ 0.005	0.373 $\pm$ 0.008	0.235 $\pm$ 0.027
	$k=10$	0.693 $\pm$ 0.005	0.355 $\pm$ 0.010	0.208 $\pm$ 0.028
	$k=20$	0.682 $\pm$ 0.005	0.338 $\pm$ 0.012	0.182 $\pm$ 0.028
	Random-pair	0.0101 $\pm$ 0.0001	0.0130 $\pm$ 0.0004	0.0081 $\pm$ 0.0004
Positive forward KL	$k=1$	0.822 $\pm$ 0.004	0.560 $\pm$ 0.009	0.521 $\pm$ 0.019

Appendix

Table 5: Recurrence between steps 20 and 100 as a function of the number of selected vocabulary entries. Values are means

\pm

sample standard deviations over 3 training seeds. Random-pair rows use

k=10

with mismatched problems.

Method	Step	Clip rate	$D_{\mathrm{KL}}(p_{T}\\|p_{S})$	$D_{\mathrm{KL}}(p_{S}\\|p_{T})$
OPSD	20	0.565 $\pm$ 0.002	0.192 $\pm$ 0.001	0.110 $\pm$ 0.002
	40	0.545 $\pm$ 0.003	1.626 $\pm$ 0.040	0.311 $\pm$ 0.002
	60	0.558 $\pm$ 0.007	2.132 $\pm$ 0.133	0.300 $\pm$ 0.014
	80	0.569 $\pm$ 0.006	2.053 $\pm$ 0.125	0.273 $\pm$ 0.013
	100	0.569 $\pm$ 0.001	2.042 $\pm$ 0.058	0.261 $\pm$ 0.003
DualOPSD w/o Residual	20	0.663 $\pm$ 0.004	0.152 $\pm$ 0.006	0.071 $\pm$ 0.003

Appendix

Table 6: Checkpoint diagnostics, reported as means

\pm

sample standard deviations over 3 training seeds. Clip rate is the fraction of positive forward KL contribution mass removed by the student cap. The KL columns report full-vocabulary divergences. At each checkpoint, the highest value for each metric is bold and the second highest is underlined.

Figure 5: Held-out validation accuracy as a function of residual weight at each Qwen3 scale. Each point represents the step-100 checkpoint trained with seed 42, evaluated using one judged response per problem on the 500-problem validation set. Stars mark the weights selected by the rule in Table 7 .

Model	$\beta$	Corr.	Acc. (%)	Trunc. (%)	Tokens
1.7B	0.1	302	60.4	1.6	1,394
	0.2	293	58.6	3.0	1,414
	0.5	285	57.0	4.0	1,359
	0.7	322	64.4	15.6	3,053
	0.8	328	65.6	6.0	1,834
	0.9	326	65.2	5.4	1,696

Appendix

Table 7: Residual weight search on the held-out validation set for three Qwen3 models. Corr. counts correct responses out of 500, and Tokens is the mean observed response length. Bold entries mark the selected settings.

$\beta$	avg@12 / pass@12	Trunc. (%)	Tokens
0	59.44 / 83.33	3.89	6.80K
0.1	55.56 / 80.00	5.28	5.68K
0.2	51.11 / 80.00	5.56	6.21K
0.5	60.56 / 83.33	5.28	7.33K
0.7	45.00 / 80.00	15.93	6.01K
1.0	34.17 / 66.67	18.61	3.85K

Appendix

Table 8: Qwen3-8B residual weight sensitivity on AIME 2024.

\beta=0

corresponds to DualOPSD w/o Residual. Tokens is the mean length of non-truncated responses. All rows use training seed 42.

Qwen3-1.7B
Method	AIME24	AIME25	HMMT25	Mean
Base	11.11 / 14.17 / 10.56	8.33 / 10.56 / 9.72	5.83 / 5.28 / 6.11	8.43 / 10.00 / 8.80
SFT	9.44 / 7.50 / 9.44	7.50 / 7.50 / 8.61	2.22 / 1.11 / 1.67	6.39 / 5.37 / 6.57
GRPO	13.06 / 9.44 / 15.00	10.28 / 10.28 / 8.61	5.00 / 4.44 / 5.00	9.44 / 8.06 / 9.54
OPSD	15.56 / 14.17 / 12.78	10.00 / 8.89 / 10.28	6.39 / 7.78 / 5.28	10.65 / 10.28 / 9.44
OPSD, matched	11.67 / 10.56 / 16.94	7.22 / 7.50 / 8.33	6.11 / 7.22 / 5.83	8.33 / 8.43 / 10.37

Appendix

Table 9: Qwen3-1.7B, 4B, and 8B avg@12 results by seed (%). Values are ordered by seed 42 / 43 / 44.

Model	Method	PFLOPs	GPU h	$\Delta$ Acc.	$\Delta$ tokens	Trunc. (%) $\downarrow$	$E\uparrow$
1.7B	OPSD	$66.62$	0.61	$+1.05$	$+1.94\mathrm{K}$	$2.90$	0.54
	OPSD, matched compute	$181.35$	2.20	$-0.03$	$+3.58\mathrm{K}$	$9.07$	-0.01
	DualOPSD w/o Residual	$128.61$	0.81	$-0.80$	$+0.67\mathrm{K}$	$1.05$	-1.21
	DualOPSD	$180.94$	1.05	$+2.53$	$+5.32\mathrm{K}$	$8.77$	0.48
4B	OPSD	$146.28$	1.01	$+4.94$	$+4.10\mathrm{K}$	$8.49$	1.20
	OPSD, matched compute	$395.12$	2.83	$+3.95$	$+6.20\mathrm{K}$	$18.80$	0.64

Appendix

Table 10: Per-scale cost and generation behavior for distillation methods.

\Delta

Acc. is the change in the avg@12 mean relative to the corresponding Base mean in percentage points.

\Delta

tokens is the change in mean observed completion tokens relative to that Base mean. Length efficiency

E

is defined in Equation 15 .

Model	Method	GPU h	PFLOPs
1.7B	OPSD	0.61	$66.62{\pm}0.32$
	DualOPSD w/o Residual	0.81	$128.61{\pm}3.75$
	DualOPSD	1.05	$180.94{\pm}0.08$
4B	OPSD	1.01	$146.28{\pm}1.06$
	DualOPSD w/o Residual	1.40	$281.41{\pm}8.23$
	DualOPSD	1.78	$394.46{\pm}9.08$

Appendix

Table 11: Training costs for 100-step distillation runs. GPU hours are measured from elapsed wall time, and PFLOPs are estimated from model calls and processed tokens.

Model	Method	Untruncated tokens	Truncation (%)
1.7B	OPSD	$3.41\mathrm{K}\pm 0.24$	$2.90\pm 0.33$
	DualOPSD w/o Residual	$2.67\mathrm{K}\pm 0.55$	$1.05\pm 0.19$
	DualOPSD	$5.21\mathrm{K}\pm 0.96$	$8.77\pm 3.23$
4B	OPSD	$4.72\mathrm{K}\pm 0.26$	$8.49\pm 2.73$
	DualOPSD w/o Residual	$5.06\mathrm{K}\pm 0.95$	$2.35\pm 0.88$
	DualOPSD	$6.82\mathrm{K}\pm 1.16$	$5.68\pm 2.52$

Appendix

Table 12: Generation length and truncation computed over all responses on the three benchmarks. Table 10 reports observed token changes including truncated responses.

Model	Teacher clipping	AIME24	AIME25	HMMT25	Mean	Length (K)	Trunc. (%)
Qwen3-1.7B	High clipping	11.94	6.48	4.35	$7.59\pm 0.48$	$2.13\pm 0.16$	$1.23\pm 0.33$
	Low clipping	13.52	10.93	5.19	$9.88\pm 0.81$	$4.40\pm 0.67$	$6.42\pm 0.63$
	No clipping	15.00	12.22	7.59	$\mathbf{11.60\pm 1.82}$	$5.22\pm 0.96$	$8.77\pm 3.23$
Qwen3-4B	High clipping	28.70	24.81	12.96	$22.16\pm 1.72$	$3.16\pm 0.67$	$5.96\pm 1.68$
	Low clipping	35.74	30.56	19.07	$28.46\pm 4.88$	$5.83\pm 1.64$	$8.09\pm 4.15$
	No clipping	42.78	37.41	24.26	$\mathbf{34.81\pm 1.80}$	$6.81\pm 1.16$	$5.68\pm 2.52$

Appendix

Table 13: Teacher clipping ablation on the DualOPSD teacher objective at each scale after 100 training steps. Benchmark entries are avg@12 percentages averaged over training seeds 42, 43, and 44. Mean is the macro average across AIME24, AIME25, and HMMT25. Length is the mean completion length of non-truncated responses. Trunc. is the truncation rate. Mean, Length, and Trunc. report means

\pm

sample standard deviations across training seeds.

Method	Step	Teacher acc. (%)	Student acc. (%)	JS retention
DualOPSD w/o Residual	20	93.93 $\pm$ 0.90	81.33 $\pm$ 1.62	0.579 $\pm$ 0.007
	40	90.73 $\pm$ 0.58	82.00 $\pm$ 1.51	0.748 $\pm$ 0.007
	60	91.80 $\pm$ 0.53	82.47 $\pm$ 2.08	0.773 $\pm$ 0.034
	80	85.87 $\pm$ 8.12	79.13 $\pm$ 5.13	0.842 $\pm$ 0.120
	100	91.07 $\pm$ 2.08	84.73 $\pm$ 3.37	0.838 $\pm$ 0.092
DualOPSD	20	94.33 $\pm$ 0.83	82.93 $\pm$ 0.83	0.689 $\pm$ 0.002

Appendix

Table 14: Qwen3-4B accuracy and reference sensitivity measurements underlying Figure 4 . Accuracy is measured on the fixed diagnostic set of 500 problems. JS retention is averaged over valid rollout problems as defined in the reference sensitivity protocol. Entries are means

\pm

sample standard deviations across three training seeds.

Step	OPSD	DualOPSD w/o Residual	DualOPSD
20	56.5 $\pm$ 0.2	66.3 $\pm$ 0.4	51.7 $\pm$ 0.5
40	54.5 $\pm$ 0.3	51.0 $\pm$ 0.4	30.9 $\pm$ 0.2
60	55.8 $\pm$ 0.7	55.3 $\pm$ 0.7	46.9 $\pm$ 2.3
80	56.9 $\pm$ 0.6	61.2 $\pm$ 9.0	51.2 $\pm$ 5.9
100	56.9 $\pm$ 0.1	61.3 $\pm$ 2.6	60.8 $\pm$ 9.3
Avg.	56.1	59.0	48.3

Appendix

Table 15: Fraction of positive forward-KL mass removed by the student cap (%). Checkpoint entries are means

\pm

sample standard deviations across three training seeds. Avg. is the unweighted mean of the five checkpoint means.

Model	OPSD	DualOPSD	Red. (%)	Retained
1.7B	0.823 $\pm$ 0.022	0.467 $\pm$ 0.049	43.2	449/423/463
4B	0.810 $\pm$ 0.006	0.266 $\pm$ 0.069	67.2	486/483/470
8B	0.830 $\pm$ 0.007	0.390 $\pm$ 0.039	53.0	469/479/457

Appendix

Table 16: Teacher–student formatting distance at step 100 for the three Qwen3 models. Lower values indicate closer agreement. Distances are means

\pm

sample standard deviations across three training seeds. Red. is the relative reduction of DualOPSD from OPSD. Retained gives the numbers of problems for which all four compared responses were untruncated, in the listed training-seed order.

Adaptive Knowledge Distillation

See all 13 papers

DAPD: Dual-Anchored Policy Distillation
Aug 3, 2026Jianyu Wu, Yizhou Wang, Encheng Su +2On-Policy Self-Distillation Language Model Distillation
Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · The Chinese University of Hong Kong +1
SCOPE-OPSD: Fisher-Conditioned Privileged Subspaces for On-Policy Self-Distillation
Sep 11, 2026Yunmeng Chen, Kunyu Wang, Peihan Li +8On-Policy Self-Distillation Language Model Distillation
Chongqing Ant Consumer Finance Co., Ltd · Alibaba Cloud Computing Co., Ltd · Chongqing University of Posts and Telecommunications
Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement
Aug 31, 2026Yi Ding, Ruqi ZhangRL for Language Models Language Model Distillation
Department of Computer Science, Purdue University, USA

Aug 3, 2026J/K move · Enter open · S save

DAPD: Dual-Anchored Policy Distillation

Jianyu Wu, Yizhou Wang, Encheng Su, Chen Tang+1

Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · The Chinese University of Hong Kong+1

On-Policy Self-Distillation Language Model Distillation RL Post-Training Policy Distillation

Open paper AskPDF arXiv

On-policy (self) distillation (OPSD) is increasingly adopted for language-model post-training. It strengthens the teacher with privileged information but can induce a privilege illusion: the student learns privilege-dependent behavior it cannot reproduce from its inference-time context, yet behaves as if the training-time privileged information remained available, ultimately degrading performance. In this paper, we identify information asymmetry between the privileged teacher and the student at inference as the root cause of this failure in OPSD. To resolve this asymmetry, we propose Dual-Anchored Policy Distillation (DAPD), a unified framework with two levels of anchoring. Dual-Path Anchoring (DPA) introduces a self-conditioned bridge and aligns reference and rollout behavior along two matched-information paths, preventing privilege-dependent behavior from being transferred to the inference-time student. Dual-Source Anchoring (DSA) applies these paths in both reference-to-rollout and rollout-to-reference directions, reducing reliance on privileged reference guidance while preserving correctness supervision. Extensive experiments show that DAPD significantly alleviates privilege illusion, outperforming OPSD on Qwen3-4B by +2.00 points on average across tasks. Notably, its gains persist across scales, reaching +2.69 at 4B and +2.78 at 32B.

Abstract

Adaptive Knowledge Distillation

Explore similar work

DAPD: Dual-Anchored Policy Distillation

SCOPE-OPSD: Fisher-Conditioned Privileged Subspaces for On-Policy Self-Distillation

Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement