OPDArena: Understanding Learning and Forgetting in On-Policy Distillation

Yuanze Xu; Tianzhu Ye; Hui Sun; Wenhui Wang; Li Dong; Furu Wei

cs.LGOct 19, 2026

OPDArena: Understanding Learning and Forgetting in On-Policy Distillation

Authors: Yuanze Xu, Tianzhu Ye, Hui Sun, Wenhui Wang, Li Dong, Furu Wei

Organizations: Microsoft Research · Nankai University

Abstract

On-policy distillation (OPD) is increasingly used for language model post-training, yet how teacher supervision shapes learning and forgetting remains unclear. We present OPDArena, a framework comparing supervised fine-tuning (SFT), reinforcement learning with verifiable rewards (RLVR), and OPD across five pretrained models and diverse teacher sources. It jointly evaluates mathematical learning, general capability retention, and predictive distribution changes. Teacher choice strongly shapes retention: students of RLVR-trained teachers improve on the target task while retaining prior capabilities, whereas students of SFT-trained teachers incur greater capability loss. Across post-training methods, out-of-domain (OOD) distribution change shows a more consistent positive association with forgetting than in-domain change. To understand how teacher choice affects these changes, we examine the additional information teachers provide relative to the student's pretrained model. Controlled teacher comparisons show that the task-specific preferences teachers acquire through post-training make student updates more concentrated in-domain.

Figures & tables

Figure 1: Training methods and joint learning–retention outcomes. (a) Training paths and feedback sources from a shared Base model. (b) In-domain accuracy gain versus OOD accuracy gain from Base; colors identify methods and shapes identify models. Ellipses summarize variation across model–teacher configurations. RLVR and Self-OPD (RLVR) combine task learning with low forgetting.

Method (Teacher)	Qwen2.5-1.5B		Qwen2.5-7B
	In-domain	OOD	In-domain	OOD
Base	13.93	35.20	26.08	49.82
SFT	11.50 $(-2.42)$	34.25 $(-0.95)$	35.87 $(+9.79)$	45.32 $(-4.50)$
RLVR	19.08 $(+5.15)$	36.23 $(+1.03)$	32.56 $(+6.48)$	50.06 $(+0.23)$
OPD (Q2.5-7B)	15.85 $(+1.93)$	34.54 $(-0.66)$	–	–
OPD (Q2.5-14B)	16.18 $(+2.26)$	34.27 $(-0.93)$	25.03 $(-1.05)$	48.47 $(-1.35)$

Table 1: Main results for four Base models: accuracy (%) with changes from Base in parentheses (percentage points). Negative OOD changes indicate forgetting. RLVR and Self-OPD (RLVR) combine in-domain gains with strong capability retention.

Figure 2: Distribution change and forgetting. Horizontal axes show in-domain (top) and OOD (bottom) JS from Base; larger values indicate greater distribution change. The vertical axis shows OOD accuracy loss from Base; higher values indicate greater forgetting. Each point pairs JS and OOD accuracy at the same checkpoint. JS is in nats, averaged equally over five in-domain or seven OOD tasks;

r

is Pearson correlation. Qwen3-1.7B is shown in Appendix Figure 14 .

Figure 3: Prediction changes from Base across post-training methods. The first two columns show in-domain and OOD JS in $10^{-2}$ nats; the last shows relative OOD influence $\rho$ . Boxes mark the lowest $\rho$ for each model.

Teacher	Teacher acc.		Student acc.
	Math	OOD	Math	OOD	$\rho$
Student: Qwen2.5-1.5B Base			13.93	35.20
Qwen3-8B Base	31.32	51.37	16.33	33.37	1.998
Qwen2.5-1.5B RLVR	19.08	36.23	20.00	36.60	0.131
Student: Qwen2.5-7B Base			26.08	49.82
Qwen3-8B Base	31.32	51.37	24.17	47.13	1.648

Table 2: Teacher and student accuracies (%) and student

\rho

at step 780. Each block compares a larger Base teacher with the student’s own RLVR teacher; shaded headings identify the student and its Base accuracies. Datasets are weighted equally within each domain.

Panel of Table 3 — Table 3: Student distribution changes and capabilities under different information sources. All A–E conditions use student step 120. JS uses shared prefixes, is relative to Base, and is reported in $10^{-2}$ nats. Parentheses show accuracy changes from Base in percentage points. Datasets are weighted equally within each domain; $\rho$ is undefined at Base.

Figure 4: Accuracy changes from Base under weight scaling. Stars mark sweet-spot coefficients that improve both accuracies over the original endpoint, as defined in Appendix F .

Figure 5: Qwen3-4B parameter distances for SFT, RLVR, and their Self-OPD students (left), and student–teacher JS for the same students (right). Each distance is normalized by its corresponding initial Base–teacher gap.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Input	AMC23	AIME24	AIME25	HMMT25	Math500	Mean
Qwen2.5-1.5B Base	Raw	20.00	1.25	0.83	0.42	47.13	13.93
Qwen2.5-1.5B Base	Template	1.88	0.42	0.00	0.00	4.58	1.37
Qwen2.5-7B Base	Raw	46.88	9.17	5.00	2.08	67.28	26.08
Qwen2.5-7B Base	Template	31.25	6.67	3.33	0.42	57.03	19.74
Qwen2.5-14B Base	Raw	51.25	10.42	7.08	2.50	73.23	28.90
Qwen3-1.7B Base	Raw	32.50	4.17	4.58	0.00	59.58	20.17

Appendix

Table 4: In-domain acc@8 (%) of the released Base models. The Mean column weights the five datasets equally. Raw and Template denote inputs without and with the corresponding Qwen chat template.

Model	Input	AMC23	AIME24	AIME25	HMMT25	Math500	Mean
Qwen2.5-1.5B Base	Raw	2359.4	5101.0	3827.6	3450.3	1074.3	3162.5
Qwen2.5-1.5B Base	Template	25725.3	25467.4	25473.3	22321.9	13228.8	22443.4
Qwen2.5-7B Base	Raw	1160.0	3131.1	3121.3	2364.9	787.3	2112.9
Qwen2.5-7B Base	Template	1541.7	1985.4	2171.4	2038.7	696.8	1686.8
Qwen2.5-14B Base	Raw	912.9	1214.1	1483.7	1317.1	614.7	1108.5
Qwen3-1.7B Base	Raw	1807.0	5101.9	4117.2	4692.0	1073.2	3358.3

Appendix

Table 5: Mean in-domain output length in tokens for the same Base-model evaluations. The Mean column weights the five datasets equally. Math500 uses a 16,384-token response limit; the other datasets use 31,744 tokens.

Model	HellaSwag	ARC-C	MMLU	IFEval	TriviaQA	BBH	NQ-Open	Macro
Qwen2.5-1.5B Base	49.22	39.45	61.40	20.70	22.00	42.59	11.05	35.20
Qwen2.5-7B Base	61.33	44.53	74.21	31.79	44.40	72.96	19.53	49.82
Qwen2.5-14B Base	62.11	50.78	78.07	31.79	55.20	79.63	–	–
Qwen3-1.7B Base	46.09	37.11	65.26	24.03	20.60	50.37	9.17	36.09
Qwen3-4B Base	53.13	41.02	75.09	31.98	34.00	66.67	16.29	45.45
Qwen3-8B Base	59.77	44.53	76.49	37.52	47.20	72.59	21.50	51.37

Appendix

Table 6: OOD accuracy (%) of the released Base models with Raw input. The macro is an unweighted mean over the seven datasets. HellaSwag, ARC-Challenge, and MMLU use candidate likelihood; the other tasks score one generated response. Response budgets are listed in Table 8 .

Setting	SFT	RLVR	OPD variants / OPSD
Train records	49,993	49,993	49,993
Epochs	1	1	1
Global batch	64	64	64
Learning rate	$10^{-5}$	$10^{-6}$	$10^{-6}$
Optimizer	AdamW	AdamW	AdamW
Adam $(\beta_{1},\beta_{2})$	$(0.9,0.999)$	$(0.9,0.999)$	$(0.9,0.999)$

Appendix

Table 7: Current OpenR1-50K training settings. OPD, Self-OPD, and OPSD share the listed optimization settings; the teacher input or token signal differs.

Domain	Dataset	Evaluation	Examples	Response budget (tokens)	Primary score
In-domain	AMC23	eight generated responses	40	31,744	acc@8
	AIME24	eight generated responses	30	31,744	acc@8
	AIME25	eight generated responses	30	31,744	acc@8
	HMMT25	eight generated responses	30	31,744	acc@8
	Math500	eight generated responses	500	16,384	acc@8
OOD	HellaSwag	candidate likelihood	256	–	normalized accuracy

Appendix

Table 8: Target evaluation protocol: fixed validation suite and response budgets in tokens. OOD category subsamples use seed 42 and are reused at every checkpoint. A dash denotes likelihood scoring without response generation. The IFEval budget applies to the response prefix used for scoring.

Figure 6: In-domain learning and OOD retention throughout training. Curves show task-group mean accuracy; dashed horizontal lines mark Base. RLVR and Self-OPD (RLVR) combine in-domain learning with sustained capability retention.

Figure 7: Task accuracies during training for Qwen2.5-1.5B Base.

Figure 8: Task accuracies during training for Qwen2.5-7B Base.

Figure 9: Task accuracies during training for Qwen3-1.7B Base.

Figure 10: Task accuracies during training for Qwen3-4B Base.

Figure 11: Task accuracies during training for Qwen3-8B Base.

Student	Teacher	Parameter distance		Prediction distance
		$d_{\theta}$	$d_{\perp}$	$d_{\mathrm{out,in}}$	$d_{\mathrm{out,OOD}}$
Qwen2.5-1.5B	SFT	0.825	0.355	0.493	0.361
	RLVR	0.894	0.744	0.010	0.354
Qwen2.5-7B	SFT	0.976	0.471	0.601	0.173
	RLVR	0.943	0.758	0.010	0.243
Qwen3-1.7B	SFT	1.003	0.516	0.705	0.187

Appendix

Table 9: Self-OPD distances to the corresponding SFT or RLVR teacher at step 780 for the four models supplementing Figure 5 . Parameter and prediction distances are normalized by the initial Base–teacher gap;

d_{\perp}

measures departure from the Base–teacher line.

Figure 12: Accuracy changes from Base for source training (dashed) and the corresponding Self-OPD student (solid). Students use a fixed endpoint teacher.

Qwen2.5-7B
Prefix source	Base	SFT@60	SFT@240	RLVR@60	RLVR@240	Teacher
SFT	1.057	0.871	0.580	1.563	1.567	0.379
RLVR	0.230	0.330	0.436	0.106	0.096	0.094
Qwen3-1.7B
SFT	0.967	0.839	0.665	1.855	1.822	0.458
RLVR	0.300	0.332	0.444	0.145	0.136	0.135

Appendix

Table 10: Mean negative log probability over the first 128 tokens of teacher responses, in nats/token. SFT/RLVR columns denote students of the corresponding teacher source; @ denotes student training step. Each teacher contributes all 256 responses; the final column is scored by the teacher that generated them.

Qwen2.5-7B
Prefix source	Tokens 1–128	Tokens 33–128
SFT	0.678	0.514
RLVR	0.136	0.122
Qwen3-1.7B
SFT	0.509	0.353
RLVR	0.165	0.132

Appendix

Table 11: Scoring gaps after correcting for teacher self-probability, in nats/token. Larger values indicate a higher predictive cost under Base relative to the generating teacher.

	In-domain accuracy			OOD accuracy
Model	Ordinary	Zeroed	$\Delta$	Ordinary	Zeroed	$\Delta$
Qwen2.5-1.5B	10.33	14.23	+3.91	34.93	34.58	-0.35
Qwen2.5-7B	28.49	34.57	+6.08	47.02	46.50	-0.52
Qwen3-1.7B	17.16	20.54	+3.38	35.59	35.71	+0.12
Qwen3-4B	34.71	38.82	+4.11	43.45	43.35	-0.10
Qwen3-8B	44.79	44.12	-0.66	47.10	47.63	+0.53

Appendix

Table 12: Step-780 accuracy (%) with ordinary feedback and first-position feedback zeroed.

\Delta

is zeroed minus ordinary accuracy, in percentage points. Evaluation leaves all output positions unchanged.

		JS to Base		JS to SFT teacher
Model	Feedback	In-domain	OOD	In-domain	OOD
Qwen2.5-1.5B	Ordinary	7.200	2.136	3.658	1.007
	First-position zeroed	4.172	1.750	4.917	0.860
Qwen2.5-7B	Ordinary	7.137	1.963	4.625	0.462
	First-position zeroed	5.355	1.620	4.779	0.753
Qwen3-1.7B	Ordinary	6.165	2.149	4.555	0.521

Appendix

Table 13: Step-780 JS in

10^{-2}

nats for the same ordinary and first-position-zeroed students. The references are the shared Base and the corresponding frozen SFT teacher.

Figure 13: Complete accuracy trajectories for ordinary Self-OPD (SFT) and its first-position feedback intervention across five models (columns), with in-domain accuracy above OOD accuracy. Both conditions use the same Base and frozen SFT teacher. Dotted horizontal lines mark Base accuracy.

Teacher supervision	Distribution change			Accuracy (%)
	$D_{\mathrm{in}}$	$D_{\mathrm{OOD}}$	$\rho$	Math ( $\Delta$ )		OOD ( $\Delta$ )
Qwen3-4B $\to$ Qwen3-1.7B
Student Base	–	–	–	20.17		36.09
A: Larger Base teacher	0.2692	0.1772	0.658	19.38	$(-0.78)$	35.96	$(-0.13)$
D: Larger RLVR teacher	4.1255	0.3643	0.088	24.21	$(+4.04)$	36.19	$(+0.10)$
E: RLVR increment	4.1056	0.3302	0.080	24.08	$(+3.92)$	36.05	$(-0.04)$

Appendix

Table 14: Additional A/D/E comparisons for Qwen3-1.7B at student step 120. JS is relative to student Base in

10^{-2}

nats. Accuracies are percentages, with changes from Base in parentheses. Domains use equal dataset weights.

Figure 14: Distribution change and forgetting for Qwen3-1.7B. Left: in-domain JS; right: OOD JS. Both panels show OOD accuracy loss at the corresponding checkpoint, with

r

denoting Pearson correlation.

Teacher	Raw	Chat template	Raw + answer boundary
Qwen2.5-7B	24.12	0.0100	0.0028
Qwen3-1.7B	15.36	0.0385	0.0034

Appendix

Table 15: Teacher end-of-sequence probability (%) before the first student response token, with the problems and reference solutions fixed.

Student	Metric	Base	Raw	Chat template	Raw + boundary
Qwen2.5-1.5B	Math	13.93	5.63	6.86	12.64
	OOD	35.20	35.21	35.06	35.09
Qwen2.5-7B	Math	26.08	8.61	22.44	25.46
	OOD	49.82	48.91	49.00	49.35
Qwen3-1.7B	Math	20.16	9.62	19.34	19.68
	OOD	36.09	35.09	36.22	35.63

Appendix

Table 16: OPSD accuracy (%) at step 780 under three teacher input formats. Students always receive Raw problems. Base is the untrained student; each metric averages its datasets equally.

Student	Teacher input	$D_{\mathrm{in}}$	$D_{\mathrm{OOD}}$	$\rho$
Qwen2.5-1.5B	Raw	2.965	0.440	0.148
	Chat template	4.978	0.472	0.095
	Raw + answer boundary	4.691	0.599	0.128
Qwen2.5-7B	Raw	2.308	0.189	0.082
	Chat template	4.789	0.192	0.040
	Raw + answer boundary	4.585	0.188	0.041

Appendix

Table 17: OPSD distribution changes at step 780 for the same runs as Table 16 . JS is measured relative to Base in

10^{-2}

nats;

\rho=D_{\mathrm{OOD}}/D_{\mathrm{in}}

Figure 15: OPSD mathematical and OOD accuracy throughout training under the three teacher input formats, one student per row. Dotted horizontal lines mark the common Base evaluation. Each curve uses measured checkpoints at 20-step intervals.

Qwen3-1.7B
Update direction	In-domain JS $\times 10^{5}$	OOD JS $\times 10^{5}$	$\rho$
A: Base teacher	1.079	0.659	0.611
D: RLVR teacher	1.072	0.079	0.074
E: RLVR increment	0.938	0.053	0.056
RLVR parameter change (step 60)	1.228	0.071	0.058
RLVR parameter change (step 780)	1.098	0.069	0.063

Appendix

Table 18: Independent-prefix measurements after calibrating in-domain JS to

10^{-5}

. Incorporating the teacher’s RLVR changes reduces relative OOD influence and brings it closer to the paired direct RLVR parameter directions.

On-Policy Distillation

On-policy distillation trains a model on its own outputs using teacher guidance, aligning supervision with the states it encounters at deployment.

See all 507 papers

Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
Apr 14, 2026Yaxuan Li, Yuxin Zuo, Bingxiang He +8Language Model Distillation On-Policy Distillation
Tsinghua University · ShanghaiTech University · University of Illinois Urbana-Champaign +1
The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes
May 11, 2026Siqi Zhu, Xuyan Ye, Hongyu Lu +2On-Policy Self-Distillation On-Policy Distillation
1UIUC · 2Renmin University of China · †Work done during an internship at UIUC. +1
Gains and Collapse in On-Policy Distillation:A Reinforcement Learning Perspective
Oct 2, 2026Han Cui, Jianhao Yan, Yun Luo +3On-Policy Distillation Language Model Distillation
Westlake University

Apr 14, 2026J/K move · Enter open · S save

Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe

Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang+7

Tsinghua University · ShanghaiTech University · University of Illinois Urbana-Champaign+1

Language Model Distillation On-Policy Distillation Knowledge Distillation Teacher-Student Learning

Open paper AskPDF arXiv

On-policy distillation (OPD) has become a core technique in the post-training of large language models, yet its training dynamics remain poorly understood. This paper provides a systematic investigation of OPD dynamics and mechanisms. We first identify that two conditions govern whether OPD succeeds or fails: (i) the student and teacher should share compatible thinking patterns; and (ii) even with consistent thinking patterns and higher scores, the teacher must offer genuinely new capabilities beyond what the student has seen during training. We validate these findings through weak-to-strong reverse distillation, showing that same-family 1.5B and 7B teachers are distributionally indistinguishable from the student's perspective. Probing into the token-level mechanism, we show that successful OPD is characterized by progressive alignment on high-probability tokens at student-visited states, a small shared token set that concentrates most of the probability mass (97%-99%). We further propose two practical strategies to recover failing OPD: off-policy cold start and teacher-aligned prompt selection. Finally, we show that OPD's apparent free lunch of dense token-level reward comes at a cost, raising the question of whether OPD can scale to long-horizon distillation.

Abstract

On-Policy Distillation

Explore similar work

Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe

The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes

Gains and Collapse in On-Policy Distillation:A Reinforcement Learning Perspective