Every Coin Has Two Sides: On the Dual Nature of Generalization in On-Policy Distillation of Large Language Models

Zhaoyi Li; Deyang Kong; Yuan Wei; Evan Yang; Ranran Shen; Mahardika Krisna Ihsani; Ming Yang; Wei Zhang

cs.CLAug 17, 2026

Every Coin Has Two Sides: On the Dual Nature of Generalization in On-Policy Distillation of Large Language Models

Authors: Zhaoyi Li, Deyang Kong, Yuan Wei, Evan Yang, Ranran Shen, Mahardika Krisna Ihsani, Ming Yang, Wei Zhang, +8 more

Organizations: University of Science and Technology of China · IQuest Research · Peking University · MBZUAI · Zhejiang University

Abstract

On-policy distillation (OPD) transfers teacher capabilities by supervising trajectories sampled from the student's own policy, yet its generalization behavior remains poorly understood, as most studies evaluate OPD on a single domain and on benchmarks close to the training data. We present a controlled study that varies one generalization factor at a time, from in-domain distribution shifts to cross-domain transfer and the multi-teacher setting. We find that OPD transfers a teacher's reasoning behavior rather than its answers to particular problems: training difficulty barely matters, and even problems the teacher never solves are useful. Transfer depends strongly on the origin relationship between teacher and student: same-origin pairs bring the student close to the teacher across languages, reasoning horizons, and even other domains, whereas cross-origin pairs mostly fit the trained distribution. This broad reach is a double-edged sword: since routing prompts to domain experts cannot confine each teacher's influence, combining them yields a mixture-dependent seesaw among their capabilities. These results clarify when OPD generalizes and offer a useful perspective for diagnosing multi-teacher OPD.

Figures & tables

Panel of Figure 1 — Figure 1: In-domain math accuracy (average over six English benchmarks) is insensitive to the difficulty of the training problems, measured by the teacher’s pass-rate. Each figure compares three BigMath subsets: easy (pass-rate $=1$ , teacher solves all four rollouts), hard (pass-rate $=0$ , teacher solves none), and random (randomly sampled from the whole set). The three subsets converge to nearly identical final accuracy across all teacher–student pairs.

Filtering Strategy	AMC2023	MATH500	AIME2025	AIME2026	BeyondAIME	OlymMATH-Hard	Average
Teacher: Polaris-7B, Student: DS-distill-1.5B
w/o dynamic sampling	$80.4\%$	$89.5\%$	$30.2\%$	$30.1\%$	$13.6\%$	$4.7\%$	$41.4\%$
pass-rate $\mathbf{=0}$	$78.8\%$	$90.1\%$	$30.5\%$	$30.0\%$	$14.7\%$	$4.4\%$	$41.4\%$
pass-rate $\mathbf{=1}$	$79.9\%$	$89.7\%$	$30.9\%$	$28.9\%$	$14.7\%$	$4.0\%$	$41.4\%$
pass-rate $\mathbf{\in[0,1)}$	$80.9\%$	$89.8\%$	$31.2\%$	$31.1\%$	$13.7\%$	$5.4\%$	$\mathbf{42.0}\%$ ( +0.6 pp )
Teacher: Light-R1-14B, Student: DS-distill-7B

Table 1: Generalization performance of OPD with student-side dynamic sampling across six math benchmarks. Discarding only the problems the student already solves (pass-rate

\in[0,1)

) gives the best average performance. Green/red mark per-benchmark gains/drops relative to the w.o. dynamic sampling baseline.

Panel of Figure 2 — Figure 2: OPD trained only on English math problems generalizes to Chinese and long-horizon math benchmarks, but the size and stability of the gains depend on model origin. Each figure pairs a same-origin and a cross-origin teacher for the same student; dashed lines mark the teachers’ accuracy.

Panel of Figure 3 — Figure 3: Cross-domain generalization in single-teacher OPD. For same-origin OPD, training on math prompts, trained student’s performance on science and code also approaches teacher’s performance and vice versa. For cross-origin OPD, cross-domain generalization is much worse: in (a) and (b) (LiveCodeBench evaluation results), the green curves (training on code prompts) are consistently and substantially higher than the blue curves (training on math prompts). Dashed lines mark the teachers’ standalone accuracy.

Panel of Figure 4 — Figure 4: The MOPD seesaw effect . Changing the mixture ratio of different teachers’ allocated prompts significantly changes the OPD students’ capabilities in different domains. For example, in subfigure (a), student’s GPQA-Diamond (scientific reasoning), LiveCodeBench (code reasoning), and IF-Eval (instruction-following) performance gradually drops down as we increase the proportion of JustRL-1.5B (math teacher) data.

Config	BeyondAIME	OlymMATH	Average
DS-distill-1.5B	$9.6\%$	$14.2\%$	$11.9\%$ ( student )
Math Teacher: JustRL-1.5B, Science/IF Teacher: Nemotron-1.5B
$\mathbf{\text{J}/\text{N}=25/2}$	$19.4\%$	$33.3\%$	$26.4\%$ ( +0.1 pp )
$\mathbf{\text{J}/\text{N}=25/8}$	$19.6\%$	$34.5\%$	$27.1\%$ ( +0.8 pp )
$\mathbf{\text{J}/\text{N}=1/1}$	$20.2\%$	$32.4\%$	$26.3\%$ ( baseline )
$\mathbf{\text{J}/\text{N}=8/25}$	$19.7\%$	$32.5\%$	$26.1\%$ ( -0.2 pp )

Table 2: MOPD math accuracy for different JustRL-1.5B/Nemotron-1.5B mixture ratios (J/N).

Panel of Figure 5 — Figure 5: A tug-of-war between the teachers can be observed during MOPD training. (a,b) highlighted red boxes show that the MOPD student’s performance first tracks the JustRL-only training curve and later drifts toward the Nemotron-only training curve , under both settings. (c) In cascaded OPD (one teacher then the other), GPQA performance rises sharply under the guidance of Nemotron-1.5B with science prompts and then drops back by the subsequent JustRL-1.5B with math prompts, decoupling the effect of multi-teacher in the temporal dimension.

Panel of Figure 6 — Figure 6: Top-K (K=16) Overlap Ratio of the teacher and student models in different OPD experiments. In Figure (a) and (b), the student models are DS-distill-1.5B and DS-distill-7B, respectively. The solid lines refer to OPD experiments with same-origin teachers; the dashed lines refer to OPD experiments with different-origin teachers.

Figure 7: The MOPD experiment results of DS-distill-7B (student), Light-R1-7B (math teacher), and Light-R1-14B (science/IF teacher). The legends refer to the expert data mix ratio of different MOPD experiments (Light-R1-14B/Light-R1-7B

\in\{1/0,1/1,8/25,4/25,2/25,0/1\}

Figure 8: MOPD on DS-distill-7B (student) with Light-R1-14B as the math expert and Polaris-7B as the science/IF expert, evaluated on four math benchmarks (AMC2023, AIME2024 (Horizon 2), AIME2026, OlymMATH-en-easy). Solid lines are MOPD runs at math/science-IF data mix ratios Light-R1-14B/Polaris-7B

\in\{8/25,1/1,25/8\}

; the dashed orange line is single-teacher OPD with only the math expert Light-R1-14B; the dashed grey line marks the science/IF expert Polaris-7B’s own math accuracy.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Lineage Root	Post-Training from the Root	Experimental Role
DS-R1-Distill-Qwen-1.5B	Qwen2.5-Math-1.5B	SFT on DeepSeek-R1 traces	Student
JustRL-DeepSeek-1.5B	DS-R1-Distill-Qwen-1.5B	Math-oriented RL	Teacher
DeepScaleR-1.5B	DS-R1-Distill-Qwen-1.5B	Math-oriented RL	Teacher
Nemotron-Research-Reasoning-Qwen-1.5B	DS-R1-Distill-Qwen-1.5B	Multi-domain prolonged RL	Teacher
Dev-1.5B	DS-R1-Distill-Qwen-1.5B	10-step science/IF OPD	Student
OpenMath-Nemotron-1.5B	Qwen2.5-Math-1.5B	SFT on OpenMathReasoning	Teacher

Appendix

Table 3: Post-training lineages of the models used in our experiments. Models sharing the same lineage root are treated as same-origin.

Model Group	Maximum Length
Qwen3-4B, Qwen3-8B, Qwen3-32B, Polaris-4B	40K
DS-distill-1.5B, JustRL-1.5B, Nemotron-1.5B	96K
Light-R1-14B, DS-distill-14B	64K
Polaris-7B, DS-distill-7B	96K

Appendix

Table 4: Maximum sequence lengths used for OPD rollout and evaluation.

Benchmark	Metric
AMC 2023	Avg@16
MATH-500	Avg@1
AIME 2025	Avg@16
AIME 2026	Avg@16
BeyondAIME	Avg@5
OlymMATH-Hard	Avg@5

Appendix

Table 5: Number of independently sampled responses used for benchmark evaluation.

Panel of Figure 9 — Figure 9: In-domain math performance (average over six English benchmarks) is largely insensitive to the difficulty of the training queries. Subfigure (a–c) training on the extremely easy GSM8K, the extremely hard DeepMath-103K, and the diverse BigMath mixture.

Panel of Figure 10 — Figure 10: Additional generalization results across three teacher–student configurations.

Panel of Figure 11 — Figure 11: Training on code and science domains and generalizing to math-related domains.

Panel of Figure 12 — Figure 12: Training on instruction-following data and generalizing on math-related benchmarks.

Configuration	BeyondAIME	OlymMATH-e(en)	OlymMATH-e(zh)	OlymMATH-h	AIME24-n2	Average
DS-distill-1.5B	$9.6\%$	$14.2\%$	$11.2\%$	$3.2\%$	$6.0\%$	$8.8\%$
JustRL-1.5B	$19.0\%$	$32.6\%$	$23.0\%$	$5.7\%$	$16.3\%$	$19.3\%$
Nemotron-1.5B	$17.6\%$	$29.2\%$	$16.6\%$	$5.6\%$	$14.3\%$	$16.7\%$
Math Teacher: JustRL-1.5B, science/IF Teacher: Nemotron-1.5B, Student: DS-distill-1.5B
$\mathbf{\text{JustRL}/\text{Nemotron}=25/0}$	$19.0\%$	$32.6\%$	$23.0\%$	$5.7\%$	$16.3\%$	$19.3\%$ ( +0.2 pp )
$\mathbf{\text{JustRL}/\text{Nemotron}=25/2}$	$19.4\%$	$33.3\%$	$23.0\%$	$5.9\%$	$14.1\%$	$19.1\%$ ( +0.0 pp )

Appendix

Table 6: MOPD performance comparison across five mathematical reasoning benchmarks (training step 200).

Explore similar work

The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes
May 11, 2026Siqi Zhu, Xuyan Ye, Hongyu Lu +2On-Policy Self-Distillation On-Policy Distillation
1UIUC · 2Renmin University of China · †Work done during an internship at UIUC. +1
Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation
Sep 2, 2026Zhiwei Zhang, Zechen Sun, Fei Zhao +6On-Policy Distillation RL Post-Training
AllSpark Team
Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe
May 5, 2026Wenjin Hou, Shangpin Peng, Weinong Wang +13Language Model Distillation Multi-Teacher Knowledge Distillation
1Zhejiang University · 2Shenzhen Loop Area Institute · 3LLM Department, Tencent

May 11, 2026, cs.AIJ/K move · Enter open · S save

The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes

Siqi Zhu, Xuyan Ye, Hongyu Lu, Weiye Shi+1

1UIUC · 2Renmin University of China · †Work done during an internship at UIUC.+1

On-Policy Self-Distillation On-Policy Distillation Learning Using Privileged Information Teacher-Student Learning

Open paper AskPDF arXiv

On-policy distillation (OPD) and on-policy self-distillation (OPSD) have emerged as promising post-training methods for large language models, offering dense token-level supervision on trajectories sampled from the model's own policy. However, existing results on their effectiveness remain mixed: while OP(S)D has shown promise in system prompt and knowledge internalization, recent studies also report instability and degradation. In this work, we present a comprehensive empirical study of when OPD and OPSD work, when they fail, and why. We find that OPD on mathematical reasoning is highly sensitive to teacher choice and loss formulation, whereas OPSD fails in our tested settings due to test-time absence of instance-specific privileged information (PI). In contrast, OPSD is effective when PI represents a shared latent rule, such as a system prompt or alignment preference. We identify three failure mechanisms: (1) distribution mismatch between teacher and student caused by conditioning on student-generated prefixes, (2) optimization instability from biased TopK reverse-KL gradients, and (3) an OPSD-specific limitation where the student learns a PI-free policy that aggregates PI-conditioned teachers, which is insufficient when PI is instance-specific. We further show that stop-gradient TopK objectives, RLVR-adapted teachers, and SFT-stabilized students mitigate these failures.

Abstract

Explore similar work

The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes

Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation

Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe