On-Policy Distillation Teaches New Skills but Not New Knowledge

Yixuan Tang; Yi Yang

cs.CLOct 7, 2026

On-Policy Distillation Teaches New Skills but Not New Knowledge

Authors: Yixuan Tang, Yi Yang

Organizations: The Hong Kong University of Science and Technology

Abstract

On-policy distillation (OPD) strengthens language-model reasoning, yet whether students acquire new factual knowledge or compositional skill for multi-step reasoning remains unknown. We separate these capabilities using a controlled synthetic framework that measures the student's initial capabilities and independently controls the teacher's additional facts, compositional skill, or both. Across four models from three families, reverse-KL OPD reliably transfers compositional skill across unseen reasoning structures, but transfers minimal factual knowledge. Decoupling the distillation recipe reveals the source of this asymmetry: replacing reverse KL with forward KL restores factual transfer, whereas student rollouts specifically improve the execution of multi-step reasoning. Experiments on recent factual QA and competition mathematics show a similar asymmetry under reverse-KL OPD, yielding notable reasoning gains without factual memory expansion. Together, these results demonstrate that on-policy distillation does not expand a model's parametric knowledge, but instead teaches it to organize and compose the knowledge it already possesses.

Figures & tables

Figure 1 : Overview of the setup. (a) Random lookup tables represent facts; chains and trees specify how their query results are combined. (b) Starting from

S_{0}

, teachers add exposure to held-out facts (

\Delta F

), branching trees (

\Delta S

), or both, with frozen

S_{0}

as the baseline. (c) A fixed teacher supervises student rollouts with reverse KL. (d) Evaluation separates factual recall on held-out facts from compositional skill over initial facts.

(a) Student and teacher capabilities
Model	Factual knowledge	Compositional skill
$S_{0}$	Initial facts	Chain execution
$T_{\emptyset}=S_{0}$	Initial facts	Chain execution
$T_{f}$	Initial + held-out facts	Chain execution
$T_{s}$	Initial facts	Chain and branching-tree execution
$T_{fs}$	Initial + held-out facts	Chain and branching-tree execution

Table 1 : Student and teacher capabilities and distillation conditions.

Figure 2 : Student performance after on-policy distillation under each teacher. Top: greedy final-answer accuracy; bottom: final-answer Pass@32 (at least one of 32 sampled completions correct). Numbers above each group show the unweighted mean across four models. All values are percentages.

Figure 3 : Effects of prefix source and KL direction. Rows show Qwen3-4B and Gemma-2-2B. Factual knowledge is evaluated after distillation from

T_{f}

; compositional skill after distillation from

T_{s}

. Accuracy measures greedy final answers; NLL is reported for reference completions in bits per completion. Reference prefixes come from teacher demonstrations.

Model	Role / Condition	Knowledge-2026	Math-2026
Qwen3-4B	Initial student $S_{0}$	11.25	17.41
	Knowledge teacher $T_{K}$	84.55	13.93
	Reasoning teacher $T_{R}$	12.24	43.78
	Distilled from $T_{K}$	9.64	19.40
	Distilled from $T_{R}$	10.75	23.88

Table 2 : Greedy accuracy (%) before and after on-policy distillation on real-world benchmarks. Shaded cells indicate each teacher’s target capability.

Panel of Figure 4 — Figure 4 : Greedy execution traces from Qwen3-4B students after distillation. (a) Seven-step chain over held-out facts: reverse KL maintains correct execution syntax but outputs incorrect lookup values ( red ), whereas forward KL restores correct lookups. (b) Unseen tree over initial facts: forward KL recalls all facts correctly but flattens the tree into a linear sweep ( red ), whereas reverse KL from $T_{s}$ executes the valid postorder. Red highlights incorrect values or misrouted operands; green marks correct final answers.

(a) Parameter Metric	Qwen3-4B	Gemma-2-2B	OLMo3-7B
Update Norm: $T_{f}$ (factual)	2.01–2.86	1.33–1.52	3.08–3.89
Update Norm: $T_{s}$ (compositional)	1.27–1.55	1.13–1.60	2.50–3.21
Update Norm: $T_{\emptyset}$ (control)	0.30–0.32	0.20–0.24	0.48–0.55
Cosine Similarity: $T_{f}$ vs. $T_{s}$	0.004–0.018	0.011–0.036	0.004–0.046
Cosine Similarity: Rev. vs. Fwd. KL ( $T_{s}$ )	0.73–0.78	0.64–0.82	0.68–0.80

Table 3: LoRA weight updates relative to

S_{0}

. (a) Update magnitude and directional alignment across models. (b) Layer-wise norm distribution across depth quartiles in Qwen3-4B. Full breakdowns across all projections and embedding drift appear in Appendix E (Table 6 ).

	Teacher $T_{f}$		Teacher $T_{s}$
Objective	Fact-Single	Comp-Unseen	Fact-Single	Comp-Unseen
$S_{0}$ (no distillation)	0.1	1.8	0.1	1.8
Reverse KL	10.9	0.7	0.1	76.8
Forward KL	54.1	0.5	0.1	76.4
GKD ( $\beta=0.5$ )	9.1	0.7	0.1	70.6
ToDi	64.3	0.5	0.1	86.4

Table 4: Greedy final-answer accuracy (%) on Qwen3-4B with alternative distillation objectives. All distillation runs use student prefixes.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

		Factual Knowledge		Compositional Skill
Model	Profile	Fact-Single	Fact-Chain	Comp- Shared	Comp- TeacherOnly	Comp- Unseen
Qwen3-0.6B	$S_{0}$ (student)	0.00	0.00	0.15	2.10	0.20
	$T_{f}$	100	100	0.20	2.05	0.45
	$T_{s}$	0.00	0.00	100	100	99.95
	$T_{fs}$	100	100	100	99.95	100
Gemma-2-2B	$S_{0}$ (student)	0.00	0.00	0.40	2.20	0.45

Appendix

Table 5 : Greedy final-answer accuracy (%) of

S_{0}

and the specialized teachers on factual knowledge and compositional skill tests. Test definitions appear in Appendix B.3 . All tests use exact match of the final four-digit answer. Green shading highlights high accuracy.

Figure 5 : Training curves for on-policy distillation in the synthetic experiments. Columns correspond to distillation from

T_{f}

(a, d),

T_{s}

(b, e), and the joint

T_{fs}

(c, f). Top row: exact-match accuracy of sampled student rollouts on the distillation prompts; bottom row: reverse KL in nats. All runs train for three epochs over 20,000 prompts.

(a) Teacher: prefix, objective	Qwen3-4B	Gemma-2-2B	OLMo3-7B
$T_{f}$ : student, reverse KL	2.01 / 2.4	1.33 / 3.9	3.08 / 2.3
$T_{f}$ : student, forward KL	2.64 / 2.5	1.49 / 3.6	3.85 / 2.0
$T_{f}$ : teacher, forward KL	2.86 / 2.5	1.52 / 3.6	3.89 / 2.1
$T_{f}$ : teacher, reverse KL	2.27 / 2.5	1.34 / 3.8	3.20 / 2.1
$T_{s}$ : student, reverse KL	1.27 / 3.3	1.13 / 3.9	2.50 / 2.6
$T_{s}$ : student, forward KL	1.55 / 2.7	1.60 / 3.6	3.21 / 2.5

Appendix

Table 6 : Complete weight updates relative to

S_{0}

. (a) Global update norm / mean effective rank. (b) Cosine similarity between updates. (c) Mean embedding drift for held-out / initial fact identifiers. (d) Mean layer norms by depth quartile. (e) Norms by projection. Ranges cover the four prefix–KL combinations; control ranges cover two prompt sets.

Explore similar work

Finding the Evidence: Discovering Decision-Supporting Tokens for On-Policy Reasoning Distillation
Jun 22, 2026Jinwei Xiao, Zhuowen Han, Yueqing Sun +6CoT Distillation On-Policy Distillation
Meituan Longcat Team · TJUNLP Lab, College of Intelligence and Computing, Tianjin University · Nanjing University
Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall
Sep 1, 2026Jacqueline He, Howard Yen, Shuyue Stella Li +9Language Model Distillation Factual Knowledge in Language Models
Meta AI · University of Washington · Princeton University
Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation
May 29, 2026Yanjiang Liu, Jie Lou, Xinyan Guan +7CoT Distillation Language Model Distillation
University of Chinese Academy of Sciences · 2Chinese Information Processing Laboratory Institute of Software, Chinese Academy of Sciences 2University of Chinese Academy of Sciences, Beijing, China · 3Xiaohongshu

Jun 22, 2026, cs.AIJ/K move · Enter open · S save

Finding the Evidence: Discovering Decision-Supporting Tokens for On-Policy Reasoning Distillation

Jinwei Xiao, Zhuowen Han, Yueqing Sun, Zhengxi Lu+5

Meituan Longcat Team · TJUNLP Lab, College of Intelligence and Computing, Tianjin University · Nanjing University

CoT Distillation On-Policy Distillation Teacher-Student Learning Knowledge Distillation Algorithmic Reasoning LLM Reasoning

Open paper AskPDF arXiv

On-policy distillation transfers reasoning ability through dense token-level supervision, yet the nature of the transferable signal remains unclear. We discover that reasoning chains contain two types of knowledge that require different discovery mechanisms: decisions (where to branch), which surface through student uncertainty, and evidence (intermediate steps that justify decisions), which hides in positions where the student is confident yet wrong. Current methods capture only decisions; the substantive knowledge in evidence tokens remains untransferred. We propose DEAR(Decision-Evidence Aware Reasoning Distillation), which first identifies decisions via student entropy, then discovers their supporting evidence through hidden-state cosine similarity to decision anchors, boosted by teacher-student divergence to prioritize the largest knowledge gaps. Across three student-teacher configurations on math and code benchmarks, DEAR consistently outperforms standard OPD, with up to +2.5pp on competition math and +5.7pp on code generation.

Abstract

Explore similar work

Finding the Evidence: Discovering Decision-Supporting Tokens for On-Policy Reasoning Distillation

Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall

Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation