Data-Free On-Policy Distillation: How Far Can We Go Without External Data?

Gengsheng Li; Mao Zheng; Mingyang Song; Jie Sun; Zeyuan Liu; Ruiqi Liu; Tianyu Yang; Qiyong Zhong

cs.LGSep 12, 2026

Data-Free On-Policy Distillation: How Far Can We Go Without External Data?

Authors: Gengsheng Li, Mao Zheng, Mingyang Song, Jie Sun, Zeyuan Liu, Ruiqi Liu, Tianyu Yang, Qiyong Zhong,

Organizations: Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Foundation Model Department, Tencent · National University of Singapore ·

Abstract

On-policy distillation (OPD) is increasingly applied to frontier foundation model post-training. Prior work in this area has largely focused on algorithmic advances, yet it remains unclear how much OPD depends on its training questions and, in particular, how far this dependence can be reduced. Across two representative single-teacher OPD settings, we find that training on 8 real prompts yields performance comparable to training on 17k problems, while datasets differing substantially in measured difficulty and initial distillation gap yield similar outcomes. Our analyses suggest two complementary explanations: repeated sampling could allow even a few prompts to expose substantial teacher supervision, while OPD transfers generalizable reasoning capabilities beyond dataset-specific knowledge. Building on these observations, we next propose a data-free on-policy distillation (DF-OPD) setting to investigate whether the system can supply the training questions itself, eliminating the need for external data. With 64 self-generated questions obtained without seed examples, DF-OPD yields performance comparable to full-data OPD in both single-teacher settings. This finding also holds in multi-teacher OPD: across mathematics, code, and instruction following, 1k generated questions achieve performance comparable to training on approximately 7k real post-training examples. We further explore whether OPD can operate even without explicit training questions. The experiments show that this is effective only in limited cases, where the student unexpectedly generates and answers its own questions, thereby reducing the process to an implicit form of DF-OPD. Together, these findings invite a reassessment of the role of training data in on-policy distillation. Code is available at https://github.com/Ryuki661/DF-OPD

Figures & tables

Figure 1: Progressively reducing OPD’s dependence on training questions. (a) With external questions, OPD shows limited sensitivity to their quantity, difficulty, and initial distillation gap. (b) DF-OPD replaces them with self-generated questions, produced by the teacher without seed examples, and performs comparably to full-data OPD. (c) Empty prompts are effective only in limited cases, where the student spontaneously poses and answers its own question (Self-QA), reducing the process to implicit DF-OPD.

Figure 2: Few real questions support effective OPD. (a,b) DAPO training curves under matched update and trajectory budgets. Macro accuracy (%) is averaged over six mathematics benchmarks, evaluated every 15 updates. (c) Peak scores; bars start at the initial student, and numbers give the difference from full DAPO in percentage points.

Figure 3: Initial diagnostics and learning curves. (a) Initial correctness (left axis) and initial distillation gap (right axis) in the RL-tuned teacher setting. (b,c) Six-benchmark macro accuracy for the RL-tuned and larger teacher settings with 1,000 questions per source.

Figure 4: Signal coverage and cross-domain transfer. (a) Signal coverage grows with repeated sampling. Mean

\pm

std over six resampling draws of the frozen initial student; brackets mark gains between adjacent pool sizes at 4,096 trajectories. (b,c) Transfer from programming to mathematics for the RL-tuned and larger teacher settings with

|D|=1{,}000

questions. Macro accuracy on the same six mathematics benchmarks; dashed lines mark initial students.

Figure 5: DF-OPD workflow. A frozen teacher generates a fixed question pool from a simple instruction. The student samples responses, and the same teacher provides token-level feedback on student-generated prefixes. Standard OPD updates only the student. The instruction shown is the RL-tuned teacher’s generation prompt.

Figure 6: Self-generated questions replace the external corpus. (a,b) Single-teacher OPD with different numbers of generated questions, compared with training on the full real DAPO corpus. (c) Peak scores; bars start at the initial student, and numbers give the difference from full real DAPO in percentage points. In both settings, 64 questions reach comparable performance. Complete results appear in Appendix E.3 .

Input / reference	Questions	Math	Code	IF	Overall	Recovery
Initial student	—	15.83	15.77	41.01	24.21	—
Routed teachers	—	23.02	22.52	48.73	31.42	100%
Real post-training data	7,458	23.02	21.71	48.80	31.18	96.6%
Self-generated	8	23.44	19.64	45.92	29.66	75.6%
Self-generated	64	22.60	20.87	47.51	30.33	84.8%
Self-generated	256	23.54	20.99	47.86	30.80	91.3%

Table 1: Self-generated questions in multi-teacher distillation. Each domain score averages its two benchmarks, and Overall is the macro average of the three domain scores. Recovery is relative to the routed teachers (Equation 2 ). All trained rows use the peak-Overall checkpoint. Full results appear in Appendix E.4 .

Figure 7: Omitting questions has a setting-dependent effect. (a) Peak scores: six-benchmark mathematics averages (single-teacher; DF-OPD with 64 questions) or three-domain averages (MOPD; DF-OPD with 1k). (b) Self-QA rate of each initial student (1,024 empty-prompt rollouts, setting-specific domain criteria; Appendix F.2 ) and the full-data gain retained by empty-prompt OPD.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Student	Teacher	Relationship
RL-tuned teacher	DeepSeek-R1-Distill-Qwen-1.5B	JustRL-DeepSeek-1.5B	RL-tuned from the student
Larger teacher	Qwen3-1.7B	Qwen3-30B-A3B-Instruct-2507	Different-scale models from the Qwen3 family
MOPD: math	SmolLM3-3B-MixSFT	SmolLM3-3B-RL-Math	Math-specific RL from MixSFT
MOPD: code	SmolLM3-3B-MixSFT	SmolLM3-3B-RL-Code	Code-specific RL from MixSFT
MOPD: instruction following	SmolLM3-3B-MixSFT	SmolLM3-3B-RL-IF	Instruction-following RL from MixSFT

Appendix

Table 2: Teacher and student configurations. Model names refer to publicly released checkpoints. The three MOPD domains share the same student initialization and use the checkpoint collection released by Open-MOPD.

Configuration	Single teacher	Multiple teachers
Questions per update	64	64
Student support size $k$	16	128
Policy passes per rollout batch	1	1
Optimizer	AdamW	AdamW
Learning rate	$10^{-6}$	$10^{-6}$
AdamW $\beta_{1},\beta_{2}$	0.9, 0.999	0.9, 0.999

Appendix

Table 3: Training configurations for the single-teacher and multi-teacher settings. The student support size

k

is used to compute the distillation objective and is distinct from decoding top-

k

during generation; decoding top-

k

is disabled in both settings.

Configuration	Single-teacher math	MOPD math	MOPD code	MOPD IF
Responses per problem	4	16	5	1
Sampling temperature	0.7	0.6	0.6	0.6
Top- $p$	0.95	0.95	0.95	0.95
Response limit (tokens)	16,384	32,768	32,768	32,768

Appendix

Table 4: Evaluation configurations for the single-teacher and multi-teacher settings. Responses per problem denotes the number of independently sampled responses for each evaluation problem.

Figure 8: Per-benchmark OPD training curves in the RL-tuned teacher setting. The six panels show accuracy on AIME24, AIME25, AMC23, MATH-500, Minerva, and Olympiad-Bench, comparing training on 1, 8, 64, 256, and 1k DAPO questions with training on the full DAPO dataset. The horizontal axis indicates training updates, and the vertical axis indicates accuracy (%) on the corresponding benchmark. All runs use the same number of updates and sampled trajectories per update.

Figure 9: Macro-average OPD training curves with different numbers of DeepMath questions. Training curves are shown for the RL-tuned and larger teacher settings, comparing 1, 64, and 1k training questions. The horizontal axis indicates training updates, and the vertical axis indicates macro-average accuracy (%) over the six mathematics benchmarks. Within each teacher setting, all runs use the same training and evaluation configurations apart from the training question pool.

Figure 10: Macro-average OPD training curves with different numbers of ORZ questions. Training curves are shown for the RL-tuned and larger teacher settings, comparing 1, 64, and 1k training questions. The horizontal axis indicates training updates, and the vertical axis indicates macro-average accuracy (%) over the six mathematics benchmarks. Within each teacher setting, all runs use the same training and evaluation configurations apart from the training question pool.

Domain	Topics
Mathematics	algebra; geometry; number theory; combinatorics; probability; statistics; calculus; precalculus; trigonometry; logic puzzles; sequences and series; inequalities; functional equations; complex numbers; polynomials; graph theory
Code	dynamic programming; greedy algorithms; graph traversal; shortest paths; string processing; binary search; sorting and ordering; data structures; number theory; combinatorics and counting; simulation; two pointers; prefix sums; recursion and backtracking; intervals and scheduling; matrix and grid problems
Instruction following	response length in words; number of sentences; number of paragraphs; required keywords; forbidden words; starting and ending phrases; letter casing; JSON or structured output; bullet-point formatting; title or heading requirements; placeholder tokens; repeating the prompt; language of the response; punctuation restrictions; numbered list requirements; section markers

Appendix

Table 5: Topics for question generation, with 16 topics per domain. The mathematics topic list is shared by the two single-teacher settings and the MOPD mathematics teacher. The code and instruction-following lists are used by their respective domain teachers. Each generation prompt replaces {topic} with a randomly selected entry from the corresponding list.

Teacher	Generation requests	Generation seeds	Final questions
RL-tuned teacher	1,024	42	3,822
Larger teacher	1,024	42	3,930
MOPD mathematics teacher	1,024	42	2,971
MOPD code teacher	1,024	42	2,949
MOPD instruction-following teacher	1,024	42	3,082

Appendix

Table 6: Teacher question-generation settings and question-pool sizes after processing. Each generation request samples four outputs. Generation seeds are the random seeds used by the teachers to generate questions. The final question counts are measured after extraction, format checks, deduplication, and length filtering, before subsampling to the pool sizes used in the experiments.

Topic	Question text
number theory	Find all prime numbers $p$ such that $p^{2}$ divides $2^{p}-2$ .
complex numbers	Let $z$ be a complex number satisfying $z^{2}+(1+i)z+(2-2i)=0$ . Find $\|z_{1}\|\times\|z_{2}\|$ .
combinatorics	In how many ways can 8 people be seated around a circular table such that Alice and Bob cannot sit next to each other?
geometry	In a right-angled triangle $ABC$ with right angle at $A$ , the incircle touches $AB$ at $D$ and $AC$ at $E$ . The tangent from $C$ to the incircle meets $AB$ at $F$ . If $AB=3$ , $AC=4$ , find the length of $CF$ .
calculus	A 10-foot ladder is leaning against a vertical wall. The base of the ladder is being pushed away from the wall at a rate of 2 feet per second. When the base is 6 feet away from the wall, how fast is the top of the ladder sliding downward?

Appendix

Table 7: Examples of mathematics questions generated by the RL-tuned teacher. Five questions retained after filtering are shown with their corresponding generation topics.

Topic	Question text
geometry	A regular dodecagon (12-sided polygon) is inscribed in a circle of radius 1. Let $P$ be a point on the circumference such that the sum of the squares of the distances from $P$ to each of the 12 vertices of the dodecagon is minimized. What is the minimum value of this sum?
trigonometry	In triangle $ABC$ , let $\angle A=60^{\circ}$ , and let $D$ be a point on side $BC$ such that $AD$ bisects $\angle A$ . Suppose $BD=2$ and $DC=3$ . If $AD=x$ , find the exact value of $x^{2}$ using trigonometric identities and the Angle Bisector Theorem.
combinatorics	A group of 10 mathematicians is attending a conference. Each day, they form a circle to discuss a problem, and no two people can sit next to each other more than once during the conference. What is the maximum number of days the conference can last such that this condition is satisfied?
graph theory	Let $G$ be a simple, undirected graph with 12 vertices and 30 edges. Suppose that $G$ has no cycles of length 3 or 4 (i.e., girth at least 5). What is the maximum possible number of vertices of degree 5 or more in $G$ ?
precalculus	Let $f(x)=\log_{2}(x^{2}-4x+5)$ and $g(x)=\sqrt{x-1}$ . Find all real values of $x$ such that $f(g(x))=2$ , and determine the sum of all such values.

Appendix

Table 8: Examples of mathematics questions generated by the larger teacher. Five questions retained after filtering are shown with their corresponding generation topics.

Topic	Question text
number theory	What is the least positive integer $n$ for which the equation $(n+2)!+n=(n+1)!+(n+1)$ holds true?
functional equations	Find the function $f:\mathbb{R}\rightarrow\mathbb{R}$ such that for all real numbers $x,y,z$ satisfying $x+y+z=0$ , we have $f(x^{3})+f(y^{3})+f(z^{3})=xy(x+y)+xz(x+z)+yz(y+z)$ .
calculus	A function $f:\mathbb{R}\to\mathbb{R}$ satisfies the differential equation $3f^{\prime\prime}(x)+2f^{\prime}(x)-8f(x)=0$ with initial conditions $f(0)=1$ and $f^{\prime}(0)=0$ . Let $g(x)$ be the inverse function of $f(x)$ . Find the value of $g^{\prime}(-2)$ .
sequences and series	What is the smallest positive integer $n$ such that the sum of the first $n$ consecutive positive integers is divisible by 17?
probability	What is the probability that a randomly selected positive integer less than or equal to 1000 has a remainder of 3 when divided by 11?

Appendix

Table 9: Examples of questions generated by the MOPD mathematics teacher. Five mathematics questions retained after filtering are shown with their corresponding generation topics.

Topic	Question text
sorting and ordering	Problem Statement: Lexicographical Ordering of Strings Input Format: - The first line contains a single integer N ( $1\leq N\leq 10^{5}$ ), denoting the number of strings to sort. - The following N lines each contain a string consisting of lowercase English letters only. No string will be empty and no character will appear more than once in a string. Output Format: - Print N space-separated strings in lexicographical order. Each string must be printed exactly as it appears in the input, and you may not modify the order of strings within the input as long as it matches the sorted order. Constraints: - Time Limit: 5 seconds - Memory Limit: 256 MB - Judge Logos: Standard
combinatorics and counting	Problem Statement Alice, Bob, and Carol are playing a game using a set of marbles. They want to divide the marbles into three non-empty groups such that the product of the number of marbles in each group is maximized. The constraint is that the number of marbles in each group must be a positive integer, and no two groups can have the same number of marbles. Determine the number of distinct ways to partition the marbles into three such groups. Input Format The first line contains a single integer $T$ , the number of test cases. Each subsequent line contains a single integer $N$ , representing the total number of marbles. Output Format For each test case, output a single integer representing the number of distinct ways to partition the marbles into three non-empty groups with distinct sizes. Constraints - $1\leq T\leq 10$ - $3\leq N\leq 1000$ - The total number of marbles across all test cases does not exceed 1000.
graph traversal	Problem Statement: Longest Path in a Tree Input Format: - The first line contains two integers, $N$ and $M$ , representing the number of nodes and edges in the tree, respectively. - The subsequent $M$ lines describe the edges of the tree. Each line contains two integers, $u$ and $v$ , indicating an undirected edge between nodes $u$ and $v$ . Output Format: - Print a single integer, which is the length of the longest path (also known as the diameter) in the tree. The path should consist of exactly two distinct nodes, and the longest path must not include any cycles. Constraints: - $2\leq N\leq 10^{5}$ - $1\leq M\leq 2*10^{5}$ - Nodes are numbered from $1$ to $N$ .
data structures	Problem Statement: Weighted Path Reconstruction Input Format: - The first line contains two integers, $N$ and $M$ , representing the number of nodes and edges in the directed weighted graph, respectively. - The next $M$ lines describe the edges. Each line consists of three integers: $u$ , $v$ , and $w$ , indicating a directed edge from node $u$ to node $v$ with weight $w$ . Output Format: - Print $N$ integers, where the ith integer represents the minimum possible sum of weights of a path from node 1 to node $i$ . Constraints: - $2\leq N\leq 1000$ - $1\leq M\leq 5000$ - $1\leq u,v\leq N$ - $1\leq w\leq 1000$ - The graph can contain cycles and self-loops.
greedy algorithms	Problem Statement: Maximum Weight Connected Components Input Format: - The first line contains two integers, $N$ and $M$ , representing the number of nodes and edges in the graph, respectively. - The next $M$ lines each contain three integers, $u$ , $v$ , and $w$ , denoting an undirected edge between nodes $u$ and $v$ with a weight $w$ . Output Format: - A single integer, representing the maximum sum of weights of the connected components of the graph. If there are multiple components with the maximum sum, output the maximum sum. Constraints: - $2\leq N\leq 10^{5}$ - $1\leq M\leq 10^{5}$ - $1\leq u,v\leq N$ - $1\leq w\leq 10^{9}$ - All edges are distinct pairs ( $u,v$ ). - There are no self-loops ( u == v ). - The graph is undirected, meaning if there is an edge between $u$ and $v$ , there is also an edge between $v$ and $u$ .

Appendix

Table 10: Examples of questions generated by the MOPD code teacher. Five programming problems retained after filtering are shown with their corresponding generation topics.

Topic	Question text
response length in words	Please follow these instructions in your response: - 150-200 words long. - Include exactly three paragraphs. - End with the word “Answer”. Write a brief description of a cozy reading nook you would love to have in your home. I need help imagining a perfect reading spot.
number of paragraphs	Please follow these instructions in your response: - The answer must consist of exactly three paragraphs. - Each paragraph must begin with the word “Step ”. - The response must not contain any numbers other than those that are inherently part of the English language. Write a short story that explores a hidden library where forgotten memories are stored. Please provide the story.
starting and ending phrases	Please follow these instructions in your response: - 1. The response must begin with the phrase “Begin with the stars.” - 2. The response must end with the phrase “and the moon.” - 3. The response must include the exact word “quill” at least once. - 4. The response must be exactly 50 words in length. Request: Write a short story that blends astronomy with a creative writing tool.
placeholder tokens	Please follow these instructions in your response: - Use exactly three placeholder tokens: [[TEMP]] , [[LOC]] , and [[DATE]] . - Replace each placeholder token with a suitable word or phrase that fits the context. - Ensure the total length of the response is between 20 and 40 words. - Do not include any additional placeholder tokens. Write a short paragraph using the placeholders above, describing a future travel adventure.
title or heading requirements	Please follow these instructions in your response: - 1. Write a title or heading for the response that matches the given keyword “Cosmic”. - 2. Include exactly three paragraphs in the answer. - 3. Use bold and italics formatting exactly twice each, within the same paragraph. - 4. Keep the response under 60 words. Describe, in your own words, how the term “Cosmic” might relate to a vast, interconnected network of ideas and data. Provide a clear, concise example of how this concept could be applied in a modern digital environment.

Appendix

Table 11: Examples of tasks generated by the MOPD instruction-following teacher. Five instruction-following tasks retained after filtering are shown with their corresponding generation topics.

Figure 11: Per-benchmark DF-OPD training curves in the RL-tuned teacher setting. The six panels show accuracy on AIME24, AIME25, AMC23, MATH-500, Minerva, and Olympiad-Bench, comparing training on 1, 8, 64, and 1k teacher-generated questions with training on the full DAPO dataset. The horizontal axis indicates training updates, and the vertical axis indicates accuracy (%) on the corresponding benchmark. All runs use the same number of training updates and sampled trajectories per update.

Figure 12: Per-benchmark DF-OPD training curves in the larger teacher setting. The six panels show accuracy on AIME24, AIME25, AMC23, MATH-500, Minerva, and Olympiad-Bench, comparing training on 1, 8, 64, and 1k teacher-generated questions with training on the full DAPO dataset. The horizontal axis indicates training updates, and the vertical axis indicates accuracy (%) on the corresponding benchmark. All runs use the same number of training updates and sampled trajectories per update.

Figure 13: MOPD training curves across three domains. The three panels show average accuracy (%) in mathematics, code, and instruction following, comparing training on 8, 64, 256, and 1k generated questions with training on 7,458 real questions. The horizontal axis indicates training updates. Each domain score is the average of its two benchmarks. Generated question counts refer to the total number of questions across the three domains.

Figure 14: Per-benchmark MOPD training curves. The six panels show accuracy on AIME25, AIME26, LiveCodeBench v5, LiveCodeBench v6, IFEval, and IFBench, comparing training on 8, 64, 256, and 1k generated questions with real-data training. The horizontal axis indicates training updates, and the vertical axis indicates accuracy (%) on the corresponding benchmark. All runs use the same training and evaluation configurations. Generated question counts refer to the total number of questions across the three domains.

Figure 15: Single-teacher DF-OPD with each of the three domain experts from MOPD. The three panels correspond to mathematics, code, and instruction following. Each experiment initializes a separate student from the same SmolLM3-3B-MixSFT checkpoint and uses supervision only from the corresponding domain teacher. The curves compare training on 64 teacher-generated questions with training on real data from the same domain, with the initial student’s score shown as a reference. The horizontal axis indicates training updates, and the vertical axis indicates average accuracy (%) across the two benchmarks in the corresponding domain. All 64 questions in each experiment belong to that domain.

Setting	Training questions	With topic	Without topic	$\Delta$
RL-tuned teacher	8	54.84	56.34	$+1.50$
RL-tuned teacher	64	56.61	56.63	$+0.02$
RL-tuned teacher	1k	56.38	56.18	$-0.20$
Larger teacher	8	49.46	48.86	$-0.60$
Larger teacher	64	49.16	49.45	$+0.29$
Larger teacher	1k	49.12	48.97	$-0.15$

Appendix

Table 12: Effect of topic guidance in the generation prompts on DF-OPD performance. We compare distillation with and without topic specifications at matched numbers of training questions. The two single-teacher settings report peak macro-average accuracy across six mathematics benchmarks. MOPD reports the peak Overall score, where Overall is the mean of the three domain scores. All scores are expressed as percentages. Each comparison uses the same training and evaluation protocols. MOPD question counts refer to the total across the three domains.

\Delta

is the score without topic guidance minus the score with topic guidance, in percentage points.

Setting	Student	Domains	Sampled responses	Self-QA responses	Self-QA rate
RL-tuned teacher	DeepSeek-R1-Distill-Qwen-1.5B	Mathematics	1,024	635	62.0%
Larger teacher	Qwen3-1.7B	Mathematics	1,024	1	0.1%
MOPD	SmolLM3-3B-MixSFT	Mathematics, code, and instruction following	1,024	121	11.8%

Appendix

Table 13: Self-QA statistics for the three initial students under empty prompts. We sample 1,024 responses from each student before training and classify them using rules based on question generation, solution cues, domain content, and request or refusal markers. Self-QA denotes the co-occurrence of self-generated questions and solution attempts; it does not measure answer correctness.

Case	Response text
1	Alright, so I need to find the vertical and horizontal asymptotes for the function $f(x)=\frac{x+6}{x^{2}+6x+11}$ . […] The discriminant $D=b^{2}-4ac=6^{2}-4(1)(11)=36-44=-8$ . Since the discriminant is negative, this quadratic doesn’t factor over the real numbers. […] In this case, the numerator is degree 1 and the denominator is degree 2. So the degrees are different, with the numerator being less. Therefore, the horizontal asymptote should be $y=0$ . […] Final Answer The function has a horizontal asymptote at $\boxed{y=0}$ and no vertical asymptotes.
2	Alright, so I’ve got this problem here: “Given the function $f(x)=x^{4}-6x^{2}+5x$ , find the equation of the tangent line at the point where $x=2$ .” […] So $f(2)=16-24+10=2$ . Okay, so the point is $(2,2)$ . […] So $f^{\prime}(x)=4x^{3}-12x+5$ . […] Therefore, the slope $m=13$ . […] Final Answer The equation of the tangent line is $\boxed{y=13x-24}$ .
3	Alright, so I’m trying to find the integral of $\arctan(x)$ with respect to $x$ . Hmm, okay. I remember from my calculus class that integration by parts is useful for integrals involving trigonometric functions or functions that have an inverse. […] So, if I let $u=\arctan(x)$ , then $du$ would be… Hmm, what’s the derivative of $\arctan(x)$ ? I think it’s $1/(x^{2}+1)$ . […] Okay, so that simplifies to $x\arctan(x)$ minus the integral of $x/(x^{2}+1)\,dx$ . […] Final Answer The integral of $\arctan(x)$ with respect to $x$ is $\boxed{x\arctan(x)-\dfrac{1}{2}\ln(x^{2}+1)+C}$ .
4	Alright, so I have this equation to solve: 2 sin theta plus 6 equals zero . I’m a bit confused about where to start. […] That would give me 2 sin theta = $-6$ . Then, maybe I can divide both sides by 2 to get sin theta by itself. So, sin theta would equal $-3$ . […] Hold on, so if sin theta equals $-3$ , which isn’t possible because sine can’t be outside that range. So does that mean there’s no solution? […] Final Answer The equation has no solution, so the solution set is $\boxed{\emptyset}$ .
5	Alright, I need to figure out how many different three-digit numbers can be formed using the digits 1, 2, 3, 4, 5, 6, 7, 8, and 9 without repeating any digits . I remember something about permutations and combinations, but I’m not completely sure. […] For the first digit (hundreds place), we have 9 options (digits 1-9). Once we’ve chosen the hundreds digit, we have 8 options left for the tens place. Then, after choosing hundreds and tens, we have 7 options left for the units place. […] Final Answer The number of different three-digit numbers is $\boxed{504}$ .

Appendix

Table 14: Self-QA examples from DeepSeek-R1-Distill-Qwen-1.5B under empty prompts. The table presents five representative responses sampled from the initial student, each containing a self-generated mathematics question and a subsequent solution attempt. The inputs use the corresponding empty template in Appendix F.1 and contain no explicit question content. Omissions are marked with […]. These examples illustrate generation behavior and do not imply that the solutions have been verified for correctness. The self-generated question is shown in bold, and the answer reached by the solution attempt is shown in blue.

Case	Response text
1	Sure! Please provide the specific question or topic you’d like me to help with, and I’ll do my best to assist you.
2	I’m sorry, but I can’t assist with that request. Please provide a clear question or topic you’d like me to help with.
3	It seems like you provided a query, but there’s no content in the message. Could you please provide a specific question or topic you’d like me to help with? I’ll do my best to assist you!
4	I’m here to help! Could you please clarify what you need assistance with? For example, are you looking for help with a specific task, question, or need guidance on something?
5	Sure! Please provide the question you’d like me to answer.

Appendix

Table 15: Examples of Qwen3-1.7B requesting a question under empty prompts. The table presents five representative responses sampled from the initial student using the corresponding empty template in Appendix F.1 . These responses ask the user to provide a question, without generating and solving one within the current rollout.

On-Policy Distillation

On-policy distillation trains a model on its own outputs using teacher guidance, aligning supervision with the states it encounters at deployment.

See all 507 papers

Rethinking On-Policy Distillation of Large Language Models II: One Training Example
Sep 3, 2026Zixuan Fu, Bingxiang He, Yuxin Zuo +10Language Model Distillation On-Policy Distillation
Tsinghua University · University of Chinese Academy of Sciences · Northeastern University +2
Ask to Distill: Learning to Synthesize Data for On-Policy Distillation
Oct 20, 2026Jiashuo Sun, Hao Zhang, Xueqiang Xu +9Language Model Distillation On-Policy Distillation
University of Illinois Urbana-Champaign, Urbana, IL, USA · The Hong Kong University of Science and Technology, Hong Kong SAR, China · Texas A&M University, College Station, TX, USA
What Matters in On-Policy Distillation? A Perspective on Data Efficiency and Data Selection
Sep 4, 2026Zhinan Hou, Jiaqi Zhang, Xunliang Cai +1Language Model Distillation On-Policy Distillation
Tsinghua University · Meituan LongCat Team

Sep 3, 2026J/K move · Enter open · S save

Rethinking On-Policy Distillation of Large Language Models II: One Training Example

Zixuan Fu, Bingxiang He, Yuxin Zuo, Haohuan Huang+9

Tsinghua University · University of Chinese Academy of Sciences · Northeastern University+2

Language Model Distillation On-Policy Distillation

Open paper AskPDF arXiv

On-policy distillation (OPD) combines student-generated rollouts with dense token-level supervision from a teacher. Existing work has mainly studied its algorithmic behavior, leaving the role of training data unclear. We examine this role at the data-minimal limit by training on a single query. One-shot OPD keeps improving for hundreds of steps and recovers most of full-data OPD's gain across task domains and model families. We explain this result through the states visited during training and the rate at which the student aligns with the teacher. We measure state coverage, the fraction of the states full-data OPD visits that a query set's rollouts reach. A single query already reaches

71.5\%

, most of it within the first 100 steps. Adding semantically distinct queries raises coverage and validation accuracy together, until 16 queries reach

98.9\%

and match full-data training. Yet alignment slows at a similar pace whether OPD trains on one query or the whole dataset, and even a fixed set of states takes hundreds of steps to absorb. OPD is therefore data-overfed but algorithm-starved. Its rollouts quickly expose broad supervision, while the student absorbs that supervision increasingly slowly. The state-coverage result extends to multi-teacher OPD, where 16 semantically diverse queries per domain match full-data MOPD. As a further stress test, content-light templates and off-domain WildChat queries also approach the real-query baseline. Task content and induced state coverage can therefore come apart. We hope these findings direct future work toward the step efficiency of OPD, and prompt a re-examination of the data and the mechanisms behind its recent successes in frontier post-training.

Abstract

On-Policy Distillation

Explore similar work

Rethinking On-Policy Distillation of Large Language Models II: One Training Example

Ask to Distill: Learning to Synthesize Data for On-Policy Distillation

What Matters in On-Policy Distillation? A Perspective on Data Efficiency and Data Selection