Organizations: Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Foundation Model Department, Tencent · National University of Singapore · Wuhan AI Research
On-policy distillation (OPD) is increasingly applied to frontier foundation model post-training. Prior work in this area has largely focused on algorithmic advances, yet it remains unclear how much OPD depends on its training questions and, in particular, how far this dependence can be reduced. Across two representative single-teacher OPD settings, we find that training on 8 real prompts yields performance comparable to training on 17k problems, while datasets differing substantially in measured difficulty and initial distillation gap yield similar outcomes. Our analyses suggest two complementary explanations: repeated sampling could allow even a few prompts to expose substantial teacher supervision, while OPD transfers generalizable reasoning capabilities beyond dataset-specific knowledge. Building on these observations, we next propose a data-free on-policy distillation (DF-OPD) setting to investigate whether the system can supply the training questions itself, eliminating the need for external data. With 64 self-generated questions obtained without seed examples, DF-OPD yields performance comparable to full-data OPD in both single-teacher settings. This finding also holds in multi-teacher OPD: across mathematics, code, and instruction following, 1k generated questions achieve performance comparable to training on approximately 7k real post-training examples. We further explore whether OPD can operate even without explicit training questions. The experiments show that this is effective only in limited cases, where the student unexpectedly generates and answers its own questions, thereby reducing the process to an implicit form of DF-OPD. Together, these findings invite a reassessment of the role of training data in on-policy distillation. Code is available at https://github.com/Ryuki661/DF-OPD
Figures & tables
Figure 1: Progressively reducing OPD’s dependence on training questions. (a) With external questions, OPD shows limited sensitivity to their quantity, difficulty, and initial distillation gap. (b) DF-OPD replaces them with self-generated questions, produced by the teacher without seed examples, and performs comparably to full-data OPD. (c) Empty prompts are effective only in limited cases, where the student spontaneously poses and answers its own question (Self-QA), reducing the process to implicit DF-OPD.
Figure 2: Few real questions support effective OPD. (a,b) DAPO training curves under matched update and trajectory budgets. Macro accuracy (%) is averaged over six mathematics benchmarks, evaluated every 15 updates. (c) Peak scores; bars start at the initial student, and numbers give the difference from full DAPO in percentage points.
Figure 3: Initial diagnostics and learning curves. (a) Initial correctness (left axis) and initial distillation gap (right axis) in the RL-tuned teacher setting. (b,c) Six-benchmark macro accuracy for the RL-tuned and larger teacher settings with 1,000 questions per source.
Figure 4: Signal coverage and cross-domain transfer. (a) Signal coverage grows with repeated sampling. Mean ± std over six resampling draws of the frozen initial student; brackets mark gains between adjacent pool sizes at 4,096 trajectories. (b,c) Transfer from programming to mathematics for the RL-tuned and larger teacher settings with ∣D∣=1,000 questions. Macro accuracy on the same six mathematics benchmarks; dashed lines mark initial students.
Figure 5: DF-OPD workflow. A frozen teacher generates a fixed question pool from a simple instruction. The student samples responses, and the same teacher provides token-level feedback on student-generated prefixes. Standard OPD updates only the student. The instruction shown is the RL-tuned teacher’s generation prompt.
Figure 6: Self-generated questions replace the external corpus. (a,b) Single-teacher OPD with different numbers of generated questions, compared with training on the full real DAPO corpus. (c) Peak scores; bars start at the initial student, and numbers give the difference from full real DAPO in percentage points. In both settings, 64 questions reach comparable performance. Complete results appear in Appendix E.3 .
Input / reference
Questions
Math
Code
IF
Overall
Recovery
Initial student
—
15.83
15.77
41.01
24.21
—
Routed teachers
—
23.02
22.52
48.73
31.42
100%
Real post-training data
7,458
23.02
21.71
48.80
31.18
96.6%
Self-generated
8
23.44
19.64
45.92
29.66
75.6%
Self-generated
64
22.60
20.87
47.51
30.33
84.8%
Self-generated
256
23.54
20.99
47.86
30.80
91.3%
Table 1: Self-generated questions in multi-teacher distillation. Each domain score averages its two benchmarks, and Overall is the macro average of the three domain scores. Recovery is relative to the routed teachers (Equation 2 ). All trained rows use the peak-Overall checkpoint. Full results appear in Appendix E.4 .
Figure 7: Omitting questions has a setting-dependent effect. (a) Peak scores: six-benchmark mathematics averages (single-teacher; DF-OPD with 64 questions) or three-domain averages (MOPD; DF-OPD with 1k). (b) Self-QA rate of each initial student (1,024 empty-prompt rollouts, setting-specific domain criteria; Appendix F.2 ) and the full-data gain retained by empty-prompt OPD.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Student
Teacher
Relationship
RL-tuned teacher
DeepSeek-R1-Distill-Qwen-1.5B
JustRL-DeepSeek-1.5B
RL-tuned from the student
Larger teacher
Qwen3-1.7B
Qwen3-30B-A3B-Instruct-2507
Different-scale models from the Qwen3 family
MOPD: math
SmolLM3-3B-MixSFT
SmolLM3-3B-RL-Math
Math-specific RL from MixSFT
MOPD: code
SmolLM3-3B-MixSFT
SmolLM3-3B-RL-Code
Code-specific RL from MixSFT
MOPD: instruction following
SmolLM3-3B-MixSFT
SmolLM3-3B-RL-IF
Instruction-following RL from MixSFT
Appendix
Table 2: Teacher and student configurations. Model names refer to publicly released checkpoints. The three MOPD domains share the same student initialization and use the checkpoint collection released by Open-MOPD.
Configuration
Single teacher
Multiple teachers
Questions per update
64
64
Student support size k
16
128
Policy passes per rollout batch
1
1
Optimizer
AdamW
AdamW
Learning rate
10−6
10−6
AdamW β1,β2
0.9, 0.999
0.9, 0.999
Appendix
Table 3: Training configurations for the single-teacher and multi-teacher settings. The student support size k is used to compute the distillation objective and is distinct from decoding top- k during generation; decoding top- k is disabled in both settings.
Configuration
Single-teacher math
MOPD math
MOPD code
MOPD IF
Responses per problem
4
16
5
1
Sampling temperature
0.7
0.6
0.6
0.6
Top- p
0.95
0.95
0.95
0.95
Response limit (tokens)
16,384
32,768
32,768
32,768
Appendix
Table 4: Evaluation configurations for the single-teacher and multi-teacher settings. Responses per problem denotes the number of independently sampled responses for each evaluation problem.
Figure 8: Per-benchmark OPD training curves in the RL-tuned teacher setting. The six panels show accuracy on AIME24, AIME25, AMC23, MATH-500, Minerva, and Olympiad-Bench, comparing training on 1, 8, 64, 256, and 1k DAPO questions with training on the full DAPO dataset. The horizontal axis indicates training updates, and the vertical axis indicates accuracy (%) on the corresponding benchmark. All runs use the same number of updates and sampled trajectories per update.
Figure 9: Macro-average OPD training curves with different numbers of DeepMath questions. Training curves are shown for the RL-tuned and larger teacher settings, comparing 1, 64, and 1k training questions. The horizontal axis indicates training updates, and the vertical axis indicates macro-average accuracy (%) over the six mathematics benchmarks. Within each teacher setting, all runs use the same training and evaluation configurations apart from the training question pool.
Figure 10: Macro-average OPD training curves with different numbers of ORZ questions. Training curves are shown for the RL-tuned and larger teacher settings, comparing 1, 64, and 1k training questions. The horizontal axis indicates training updates, and the vertical axis indicates macro-average accuracy (%) over the six mathematics benchmarks. Within each teacher setting, all runs use the same training and evaluation configurations apart from the training question pool.
Domain
Topics
Mathematics
algebra; geometry; number theory; combinatorics; probability; statistics; calculus; precalculus; trigonometry; logic puzzles; sequences and series; inequalities; functional equations; complex numbers; polynomials; graph theory
Code
dynamic programming; greedy algorithms; graph traversal; shortest paths; string processing; binary search; sorting and ordering; data structures; number theory; combinatorics and counting; simulation; two pointers; prefix sums; recursion and backtracking; intervals and scheduling; matrix and grid problems
Instruction following
response length in words; number of sentences; number of paragraphs; required keywords; forbidden words; starting and ending phrases; letter casing; JSON or structured output; bullet-point formatting; title or heading requirements; placeholder tokens; repeating the prompt; language of the response; punctuation restrictions; numbered list requirements; section markers
Appendix
Table 5: Topics for question generation, with 16 topics per domain. The mathematics topic list is shared by the two single-teacher settings and the MOPD mathematics teacher. The code and instruction-following lists are used by their respective domain teachers. Each generation prompt replaces {topic} with a randomly selected entry from the corresponding list.
Teacher
Generation requests
Generation seeds
Final questions
RL-tuned teacher
1,024
42
3,822
Larger teacher
1,024
42
3,930
MOPD mathematics teacher
1,024
42
2,971
MOPD code teacher
1,024
42
2,949
MOPD instruction-following teacher
1,024
42
3,082
Appendix
Table 6: Teacher question-generation settings and question-pool sizes after processing. Each generation request samples four outputs. Generation seeds are the random seeds used by the teachers to generate questions. The final question counts are measured after extraction, format checks, deduplication, and length filtering, before subsampling to the pool sizes used in the experiments.
Topic
Question text
number theory
Find all prime numbers p such that p2 divides 2p−2 .
complex numbers
Let z be a complex number satisfying z2+(1+i)z+(2−2i)=0 . Find ∣z1∣×∣z2∣ .
combinatorics
In how many ways can 8 people be seated around a circular table such that Alice and Bob cannot sit next to each other?
geometry
In a right-angled triangle ABC with right angle at A , the incircle touches AB at D and AC at E . The tangent from C to the incircle meets AB at F . If AB=3 , AC=4 , find the length of CF .
calculus
A 10-foot ladder is leaning against a vertical wall. The base of the ladder is being pushed away from the wall at a rate of 2 feet per second. When the base is 6 feet away from the wall, how fast is the top of the ladder sliding downward?
Appendix
Table 7: Examples of mathematics questions generated by the RL-tuned teacher. Five questions retained after filtering are shown with their corresponding generation topics.
Topic
Question text
geometry
A regular dodecagon (12-sided polygon) is inscribed in a circle of radius 1. Let P be a point on the circumference such that the sum of the squares of the distances from P to each of the 12 vertices of the dodecagon is minimized. What is the minimum value of this sum?
trigonometry
In triangle ABC , let ∠A=60∘ , and let D be a point on side BC such that AD bisects ∠A . Suppose BD=2 and DC=3 . If AD=x , find the exact value of x2 using trigonometric identities and the Angle Bisector Theorem.
combinatorics
A group of 10 mathematicians is attending a conference. Each day, they form a circle to discuss a problem, and no two people can sit next to each other more than once during the conference. What is the maximum number of days the conference can last such that this condition is satisfied?
graph theory
Let G be a simple, undirected graph with 12 vertices and 30 edges. Suppose that G has no cycles of length 3 or 4 (i.e., girth at least 5). What is the maximum possible number of vertices of degree 5 or more in G ?
precalculus
Let f(x)=log2(x2−4x+5) and g(x)=x−1 . Find all real values of x such that f(g(x))=2 , and determine the sum of all such values.
Appendix
Table 8: Examples of mathematics questions generated by the larger teacher. Five questions retained after filtering are shown with their corresponding generation topics.
Topic
Question text
number theory
What is the least positive integer n for which the equation (n+2)!+n=(n+1)!+(n+1) holds true?
functional equations
Find the function f:R→R such that for all real numbers x,y,z satisfying x+y+z=0 , we have f(x3)+f(y3)+f(z3)=xy(x+y)+xz(x+z)+yz(y+z) .
calculus
A function f:R→R satisfies the differential equation 3f′′(x)+2f′(x)−8f(x)=0 with initial conditions f(0)=1 and f′(0)=0 . Let g(x) be the inverse function of f(x) . Find the value of g′(−2) .
sequences and series
What is the smallest positive integer n such that the sum of the first n consecutive positive integers is divisible by 17?
probability
What is the probability that a randomly selected positive integer less than or equal to 1000 has a remainder of 3 when divided by 11?
Appendix
Table 9: Examples of questions generated by the MOPD mathematics teacher. Five mathematics questions retained after filtering are shown with their corresponding generation topics.
Topic
Question text
sorting and ordering
Problem Statement: Lexicographical Ordering of Strings Input Format: - The first line contains a single integer N ( 1≤N≤105 ), denoting the number of strings to sort. - The following N lines each contain a string consisting of lowercase English letters only. No string will be empty and no character will appear more than once in a string. Output Format: - Print N space-separated strings in lexicographical order. Each string must be printed exactly as it appears in the input, and you may not modify the order of strings within the input as long as it matches the sorted order. Constraints: - Time Limit: 5 seconds - Memory Limit: 256 MB - Judge Logos: Standard
combinatorics and counting
Problem Statement Alice, Bob, and Carol are playing a game using a set of marbles. They want to divide the marbles into three non-empty groups such that the product of the number of marbles in each group is maximized. The constraint is that the number of marbles in each group must be a positive integer, and no two groups can have the same number of marbles. Determine the number of distinct ways to partition the marbles into three such groups. Input Format The first line contains a single integer T , the number of test cases. Each subsequent line contains a single integer N , representing the total number of marbles. Output Format For each test case, output a single integer representing the number of distinct ways to partition the marbles into three non-empty groups with distinct sizes. Constraints - 1≤T≤10 - 3≤N≤1000 - The total number of marbles across all test cases does not exceed 1000.
graph traversal
Problem Statement: Longest Path in a Tree Input Format: - The first line contains two integers, N and M , representing the number of nodes and edges in the tree, respectively. - The subsequent M lines describe the edges of the tree. Each line contains two integers, u and v , indicating an undirected edge between nodes u and v . Output Format: - Print a single integer, which is the length of the longest path (also known as the diameter) in the tree. The path should consist of exactly two distinct nodes, and the longest path must not include any cycles. Constraints: - 2≤N≤105 - 1≤M≤2∗105 - Nodes are numbered from 1 to N .
data structures
Problem Statement: Weighted Path Reconstruction Input Format: - The first line contains two integers, N and M , representing the number of nodes and edges in the directed weighted graph, respectively. - The next M lines describe the edges. Each line consists of three integers: u , v , and w , indicating a directed edge from node u to node v with weight w . Output Format: - Print N integers, where the ith integer represents the minimum possible sum of weights of a path from node 1 to node i . Constraints: - 2≤N≤1000 - 1≤M≤5000 - 1≤u,v≤N - 1≤w≤1000 - The graph can contain cycles and self-loops.
greedy algorithms
Problem Statement: Maximum Weight Connected Components Input Format: - The first line contains two integers, N and M , representing the number of nodes and edges in the graph, respectively. - The next M lines each contain three integers, u , v , and w , denoting an undirected edge between nodes u and v with a weight w . Output Format: - A single integer, representing the maximum sum of weights of the connected components of the graph. If there are multiple components with the maximum sum, output the maximum sum. Constraints: - 2≤N≤105 - 1≤M≤105 - 1≤u,v≤N - 1≤w≤109 - All edges are distinct pairs ( u,v ). - There are no self-loops ( u == v ). - The graph is undirected, meaning if there is an edge between u and v , there is also an edge between v and u .
Appendix
Table 10: Examples of questions generated by the MOPD code teacher. Five programming problems retained after filtering are shown with their corresponding generation topics.
Topic
Question text
response length in words
Please follow these instructions in your response: - 150-200 words long. - Include exactly three paragraphs. - End with the word “Answer”. Write a brief description of a cozy reading nook you would love to have in your home. I need help imagining a perfect reading spot.
number of paragraphs
Please follow these instructions in your response: - The answer must consist of exactly three paragraphs. - Each paragraph must begin with the word “Step ”. - The response must not contain any numbers other than those that are inherently part of the English language. Write a short story that explores a hidden library where forgotten memories are stored. Please provide the story.
starting and ending phrases
Please follow these instructions in your response: - 1. The response must begin with the phrase “Begin with the stars.” - 2. The response must end with the phrase “and the moon.” - 3. The response must include the exact word “quill” at least once. - 4. The response must be exactly 50 words in length. Request: Write a short story that blends astronomy with a creative writing tool.
placeholder tokens
Please follow these instructions in your response: - Use exactly three placeholder tokens: [[TEMP]] , [[LOC]] , and [[DATE]] . - Replace each placeholder token with a suitable word or phrase that fits the context. - Ensure the total length of the response is between 20 and 40 words. - Do not include any additional placeholder tokens. Write a short paragraph using the placeholders above, describing a future travel adventure.
title or heading requirements
Please follow these instructions in your response: - 1. Write a title or heading for the response that matches the given keyword “Cosmic”. - 2. Include exactly three paragraphs in the answer. - 3. Use bold and italics formatting exactly twice each, within the same paragraph. - 4. Keep the response under 60 words. Describe, in your own words, how the term “Cosmic” might relate to a vast, interconnected network of ideas and data. Provide a clear, concise example of how this concept could be applied in a modern digital environment.
Appendix
Table 11: Examples of tasks generated by the MOPD instruction-following teacher. Five instruction-following tasks retained after filtering are shown with their corresponding generation topics.
Figure 11: Per-benchmark DF-OPD training curves in the RL-tuned teacher setting. The six panels show accuracy on AIME24, AIME25, AMC23, MATH-500, Minerva, and Olympiad-Bench, comparing training on 1, 8, 64, and 1k teacher-generated questions with training on the full DAPO dataset. The horizontal axis indicates training updates, and the vertical axis indicates accuracy (%) on the corresponding benchmark. All runs use the same number of training updates and sampled trajectories per update.
Figure 12: Per-benchmark DF-OPD training curves in the larger teacher setting. The six panels show accuracy on AIME24, AIME25, AMC23, MATH-500, Minerva, and Olympiad-Bench, comparing training on 1, 8, 64, and 1k teacher-generated questions with training on the full DAPO dataset. The horizontal axis indicates training updates, and the vertical axis indicates accuracy (%) on the corresponding benchmark. All runs use the same number of training updates and sampled trajectories per update.
Figure 13: MOPD training curves across three domains. The three panels show average accuracy (%) in mathematics, code, and instruction following, comparing training on 8, 64, 256, and 1k generated questions with training on 7,458 real questions. The horizontal axis indicates training updates. Each domain score is the average of its two benchmarks. Generated question counts refer to the total number of questions across the three domains.
Figure 14: Per-benchmark MOPD training curves. The six panels show accuracy on AIME25, AIME26, LiveCodeBench v5, LiveCodeBench v6, IFEval, and IFBench, comparing training on 8, 64, 256, and 1k generated questions with real-data training. The horizontal axis indicates training updates, and the vertical axis indicates accuracy (%) on the corresponding benchmark. All runs use the same training and evaluation configurations. Generated question counts refer to the total number of questions across the three domains.
Figure 15: Single-teacher DF-OPD with each of the three domain experts from MOPD. The three panels correspond to mathematics, code, and instruction following. Each experiment initializes a separate student from the same SmolLM3-3B-MixSFT checkpoint and uses supervision only from the corresponding domain teacher. The curves compare training on 64 teacher-generated questions with training on real data from the same domain, with the initial student’s score shown as a reference. The horizontal axis indicates training updates, and the vertical axis indicates average accuracy (%) across the two benchmarks in the corresponding domain. All 64 questions in each experiment belong to that domain.
Setting
Training questions
With topic
Without topic
Δ
RL-tuned teacher
8
54.84
56.34
+1.50
RL-tuned teacher
64
56.61
56.63
+0.02
RL-tuned teacher
1k
56.38
56.18
−0.20
Larger teacher
8
49.46
48.86
−0.60
Larger teacher
64
49.16
49.45
+0.29
Larger teacher
1k
49.12
48.97
−0.15
Appendix
Table 12: Effect of topic guidance in the generation prompts on DF-OPD performance. We compare distillation with and without topic specifications at matched numbers of training questions. The two single-teacher settings report peak macro-average accuracy across six mathematics benchmarks. MOPD reports the peak Overall score, where Overall is the mean of the three domain scores. All scores are expressed as percentages. Each comparison uses the same training and evaluation protocols. MOPD question counts refer to the total across the three domains. Δ is the score without topic guidance minus the score with topic guidance, in percentage points.
Setting
Student
Domains
Sampled responses
Self-QA responses
Self-QA rate
RL-tuned teacher
DeepSeek-R1-Distill-Qwen-1.5B
Mathematics
1,024
635
62.0%
Larger teacher
Qwen3-1.7B
Mathematics
1,024
1
0.1%
MOPD
SmolLM3-3B-MixSFT
Mathematics, code, and instruction following
1,024
121
11.8%
Appendix
Table 13: Self-QA statistics for the three initial students under empty prompts. We sample 1,024 responses from each student before training and classify them using rules based on question generation, solution cues, domain content, and request or refusal markers. Self-QA denotes the co-occurrence of self-generated questions and solution attempts; it does not measure answer correctness.
Case
Response text
1
Alright, so I need to find the vertical and horizontal asymptotes for the function f(x)=x2+6x+11x+6 . […] The discriminant D=b2−4ac=62−4(1)(11)=36−44=−8 . Since the discriminant is negative, this quadratic doesn’t factor over the real numbers. […] In this case, the numerator is degree 1 and the denominator is degree 2. So the degrees are different, with the numerator being less. Therefore, the horizontal asymptote should be y=0 . […] Final Answer The function has a horizontal asymptote at y=0 and no vertical asymptotes.
2
Alright, so I’ve got this problem here: “Given the function f(x)=x4−6x2+5x , find the equation of the tangent line at the point where x=2 .” […] So f(2)=16−24+10=2 . Okay, so the point is (2,2) . […] So f′(x)=4x3−12x+5 . […] Therefore, the slope m=13 . […] Final Answer The equation of the tangent line is y=13x−24 .
3
Alright, so I’m trying to find the integral of arctan(x) with respect to x . Hmm, okay. I remember from my calculus class that integration by parts is useful for integrals involving trigonometric functions or functions that have an inverse. […] So, if I let u=arctan(x) , then du would be… Hmm, what’s the derivative of arctan(x) ? I think it’s 1/(x2+1) . […] Okay, so that simplifies to xarctan(x) minus the integral of x/(x2+1)dx . […] Final Answer The integral of arctan(x) with respect to x is xarctan(x)−21ln(x2+1)+C .
4
Alright, so I have this equation to solve: 2 sin theta plus 6 equals zero . I’m a bit confused about where to start. […] That would give me 2 sin theta = −6 . Then, maybe I can divide both sides by 2 to get sin theta by itself. So, sin theta would equal −3 . […] Hold on, so if sin theta equals −3 , which isn’t possible because sine can’t be outside that range. So does that mean there’s no solution? […] Final Answer The equation has no solution, so the solution set is ∅ .
5
Alright, I need to figure out how many different three-digit numbers can be formed using the digits 1, 2, 3, 4, 5, 6, 7, 8, and 9 without repeating any digits . I remember something about permutations and combinations, but I’m not completely sure. […] For the first digit (hundreds place), we have 9 options (digits 1-9). Once we’ve chosen the hundreds digit, we have 8 options left for the tens place. Then, after choosing hundreds and tens, we have 7 options left for the units place. […] Final Answer The number of different three-digit numbers is 504 .
Appendix
Table 14: Self-QA examples from DeepSeek-R1-Distill-Qwen-1.5B under empty prompts. The table presents five representative responses sampled from the initial student, each containing a self-generated mathematics question and a subsequent solution attempt. The inputs use the corresponding empty template in Appendix F.1 and contain no explicit question content. Omissions are marked with […]. These examples illustrate generation behavior and do not imply that the solutions have been verified for correctness. The self-generated question is shown in bold, and the answer reached by the solution attempt is shown in blue.
Case
Response text
1
Sure! Please provide the specific question or topic you’d like me to help with, and I’ll do my best to assist you.
2
I’m sorry, but I can’t assist with that request. Please provide a clear question or topic you’d like me to help with.
3
It seems like you provided a query, but there’s no content in the message. Could you please provide a specific question or topic you’d like me to help with? I’ll do my best to assist you!
4
I’m here to help! Could you please clarify what you need assistance with? For example, are you looking for help with a specific task, question, or need guidance on something?
5
Sure! Please provide the question you’d like me to answer.
Appendix
Table 15: Examples of Qwen3-1.7B requesting a question under empty prompts. The table presents five representative responses sampled from the initial student using the corresponding empty template in Appendix F.1 . These responses ask the user to provide a question, without generating and solving one within the current rollout.
On-policy distillation (OPD) combines student-generated rollouts with dense token-level supervision from a teacher. Existing work has mainly studied its algorithmic behavior, leaving the role of training data unclear. We examine this role at the data-minimal limit by training on a single query. One-shot OPD keeps improving for hundreds of steps and recovers most of full-data OPD's gain across task domains and model families. We explain this result through the states visited during training and the rate at which the student aligns with the teacher. We measure state coverage, the fraction of the states full-data OPD visits that a query set's rollouts reach. A single query already reaches 71.5%, most of it within the first 100 steps. Adding semantically distinct queries raises coverage and validation accuracy together, until 16 queries reach 98.9% and match full-data training. Yet alignment slows at a similar pace whether OPD trains on one query or the whole dataset, and even a fixed set of states takes hundreds of steps to absorb. OPD is therefore data-overfed but algorithm-starved. Its rollouts quickly expose broad supervision, while the student absorbs that supervision increasingly slowly. The state-coverage result extends to multi-teacher OPD, where 16 semantically diverse queries per domain match full-data MOPD. As a further stress test, content-light templates and off-domain WildChat queries also approach the real-query baseline. Task content and induced state coverage can therefore come apart. We hope these findings direct future work toward the step efficiency of OPD, and prompt a re-examination of the data and the mechanisms behind its recent successes in frontier post-training.
Zixuan Fu, Bingxiang He, Yuxin Zuo +10
Tsinghua University · University of Chinese Academy of Sciences · Northeastern University +2
On-policy distillation (OPD) can succeed with surprisingly few training questions, motivating a view of questions as carriers of distributional knowledge transfer. How can we assess whether a newly generated question is valuable for distillation, and learn to generate better questions---without access to its correct answer? We introduce Ask-to-Distill (\method), a framework that treats synthetic questions as distribution elicitors and optimizes the teacher supervision they expose along student-generated trajectories. A student-initialized synthesizer is optimized using a distributional surrogate for question value, computed from a frozen student--teacher pair without reference answers, correctness rewards, or per-candidate student updates. The reward combines teacher--student disagreement with teacher probability mass on the student's candidate support, together with length and diversity controls. The generated questions are filtered and deduplicated into a prompt-only bank for standard OPD. Across mathematical reasoning, code generation, and agentic tasks, our method improves average performance and teacher-gap recovery over unoptimized student-generated prompts and KL-filtered real questions, while matching or exceeding real-bank OPD in the evaluated settings. Signal analyses further show that optimized questions can elicit denser supervision and more total signal from shorter student responses, supporting elicited distributional supervision as an objective for synthetic distillation data. Our code, data and models are publicly available at https://github.com/GasolSun36/Ask_to_Distill.
Jiashuo Sun, Hao Zhang, Xueqiang Xu +9
University of Illinois Urbana-Champaign, Urbana, IL, USA · The Hong Kong University of Science and Technology, Hong Kong SAR, China · Texas A&M University, College Station, TX, USA
On-Policy Distillation (OPD) has emerged as a widely adopted post-training paradigm for enhancing large language models in reasoning domains. However, the data-centric mechanisms in OPD remain relatively underexplored. This paper presents an empirical study of data efficiency and data selection in OPD. We begin by investigating an extreme setting: training OPD on only one example, namely 1-shot OPD. Surprisingly, we find that 1-shot OPD is consistently effective across all sampled training examples and harder examples often yield superior performance gain. We next investigate what actually drives the student model's improvement in the training data. Our analysis reveals that the improvement is not driven by high token entropy, but the longer CoT paths which hard problem naturally generate. Training on longer CoT can help maintain closer alignment with the teacher over a long reasoning horizon, and learn critical thinking patterns usually missing in short CoTs, such as reflection (e.g., "Alternatively"). Based on these insights, we propose a simple data selection method that selects only hard examples for training, where even "unsolvable" examples that completely exceed the teacher's capability can be successfully used. Our experiments conducted on five models ranging from 1.5B to 8B show that training the student model on only 8 selected hard examples matches the performance of the 17K dataset baseline. Code and models will be publicly available.