Organizations: University of Illinois Urbana-Champaign, Urbana, IL, USA · The Hong Kong University of Science and Technology, Hong Kong SAR, China · Texas A&M University, College Station, TX, USA
On-policy distillation (OPD) can succeed with surprisingly few training questions, motivating a view of questions as carriers of distributional knowledge transfer. How can we assess whether a newly generated question is valuable for distillation, and learn to generate better questions---without access to its correct answer? We introduce Ask-to-Distill (\method), a framework that treats synthetic questions as distribution elicitors and optimizes the teacher supervision they expose along student-generated trajectories. A student-initialized synthesizer is optimized using a distributional surrogate for question value, computed from a frozen student--teacher pair without reference answers, correctness rewards, or per-candidate student updates. The reward combines teacher--student disagreement with teacher probability mass on the student's candidate support, together with length and diversity controls. The generated questions are filtered and deduplicated into a prompt-only bank for standard OPD. Across mathematical reasoning, code generation, and agentic tasks, our method improves average performance and teacher-gap recovery over unoptimized student-generated prompts and KL-filtered real questions, while matching or exceeding real-bank OPD in the evaluated settings. Signal analyses further show that optimized questions can elicit denser supervision and more total signal from shorter student responses, supporting elicited distributional supervision as an objective for synthetic distillation data. Our code, data and models are publicly available at https://github.com/GasolSun36/Ask_to_Distill.
Figures & tables
Figure 1: A2D learns to ask questions where the teacher can help. (a) Schematic. A student-initialized synthesizer is RL-trained against a frozen student–teacher pair to elicit disagreement δ that remains within the student’s top- K support (compatibility ω ). Real questions mostly elicit little disagreement or disagreement outside the student’s reach, which KL filtering cannot separate. Insets: typical student and teacher next-token distributions per region. (b) Generated questions yield 20% shorter responses and 28% more total signal than DAPO-17k-Math.
Figure 2: Overview of A2D . A student-initialized synthesizer is optimized using learning-value signals from a frozen student–teacher pair. The generated questions are filtered and deduplicated into a prompt-only synthetic bank, which is then used for on-policy distillation with a frozen teacher providing supervision on the student’s top- K candidates.
Table 1: Results across six mathematical reasoning benchmarks for three student–teacher pairs. Within each block and metric, shading is relative to Student-Generated OPD (white): blue indicates improvement and red indicates degradation. Positive and negative changes are normalised separately across OPD methods and mapped nonlinearly (exponent 1.5) to muted palettes. Bold indicates the best result among OPD methods, including ties. The baselines compare real-bank OPD, untrained student-generated prompts, KL-filtered selection, and two additional synthetic-data methods.
Table 2: Results across six code benchmarks for three student–teacher pairs. Within each block and metric, shading is relative to Student-Generated OPD (white): blue indicates improvement and red indicates degradation. Positive and negative changes are normalised separately across OPD methods and mapped nonlinearly (exponent 1.5) to muted palettes. Bold indicates the best result among OPD methods, including ties. The baselines compare real-bank OPD, untrained student-generated prompts, KL-filtered selection, and two additional code-generation methods.
Figure 3: Downstream performance during training for student- and teacher-initialized question synthesizers, with and without pair-specific RL, using Qwen3-4B (non-thinking) as the student and Qwen3-4B-Instruct-2507 as the teacher, evaluated on LCB, APPS-Intro, and APPS-Interview. Dashed lines show the student and teacher reference scores. RL improves both initializations; student initialization with RL is competitive across benchmarks and exceeds the teacher reference on LCB, while remaining below it on APPS-Intro and APPS-Interview.
Table 3: Results across the three τ2 -Bench domains. Within each metric, shading is relative to Student-Generated OPD (white): blue indicates improvement and red indicates degradation. Positive and negative changes are normalised separately across OPD methods and mapped nonlinearly (exponent 1.5) to muted palettes. Bold indicates the best result among OPD methods, including ties. The three baselines isolate the effects of synthesizer training and learning-value-based question generation.
Figure 4: Reward-component ablation on AIME24, AIME25, and AIME26. Numbers above bars indicate accuracy (%). Removing compatibility, the training-time diversity term, or the explicit length penalty reduces accuracy on all three benchmarks.
Training variant
Avg.
Gain over round 1
Recovery (%)
Student
29.0
-
0.0
Teacher
55.6
-
100.0
A2D : round 1
40.5
0.0
43.3
A2D : round 2
40.8
+0.3
44.3
A2D : final round ( R=3 )
41.0
+0.5
44.8
Table 4: Iterative training on code pair A, with student Qwen3-4B (non-thinking) and teacher Qwen3-30B-A3B-Instruct-2507. Avg. is the mean of LiveCodeBench v5_v6, APPS-Intro and APPS-Interview. Recovery =(Avg−student)/(teacher−student) . Dashes indicate entries that are not applicable.
Figure 5: Average accuracy versus cumulative training tokens for (a) R1-Distill-1.5B on JustRL and (b) R1-Distill-7B on Skywork. Gray squares show Vanilla checkpoints and the dashed line its best-so-far accuracy; blue circles show A2D checkpoints. Shaded regions indicate intervals where A2D is above (gain) or below (loss) Vanilla’s best-so-far reference. Average accuracy is computed over AIME24, AIME25, and AMC23, and therefore differs from the six-benchmark average in Table 1 .
Figure 6: Simple prompts can elicit dense distillation signals. Comparison of (a) a real question from DAPO-17k-Math and (b) a synthetic question generated by our method. Darker token backgrounds indicate higher support-aware scores st ; the strips summarize the complete rollouts, with boxes marking the displayed excerpts. Although both student answers are incorrect, the apparently simpler synthetic question elicits a higher mean st (0.172 versus 0.117; approximately 1.5× ) with a shorter rollout (1,241 versus 2,002 tokens), illustrating that surface complexity alone does not characterize the density of elicited supervision.
Figure 7: Cross-domain transfer between math and code. Gains over the initial student after source-domain OPD, without additional target-domain training. (a) Code-to-math transfer with Qwen3-4B (non-thinking) distilled from Qwen3-4B-Instruct-2507. (b) Math-to-code transfer with R1-Distill-7B distilled from Skywork-OR1-Math-7B. A2D uses optimized synthetic source-domain questions, while Vanilla OPD uses real source-domain questions. Bars report gains in percentage points; shaded groups show unweighted averages across the displayed benchmarks. Both methods improve all seven evaluated pair–benchmark combinations, and A2D consistently yields larger gains.
Figure 8: Optimized questions elicit more signal with fewer tokens. We compare student rollouts on 1,000 questions each from DAPO-17k-Math and our synthetic bank. (a) Mean response length and cumulative support-aware score ∑tst , normalized to the real-question baseline (100%); labels above bars show raw means. Synthetic questions yield 20% shorter responses and 28% more cumulative signal. (b) Mean token-level score st across relative response-position bins. Synthetic questions maintain higher scores throughout the response, indicating that the signal-density advantage extends across the trajectory.
Figure 9: Teacher–student divergence is positively associated with learning value. Left: Raw and partial Spearman correlations of mean divergence δˉ and mean score sˉ with one-step learning value over 1,500 questions. Partial correlations control for response length, pass rate, and entropy; error bars indicate 95% confidence intervals. Middle: Probe KL decrease increases across question quintiles ranked by sˉ , with the highest quintile yielding 2.15× the decrease of the lowest. Right: Final average accuracy over six benchmarks for training on five score buckets, annotated with their mean δˉ . Points show means over three seeds; error bars and shading indicate ± 1 standard deviation.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
#Problems
k
Temperature
Top- p
Top- k
Max tokens
AIME24 / AIME25 / AIME26
30 each
16
0.7
0.95
–
31,744
AMC23
40
16
0.7
0.95
–
31,744
BRUMO25
30
16
0.7
0.95
–
31,744
MATH500
500
4
0.7
0.95
–
31,744
LiveCodeBench (v5–v6)
342
4
0.7
0.8
20
8,192
APPS-Intro
1,000
1
0.2
0.95
20
4,096
Appendix
Table 5: Evaluation settings. k is the number of sampled responses per problem; “Max tokens” is the maximum generation length.
Figure 10: Accuracy gains of high- over low-divergence training groups. Each cell reports the gain in percentage points, its 95% confidence interval, and p -value.
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.
On-policy distillation (OPD) is increasingly applied to frontier foundation model post-training. Prior work in this area has largely focused on algorithmic advances, yet it remains unclear how much OPD depends on its training questions and, in particular, how far this dependence can be reduced. Across two representative single-teacher OPD settings, we find that training on 8 real prompts yields performance comparable to training on 17k problems, while datasets differing substantially in measured difficulty and initial distillation gap yield similar outcomes. Our analyses suggest two complementary explanations: repeated sampling could allow even a few prompts to expose substantial teacher supervision, while OPD transfers generalizable reasoning capabilities beyond dataset-specific knowledge. Building on these observations, we next propose a data-free on-policy distillation (DF-OPD) setting to investigate whether the system can supply the training questions itself, eliminating the need for external data. With 64 self-generated questions obtained without seed examples, DF-OPD yields performance comparable to full-data OPD in both single-teacher settings. This finding also holds in multi-teacher OPD: across mathematics, code, and instruction following, 1k generated questions achieve performance comparable to training on approximately 7k real post-training examples. We further explore whether OPD can operate even without explicit training questions. The experiments show that this is effective only in limited cases, where the student unexpectedly generates and answers its own questions, thereby reducing the process to an implicit form of DF-OPD. Together, these findings invite a reassessment of the role of training data in on-policy distillation. Code is available at https://github.com/Ryuki661/DF-OPD
Gengsheng Li, Mao Zheng, Mingyang Song +10
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Foundation Model Department, Tencent +2
On-policy distillation transfers reasoning capabilities by training a student model on its own generated trajectories using token-level feedback from a teacher. However, we identify a critical bottleneck, Supervision Fidelity Decay (SFD): as student-generated prefixes lengthen, the teacher's next-token distribution becomes less confident and less discriminative. Consequently, the teacher-dependent corrective signal in reverse-KL distillation weakens, causing student drift to compound across long reasoning chains. To mitigate SFD, we introduce Lookahead Group Reward (\ours{}). Building on the insight that next-step teacher confidence reflects the discriminative strength of future reverse-KL supervision, \ours{} evaluates the student's top-K candidate tokens by the teacher confidence they induce at the subsequent step and assigns a group-normalized reward. To maintain computational efficiency, we further design an entropy-triggered tree-attention mechanism. Across six math and code benchmarks, \ours{} improves mean@8 by 2.57 points over OPD for a 7B student, with gains increasing in longer-generation and reaching +4.92 points on AIME-26 at 39k tokens.
Yanjiang Liu, Jie Lou, Xinyan Guan +7
University of Chinese Academy of Sciences · 2Chinese Information Processing Laboratory Institute of Software, Chinese Academy of Sciences 2University of Chinese Academy of Sciences, Beijing, China · 3Xiaohongshu
On-policy distillation (OPD) consolidates expert capabilities into a single student by matching teacher and student final-layer distributions on student-generated rollouts. We investigate whether depthwise prediction transitions, the changes in predictions between layers, can provide additional task-relevant supervision. Our diagnostic study finds that stronger teacher--student agreement in intermediate transitions is associated with higher answer correctness within questions, with correct responses exhibiting lower mean transition KL across all measured intermediate layer pairs. These observations motivate On-Policy Transition Distillation (OPTD), which augments final-layer distribution matching by aligning teacher and student layer-to-layer log-probability increments on student-generated rollouts. Evaluations across three backbones and eight benchmarks spanning mathematics, coding, and instruction following show that OPTD achieves the best or tied-best overall performance in both dual- and three-domain distillation. In the three-domain setting, OPTD recovers ≈90% of the overall teacher--student capability gap, compared with ≈75% for OPD.
Runyang You, Yongqi Li, Tiezheng Yu +1
The Hong Kong Polytechnic University · Huawei Technologies Ltd.