Ask to Distill: Learning to Synthesize Data for On-Policy Distillation

Jiashuo Sun; Hao Zhang; Xueqiang Xu; Heng Wang; Yixuan Xie; Pengcheng Jiang; Zhenrui Yue; Jiajun Fan

cs.AIOct 20, 2026

Ask to Distill: Learning to Synthesize Data for On-Policy Distillation

Authors: Jiashuo Sun, Hao Zhang, Xueqiang Xu, Heng Wang, Yixuan Xie, Pengcheng Jiang, Zhenrui Yue, Jiajun Fan,

Organizations: University of Illinois Urbana-Champaign, Urbana, IL, USA · The Hong Kong University of Science and Technology, Hong Kong SAR, China · Texas A&M University, College Station, TX, USA

Abstract

On-policy distillation (OPD) can succeed with surprisingly few training questions, motivating a view of questions as carriers of distributional knowledge transfer. How can we assess whether a newly generated question is valuable for distillation, and learn to generate better questions---without access to its correct answer? We introduce Ask-to-Distill (\method), a framework that treats synthetic questions as distribution elicitors and optimizes the teacher supervision they expose along student-generated trajectories. A student-initialized synthesizer is optimized using a distributional surrogate for question value, computed from a frozen student--teacher pair without reference answers, correctness rewards, or per-candidate student updates. The reward combines teacher--student disagreement with teacher probability mass on the student's candidate support, together with length and diversity controls. The generated questions are filtered and deduplicated into a prompt-only bank for standard OPD. Across mathematical reasoning, code generation, and agentic tasks, our method improves average performance and teacher-gap recovery over unoptimized student-generated prompts and KL-filtered real questions, while matching or exceeding real-bank OPD in the evaluated settings. Signal analyses further show that optimized questions can elicit denser supervision and more total signal from shorter student responses, supporting elicited distributional supervision as an objective for synthetic distillation data. Our code, data and models are publicly available at https://github.com/GasolSun36/Ask_to_Distill.

Figures & tables

Figure 1: A2D learns to ask questions where the teacher can help. (a) Schematic. A student-initialized synthesizer is RL-trained against a frozen student–teacher pair to elicit disagreement $\delta$ that remains within the student’s top- $K$ support (compatibility $\omega$ ). Real questions mostly elicit little disagreement or disagreement outside the student’s reach, which KL filtering cannot separate. Insets: typical student and teacher next-token distributions per region. (b) Generated questions yield 20% shorter responses and 28% more total signal than DAPO-17k-Math.

Figure 2: Overview of A2D . A student-initialized synthesizer is optimized using learning-value signals from a frozen student–teacher pair. The generated questions are filtered and deduplicated into a prompt-only synthetic bank, which is then used for on-policy distillation with a frozen teacher providing supervision on the student’s top- $K$ candidates.

Method	AIME24	AIME25	AIME26	AMC23	BRUMO25	MATH500	Avg.	Recovery (%)
Qwen3-1.7B (Non-thinking) Teacher: Qwen3-4B-Non-Thinking-RL-Math
Student	10.0	8.3	8.3	37.5	17.9	72.0	25.7	0.0
Teacher †	41.2	33.1	34.4	57.5	45.8	78.3	48.4	100.0
Student-Generated OPD	27.5	22.7	21.3	61.6	35.0	78.4	41.1	67.9
KL-Filtered OPD	33.5	26.7	23.3	64.4	36.7	80.2	44.1	81.3
Vanilla OPD ( Li et al., 2026b )	31.5	23.5	20.0	71.1	31.7	80.4	43.0	76.4

Table 1: Results across six mathematical reasoning benchmarks for three student–teacher pairs. Within each block and metric, shading is relative to Student-Generated OPD (white): blue indicates improvement and red indicates degradation. Positive and negative changes are normalised separately across OPD methods and mapped nonlinearly (exponent 1.5) to muted palettes. Bold indicates the best result among OPD methods, including ties. The baselines compare real-bank OPD, untrained student-generated prompts, KL-filtered selection, and two additional synthetic-data methods.

Method	LCB	MBPP+	APPS-Intro	APPS-Interv.	APPS-Comp.	CodeContests	Avg.	Recovery (%)
Qwen3-4B (Non-thinking) Teacher: Qwen3-4B-Instruct-2507
Student	21.6	68.8	46.4	19.1	8.8	7.3	28.7	0.0
Teacher †	33.7	72.0	65.6	41.0	19.7	18.8	41.8	100.0
Student-Generated OPD	30.2	67.2	55.8	27.8	10.4	9.7	33.5	36.9
KL-Filtered OPD	31.1	67.7	58.2	30.0	11.5	11.5	35.0	48.2
Vanilla OPD ( Li et al., 2026b )	31.0	68.5	57.1	27.6	10.7	12.1	34.5	44.4

Table 2: Results across six code benchmarks for three student–teacher pairs. Within each block and metric, shading is relative to Student-Generated OPD (white): blue indicates improvement and red indicates degradation. Positive and negative changes are normalised separately across OPD methods and mapped nonlinearly (exponent 1.5) to muted palettes. Bold indicates the best result among OPD methods, including ties. The baselines compare real-bank OPD, untrained student-generated prompts, KL-filtered selection, and two additional code-generation methods.

Figure 3: Downstream performance during training for student- and teacher-initialized question synthesizers, with and without pair-specific RL, using Qwen3-4B (non-thinking) as the student and Qwen3-4B-Instruct-2507 as the teacher, evaluated on LCB, APPS-Intro, and APPS-Interview. Dashed lines show the student and teacher reference scores. RL improves both initializations; student initialization with RL is competitive across benchmarks and exceeds the teacher reference on LCB, while remaining below it on APPS-Intro and APPS-Interview.

Method	Airline	Retail	Telecom	Avg.	Recovery (%)
Qwen3-4B (Non-thinking) Teacher: Qwen3-4B-Thinking-2507 (Non-thinking)
Student	15.0	17.5	10.6	14.4	0.0
Teacher †	47.5	48.1	34.4	43.3	100.0
Student-Generated OPD	42.5	42.5	26.9	37.3	79.2
KL-Filtered OPD	45.0	48.8	28.8	40.9	91.7
Vanilla OPD	45.0	46.9	29.4	40.4	90.0

Table 3: Results across the three

\tau^{2}

-Bench domains. Within each metric, shading is relative to Student-Generated OPD (white): blue indicates improvement and red indicates degradation. Positive and negative changes are normalised separately across OPD methods and mapped nonlinearly (exponent 1.5) to muted palettes. Bold indicates the best result among OPD methods, including ties. The three baselines isolate the effects of synthesizer training and learning-value-based question generation.

Figure 4: Reward-component ablation on AIME24, AIME25, and AIME26. Numbers above bars indicate accuracy (%). Removing compatibility, the training-time diversity term, or the explicit length penalty reduces accuracy on all three benchmarks.

Training variant	Avg.	Gain over round 1	Recovery (%)
Student	29.0	-	0.0
Teacher	55.6	-	100.0
A2D : round 1	40.5	0.0	43.3
A2D : round 2	40.8	+0.3	44.3
A2D : final round ( $R=3$ )	41.0	+0.5	44.8

Table 4: Iterative training on code pair A, with student Qwen3-4B (non-thinking) and teacher Qwen3-30B-A3B-Instruct-2507. Avg. is the mean of LiveCodeBench v5_v6, APPS-Intro and APPS-Interview. Recovery

=(\text{Avg}-\text{student})/(\text{teacher}-\text{student})

. Dashes indicate entries that are not applicable.

Figure 5: Average accuracy versus cumulative training tokens for (a) R1-Distill-1.5B on JustRL and (b) R1-Distill-7B on Skywork. Gray squares show Vanilla checkpoints and the dashed line its best-so-far accuracy; blue circles show A2D checkpoints. Shaded regions indicate intervals where A2D is above (gain) or below (loss) Vanilla’s best-so-far reference. Average accuracy is computed over AIME24, AIME25, and AMC23, and therefore differs from the six-benchmark average in Table 1 .

Figure 6: Simple prompts can elicit dense distillation signals. Comparison of (a) a real question from DAPO-17k-Math and (b) a synthetic question generated by our method. Darker token backgrounds indicate higher support-aware scores $s_{t}$ ; the strips summarize the complete rollouts, with boxes marking the displayed excerpts. Although both student answers are incorrect, the apparently simpler synthetic question elicits a higher mean $s_{t}$ (0.172 versus 0.117; approximately $1.5\times$ ) with a shorter rollout (1,241 versus 2,002 tokens), illustrating that surface complexity alone does not characterize the density of elicited supervision.

Figure 7: Cross-domain transfer between math and code. Gains over the initial student after source-domain OPD, without additional target-domain training. (a) Code-to-math transfer with Qwen3-4B (non-thinking) distilled from Qwen3-4B-Instruct-2507. (b) Math-to-code transfer with R1-Distill-7B distilled from Skywork-OR1-Math-7B. A2D uses optimized synthetic source-domain questions, while Vanilla OPD uses real source-domain questions. Bars report gains in percentage points; shaded groups show unweighted averages across the displayed benchmarks. Both methods improve all seven evaluated pair–benchmark combinations, and A2D consistently yields larger gains.

Figure 8: Optimized questions elicit more signal with fewer tokens. We compare student rollouts on 1,000 questions each from DAPO-17k-Math and our synthetic bank. (a) Mean response length and cumulative support-aware score

\sum_{t}s_{t}

, normalized to the real-question baseline (100%); labels above bars show raw means. Synthetic questions yield 20% shorter responses and 28% more cumulative signal. (b) Mean token-level score

s_{t}

across relative response-position bins. Synthetic questions maintain higher scores throughout the response, indicating that the signal-density advantage extends across the trajectory.

Figure 9: Teacher–student divergence is positively associated with learning value. Left: Raw and partial Spearman correlations of mean divergence

\bar{\delta}

and mean score

\bar{s}

with one-step learning value over 1,500 questions. Partial correlations control for response length, pass rate, and entropy; error bars indicate 95% confidence intervals. Middle: Probe KL decrease increases across question quintiles ranked by

\bar{s}

, with the highest quintile yielding

2.15\times

the decrease of the lowest. Right: Final average accuracy over six benchmarks for training on five score buckets, annotated with their mean

\bar{\delta}

. Points show means over three seeds; error bars and shading indicate

\pm

1 standard deviation.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	#Problems	$k$	Temperature	Top- $p$	Top- $k$	Max tokens
AIME24 / AIME25 / AIME26	30 each	16	0.7	0.95	–	31,744
AMC23	40	16	0.7	0.95	–	31,744
BRUMO25	30	16	0.7	0.95	–	31,744
MATH500	500	4	0.7	0.95	–	31,744
LiveCodeBench (v5–v6)	342	4	0.7	0.8	20	8,192
APPS-Intro	1,000	1	0.2	0.95	20	4,096

Appendix

Table 5: Evaluation settings.

k

is the number of sampled responses per problem; “Max tokens” is the maximum generation length.

Figure 10: Accuracy gains of high- over low-divergence training groups. Each cell reports the gain in percentage points, its 95% confidence interval, and

p

-value.

Language Model Distillation

Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.

See all 508 papers

Data-Free On-Policy Distillation: How Far Can We Go Without External Data?
Sep 12, 2026Gengsheng Li, Mao Zheng, Mingyang Song +10On-Policy Distillation Knowledge Distillation
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Foundation Model Department, Tencent +2
Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation
May 29, 2026Yanjiang Liu, Jie Lou, Xinyan Guan +7CoT Distillation Language Model Distillation
University of Chinese Academy of Sciences · 2Chinese Information Processing Laboratory Institute of Software, Chinese Academy of Sciences 2University of Chinese Academy of Sciences, Beijing, China · 3Xiaohongshu
On-Policy Transition Distillation
Oct 20, 2026Runyang You, Yongqi Li, Tiezheng Yu +1Language Model Distillation On-Policy Distillation
The Hong Kong Polytechnic University · Huawei Technologies Ltd.

Sep 12, 2026J/K move · Enter open · S save

Data-Free On-Policy Distillation: How Far Can We Go Without External Data?

Gengsheng Li, Mao Zheng, Mingyang Song, Jie Sun+9

Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Foundation Model Department, Tencent+2

On-Policy Distillation Knowledge Distillation Data-Free Knowledge Distillation Synthetic Data Generation

Open paper AskPDF arXiv

On-policy distillation (OPD) is increasingly applied to frontier foundation model post-training. Prior work in this area has largely focused on algorithmic advances, yet it remains unclear how much OPD depends on its training questions and, in particular, how far this dependence can be reduced. Across two representative single-teacher OPD settings, we find that training on 8 real prompts yields performance comparable to training on 17k problems, while datasets differing substantially in measured difficulty and initial distillation gap yield similar outcomes. Our analyses suggest two complementary explanations: repeated sampling could allow even a few prompts to expose substantial teacher supervision, while OPD transfers generalizable reasoning capabilities beyond dataset-specific knowledge. Building on these observations, we next propose a data-free on-policy distillation (DF-OPD) setting to investigate whether the system can supply the training questions itself, eliminating the need for external data. With 64 self-generated questions obtained without seed examples, DF-OPD yields performance comparable to full-data OPD in both single-teacher settings. This finding also holds in multi-teacher OPD: across mathematics, code, and instruction following, 1k generated questions achieve performance comparable to training on approximately 7k real post-training examples. We further explore whether OPD can operate even without explicit training questions. The experiments show that this is effective only in limited cases, where the student unexpectedly generates and answers its own questions, thereby reducing the process to an implicit form of DF-OPD. Together, these findings invite a reassessment of the role of training data in on-policy distillation. Code is available at https://github.com/Ryuki661/DF-OPD

Abstract

Language Model Distillation

Explore similar work

Data-Free On-Policy Distillation: How Far Can We Go Without External Data?

Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation

On-Policy Transition Distillation