PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competence

Yuanda Xu; Hejian Sang; Zhengze Zhou; Ran He; Zhipeng Wang

cs.AIMar 11, 2026

PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competence

Authors: Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang

Organizations: LinkedIn Corporation

Abstract

Standard LLM distillation treats all training problems equally -- wasting compute on problems the student has already mastered or cannot yet solve. We empirically show that this inefficiency has a precise gradient-level signature: the cross-problem gradient signal-to-noise ratio (SNR) follows a bell curve over student pass rate, collapsing at both extremes. We propose PACED, which weights each problem by $w(p) = p(1{-}p)$ where $p$ is the student's empirical pass rate -- concentrating training on the zone of proximal development. This requires only student rollouts, no architectural changes, and no hyperparameters. We prove the Beta kernel $w(p) = p^α(1{-}p)^β$ is the leading-order optimal weight family arising from the SNR boundary-collapse structure, and is minimax-robust under misspecification (worst-case efficiency loss $O(δ^2)$ ). Across Qwen3, Qwen2.5, and Llama-3 families, PACED sets a new state of the art in our experimental setting on MATH-500, AIME~~2024, and AIME~~2025, improving over unweighted distillation by up to $\mathbf{+8.2}$ and over the strong AKL baseline by up to $\mathbf{+3.6}$ , while reducing forgetting to $\mathbf{1.4\%}$ and $\mathbf{0.6\%}$ in distillation and self-distillation. A two-stage forward-then-reverse KL schedule pushes gains further to $\mathbf{+5.8}$ over standard forward KL on the hardest benchmark.

Figures & tables

Figure 1: Overview of Paced . Left: The pipeline—an expert provides reference solutions, and the student learns via a distillation loss weighted by pass rate. Right: The Beta-kernel weighting

w(p)=p^{\alpha}(1-p)^{\beta}

concentrates training on the zone of proximal development, suppressing trivial and intractable problems.

Feature	Self-Dist.	AdaRFT	AdaKD	AKL	Paced
Adaptive weighting / curriculum		✓	✓	✓	✓
Student-side competence signal		✓			✓
Implicit forgetting reduction	✓				✓
Loss-agnostic			✓		✓
Theoretically grounded				✓	✓

Table 1: Method-feature comparison. ✓ = primary design characteristic.

Method	MATH-500 ( $\uparrow$ )	AIME 2024 ( $\uparrow$ )	AIME 2025 ( $\uparrow$ )
Base	69.4 $\scriptstyle\,\pm\,0.4$ %	11.5 $\scriptstyle\,\pm\,0.9$ %	7.6 $\scriptstyle\,\pm\,0.7$ %
Forward KL (unweighted)	76.8 $\scriptstyle\,\pm\,0.3$ %	21.2 $\scriptstyle\,\pm\,1.3$ %	17.0 $\scriptstyle\,\pm\,0.9$ %
Hard Filter Forward KL	78.5 $\scriptstyle\,\pm\,0.6$ %	23.7 $\scriptstyle\,\pm\,0.9$ %	18.8 $\scriptstyle\,\pm\,0.6$ %
AKL	77.6 $\scriptstyle\,\pm\,0.4$ %	23.9 $\scriptstyle\,\pm\,1.2$ %	19.1 $\scriptstyle\,\pm\,0.8$ %
Paced Forward KL	79.4 $\scriptstyle\,\pm\,0.5$ %	25.1 $\scriptstyle\,\pm\,1.0$ %	20.6 $\scriptstyle\,\pm\,0.7$ %

Table 2: Distillation track (Qwen3-8B GRPO

\rightarrow

Qwen3-1.7B, forward KL family): reasoning performance (8-sample mean accuracy).

\uparrow

= higher is better.

Method	MATH-500 ( $\uparrow$ )	AIME 2024 ( $\uparrow$ )	AIME 2025 ( $\uparrow$ )
Base	83.9 $\scriptstyle\,\pm\,0.6$ %	19.6 $\scriptstyle\,\pm\,1.0$ %	11.5 $\scriptstyle\,\pm\,0.7$ %
Reverse KL (unweighted)	88.9 $\scriptstyle\,\pm\,0.5$ %	25.3 $\scriptstyle\,\pm\,1.2$ %	16.9 $\scriptstyle\,\pm\,1.1$ %
Hard Filter Reverse KL	92.0 $\scriptstyle\,\pm\,0.5$ %	28.9 $\scriptstyle\,\pm\,1.3$ %	22.0 $\scriptstyle\,\pm\,0.9$ %
AKL	91.4 $\scriptstyle\,\pm\,0.5$ %	28.2 $\scriptstyle\,\pm\,0.8$ %	21.5 $\scriptstyle\,\pm\,0.6$ %
Paced Reverse KL	93.7 $\scriptstyle\,\pm\,0.6$ %	31.6 $\scriptstyle\,\pm\,1.1$ %	25.1 $\scriptstyle\,\pm\,0.7$ %

Table 3: Self-distillation track (Qwen2.5-Math-7B-Instruct, reverse KL family): reasoning performance (8-sample mean accuracy).

Method	MMLU ( $\uparrow$ )	Forgetting ( $\downarrow$ )	Weighting
Base	51.2 $\scriptstyle\,\pm\,0.2$ %	–	–
Forward KL (unweighted)	48.3 $\scriptstyle\,\pm\,0.3$ %	2.9%	None
Hard Filter Forward KL	49.8 $\scriptstyle\,\pm\,0.5$ %	1.4%	Hard
AKL	49.5 $\scriptstyle\,\pm\,0.4$ %	1.7%	Token-level
Paced Forward KL	49.8 $\scriptstyle\,\pm\,0.4$ %	1.4%	Beta

Table 4: Retention in distillation track (Qwen3 forward KL family): MMLU and forgetting (

\Delta

from base).

Method	MMLU ( $\uparrow$ )	Forgetting ( $\downarrow$ )	Weighting
Base	70.6 $\scriptstyle\,\pm\,0.5$ %	–	–
Reverse KL (unweighted)	68.4 $\scriptstyle\,\pm\,0.4$ %	2.2%	None
Hard Filter Reverse KL	70.1 $\scriptstyle\,\pm\,0.4$ %	0.5%	Hard
AKL	69.8 $\scriptstyle\,\pm\,0.3$ %	0.8%	Token-level
Paced Reverse KL	70.0 $\scriptstyle\,\pm\,0.3$ %	0.6%	Beta

Table 5: Retention in self-distillation track (Qwen2.5 reverse KL family): MMLU and forgetting (

\Delta

from base).

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2: Cross-problem gradient SNR vs. student pass rate at two training stages (Qwen3-1.7B, forward KL). Left: Step 1. Right: Step 20. Each problem contributes one gradient (from its fixed teacher reference); $K{=}10$ student rollouts are used only for pass-rate estimation. Problems are grouped into equal-width pass-rate bins. Both empirical and theoretical values are normalized by dividing by the respective bin maximum (Eq. 6 ), so the tallest bar is $1.0$ in both cases. The bell-shaped profile persists across training stages, confirming that the boundary collapse of cross-problem gradient coherence is a structural property of the distillation landscape, not a transient artifact of initialization.

Panel of Figure 3 — Figure 3: Prompt example for student and teacher policies. Both policies share the same model family but differ in conditioning context. The teacher receives the expert solution $y_{\mathcal{E}}$ as additional context, while the student receives only the original problem. This contextual asymmetry enables black-box expert guidance to be transferred into white-box teacher logits for distillation.

Parameter	Value
General
Models	Qwen2.5-Math-7B-Instruct (self-distillation), Qwen3-1.7B (teacher: Qwen3-8B GRPO )
Data
Training prompts	DAPO-Math-17k [ Yu et al., 2025 ]
Max prompt length (student)	1,024 tokens (problem only)
Max prompt length (teacher)	3,072 tokens (problem + expert solution)

Appendix

Table 6: Hyperparameters for Paced . Shared settings are listed once, with setting-specific differences noted explicitly.

$\alpha$	$\beta$	MATH-500 ( $\uparrow$ )	Forgetting on MMLU ( $\downarrow$ )
1	1	79.4%	1.4%
1	2	80.8%	2.1%
2	1	76.9%	3.0%
1	3	80.3%	3.6%
3	1	75.7%	2.9%

Appendix

Table 7: Ablation on pass-rate weight exponents

w(p)=p^{\alpha}(1-p)^{\beta}

using forward KL divergence as the distillation loss (Qwen3-1.7B).

$K$	MATH-500 ( $\uparrow$ )	AIME 2024 ( $\uparrow$ )	AIME 2025 ( $\uparrow$ )	MMLU Fgt. ( $\downarrow$ )
4	78.0%	23.3%	19.5%	1.7%
8	79.4%	25.1%	20.6%	1.4%
16	80.1%	26.3%	21.8%	1.5%

Appendix

Table 8: Sensitivity to number of rollouts

K

for pass-rate estimation. All results use Qwen3-1.7B with forward KL and default exponents

(\alpha{=}\beta{=}1)

Recompute interval	MATH-500 ( $\uparrow$ )	AIME 2024 ( $\uparrow$ )	AIME 2025 ( $\uparrow$ )	MMLU Fgt. ( $\downarrow$ )
Single-pass	79.4%	25.1%	20.6%	1.4%
Every 200 steps	79.9%	26.6%	22.5%	1.6%
Every 100 steps	80.2%	26.9%	23.9%	1.4%
Every 50 steps	80.4%	28.5%	23.3%	1.5%

Appendix

Table 9: Effect of periodic pass-rate recomputation on Qwen3-1.7B (forward KL,

\alpha{=}\beta{=}1

). “Single-pass” estimates pass rates once before training; “Every

N

steps” recomputes pass rates and updates weights at the specified interval.

Stage 1	Stage 2	MATH-500 ( $\uparrow$ )	AIME 2024 ( $\uparrow$ )	AIME 2025 ( $\uparrow$ )	MMLU Fgt. ( $\downarrow$ )
Paced KL	Paced KL	79.7%	25.6%	21.1%	1.3%
Paced RevKL	Paced RevKL	78.8%	23.5%	19.4%	1.2%
Paced RevKL	Paced KL	76.9%	23.0%	18.2%	2.5%
Paced KL	Paced RevKL	81.4%	26.1%	22.8%	1.1%

Appendix

Table 10: Two-stage order comparison on Qwen3 with the same pass-rate weighting

w(p)=p(1-p)

. Pass rates are recomputed once between stages; the first half of training steps uses Stage 1 and the second half uses Stage 2. Results are reported as 8-sample mean accuracy. The first two rows give the corresponding single-loss references under the same midpoint-recompute setup, and the last two rows isolate schedule order.

Schedule	Stage 1 ratio	MATH-500 ( $\uparrow$ )	AIME 2024 ( $\uparrow$ )	AIME 2025 ( $\uparrow$ )	MMLU Fgt. ( $\downarrow$ )
KL $\rightarrow$ RevKL	25%	78.9%	22.4%	19.3%	1.6%
KL $\rightarrow$ RevKL	50%	81.4%	26.1%	22.8%	1.1%
KL $\rightarrow$ RevKL	75%	80.1%	26.9%	20.6%	1.1%

Appendix

Table 11: Ablation on the fraction of training steps allocated to Stage 1 for two-stage distillation on Qwen3 under the KL

\rightarrow

RevKL schedule. The first

x

% of steps use Paced KL (Stage 1) and the remaining steps use Paced RevKL (Stage 2). Results are 8-sample mean accuracy.

Training Stage	Low $p$ ( $<\!0.2$ )	Med $p$ ( $0.2$ – $0.8$ )	High $p$ ( $>\!0.8$ )	Avg pass rate $\bar{p}$
Step 0 (Init)	17%	51%	32%	0.61
Step 100	12%	32%	56%	0.70
Step 200	9%	24%	67%	0.78
Step 300	5%	21%	74%	0.84

Appendix

Table 12: Evolution of the pass-rate distribution and average pass rate

\bar{p}

across training. The distillation signal peaks when most problems enter the

p\in[0.2,0.8]

zone.

Method	MATH-500 ( $\uparrow$ )	AIME 2024 ( $\uparrow$ )	AIME 2025 ( $\uparrow$ )	MMLU Fgt. ( $\downarrow$ )
Forward KL (unweighted)	72.3%	22.9%	5.1%	3.5%
Hard Filter Forward KL	75.2%	25.2%	7.8%	1.5%
Paced Forward KL	76.7%	27.7%	9.2%	1.4%

Appendix

Table 13: Distillation from Llama-3.3-70B-Instruct to Llama-3.1-8B-Instruct (forward KL family): reasoning performance (8-sample mean accuracy) and retention.

Language Model Distillation

Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.

See all 489 papers

Not Every Rollout Needs Distillation: Correctness-Filtered On-Policy Self-Distillation
Oct 19, 2026Harrison Zhao, Lei Huang, Fei Liu +2On-Policy Self-Distillation Language Model Distillation
Meta
Dr. OPD: Learning What to Follow for Optimal On-Policy Distillation of Large Language Models
Sep 29, 2026Zhenyu Wang, Tianze Wang, Linjun Zhang +1Language Model Distillation Bilevel Optimization
Department of Statistics, Rutgers University
PAST: Privileged Adaptation from Complete Student Trajectories for On-Policy Self-Distillation
Aug 9, 2026Yangyang Feng, Zhuoyan Feng, Junlan ChenOn-Policy Self-Distillation Knowledge Distillation
1The Hong Kong University of Science and Technology (Guangzhou) · 2Sun Yat-sen University

Oct 19, 2026J/K move · Enter open · S save

Not Every Rollout Needs Distillation: Correctness-Filtered On-Policy Self-Distillation

Harrison Zhao, Lei Huang, Fei Liu, Wen Shi+1

Abstract

Language Model Distillation

Explore similar work

Not Every Rollout Needs Distillation: Correctness-Filtered On-Policy Self-Distillation

Dr. OPD: Learning What to Follow for Optimal On-Policy Distillation of Large Language Models

PAST: Privileged Adaptation from Complete Student Trajectories for On-Policy Self-Distillation