When Clipping Reverses Correction: Failure Dynamics of Pointwise Forward-KL On-Policy Self-Distillation

Di Huang; Hao Li; Yixin Chen; Fuhai Li

cs.CLSep 30, 2026

When Clipping Reverses Correction: Failure Dynamics of Pointwise Forward-KL On-Policy Self-Distillation

Authors: Di Huang, Hao Li, Yixin Chen, Fuhai Li

Organizations: Department of Computer Science Washington University in St. Louis · Department of Pediatrics Washington University in St. Louis

Abstract

On-policy self-distillation (OPSD) trains a student on its own generated responses using feedback from the same model conditioned on privileged information. On mathematical reasoning, the original OPSD study finds that stylistic tokens can dominate the training signal over math-related tokens, and that pointwise clipping of the forward KL objective stabilizes training. Pointwise clipping caps each vocabulary-wise forward KL term at a fixed threshold before summing over the vocabulary. Follow-up studies have adopted this clipping, but its effect on training has not been directly examined. In matched training runs differing only in whether clipping is applied, we observe that clipped runs produce substantially more repetitions that persist to the end of the response than their unclipped counterparts. We trace this failure to the clipped objective. We prove that the clipped objective can fail to correct the student toward the teacher and can instead push clipped and unclipped token probabilities away from its teacher. Our training runs agree with this analysis: inside repetitions, the clipped student places less probability than its teacher on leaving the repetition, and more on continuing it, whereas the unclipped runs stay close to their teachers.

Figures & tables

Family	Reference	Student/teacher thinking	Response cap
F	yes	off/on	18,432
M	yes	on/off	18,432
K	yes	off/off	8,192
L	yes	on/on	18,432

Table 1: Paired training families. All use Qwen3-4B, teacher top-128 support,

T=1.1

, 30 problems per update with one rollout per problem, learning rate of

1\times 10^{-6}

, forward-KL distillation, and one trajectory per run. Clipped runs use

\tau=0.05

and their unclipped runs do not.

Term	Definition
Periodic segment	A maximal contiguous sequence $x_{s},\ldots,x_{e}$ , where $s$ and $e$ are its start and end positions, with period $\ell$ and length $L=e-s+1>\ell$ . $L$ satisfies $x_{t}=x_{t-\ell}$ for $s+\ell\leq t\leq e$ .
Repetition	A periodic segment with at least two complete period $\lfloor L/l\rfloor\geq 2$ .
Mature repetition	A repetition $\lfloor L/l\rfloor\geq 3$ with $L\geq 30$ tokens.
Terminal loop	Mature repetition that reaches the last response token.
Copy token	At a position $t\geq s+\ell$ while the periodic continuation remains, the token $c_{t}=x_{t-\ell}$ that continues the period. Its teacher and student probabilities are $p_{t,\mathrm{copy}}$ and $q_{t,\mathrm{copy}}$ , respectively.
First copy	Copy token belong to the second completed occurrence of the period, at positions $s+\ell,\ldots,s+2\ell-1$ .

Table 2: Definitions used to characterize token repetition and analyze its continuation. The upper block defines periodic segments, repetitions, mature repetitions, and terminal loops. The lower block defines the first copy, copy token, and exit mass.

Figure 1: Loop outcome over training, one trajectory per run. Each point is the number of the 30 rollouts sampled at that update that end in a terminal loop. Clipped runs in red, unclipped twins in blue. Shaded windows are three phases: early (updates 1–25, gray), ignition (F 51–75, M 76–100, K 151–175, L 101–125, orange), and late (updates 176–200, pink).

Quantity	Arm	F	M	K	L
$n(N\geq 2)$	clipped	$1195\!{\scriptscriptstyle\to}\!4954\!{\scriptscriptstyle\to}\!2814$	$3852\!{\scriptscriptstyle\to}\!2666\!{\scriptscriptstyle\to}\!1304$	$1163\!{\scriptscriptstyle\to}\!3757\!{\scriptscriptstyle\to}\!4024$	$5952\!{\scriptscriptstyle\to}\!5704\!{\scriptscriptstyle\to}\!6374$
	twin	$1237\!{\scriptscriptstyle\to}\!2395\!{\scriptscriptstyle\to}\!3154$	$3288\!{\scriptscriptstyle\to}\!975\!{\scriptscriptstyle\to}\!1496$	$1130\!{\scriptscriptstyle\to}\!1042\!{\scriptscriptstyle\to}\!1043$	$3991\!{\scriptscriptstyle\to}\!2642\!{\scriptscriptstyle\to}\!3121$
$N(\text{mat.}\mid N\geq 2)$	clipped	$1\!{\scriptscriptstyle\to}\!790\!{\scriptscriptstyle\to}\!749$	$6\!{\scriptscriptstyle\to}\!409\!{\scriptscriptstyle\to}\!69$	$4\!{\scriptscriptstyle\to}\!170\!{\scriptscriptstyle\to}\!176$	$16\!{\scriptscriptstyle\to}\!195\!{\scriptscriptstyle\to}\!328$
	twin	$5\!{\scriptscriptstyle\to}\!3\!{\scriptscriptstyle\to}\!7$	$16\!{\scriptscriptstyle\to}\!1\!{\scriptscriptstyle\to}\!8$	$6\!{\scriptscriptstyle\to}\!4\!{\scriptscriptstyle\to}\!8$	$7\!{\scriptscriptstyle\to}\!20\!{\scriptscriptstyle\to}\!9$
$P(\text{mat.}\mid N\geq 2)$	clipped	$.001\!{\scriptscriptstyle\to}\!.159\!{\scriptscriptstyle\to}\!.266$	$.002\!{\scriptscriptstyle\to}\!.153\!{\scriptscriptstyle\to}\!.053$	$.003\!{\scriptscriptstyle\to}\!.045\!{\scriptscriptstyle\to}\!.044$	$.003\!{\scriptscriptstyle\to}\!.034\!{\scriptscriptstyle\to}\!.051$
	twin	$.004\!{\scriptscriptstyle\to}\!.001\!{\scriptscriptstyle\to}\!.002$	$.005\!{\scriptscriptstyle\to}\!.001\!{\scriptscriptstyle\to}\!.005$	$.005\!{\scriptscriptstyle\to}\!.004\!{\scriptscriptstyle\to}\!.008$	$.002\!{\scriptscriptstyle\to}\!.008\!{\scriptscriptstyle\to}\!.003$

Table 3: Progression of repetition by training phase (period 2–50). Rows:

n(N\geq 2)

, the number of repetitions;

N(\text{mat.}\mid N\geq 2)

, the mature repetitions among repetitions;

P(\text{mat.}\mid N\geq 2)

the proportion of repetitions that mature;

N(\text{term.}\mid\text{mat.})

, the terminal loops among mature repetition. Each cell reads early

\to

ignition

\to

late, the three phases of Figure 1 , 750 rollouts each.

Figure 2: The copy token inside repetitions (period 2–50), with the student’s probabilities taken before the sampler’s truncation. (a) Student/teacher exit-mass ratio, one point per block of five updates (1–5, …, 196–200). Lines without markers give the ratio of the teacher’s exit mass on clipped tokens, or on over-threshold tokens in the twins (right axis). (b) Copied positions between the first copy and maturity: fraction with

q_{\mathrm{copy}}>p_{\mathrm{copy}}

. Dark bar:

q_{\mathrm{copy}}<p_{\mathrm{copy}}/P_{A}

, where the logit gradient of exact KL points toward lowering the copy logit and that of the clipped objective toward raising it. Light bar: both point toward lowering it. (c) The same copied positions, plus observed stopping positions before maturity: mean

-\partial D/\partial z_{\mathrm{copy}}

. Black and red: exact KL (hypothetical) and clipped KL, respectively, evaluated on the same positions and probabilities from the clipped run. Positive values point toward raising the copy logit. Phases as in Table 3 .

Figure 3: AIME 2025 across training. (a) Avg@12, the mean correct ratio over 12 samples for each of 30 problems at the checkpoint saved after each 25 updates. (b) The ratio of the 360 responses per checkpoint that end in a terminal loop. Solid curves with circles are the clipped runs, dashed curves with squares their unclipped twins. Each trained curve is one realized trajectory. The black arrow on each vertical axis marks the base model, the fixed starting checkpoint averaged over eight evaluation-sampling seeds.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: A simple example of updating direction of q under different losses(Forward KL v.s. Clipped Loss Updates.

Component	Executed value
Base checkpoint	Qwen/Qwen3-4B
Optimizer	torch.optim.AdamW ; learning rate $10^{-6}$ ; $\beta=(0.9,0.999)$ ; $\epsilon=10^{-8}$ ; weight decay $0.01$
Schedule	Zero warmup. $10^{-6}$ constant.
Updates and checkpoints	200 updates; a checkpoint is saved every 25 updates
Seed	Training and data-loader seeds are 42
Global update	30 problems, one rollout each

Appendix

Table 4: Training Configuration.

Figure 5: AIME 2024 across training, in the layout of Figure 3 . (a) Avg@12, the mean correct ratio over 12 samples for each of 30 problems at the checkpoint saved after each 25 updates. (b) The ratio of the 360 responses per checkpoint that end in a terminal periodic loop. Solid curves with circles are the clipped runs, dashed curves with squares their unclipped twins. Each trained curve is one realized trajectory. The black arrow on each vertical axis marks the base model, averaged over eight evaluation-sampling seeds. The values appear in Table 6 .

(a) avg@12
Run	25	50	75	100	125	150	175	200
F	$.661$	$.653$	$.589$	$.550$	$.581$	$.525$	$.531$	$.544$
F-no-clip	$.622$	$.644$	$.578$	$.544$	$.619$	$.561$	$.575$	$.572$
M	$.592$	$.481$	$.378$	$.122$	$.050$	$.067$	$.031$	$.022$
M-no-clip	$.047$	$.000$	$.000$	$.000$	$.000$	$.000$	$.000$	$.000$
K	$.664$	$.653$	$.647$	$.619$	$.622$	$.550$	$.628$	$.600$

Appendix

Table 5: Complete AIME 2025 evaluation at every checkpoint (columns: training update). (a) Avg@12, the mean correct ratio over 12 samples for each of 30 problems. (b) Terminal loop rate, the ratio of the 360 responses that end in a terminal periodic loop. (c) Pass@12, the ratio of the 30 problems solved by at least one of the 12 samples. Each trained row is one realized trajectory. The base row gives the mean and, in parentheses, the standard deviation over eight evaluation-sampling seeds of the fixed starting checkpoint.

(a) avg@12
Run	25	50	75	100	125	150	175	200
F	$.739$	$.756$	$.622$	$.617$	$.553$	$.539$	$.547$	$.533$
F-no-clip	$.728$	$.717$	$.694$	$.658$	$.672$	$.667$	$.683$	$.661$
M	$.672$	$.572$	$.447$	$.125$	$.069$	$.069$	$.036$	$.031$
M-no-clip	$.056$	$.000$	$.000$	$.000$	$.000$	$.000$	$.000$	$.000$
K	$.742$	$.733$	$.664$	$.681$	$.678$	$.700$	$.681$	$.700$

Appendix

Table 6: Complete AIME 2024 evaluation at every checkpoint (columns: training update). (a) Avg@12, the mean correct ratio over 12 samples for each of 30 problems. (b) Terminal loop rate, the ratio of the 360 responses that end in a terminal periodic loop. (c) Pass@12, the ratio of the 30 problems solved by at least one of the 12 samples. Each trained row is one realized trajectory. The base row gives the mean and, in parentheses, the standard deviation over eight evaluation-sampling seeds of the fixed starting checkpoint.

On-Policy Self-Distillation

On-policy self-distillation trains a model on its own generated trajectories using a teacher view to provide dense supervision.

See all 157 papers

Aug 26, 2026

One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation

On-policy distillation trains a language model on its own generations while a teacher scores them token by token. It combines the dense supervision of imitation learning with the on-policy sampling of reinforcement learning. But it requires a second, larger model to act as teacher. On-Policy Self-Distillation (OPSD) removes that cost. The teacher is the model itself, conditioned on privileged information the student will not have at test time, such as a reference solution, a plan, or environment feedback. The teacher is no stronger than the student, only better informed. Early results were promising, with accuracy comparable to reinforcement learning at a fraction of the generated tokens. But the same asymmetry that produces the signal also biases it. One failure mode now dominates the field: collapse, the progressive narrowing of the set of reasoning paths the model can produce. Collapse is not specific to OPSD, though privileged information aggravates it. This review treats collapse as a symptom governed by three levers: (i) where the signal is applied, that is, how tokens are weighted; (ii) what the teacher is shown, that is, the nature of the privileged information; and (iii) when the signal changes, that is, the teacher's dynamics and the decay of guidance. We restrict our scope to mathematical reasoning, where the method originated and where its failure modes are best documented. We report no new experiments. The contribution is structural: a shared vocabulary for phenomena named differently across papers, and a clear line between what is settled and what is still disputed.

Justin Robert, Raheel Qader

OVHai LLM

On-Policy Self-Distillation On-Policy Distillation Mode Collapse

Ask PDF

Oct 19, 2026

Not Every Rollout Needs Distillation: Correctness-Filtered On-Policy Self-Distillation

On-policy self-distillation (OPSD) supervises student-generated trajectories with a reference-conditioned version of the same base model. We ask whether every trajectory supplies distinct supervision that warrants its distillation cost. Gradient measurements on Qwen3-1.7B at initialization reveal that correct and incorrect responses have similar individual gradient magnitudes, but incorrect responses have approximately three times the coherent gradient share. Their estimated population-mean directions also largely overlap. A signal--noise model fitted at a small group size tracks alignment at larger group sizes without refitting. Motivated by these measurements, we introduce correctness-filtered OPSD (C-OPSD), which generates one response per problem and distills only a subset of incorrect responses. Removing unselected rows before both distillation forwards avoids their teacher/student forward and student backward work. Across Qwen3-1.7B, 4B, and 8B on AIME24, AIME25, and HMMT25, C-OPSD achieves on-par performance with full OPSD. In a measured micro-batch-size-four implementation, executed rows decrease by approximately 68%, and with retained rows packed into full micro-batches, training takes about 40% less time while rollout generation remains unchanged. Together, the measurements and training results connect outcome-conditioned supervision redundancy to more compute-efficient distillation.

Harrison Zhao, Lei Huang, Fei Liu +2

E $^2$ -OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation

On-policy self-distillation (OPSD) provides dense token-level supervision without a second model: one network acts as teacher with the reference solution and as student with only the problem. We identify a specific failure mode of this recipe. During training, student token entropy rises past the teacher's and remains elevated, a pattern we call entropy overshoot. We trace it to both sides of distillation. The reference-conditioned teacher is confident along its answer-directed reasoning path, but this confidence transfers poorly to student-generated prefixes, making its supervision overly tied to answer-specific cues rather than reusable reasoning patterns; meanwhile, the forward KL used by OPSD continually diffuses the student's predictive distribution without pulling it back. We introduce E

^2

-OPSD to address both causes. Exemplar-guided teaching replaces the current answer with a retrieved solved neighboring problem, providing transferable reasoning guidance without revealing the destination and better matching student-reachable states. Entropy-aware distillation uses the student-teacher entropy gap to determine the direction and strength of each token's correction. E

^2

-OPSD improves math reasoning by up to 4.3 points in mean@16 over OPSD, while out-of-domain evaluations show gains over the corresponding base models of up to 4.9 points in mean@16 and 5.5 points in pass@8. Despite these gains, E

^2

-OPSD remains simple, requiring no additional forward passes or networks. Code repo: https://github.com/orionfei/E2-OPSD.

Yifei Liu, Minghao Fang, Xinyu Gu +6

The Chinese University of Hong Kong · Zhejiang University · Shanghai Artificial Intelligence Laboratory +5

On-Policy Self-Distillation Mathematical Reasoning Uncertainty-Aware Knowledge Distillation Knowledge Distillation

Ask PDF

Abstract

On-Policy Self-Distillation

Explore similar work

One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation

Not Every Rollout Needs Distillation: Correctness-Filtered On-Policy Self-Distillation

E2^22-OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation

E $^2$ -OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation