When Clipping Reverses Correction: Failure Dynamics of Pointwise Forward-KL On-Policy Self-Distillation

Di Huang; Hao Li; Yixin Chen; Fuhai Li

cs.CLSep 30, 2026

When Clipping Reverses Correction: Failure Dynamics of Pointwise Forward-KL On-Policy Self-Distillation

Authors: Di Huang, Hao Li, Yixin Chen, Fuhai Li

Organizations: Department of Computer Science Washington University in St. Louis · Department of Pediatrics Washington University in St. Louis

Abstract

On-policy self-distillation (OPSD) trains a student on its own generated responses using feedback from the same model conditioned on privileged information. On mathematical reasoning, the original OPSD study finds that stylistic tokens can dominate the training signal over math-related tokens, and that pointwise clipping of the forward KL objective stabilizes training. Pointwise clipping caps each vocabulary-wise forward KL term at a fixed threshold before summing over the vocabulary. Follow-up studies have adopted this clipping, but its effect on training has not been directly examined. In matched training runs differing only in whether clipping is applied, we observe that clipped runs produce substantially more repetitions that persist to the end of the response than their unclipped counterparts. We trace this failure to the clipped objective. We prove that the clipped objective can fail to correct the student toward the teacher and can instead push clipped and unclipped token probabilities away from its teacher. Our training runs agree with this analysis: inside repetitions, the clipped student places less probability than its teacher on leaving the repetition, and more on continuing it, whereas the unclipped runs stay close to their teachers.

Figures & tables

Family	Reference	Student/teacher thinking	Response cap
F	yes	off/on	18,432
M	yes	on/off	18,432
K	yes	off/off	8,192
L	yes	on/on	18,432

Table 1: Paired training families. All use Qwen3-4B, teacher top-128 support,

T=1.1

, 30 problems per update with one rollout per problem, learning rate of

1\times 10^{-6}

, forward-KL distillation, and one trajectory per run. Clipped runs use

\tau=0.05

and their unclipped runs do not.

Term	Definition
Periodic segment	A maximal contiguous sequence $x_{s},\ldots,x_{e}$ , where $s$ and $e$ are its start and end positions, with period $\ell$ and length $L=e-s+1>\ell$ . $L$ satisfies $x_{t}=x_{t-\ell}$ for $s+\ell\leq t\leq e$ .
Repetition	A periodic segment with at least two complete period $\lfloor L/l\rfloor\geq 2$ .
Mature repetition	A repetition $\lfloor L/l\rfloor\geq 3$ with $L\geq 30$ tokens.
Terminal loop	Mature repetition that reaches the last response token.
Copy token	At a position $t\geq s+\ell$ while the periodic continuation remains, the token $c_{t}=x_{t-\ell}$ that continues the period. Its teacher and student probabilities are $p_{t,\mathrm{copy}}$ and $q_{t,\mathrm{copy}}$ , respectively.
First copy	Copy token belong to the second completed occurrence of the period, at positions $s+\ell,\ldots,s+2\ell-1$ .

Table 2: Definitions used to characterize token repetition and analyze its continuation. The upper block defines periodic segments, repetitions, mature repetitions, and terminal loops. The lower block defines the first copy, copy token, and exit mass.

Figure 1: Loop outcome over training, one trajectory per run. Each point is the number of the 30 rollouts sampled at that update that end in a terminal loop. Clipped runs in red, unclipped twins in blue. Shaded windows are three phases: early (updates 1–25, gray), ignition (F 51–75, M 76–100, K 151–175, L 101–125, orange), and late (updates 176–200, pink).

Quantity	Arm	F	M	K	L
$n(N\geq 2)$	clipped	$1195\!{\scriptscriptstyle\to}\!4954\!{\scriptscriptstyle\to}\!2814$	$3852\!{\scriptscriptstyle\to}\!2666\!{\scriptscriptstyle\to}\!1304$	$1163\!{\scriptscriptstyle\to}\!3757\!{\scriptscriptstyle\to}\!4024$	$5952\!{\scriptscriptstyle\to}\!5704\!{\scriptscriptstyle\to}\!6374$
	twin	$1237\!{\scriptscriptstyle\to}\!2395\!{\scriptscriptstyle\to}\!3154$	$3288\!{\scriptscriptstyle\to}\!975\!{\scriptscriptstyle\to}\!1496$	$1130\!{\scriptscriptstyle\to}\!1042\!{\scriptscriptstyle\to}\!1043$	$3991\!{\scriptscriptstyle\to}\!2642\!{\scriptscriptstyle\to}\!3121$
$N(\text{mat.}\mid N\geq 2)$	clipped	$1\!{\scriptscriptstyle\to}\!790\!{\scriptscriptstyle\to}\!749$	$6\!{\scriptscriptstyle\to}\!409\!{\scriptscriptstyle\to}\!69$	$4\!{\scriptscriptstyle\to}\!170\!{\scriptscriptstyle\to}\!176$	$16\!{\scriptscriptstyle\to}\!195\!{\scriptscriptstyle\to}\!328$
	twin	$5\!{\scriptscriptstyle\to}\!3\!{\scriptscriptstyle\to}\!7$	$16\!{\scriptscriptstyle\to}\!1\!{\scriptscriptstyle\to}\!8$	$6\!{\scriptscriptstyle\to}\!4\!{\scriptscriptstyle\to}\!8$	$7\!{\scriptscriptstyle\to}\!20\!{\scriptscriptstyle\to}\!9$
$P(\text{mat.}\mid N\geq 2)$	clipped	$.001\!{\scriptscriptstyle\to}\!.159\!{\scriptscriptstyle\to}\!.266$	$.002\!{\scriptscriptstyle\to}\!.153\!{\scriptscriptstyle\to}\!.053$	$.003\!{\scriptscriptstyle\to}\!.045\!{\scriptscriptstyle\to}\!.044$	$.003\!{\scriptscriptstyle\to}\!.034\!{\scriptscriptstyle\to}\!.051$
	twin	$.004\!{\scriptscriptstyle\to}\!.001\!{\scriptscriptstyle\to}\!.002$	$.005\!{\scriptscriptstyle\to}\!.001\!{\scriptscriptstyle\to}\!.005$	$.005\!{\scriptscriptstyle\to}\!.004\!{\scriptscriptstyle\to}\!.008$	$.002\!{\scriptscriptstyle\to}\!.008\!{\scriptscriptstyle\to}\!.003$

Table 3: Progression of repetition by training phase (period 2–50). Rows:

n(N\geq 2)

, the number of repetitions;

N(\text{mat.}\mid N\geq 2)

, the mature repetitions among repetitions;

P(\text{mat.}\mid N\geq 2)

the proportion of repetitions that mature;

N(\text{term.}\mid\text{mat.})

, the terminal loops among mature repetition. Each cell reads early

\to

ignition

\to

late, the three phases of Figure 1 , 750 rollouts each.

Figure 2: The copy token inside repetitions (period 2–50), with the student’s probabilities taken before the sampler’s truncation. (a) Student/teacher exit-mass ratio, one point per block of five updates (1–5, …, 196–200). Lines without markers give the ratio of the teacher’s exit mass on clipped tokens, or on over-threshold tokens in the twins (right axis). (b) Copied positions between the first copy and maturity: fraction with

q_{\mathrm{copy}}>p_{\mathrm{copy}}

. Dark bar:

q_{\mathrm{copy}}<p_{\mathrm{copy}}/P_{A}

, where the logit gradient of exact KL points toward lowering the copy logit and that of the clipped objective toward raising it. Light bar: both point toward lowering it. (c) The same copied positions, plus observed stopping positions before maturity: mean

-\partial D/\partial z_{\mathrm{copy}}

. Black and red: exact KL (hypothetical) and clipped KL, respectively, evaluated on the same positions and probabilities from the clipped run. Positive values point toward raising the copy logit. Phases as in Table 3 .

Figure 3: AIME 2025 across training. (a) Avg@12, the mean correct ratio over 12 samples for each of 30 problems at the checkpoint saved after each 25 updates. (b) The ratio of the 360 responses per checkpoint that end in a terminal loop. Solid curves with circles are the clipped runs, dashed curves with squares their unclipped twins. Each trained curve is one realized trajectory. The black arrow on each vertical axis marks the base model, the fixed starting checkpoint averaged over eight evaluation-sampling seeds.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: A simple example of updating direction of q under different losses(Forward KL v.s. Clipped Loss Updates.

Component	Executed value
Base checkpoint	Qwen/Qwen3-4B
Optimizer	torch.optim.AdamW ; learning rate $10^{-6}$ ; $\beta=(0.9,0.999)$ ; $\epsilon=10^{-8}$ ; weight decay $0.01$
Schedule	Zero warmup. $10^{-6}$ constant.
Updates and checkpoints	200 updates; a checkpoint is saved every 25 updates
Seed	Training and data-loader seeds are 42
Global update	30 problems, one rollout each

Appendix

Table 4: Training Configuration.

Figure 5: AIME 2024 across training, in the layout of Figure 3 . (a) Avg@12, the mean correct ratio over 12 samples for each of 30 problems at the checkpoint saved after each 25 updates. (b) The ratio of the 360 responses per checkpoint that end in a terminal periodic loop. Solid curves with circles are the clipped runs, dashed curves with squares their unclipped twins. Each trained curve is one realized trajectory. The black arrow on each vertical axis marks the base model, averaged over eight evaluation-sampling seeds. The values appear in Table 6 .

(a) avg@12
Run	25	50	75	100	125	150	175	200
F	$.661$	$.653$	$.589$	$.550$	$.581$	$.525$	$.531$	$.544$
F-no-clip	$.622$	$.644$	$.578$	$.544$	$.619$	$.561$	$.575$	$.572$
M	$.592$	$.481$	$.378$	$.122$	$.050$	$.067$	$.031$	$.022$
M-no-clip	$.047$	$.000$	$.000$	$.000$	$.000$	$.000$	$.000$	$.000$
K	$.664$	$.653$	$.647$	$.619$	$.622$	$.550$	$.628$	$.600$

Appendix

Table 5: Complete AIME 2025 evaluation at every checkpoint (columns: training update). (a) Avg@12, the mean correct ratio over 12 samples for each of 30 problems. (b) Terminal loop rate, the ratio of the 360 responses that end in a terminal periodic loop. (c) Pass@12, the ratio of the 30 problems solved by at least one of the 12 samples. Each trained row is one realized trajectory. The base row gives the mean and, in parentheses, the standard deviation over eight evaluation-sampling seeds of the fixed starting checkpoint.

(a) avg@12
Run	25	50	75	100	125	150	175	200
F	$.739$	$.756$	$.622$	$.617$	$.553$	$.539$	$.547$	$.533$
F-no-clip	$.728$	$.717$	$.694$	$.658$	$.672$	$.667$	$.683$	$.661$
M	$.672$	$.572$	$.447$	$.125$	$.069$	$.069$	$.036$	$.031$
M-no-clip	$.056$	$.000$	$.000$	$.000$	$.000$	$.000$	$.000$	$.000$
K	$.742$	$.733$	$.664$	$.681$	$.678$	$.700$	$.681$	$.700$

Appendix

Table 6: Complete AIME 2024 evaluation at every checkpoint (columns: training update). (a) Avg@12, the mean correct ratio over 12 samples for each of 30 problems. (b) Terminal loop rate, the ratio of the 360 responses that end in a terminal periodic loop. (c) Pass@12, the ratio of the 30 problems solved by at least one of the 12 samples. Each trained row is one realized trajectory. The base row gives the mean and, in parentheses, the standard deviation over eight evaluation-sampling seeds of the fixed starting checkpoint.

On-Policy Self-Distillation

On-policy self-distillation trains a model on its own generated trajectories using a teacher view to provide dense supervision.

See all 157 papers

One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation
Aug 26, 2026Justin Robert, Raheel QaderOn-Policy Self-Distillation On-Policy Distillation
OVHai LLM
Not Every Rollout Needs Distillation: Correctness-Filtered On-Policy Self-Distillation
Oct 19, 2026Harrison Zhao, Lei Huang, Fei Liu +2On-Policy Self-Distillation Language Model Distillation
Meta
E $^2$ -OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation
Oct 4, 2026Yifei Liu, Minghao Fang, Xinyu Gu +6On-Policy Self-Distillation Mathematical Reasoning
The Chinese University of Hong Kong · Zhejiang University · Shanghai Artificial Intelligence Laboratory +5

Aug 26, 2026J/K move · Enter open · S save

One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation

Justin Robert, Raheel Qader

OVHai LLM

On-Policy Self-Distillation On-Policy Distillation Mode Collapse

Open paper AskPDF arXiv

On-policy distillation trains a language model on its own generations while a teacher scores them token by token. It combines the dense supervision of imitation learning with the on-policy sampling of reinforcement learning. But it requires a second, larger model to act as teacher. On-Policy Self-Distillation (OPSD) removes that cost. The teacher is the model itself, conditioned on privileged information the student will not have at test time, such as a reference solution, a plan, or environment feedback. The teacher is no stronger than the student, only better informed. Early results were promising, with accuracy comparable to reinforcement learning at a fraction of the generated tokens. But the same asymmetry that produces the signal also biases it. One failure mode now dominates the field: collapse, the progressive narrowing of the set of reasoning paths the model can produce. Collapse is not specific to OPSD, though privileged information aggravates it. This review treats collapse as a symptom governed by three levers: (i) where the signal is applied, that is, how tokens are weighted; (ii) what the teacher is shown, that is, the nature of the privileged information; and (iii) when the signal changes, that is, the teacher's dynamics and the decay of guidance. We restrict our scope to mathematical reasoning, where the method originated and where its failure modes are best documented. We report no new experiments. The contribution is structural: a shared vocabulary for phenomena named differently across papers, and a clear line between what is settled and what is still disputed.

Abstract

On-Policy Self-Distillation

Explore similar work

One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation

Not Every Rollout Needs Distillation: Correctness-Filtered On-Policy Self-Distillation

E2^22-OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation

E $^2$ -OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation