LoOPSD: On-Policy Self-Distillation with Looped Teachers

Wei Fan; Qingyue Yang; Chao-Hong Tan; Qian Chen; Wen Wang; Xiangang Li; Kejiang Chen; Weiming Zhang

cs.LGOct 13, 2026

LoOPSD: On-Policy Self-Distillation with Looped Teachers

Authors: Wei Fan, Qingyue Yang, Chao-Hong Tan, Qian Chen, Wen Wang, Xiangang Li, Kejiang Chen, Weiming Zhang,

Organizations: Anhui Province Key Laboratory of Digital Security · Independent Researcher · Token Foundry, Alibaba Group

Abstract

On-policy self-distillation improves a model using guidance from the model itself, but it still requires a teacher that provides more useful predictions than the student. Existing approaches obtain such guidance by giving the teacher additional context or instructions. We ask whether it can instead come from changing how the same model uses its own layers. To address this, we introduce LoOPSD, which searches over a model's layer execution order to construct a stronger teacher and distills its advantage into the student. Specifically, we keep the model weights unchanged while searching for layers to repeat, and use the resulting teacher's predictions on student-generated prefixes to guide learning. We then alternate search and distillation, rebuilding the teacher from the updated student in each round so that the student progressively absorbs the gains from alternative layer execution. Experiments on mathematical and logical reasoning tasks show improvements across models. On GSM8K with a 400-token thinking budget, LoOPSD raises Qwen3-1.7B accuracy from 42.30% to 79.61%; subsequent distillation rounds yield further gains beyond the first round. Together, these findings show that by searching layer execution to construct teachers, we can turn a model's additional computation into improved student capabilities, opening a new avenue for self-distillation to derive supervision from the model's own computation.

Figures & tables

Figure 1: Sources of supervision for reasoning models. LoOPSD constructs a teacher from the student’s starting weights by selecting an alternative layer route with labeled examples. The resulting predictions provide supervision without an external teacher model.

Figure 2: Overview of LoOPSD. (1) Teacher construction: Using a small labeled set, we try different choices of layers to repeat, then select the best-performing route to define the teacher. (2) On-policy distillation: The student generates responses using the original layer order. The frozen teacher evaluates the same response prefixes and provides token-level supervision to update the student. (3) Iteration and final model: The updated student is used to build the teacher for the next round. After $K$ rounds, the final student uses the original layer order.

Method	GSM8K	GSM-Plus	SVAMP	MATH-500	ProofWriter
Qwen3-1.7B
Original	42.30	31.70	71.33	33.20	42.50
Elastic Reasoning	46.25	30.20	73.00	34.00	45.00
CRISP	76.65	58.50	88.33	37.80	43.40
LoOPSD (R1 $\to$ Final)	78.09 $\to$ 79.61	56.50 $\to$ 61.40	87.67 $\to$ 87.67	44.40 $\to$ 44.80	42.10 $\to$ 56.20
LoOPSD † (R1 $\to$ Final)	78.32 $\to$ 79.68	59.30 $\to$ 61.80	88.67 $\to$ 91.67	37.20 $\to$ 48.20	46.80 $\to$ 54.10

Table 1: Accuracy (%) across four models and five benchmarks, using greedy decoding with 400 thinking and 128 answer tokens. For LoOPSD variants, entries report first-round (R1)

\to

final accuracy. R1 denotes the student after the first distillation round; final denotes the checkpoint obtained by the iterative continuation procedure. GSM-Plus evaluates the corresponding GSM8K-trained checkpoints.

Method	Phi-4-reasoning-plus	Qwen3-14B
Original	89.54	83.40
Elastic Reasoning	89.69	85.14
CRISP	92.95	92.57
LoOPSD (R1 $\to$ Final)	91.89 $\to$ 94.92	90.07 $\to$ 94.39
LoOPSD † (R1 $\to$ Final)	95.07 $\to$ 93.18	93.63 $\to$ 93.63

Table 2: GSM8K accuracy (%) of 14B-scale reasoning models with a 400-token thinking budget and greedy decoding. LoOPSD entries report first-round (R1)

\to

final accuracy. All students use standard execution and their original prompt. LoOPSD achieves the highest final accuracy for both model families.

Figure 3: Accuracy across thinking budgets on GSM8K. (a,b) Qwen3-1.7B and DeepSeek-R1-Distill-Llama-8B students obtained separately for each target search budget and evaluated at that budget. (c) Qwen3-4B students obtained with a 400-token search budget, with each checkpoint fixed across test budgets. LoOPSD improves reasoning across a range of budgets, and fixed students adapt to different test budgets.

Figure 4: Accuracy and generation length of different methods on GSM8K with Qwen3-4B. All models use a 400-token thinking budget. (a) Accuracy versus average total generated tokens. (b) Average thinking and answer lengths. LoOPSD improves accuracy while shortening generation; LoOPSD † achieves the highest accuracy with the fewest thinking tokens.

	Total labeled examples
Method	200	400	1100
Original	42.30 (shared baseline)
Matched-label GRPO	43.90	43.52	44.05
LoOPSD	78.01	78.39	78.62

Table 3: Label efficiency on GSM8K with Qwen3-1.7B. LoOPSD and matched-label GRPO each receive 200 updates; label totals include 100 validation examples, and LoOPSD uses a fixed question-only distillation pool. LoOPSD already obtains most of its observed gain with 200 labels.

Figure 5: Iterative learning on GSM8K with Qwen3-4B. (a) Each round shows the looped teacher constructed from the current model and the standard-path student obtained after distillation. We compare LoOPSD, LoOPSD † , CRISP, and standard-path self-distillation across six 200-update rounds, evaluated at a 128-token thinking budget. (b) First-round and final students from the respective LoOPSD and LoOPSD † training trajectories, each held fixed across test budgets. Panel (a) includes matched-update standard-path controls under both the original and concise teacher prompts, while panel (b) shows the additional gains obtained across LoOPSD iterations.

Figure 6: Layerwise answer readout on ProofWriter with Qwen3-4B under shared contexts and a fixed output projection. Shading marks repeated teacher execution. Student gains concentrate in later layers.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Teacher construction on GSM8K with Qwen3-1.7B and ProofWriter with Qwen3-4B, at a 400-token thinking budget. Heatmaps show search accuracy changes in percentage points; each panel has an independent color bar. Stars mark validation-selected nonstandard routes. Validation selects a repeated-layer route under both prompts on ProofWriter, while retaining standard execution under the concise prompt on GSM8K.

Figure 8: Teacher construction on MATH-500 with Qwen3-8B and SVAMP with DeepSeek-R1-8B, using the same 400-token protocol. Heatmaps show search accuracy changes; independent bars preserve each panel’s measured range. Stars mark validation-selected nonstandard routes, which are selected under both the original and concise prompts in these two settings.

	SVAMP		GSM-Plus
Method	Accuracy	Think tokens	Accuracy	Think tokens
Qwen3-1.7B
Original	71.33	299.7	31.70	378.8
Elastic Reasoning	75.00	294.4	30.20	378.1
CRISP	87.67	251.8	58.50	353.5
LoOPSD	85.33	373.0	61.40	392.7

Appendix

Table 4: Transfer of GSM8K-trained students to additional math datasets. Accuracy (%) and mean thinking tokens at a 400-token thinking cap. All methods use standard execution, the original prompt, and greedy decoding.

Figure 9: Initial teachers and final students on GSM8K with Qwen3-1.7B. Each initial teacher is connected to the final student obtained from the corresponding training configuration. Final LoOPSD students outperform their initial teachers at all six thinking budgets.

Figure 11: Gold-label log probability across layer executions on ProofWriter with Qwen3-4B. Higher values indicate greater probability assigned to the correct label. Shading marks repeated teacher execution. LoOPSD students assign higher probability to the correct label in later layers.

	Qwen3-1.7B				Qwen3-4B
Method	$0$	$0.3$	$0.5$	$1.0$	$0$	$0.3$	$0.5$	$1.0$
Original	42.38	42.99	41.55	41.70	74.91	73.62	75.89	73.92
Elastic	44.43	44.12	44.20	43.59	76.27	75.66	76.12	75.36
CRISP	76.65	78.17	79.23	77.26	87.64	88.40	88.02	87.57
LoOPSD	79.61	79.45	79.38	77.03	89.69	90.37	90.37	91.21
LoOPSD †	79.68	80.52	81.88	80.82	91.05	91.51	91.66	91.05

Appendix

Table 5: GSM8K accuracy (%) at different thinking temperatures. We evaluate the same checkpoints with a 400-token thinking budget and a 128-token answer budget, sampling one response per question. LoOPSD improvements persist across decoding temperatures.

On-Policy Self-Distillation

On-policy self-distillation trains a model on its own generated trajectories using a teacher view to provide dense supervision.

See all 162 papers

On-Policy Self-Distillation without Any Supervision
Aug 6, 2026Yijiang Li, Bingyang Wang, Yijun Liang +3On-Policy Self-Distillation Language Model Self-Improvement
UC San Diego · Georgia Institute of Technology · University of Maryland, College Park +1
Recursive Self-Improvement via On-Policy Distillation for Reasoning
Sep 25, 2026Shangjian Yin, Zehao Zhao, Kavosh Asadi +7Language Model Self-Improvement On-Policy Self-Distillation
Meta AI · University of California, Riverside · Work done at Meta
Teach Yourself Where to Look: On-Policy Attention Self-Distillation for Reasoning
Sep 27, 2026Safaeid Hossain Arib, Rabeya Akter, Ismam Nur Swapnil +3Numerical Reasoning in Language Models Attention-Guided Distillation
ACI PLC, Bangladesh · University of Dhaka, Bangladesh · BRAC University, Bangladesh +2

Aug 6, 2026J/K move · Enter open · S save

On-Policy Self-Distillation without Any Supervision

Yijiang Li, Bingyang Wang, Yijun Liang, Yunjie Tian+2

UC San Diego · Georgia Institute of Technology · University of Maryland, College Park+1

On-Policy Self-Distillation Language Model Self-Improvement Self-Consistency in Language Models

Open paper AskPDF arXiv

On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for post-training large language models (LLMs). However, existing methods still rely heavily on external supervision, including ground-truth signals, environmental feedback, or guidance from larger models, and therefore fall short of genuine "self"-distillation. In this study, we show that on-policy self-distillation can be achieved using only a model's own generations via internal consistency. We propose unsupervised on-policy self-distillation (U-OPSD). U-OPSD first samples multiple rollouts and constructs a pseudo solution by majority vote under a self-consistency threshold. It then conditions the model's distribution on the pseudo-solution and distills itself on the disagreeing completions, allowing the model to correct itself precisely where it is confidently wrong. Across diverse benchmarks, base models, and training settings, U-OPSD consistently improves over the base models and matches or surpasses supervised methods with ground truth (GT) such as OPSD and GRPO. On five mathematical reasoning benchmarks, i.e., AIME24, AIME25, HMMT25, MATH500, and AMC23, U-OPSD improves over the base model by 8.5% and 10.7% on Qwen3 non-thinking mode at 4B and 8B scales, and outperforms OPSD by 3.2% and 2.3% on average, respectively. In thinking mode, U-OPSD stays on par with OPSD, ahead by 0.9% at 4B and level at 8B and surpassing GRPO by 0.7% and 1.1%, respectively. Code is available at https://github.com/williamium3000/u-opsd.

Abstract

On-Policy Self-Distillation

Explore similar work

On-Policy Self-Distillation without Any Supervision

Recursive Self-Improvement via On-Policy Distillation for Reasoning

Teach Yourself Where to Look: On-Policy Attention Self-Distillation for Reasoning