cs.LGOct 13, 2026

LoOPSD: On-Policy Self-Distillation with Looped Teachers

Authors: Wei Fan, Qingyue Yang, Chao-Hong Tan, Qian Chen, Wen Wang, Xiangang Li, Kejiang Chen, Weiming Zhang, 

Organizations: Anhui Province Key Laboratory of Digital Security · Independent Researcher · Token Foundry, Alibaba Group

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

On-policy self-distillation improves a model using guidance from the model itself, but it still requires a teacher that provides more useful predictions than the student. Existing approaches obtain such guidance by giving the teacher additional context or instructions. We ask whether it can instead come from changing how the same model uses its own layers. To address this, we introduce LoOPSD, which searches over a model's layer execution order to construct a stronger teacher and distills its advantage into the student. Specifically, we keep the model weights unchanged while searching for layers to repeat, and use the resulting teacher's predictions on student-generated prefixes to guide learning. We then alternate search and distillation, rebuilding the teacher from the updated student in each round so that the student progressively absorbs the gains from alternative layer execution. Experiments on mathematical and logical reasoning tasks show improvements across models. On GSM8K with a 400-token thinking budget, LoOPSD raises Qwen3-1.7B accuracy from 42.30% to 79.61%; subsequent distillation rounds yield further gains beyond the first round. Together, these findings show that by searching layer execution to construct teachers, we can turn a model's additional computation into improved student capabilities, opening a new avenue for self-distillation to derive supervision from the model's own computation.

Figures & tables

Figure 1: Sources of supervision for reasoning models. LoOPSD constructs a teacher from the student’s starting weights by selecting an alternative layer route with labeled examples. The resulting predictions provide supervision without an external teacher model.
Figure 2: Overview of LoOPSD. (1) Teacher construction: Using a small labeled set, we try different choices of layers to repeat, then select the best-performing route to define the teacher. (2) On-policy distillation: The student generates responses using the original layer order. The frozen teacher evaluates the same response prefixes and provides token-level supervision to update the student. (3) Iteration and final model: The updated student is used to build the teacher for the next round. After KK rounds, the final student uses the original layer order.
MethodGSM8KGSM-PlusSVAMPMATH-500ProofWriter
Qwen3-1.7B
Original42.3031.7071.3333.2042.50
Elastic Reasoning46.2530.2073.0034.0045.00
CRISP76.6558.5088.3337.8043.40
LoOPSD (R1 →\to Final)78.09 →\to 79.6156.50 →\to 61.4087.67 →\to 87.6744.40 →\to 44.8042.10 →\to 56.20
LoOPSD † (R1 →\to Final)78.32 →\to 79.6859.30 →\to 61.8088.67 →\to 91.6737.20 →\to 48.2046.80 →\to 54.10
Table 1: Accuracy (%) across four models and five benchmarks, using greedy decoding with 400 thinking and 128 answer tokens. For LoOPSD variants, entries report first-round (R1) →\to final accuracy. R1 denotes the student after the first distillation round; final denotes the checkpoint obtained by the iterative continuation procedure. GSM-Plus evaluates the corresponding GSM8K-trained checkpoints.
MethodPhi-4-reasoning-plusQwen3-14B
Original89.5483.40
Elastic Reasoning89.6985.14
CRISP92.9592.57
LoOPSD (R1 →\to Final)91.89 →\to 94.9290.07 →\to 94.39
LoOPSD † (R1 →\to Final)95.07 →\to 93.1893.63 →\to 93.63
Table 2: GSM8K accuracy (%) of 14B-scale reasoning models with a 400-token thinking budget and greedy decoding. LoOPSD entries report first-round (R1) →\to final accuracy. All students use standard execution and their original prompt. LoOPSD achieves the highest final accuracy for both model families.
Figure 3: Accuracy across thinking budgets on GSM8K. (a,b) Qwen3-1.7B and DeepSeek-R1-Distill-Llama-8B students obtained separately for each target search budget and evaluated at that budget. (c) Qwen3-4B students obtained with a 400-token search budget, with each checkpoint fixed across test budgets. LoOPSD improves reasoning across a range of budgets, and fixed students adapt to different test budgets.
Figure 4: Accuracy and generation length of different methods on GSM8K with Qwen3-4B. All models use a 400-token thinking budget. (a) Accuracy versus average total generated tokens. (b) Average thinking and answer lengths. LoOPSD improves accuracy while shortening generation; LoOPSD † achieves the highest accuracy with the fewest thinking tokens.
 Total labeled examples
Method2004001100
Original42.30 (shared baseline)
Matched-label GRPO43.9043.5244.05
LoOPSD78.0178.3978.62
Table 3: Label efficiency on GSM8K with Qwen3-1.7B. LoOPSD and matched-label GRPO each receive 200 updates; label totals include 100 validation examples, and LoOPSD uses a fixed question-only distillation pool. LoOPSD already obtains most of its observed gain with 200 labels.
Figure 5: Iterative learning on GSM8K with Qwen3-4B. (a) Each round shows the looped teacher constructed from the current model and the standard-path student obtained after distillation. We compare LoOPSD, LoOPSD † , CRISP, and standard-path self-distillation across six 200-update rounds, evaluated at a 128-token thinking budget. (b) First-round and final students from the respective LoOPSD and LoOPSD † training trajectories, each held fixed across test budgets. Panel (a) includes matched-update standard-path controls under both the original and concise teacher prompts, while panel (b) shows the additional gains obtained across LoOPSD iterations.
Figure 6: Layerwise answer readout on ProofWriter with Qwen3-4B under shared contexts and a fixed output projection. Shading marks repeated teacher execution. Student gains concentrate in later layers.
Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Teacher construction on GSM8K with Qwen3-1.7B and ProofWriter with Qwen3-4B, at a 400-token thinking budget. Heatmaps show search accuracy changes in percentage points; each panel has an independent color bar. Stars mark validation-selected nonstandard routes. Validation selects a repeated-layer route under both prompts on ProofWriter, while retaining standard execution under the concise prompt on GSM8K.
Figure 8: Teacher construction on MATH-500 with Qwen3-8B and SVAMP with DeepSeek-R1-8B, using the same 400-token protocol. Heatmaps show search accuracy changes; independent bars preserve each panel’s measured range. Stars mark validation-selected nonstandard routes, which are selected under both the original and concise prompts in these two settings.
 SVAMPGSM-Plus
MethodAccuracyThink tokensAccuracyThink tokens
Qwen3-1.7B
Original71.33299.731.70378.8
Elastic Reasoning75.00294.430.20378.1
CRISP87.67251.858.50353.5
LoOPSD85.33373.061.40392.7
Appendix
Table 4: Transfer of GSM8K-trained students to additional math datasets. Accuracy (%) and mean thinking tokens at a 400-token thinking cap. All methods use standard execution, the original prompt, and greedy decoding.
Figure 9: Initial teachers and final students on GSM8K with Qwen3-1.7B. Each initial teacher is connected to the final student obtained from the corresponding training configuration. Final LoOPSD students outperform their initial teachers at all six thinking budgets.
Figure 11: Gold-label log probability across layer executions on ProofWriter with Qwen3-4B. Higher values indicate greater probability assigned to the correct label. Shading marks repeated teacher execution. LoOPSD students assign higher probability to the correct label in later layers.
 Qwen3-1.7BQwen3-4B
Method000.30.30.50.51.01.0000.30.30.50.51.01.0
Original42.3842.9941.5541.7074.9173.6275.8973.92
Elastic44.4344.1244.2043.5976.2775.6676.1275.36
CRISP76.6578.1779.2377.2687.6488.4088.0287.57
LoOPSD79.6179.4579.3877.0389.6990.3790.3791.21
LoOPSD †79.6880.5281.8880.8291.0591.5191.6691.05
Appendix
Table 5: GSM8K accuracy (%) at different thinking temperatures. We evaluate the same checkpoints with a 400-token thinking budget and a 128-token answer budget, sampling one response per question. LoOPSD improvements persist across decoding temperatures.
More in this topic

On-Policy Self-Distillation

On-policy self-distillation trains a model on its own generated trajectories using a teacher view to provide dense supervision.

See all 161 papers

Explore similar work

CardsList
Aug 6, 2026

On-Policy Self-Distillation without Any Supervision

On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for post-training large language models (LLMs). However, existing methods still rely heavily on external supervision, including ground-truth signals, environmental feedback, or guidance from larger models, and therefore fall short of genuine "self"-distillation. In this study, we show that on-policy self-distillation can be achieved using only a model's own generations via internal consistency. We propose unsupervised on-policy self-distillation (U-OPSD). U-OPSD first samples multiple rollouts and constructs a pseudo solution by majority vote under a self-consistency threshold. It then conditions the model's distribution on the pseudo-solution and distills itself on the disagreeing completions, allowing the model to correct itself precisely where it is confidently wrong. Across diverse benchmarks, base models, and training settings, U-OPSD consistently improves over the base models and matches or surpasses supervised methods with ground truth (GT) such as OPSD and GRPO. On five mathematical reasoning benchmarks, i.e., AIME24, AIME25, HMMT25, MATH500, and AMC23, U-OPSD improves over the base model by 8.5% and 10.7% on Qwen3 non-thinking mode at 4B and 8B scales, and outperforms OPSD by 3.2% and 2.3% on average, respectively. In thinking mode, U-OPSD stays on par with OPSD, ahead by 0.9% at 4B and level at 8B and surpassing GRPO by 0.7% and 1.1%, respectively. Code is available at https://github.com/williamium3000/u-opsd.

Yijiang Li, Bingyang Wang, Yijun Liang +3

UC San Diego · Georgia Institute of Technology · University of Maryland, College Park +1

On-Policy Self-DistillationLanguage Model Self-ImprovementSelf-Consistency in Language Models
AskPDF
Sep 25, 2026

Recursive Self-Improvement via On-Policy Distillation for Reasoning

On-policy distillation (OPD) trains a student model by having it generate trajectories, then matching its next-token predictions with an external teacher's next-token predictions. This provides dense, token-level supervision to the student. On-policy self-distillation (OPSD) eliminates the need for the external teacher. Specifically, a second frozen copy of the student model, now given the ground truth in its context, serves as the teacher. The student model only receives the problem and learns to mimic the privileged teacher model, while the teacher remains frozen throughout training. Previous work showed that freezing the teacher is useful for training stability, but we argue that this can prevent the teacher from incorporating the improvements learned by the student during training. Our primary contribution is to address this limitation with a recursive framework built around two complementary components. First, we let the privileged teacher co-evolve with the student so that revision learned in one round can guide the next, a process we refer to as Dynamic Co-Evolution (DCE). Second, because stronger revision can also make responses too verbose and self-critical, we additionally train on shorter, verified rewrites of the model's own on-policy responses. We call this complementary objective Self-Refined Concise Learning (SRCL). Overall, our comprehensive evaluations show that DCE+SRCL outperforms OPSD across multiple model scales and four competition-level mathematics benchmarks. Specifically, on Qwen3-8B, DCE+SRCL reaches 65.97% Average@12, outperforming OPSD by 35.62 percentage points while reducing mean output length by 7.80% relative to DCE alone.

Shangjian Yin, Zehao Zhao, Kavosh Asadi +7

Meta AI · University of California, Riverside · Work done at Meta

Language Model Self-ImprovementOn-Policy Self-DistillationSelf-DistillationMathematical Reasoning
AskPDF
Sep 27, 2026

Teach Yourself Where to Look: On-Policy Attention Self-Distillation for Reasoning

On-policy self-distillation trains reasoning models on their own trajectories using dense token distribution guidance from a privileged teacher with access to a verified solution. This supervision transfers what the teacher predicts without directly transferring where it attends within the preceding context. We introduce On-Policy Attention Self-Distillation (OPASD), which complements token-level supervision with solution-conditioned attention distillation. Because the privileged teacher can attend to verified solution tokens unavailable to the student, OPASD projects teacher attention onto student-visible positions and renormalizes the resulting distribution before alignment. Across three model sizes and four competition-level mathematics benchmarks, OPASD consistently outperforms token-only OPSD, improving average accuracy by 4.98 to 8.40 percentage points. OPASD also avoids the response-length inflation and performance degradation observed with token-only distillation, reducing generated rollout tokens by 73.9% and estimated model compute by 72.6% while training 1.53x faster. These results show that solution-conditioned attention provides a complementary supervision signal that makes on-policy self-distillation more accurate, stable, and compute-efficient.

Safaeid Hossain Arib, Rabeya Akter, Ismam Nur Swapnil +3

ACI PLC, Bangladesh · University of Dhaka, Bangladesh · BRAC University, Bangladesh +2

Numerical Reasoning in Language ModelsAttention-Guided DistillationOn-Policy Self-DistillationKnowledge Distillation
AskPDF