Current post-training methods in verifiable settings fall into two categories. Reinforcement learning (RLVR) relies on binary rewards, which are broadly applicable and powerful, but provide only sparse supervision during training. Distillation provides dense token-level supervision, typically obtained from an external teacher or using high-quality demonstrations. Collecting such supervision can be costly or unavailable. We propose Self-Distillation Zero (SD-Zero), a method that is substantially more training sample-efficient than RL and does not require an external teacher or high-quality demonstrations. SD-Zero trains a single model to play two roles: a Generator, which produces an initial response, and a Reviser, which conditions on that response and its binary reward to produce an improved response. We then perform on-policy self-distillation to distill the reviser into the generator, using the reviser's token distributions conditioned on the generator's response and its reward as supervision. In effect, SD-Zero trains the model to transform binary rewards into dense token-level self-supervision. On math and code reasoning benchmarks with Qwen3-4B-Instruct and Olmo-3-7B-Instruct, SD-Zero improves performance by at least 10% over the base models and outperforms strong baselines, including Rejection Fine-Tuning (RFT), GRPO, and Self-Distillation Fine-Tuning (SDFT), under the same question set and training sample budget. Extensive ablation studies show two novel characteristics of our proposed algorithm: (a) token-level self-localization, where the reviser can identify the key tokens that need to be revised in the generator's response based on reward, and (b) iterative self-evolution, where the improving ability to revise answers can be distilled back into generation performance with regular teacher synchronization. Code: https://github.com/princeton-pli/Self-Distillation-Zero.
Figures & tables
Figure 1: We introduce SD-Zero , an on-policy self-distillation paradigm where the teacher ( reviser ) can condition on incorrect student ( generator ) attempts.
Figure 2: Overview of SD-Zero . In Phase 1 ( SRT ) , we collect 6K outcome-conditioned self-revision traces by sampling an initial response from the base model, prompting the model to self-revise its incorrect response, and keeping the correct self-revision. In Phase 2 (Self-Distillation) , we conduct on-policy self-distillation with the self-revise ( SRT ) model acting as both student and teacher: the student generates an on-policy response, and the teacher generates a revised version conditioned on that response and its outcome reward. Throughout the Self-Distillation phase, the model bootstraps performance through internalizing self-revision behavior. See Algorithm 1 for details.
Math
Code
Avg.
AIME24
AIME25
HMMT25
AMOBench
OpenR1
MATH
Codeforces
LCB
Qwen3-4B-Instruct
59.6
45.8
26.7
9.8
55.8
91.0
48.0
61.8
49.8
SFT
61.7
46.3
31.3
7.3
56.1
91.3
49.1
57.2
50.0
RFT
64.2
52.1
37.1
11.3
59.3
91.9
50.9
68.0
54.3
GRPO
62.5
50.0
30.4
11.0
62.9
93.5
52.2
62.6
53.1
SDFT
63.3
47.9
32.9
9.0
57.0
91.1
49.3
59.2
51.2
Table 1: Performance comparison of SD-Zero and SRT (Self-Revision Training, Phase 1) against baseline post-training methods on math and code reasoning benchmarks, reported as avg@8. Across both Qwen3-4B-Instruct and Olmo-3-7B-Instruct, SRT and especially SD-Zero achieve the strongest overall results; “LCB” denotes LiveCodeBench. All methods are compute equalized (see Appendix C.2 for more details).
Figure 3: Comparison of outcome-conditioned self-revision capability on AIME24, Qwen3-4B-Instruct. SRT unlocks self-revision behaviors, while final SD-Zero model preserves this advantage while improving token efficiency.
Figure 4: Reviser converts binary outcome reward into dense token-level reward. Left: Comparison of token-level KL reward distributions for correct ( r=1 ) and incorrect ( r=0 ) student generations. Incorrect trajectories concentrate larger rewards on a small number of tokens, whereas correct trajectories receive a flatter reward distribution that mainly preserves the response. Right: Visualization of token-level KL rewards for an incorrect student generation ( r=0 ). The teacher policy converts binary reward r=0 to dense rewards that localize mistake-relevant tokens and guide targeted correction.
Figure 5: Self-evolved reasoning through teacher synchronization in SD-Zero . SD-Zero can iteratively improve the model by reusing its own learned self-revision behavior as supervision.
Figure 6: Evolution of self-revision behavior across training phases for Qwen3-4B-Instruct on OpenR1-Math. Left: Example reasoning traces from different model phases. The base model fails without revision, whereas the SRT model performs explicit self-revision and the SD-Zero model exhibits more internalized self-guidance, identifying pitfalls and directing itself to the correct answer. Right: Training dynamics of self-revision behavior. Response length and explicit self-revision keywords rise during the SRT phase and fall during the Self-Distillation phase, indicating a shift from overt revision to more internalized, token-efficient reasoning.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Pr←{“Let me rephrase the above solution.”,“Wait, this response is not correct, let me start over.”,r=1,r=0
Appendix
Algorithm 1 SD-Zero Training Pipeline
Method
Sampling
Signal
Teacher
Teacher can condition on wrong attempt
SFT / Distillation
× off-policy
✓ dense
× external
—
On-Policy Distillation
✓ on-policy
✓ dense
× external
—
RLVR (e.g., GRPO)
✓ on-policy
× sparse
—
—
OPSD / SDFT / SDPO
✓ on-policy
✓ dense
✓ self
×
SD-Zero (ours)
✓ on-policy
✓ dense
✓ self
✓
Appendix
Table 2: Comparison of post-training methods for LLMs. SD-Zero is the only self-distillation method whose teacher can condition on incorrect student attempts, enabling it to convert sparse binary rewards into dense token-level supervision without requiring gold demonstrations.
Method
# Training
# Training rollouts
# Training
Total
questions
per question
epochs
# generations
RFT
15K
-
1
60K
GRPO
15K
4
1
60K
SDFT
15K
4
1
60K
SRT phase
6K
-
1
40K
Self-Distillation phase
9K
1
1
9K
Appendix
Table 3: Comparison of training data and generation budgets across different training paradigms. For RFT, GRPO, and SDFT, we assume 15K training questions and 4 sampled rollouts per question, resulting in 60K total generations before any filtering. In our method, the SRT phase starts from 10K seed questions, generates 1 initial response and 3 self-revisions per question (40K generations in total), and retains 6K correct self-revision traces for training. The Self-Distillation phase directly uses 9K additional question-answer pairs, requiring 9K generations. In total, SD-Zero uses 15K training questions and 49K generations.
Parameters
SFT
General
Model
Qwen/Qwen3-4B-Instruct-2507
Thinking
False
Data
Prompt format
Chat template (system + user)
Completion-only loss
True
Appendix
Table 4: Hyperparameters used for SFT/ RFT baselines and SD-Zero SRT phase on Qwen3-4B-Instruct.
Parameters
GRPO
General
Model
Qwen/Qwen3-4B-Instruct-2507
Thinking
False
Data
Max. prompt length
1024
Max. response length
8192
Appendix
Table 5: Hyperparameters used in GRPO baseline on Qwen3-4B-Instruct.
Parameters
SD-Zero
SDFT
General
Policy Model
Qwen/Qwen3-4B-Instruct-2507
Qwen/Qwen3-4B-Instruct-2507
Teacher model
Qwen/Qwen3-4B-Instruct-2507
Qwen/Qwen3-4B-Instruct-2507
Thinking
False
False
Data
Max. teacher prompt length
32768
32768
Appendix
Table 6: Hyperparameters used for SD-Zero Self-Distillation phase and SDFT on Qwen3-4B-Instruct.
AIME24
AIME25
HMMT25
AMOBench
OpenR1
MATH
Average
Qwen3-4B-Instruct
76.7
76.7
50.0
30.0
69.2
97.6
66.7
SFT
83.3
73.3
46.7
18.0
69.2
97.0
64.6
RFT
86.7
80.0
53.3
30.0
70.2
98.4
69.8
GRPO
80.0
70.0
46.7
28.0
70.0
98.0
65.4
SDFT
80.0
70.0
50.0
26.0
70.0
97.0
65.5
SRT (Ours)
86.7
80.0
60.0
28.0
70.2
97.6
70.4
Appendix
Table 7: Pass@8 performance comparison of SD-Zero and SRT (Self-Revision Training, Phase 1) against baseline post-training methods on math reasoning benchmarks. Across both Qwen3-4B-Instruct and Olmo-3-7B-Instruct, SRT and especially SD-Zero achieve the strongest overall results.
AIME24
AIME25
HMMT25
AMOBench
OpenR1
MATH
Avg.
Base model
59.6
45.8
26.7
9.8
55.8
91.0
48.1
SDFT
63.3
47.9
29.9
9.0
57.0
91.1
49.7
SDFT (final answers only)
62.5
47.1
29.9
9.3
57.0
91.0
49.5
SD-Zero (Ours)
68.3
60.0
45.4
16.0
60.4
93.6
57.3
Appendix
Table 8: SDFT with final-answer-only supervision on Qwen3-4B-Instruct evaluated on math benchmarks. When given access only to final answers rather than gold solutions, SDFT performs only marginally better than the base model and remains far below SD-Zero , showing that SD-Zero is much more effective under outcome-only supervision.
Method
First Attempt
Revised Attempt
First Attempt
Revised Attempt
Correction
Tokens
Tokens
Accuracy (%)
Accuracy (%)
Rate (%)
Base Model
3708
5098
59.6
60.7
2.7
GRPO
4432
5499
65.2
66.9
4.9
SDFT
3630
5099
63.3
64.7
3.8
SRT
8458
8137
66.7
71.7
15.0
SD-Zero
3518
3314
68.3
73.6
16.7
Appendix
Table 9: Comparison of self-revision effectiveness and token costs across different models on AIME24 with Qwen3-4B-Instruct as the base model. The Correction Rate is computed as (Revised Attempt Accuracy−First Attempt Accuracy)/(100−First Attempt Accuracy) , i.e., the net fraction of initially incorrect answers corrected after revision.
AIME24
AIME25
HMMT25
AMOBench
OpenR1
MATH
Avg.
GRPO (4 generations, 1 epoch)
62.5
50.0
30.4
11.0
62.9
93.5
51.7
GRPO (8 generations, 0.5 epoch)
58.8
50.0
30.4
9.8
61.5
93.0
50.6
GRPO (8 generations, 1 epoch)
61.7
52.1
31.3
12.0
63.4
93.1
52.3
SD-Zero (Ours)
68.3
60.0
45.4
16.0
60.4
93.6
57.3
Appendix
Table 10: Comparison with GRPO under larger rollout budgets on Qwen3-4B-Instruct evaluated on math benchmarks. We compare SD-Zero against GRPO variants that use more sampled generations per prompt and different effective training budgets. Even when GRPO is given substantially more exploration through 8 generations, its gains remain limited, whereas SD-Zero achieves the best average performance and outperforms all GRPO settings on most benchmarks. This suggests that the advantage of our method comes not simply from sampling more trajectories, but from turning binary outcome feedback into more informative self-revision-based supervision.
Method
Average Generation
Generate-then-Revise on AIME24
Accuracy (%)
First Attempt
Revised Attempt
Correction
Accuracy (%)
Accuracy (%)
Rate (%)
Base Model
49.8
59.6
60.7
2.7
SRT
57.6
66.7
71.7
15.0
SRT ( Lgeneration Only)
56.4
65.4
67.9
7.2
SRT ( Lrevision Only)
52.2
62.1
66.7
12.1
Appendix
Table 11: Ablation of the two loss terms in the SRT objective. We train Qwen3-4B-Instruct with SRT using only Lgeneration or only Lrevision , and evaluate both overall generation quality and self-revision ability. Both terms are important: using either term alone underperforms the full SRT objective in average generation accuracy and in generate-then-revise performance. In particular, Lgeneration alone yields relatively weak correction after revision, while Lrevision alone improves correction rate but leads to substantially worse first-attempt and overall generation accuracy, showing that the two terms play complementary roles in strengthening both generation and self-revision.
Method
Average Generation
Generate-then-Revise on AIME24
Accuracy (%)
First Attempt
Revised Attempt
Correction
Accuracy (%)
Accuracy (%)
Rate (%)
Base Model
49.8
59.6
60.7
2.7
SRT
57.6
66.7
71.7
15.0
SD-Zero
60.3
68.3
73.6
16.7
SD-Zero (Phase 2 Only)
51.4
61.2
62.2
2.6
Appendix
Table 12: Ablation of applying SD-Zero without the SRT phase. We train a Phase-2-only variant on Qwen3-4B-Instruct by running SD-Zero directly on the base model, without first inducing self-revision through SRT . This variant shows only marginal improvement over the base model in both overall generation accuracy and generate-then-revise performance, and its correction rate remains nearly unchanged. In contrast, the full two-phase method substantially improves both first-attempt generation and revision effectiveness. These results suggest that the SRT phase is necessary to first elicit self-revision capability, which SD-Zero can then refine and distill into stronger reasoning.
Data Split
Average Performance (%)
SRT
Self-Distillation
SRT Model
SD-Zero Model
6K
9K
57.6
60.3
9K
6K
57.8
59.1
7.5K
7.5K
57.8
59.8
Appendix
Table 13: Ablation of data split between SRT and Self-Distillation phases. On Qwen3-4B-Instruct, we vary how the training data is allocated across the two phases while keeping the total data budget fixed. Allocating more data to SRT slightly improves SRT model performance, but does not lead to the best final SD-Zero accuracy. The best overall result is achieved by assigning more data to Self-Distillation, suggesting that once SRT has sufficiently elicited self-revision capability, additional data is more effectively used in Self-Distillation phase to distill and refine this behavior into stronger generation.
Method
AIME24
AIME25
HMMT25
Qwen3-4B
0.735
0.647
0.458
+ SDFT ( enable_thinking=True )
0.637
0.539
0.375
+ SDFT ( enable_thinking=False )
0.733
0.616
0.466
Appendix
Table 14: Effect of the student’s thinking mode on SDFT training (Qwen3-4B, avg@16). enable_thinking refers to whether thinking is active during training; all models are evaluated with thinking enabled. Red entries denote degradation relative to the base model.
Training Source
pass@1
pass@2
pass@4
No training
0.410
0.538
0.639
LLaMA-70B (6.5K, filtered)
0.605
0.725
0.814
Qwen2.5-7B (6.2K, filtered)
0.552
0.676
0.768
GPT-4o (5.4K, filtered)
0.617
0.711
0.775
SFT on GRPO data (8K, unfiltered)
0.878
0.904
0.922
SFT on GRPO data (6.5K, unfiltered)
0.883
0.910
0.919
Appendix
Table 15: Countdown pass@ k after fine-tuning Qwen2.5-7B on different data sources. On-policy sources (SFT on GRPO data, self-revision) consistently outperform off-policy ones (LLaMA-70B, GPT-4o). Among methods that do not require a prior RL stage, filtered self-revision achieves the highest pass@1.
AIME24
AIME25
Countdown
IFBench
IFEval
MMLU
MMLU-Pro
Method
@1
@128
@1
@128
@1
@128
@1
@128
@1
@128
@1
@128
@1
@128
Base
.07
.27
.07
.43
.41
.90
.27
.58
.71
.90
.74
.91
.56
.82
GRPO (Countdown)
.08
.30
.08
.40
.91
.98
.27
.61
.72
.91
.74
.92
.56
.84
Countdown (A)
.05
.27
.05
.43
.39
.91
.23
.54
.52
.85
.69
.96
.46
.91
Countdown (A) + GRPO
.06
.23
.07
.40
.92
.98
.24
.55
.53
.84
.71
.97
.50
.93
Countdown (A ′ )
.07
.30
.06
.47
.40
.88
.19
.62
.60
.91
.71
.97
.50
.94
Appendix
Table 16: OOD evaluation (pass@1 / pass@128) after training on Countdown with LLaMA-70B answers (A) or self-revised answers (A ′ ), optionally followed by GRPO. A ′ +GRPO achieves the highest OOD scores.
Reinforcement learning with verifiable rewards (RLVR) provides reliable outcome supervision for language model reasoning, but scalar trajectory rewards offer limited guidance for token-level credit assignment. Existing self-distillation methods incorporate privileged teacher information through fixed strategies: direct distribution matching may destabilize successful behavior, while magnitude modulation provides limited corrective guidance after failure. We propose Hybrid Hindsight Self-Distillation (H2SD), which adapts both teacher context and update strategy to trajectory correctness. For successful trajectories, the teacher conditions on the verified student response and a rephrasing instruction to rescore the original tokens. These scores modulate reward-derived updates while preserving advantage signs. For failed trajectories, a verified reference hint guides corrective supervision through reverse KL distillation. Across Sudoku, Calcudoku, Arrow Maze, and mathematical reasoning with two Qwen3 models, H2SD achieves an overall average accuracy of 43.33%, outperforming the strongest baseline by 11.43 percentage points while maintaining stable optimization.
Qiye Cai, Yichuan Ma, Weitao Ma +6
Shanghai Artificial Intelligence Laboratory · Harbin Institute of Technology · Fudan University +1
Reinforcement learning with verifiable rewards (RLVR) has become a standard paradigm for improving large language models (LLMs) on various tasks, yet its sparse outcome rewards lack token-level credit assignment for intermediate steps. To address this, on-policy self-distillation (OPSD) leverages a self-teacher with privileged context to provide additional dense learning signals. However, because the self-teacher is often overconfident and imposes excessive penalties on long reasoning trajectories, OPSD frequently struggles in practice. To mitigate this, we propose self-instructing policy optimization (SIPO) with a contrastive self-teacher to provide dense credit. At each iteration, SIPO samples multiple rollouts per prompt from the current policy, scores them with environment rewards, and constructs two teacher contexts for each rollout by pairing the reference answer with mistakes made within the group. The model then re-evaluates its own responses under both contexts, using the difference between the two teacher log-probabilities as token-level feedback, so that biases shared by both contexts are expected to largely cancel. The resulting objective yields a token-level advantage for every rollout: the reward still sets the main direction of each update while the self-teacher redistributes credit across tokens. Even in groups where every rollout fails and group-relative advantages vanish, SIPO still provides a learning signal. By preserving direct optimization of the task reward while providing dense, token-level feedback, this approach bridges reinforcement learning and on-policy self-distillation. Extensive experiments across multiple reasoning and code-generation benchmarks demonstrate that SIPO outperforms both RLVR and OPSD baselines without an external teacher or additional generation.
Zhenrui Yue, Huimin Zeng, Yueqi Wang +8
University of Illinois · UC San Diego · Rochester Institute of Technology +2
Can post-trained large language models (LLMs) further improve themselves using only unlabeled prompts, without external teachers or feedback from tools? We study this setting starting only from unlabeled seed questions with no ground-truth solutions, across three reasoning domains: math, science, and coding. We propose Self-Verified Distillation, a simple post-training refinement algorithm in which the model generates candidate solutions to these seed questions, filters them using prompt-based self-verification, and trains on the resulting self-curated dataset. Inspired by the UQ benchmark's use of multiple validators to screen candidate answers to hard unsolved questions, we adapt this validation-based filtering idea to self-training: the model filters its own generated solutions through a three-stage cascade of cycle-consistency, factuality, and correctness checks, accepting a solution only if it passes all stages with unanimous judge votes. We find that sampling more candidate generations and using a larger verification budget during training data construction produces higher-quality self-curated data and, in turn, better reasoning models. We then train Qwen3 models at multiple scales with Self-Verified Distillation and obtain gains across all three domains. For Qwen3-4B, our method improves aggregate held-out pass@1 by +16.7 points in math (AIME26 and HMMT), +11.1 points in science (GPQA Diamond and HLE), and +8.3 points in coding (LCBv5 and LCBv6), with gains also extending to 0.6B and 8B models. Compared to our test-time-only baseline (UQ-TTC), which improves performance by spending extra compute at inference time, Self-Verified Distillation achieves better performance in most settings while requiring only a single inference call at test time.