On-policy self-distillation provides dense supervision for reasoning models via a privileged self-teacher. However, this supervision reflects local token preferences rather than the expected return of an unguided student continuation. Recent methods combine this signal with GRPO. They use GRPO for update directions and the teacher for signal scaling. However, we demonstrate a more effective integration within PPO. Specifically, we identify complementary positional dynamics: privileged information enhances a critic's value-prediction accuracy toward the response's end, whereas self-teacher guidance is strong initially, drops mid-generation, and rebounds finally. These insights motivate Value-Augmented Self-Distillation Policy Optimization (VASPO), wherein a self-teacher and critic evaluate identical privileged information at student-generated prefixes. The critic computes token-level PPO advantages from predicted continuation returns, while the self-teacher modulates selected positive advantages via bounded probability ratios. Across six mathematical reasoning benchmarks, VASPO improves mean accuracy over PPO by 2.59 and 1.81 percentage points on Qwen3-4B and OLMo3-7B variants, respectively, and achieves the highest mean accuracy among evaluated methods on Knights and Knaves. Our code is available at https://github.com/cii030/VASPO.
Figures & tables
Figure 1: Training signals in GRPO, PPO, OPSD, and VASPO. VASPO shares privileged information for token guidance and value estimation.
Figure 2: Privileged signals by relative response position. All responses: (a) solution-induced reduction in critic squared error (positive is better); (b) DKL(p∥q) between the clean actor and privileged self-teacher. Shading shows 95% confidence intervals.
Figure 3: Overview of VASPO training and deployment. Shared privileged information (PI) supplies critic values for GAE and teacher weights for advantage modulation. The actor remains prompt-only.
Figure 4: Training dynamics of Qwen3-4B-Instruct-2507. Top: DeepMath training reward, AIME 2024 and AIME 2025 avg@16 accuracy, and critic explained variance. Bottom: KK training reward, 4–8-role and 11-role avg@4 accuracy, and critic explained variance.
Method
AIME24
AIME25
AIME26
HMMT25
Minerva
Olympiad
Avg.
avg@32
avg@8
Qwen3-4B-Instruct-2507
Vanilla
51.04
40.83
48.85
25.83
60.43
71.55
49.76
DAPO
53.85
42.08
49.90
28.33
62.59
72.81
51.59
PPO
51.67
42.71
48.65
26.35
63.14
70.96
50.58
OPSD
29.58
26.46
31.77
12.29
54.14
54.56
34.80
Table 1: Mathematical reasoning accuracy (%). Bold/underlined: best/second-best available scores per backbone.
ID
OOD
Method
4–8 Roles
9 Roles
10 Roles
11 Roles
Avg.
Vanilla
72.99
67.94
54.44
37.81
58.30
DAPO
73.71
68.81
57.00
40.00
59.88
PPO
74.33
65.59
57.69
39.14
59.19
RLSD
73.54
70.19
57.50
40.12
60.34
VASPO
75.31
69.25
58.06
40.62
60.81
Table 2: Knights and Knaves accuracy (avg@16, %) on Qwen3-4B-Instruct-2507. Avg.: mean across the four role groups. Bold/underlined: best/second-best.
Mathematical Reasoning
Logical Reasoning: Knights and Knaves
Method
AIME2024
AIME2025
AIME2026
Olympiad
Avg.
4–8 Roles
9 Roles
10 Roles
11 Roles
Avg.
VASPO (full)
56.87
43.96
51.15
73.12
56.28
75.31
69.25
58.06
40.62
60.81
− privileged critic
29.58
26.46
31.77
54.56
35.59
47.20
25.19
15.06
5.75
23.30
− self-teacher modulation
55.73
43.44
50.73
73.07
55.74
73.95
69.25
56.13
39.25
59.65
− both reader components
51.67
42.71
48.65
70.96
53.50
74.33
65.59
57.69
39.14
59.19
− token-level mask
55.41
41.56
50.42
72.40
54.95
74.84
67.19
56.94
39.44
59.60
Table 3: Ablation of critic, self-teacher and privileged-information content in VASPO (Qwen3-4B-Instruct-2507; accuracy, %). Rows denote changes relative to the full method. Mathematics uses avg@32 on AIME2024/2025/2026 and avg@8 on OlympiadBench; KK uses avg@16, with 4–8 roles and 9/10/11 roles.
Figure 5: Fusion strength on Qwen3-4B-Instuct-2507. AIME 2024 and AIME 2025 avg@16 accuracy over DeepMath training steps. The probability-ratio clip width ϵw=0 disables teacher modulation.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Critic pretraining dynamics. Logged loss and explained variance for clean, answer-conditioned, and solution-conditioned critics. Panels (a)–(b): mathematics, with 12 recorded checkpoints from step 5 to 555. Panels (c)–(d): KK, with nine recorded checkpoints from step 5 to 85. Points show recorded values, and solid lines connect consecutive observations without additional smoothing.
Table 4: Dataset and model assets. Licenses and question counts describe the linked distributions. “Not specified” means that the linked distribution does not state a license.
Parameter
Value
Training framework
verl
Training hardware
8× H200 GPUs
Parameter updates
Full-parameter
Optimizer
AdamW
Weight decay
0.01
Critic pretraining epochs
2
Appendix
Table 5: Training and decoding configuration for VASPO and matched reader ablations.
Figure 7: Training-time profile of VASPO with Qwen3-4B-Instruct-2507. Mean recorded time in seconds per step over 200 training steps. Bars show individual component timers; the separately recorded end-to-end step time is 412.28 s. The component timers are not an additive decomposition of this total.
Setting
Value or procedure
Critic architecture
Actor backbone with a scalar value head; matched initialization across clean and privileged variants
Critic regression
Monte Carlo terminal return; PPO clipped value loss
Critic update schedule
Pre-update once per rollout batch, then re-score values and recompute GAE before fusion
Fusion
Full clipped probability-ratio reweighting; ϵw=0.2
Token selection
Raw Atz>0 and ∣tanh(dt/τ)∣>δ ; τ=δ=0.02
Student probability source
Behavior policy
Appendix
Table 6: Critic, teacher, and policy-update settings for VASPO. Matched controls retain the applicable settings when a reader is disabled. Optional implementation variants are described in Appendix F .
Figure 8: A pooled trend hides different response groups. Saved-response critic error reduction (left) and teacher KL (right). Shading shows pointwise 95% problem-bootstrap intervals; n counts problems, which can occur in both correctness groups. There are 6,659 correct and 1,629 incorrect responses (8,288 total). Positive critic values favor solution conditioning; teacher KL measures disagreement, not useful guidance.
Figure 9: Prediction accuracy and calibration need not improve together. Three existing critics evaluated on the same 128 problems, 384 prefixes, and 3,072 independent continuations. Markers and bars show estimates and pointwise 95% problem-bootstrap intervals. Suffix labels agree with full-response rescoring for all 3,072 continuations. Arrows indicate the better direction.
Figure 10: Later critic gains depend on the source response. Clean-minus-solution Brier improvement by source correctness (a) and length (b), with paired positional contrasts C in (c,d). Positive values favor the solution critic. Bars are pointwise 95% problem-bootstrap intervals; n counts problems. Horizontal offsets in (a,b) separate overlapping points; the sampled positions are 20%, 50%, and 80%. Each row shares axis limits. Length strata use fit-split tertiles (1,008 and 1,779.33 tokens, upper bounds included). All evaluation labels agree with full-response rescoring.
Self-distillation is an emerging approach that trains a model to imitate its own predictions when conditioned on privileged information, such as correct answers or environmental feedback. Recent methods have reported better sample efficiency and generalization than reinforcement learning (RL), but it remains unclear whether these gains extend beyond in-distribution evaluations and whether they stem from privileged distillation. In this paper, we revisit two representative methods, Self-Distillation Policy Optimization (SDPO) and On-Policy Self-Distillation (OPSD), across code, science, tool use, and math reasoning tasks. We find that their reported gains do not consistently generalize to unseen problems and are largely explained by overfitting or transferring existing reasoning behavior, rather than effectively using privileged information. Moreover, self-distillation gradients show weak alignment with policy gradients that directly optimize expected reward, questioning whether they provide an effective direction for policy improvement. Our experiments across Qwen3 models (1.7B-14B) and OLMo-3-7B suggest that effectively leveraging privileged information to improve reasoning remains an open challenge. Our code is available at https://github.com/sleepypandu/does-sd-improve-reasoning
Jason Ken Adhinarta, Raina Wu, Ashutosh Sharma +5
Massachusetts Institute of Technology · MIT-IBM Computing Research Lab
Reinforcement learning with verifiable rewards (RLVR) has become a standard paradigm for improving large language models (LLMs) on various tasks, yet its sparse outcome rewards lack token-level credit assignment for intermediate steps. To address this, on-policy self-distillation (OPSD) leverages a self-teacher with privileged context to provide additional dense learning signals. However, because the self-teacher is often overconfident and imposes excessive penalties on long reasoning trajectories, OPSD frequently struggles in practice. To mitigate this, we propose self-instructing policy optimization (SIPO) with a contrastive self-teacher to provide dense credit. At each iteration, SIPO samples multiple rollouts per prompt from the current policy, scores them with environment rewards, and constructs two teacher contexts for each rollout by pairing the reference answer with mistakes made within the group. The model then re-evaluates its own responses under both contexts, using the difference between the two teacher log-probabilities as token-level feedback, so that biases shared by both contexts are expected to largely cancel. The resulting objective yields a token-level advantage for every rollout: the reward still sets the main direction of each update while the self-teacher redistributes credit across tokens. Even in groups where every rollout fails and group-relative advantages vanish, SIPO still provides a learning signal. By preserving direct optimization of the task reward while providing dense, token-level feedback, this approach bridges reinforcement learning and on-policy self-distillation. Extensive experiments across multiple reasoning and code-generation benchmarks demonstrate that SIPO outperforms both RLVR and OPSD baselines without an external teacher or additional generation.
Zhenrui Yue, Huimin Zeng, Yueqi Wang +8
University of Illinois · UC San Diego · Rochester Institute of Technology +2
On-policy distillation is an efficient alternative to reinforcement learning, offering dense token-level training signals. However, its reliance on a stronger external teacher has driven recent work on on-policy self-distillation, where the same model serves as both teacher and student under different prompt contexts. Yet, existing self-distillation methods largely reduce learning to KL matching toward the context-augmented teacher model. This approach often suffers from training instability and can degrade reasoning performance over time. Moreover, self-distillation from the same model with prompt augmentation lacks the exploratory diversity provided by a genuine external teacher. To address these limitations, we move beyond fixed-teacher KL matching and propose Preference-Based Self-Distillation (PBSD), which revisits on-policy self-distillation through a reward-regularized perspective. Instead of directly matching the teacher distribution, we derive a reward-regularized objective whose analytic optimum is a reward-reweighted teacher distribution, yielding a target policy provably superior to the original teacher under this objective. Practically, PBSD optimizes preference gaps between teacher and student samples while maintaining on-policy student sampling. We support this framework with a statistical analysis of the induced preference-learning problem, formally establishing when on policy self-distillation is preferable to learning from an external teacher in our setting. Experiments on mathematical reasoning and tool-use benchmarks across multiple model scales demonstrate that PBSD consistently achieves the strongest average performance among comparable baselines, showing improved training stability over prior self-distillation baselines while preserving token efficiency.