On-policy self-distillation provides dense supervision for reasoning models via a privileged self-teacher. However, this supervision reflects local token preferences rather than the expected return of an unguided student continuation. Recent methods combine this signal with GRPO. They use GRPO for update directions and the teacher for signal scaling. However, we demonstrate a more effective integration within PPO. Specifically, we identify complementary positional dynamics: privileged information enhances a critic's value-prediction accuracy toward the response's end, whereas self-teacher guidance is strong initially, drops mid-generation, and rebounds finally. These insights motivate Value-Augmented Self-Distillation Policy Optimization (VASPO), wherein a self-teacher and critic evaluate identical privileged information at student-generated prefixes. The critic computes token-level PPO advantages from predicted continuation returns, while the self-teacher modulates selected positive advantages via bounded probability ratios. Across six mathematical reasoning benchmarks, VASPO improves mean accuracy over PPO by 2.59 and 1.81 percentage points on Qwen3-4B and OLMo3-7B variants, respectively, and achieves the highest mean accuracy among evaluated methods on Knights and Knaves. Our code is available at https://github.com/cii030/VASPO.
Figures & tables
Figure 1: Training signals in GRPO, PPO, OPSD, and VASPO. VASPO shares privileged information for token guidance and value estimation.
Figure 2: Privileged signals by relative response position. All responses: (a) solution-induced reduction in critic squared error (positive is better); (b) DKL(p∥q) between the clean actor and privileged self-teacher. Shading shows 95% confidence intervals.
Figure 3: Overview of VASPO training and deployment. Shared privileged information (PI) supplies critic values for GAE and teacher weights for advantage modulation. The actor remains prompt-only.
Figure 4: Training dynamics of Qwen3-4B-Instruct-2507. Top: DeepMath training reward, AIME 2024 and AIME 2025 avg@16 accuracy, and critic explained variance. Bottom: KK training reward, 4–8-role and 11-role avg@4 accuracy, and critic explained variance.
Method
AIME24
AIME25
AIME26
HMMT25
Minerva
Olympiad
Avg.
avg@32
avg@8
Qwen3-4B-Instruct-2507
Vanilla
51.04
40.83
48.85
25.83
60.43
71.55
49.76
DAPO
53.85
42.08
49.90
28.33
62.59
72.81
51.59
PPO
51.67
42.71
48.65
26.35
63.14
70.96
50.58
OPSD
29.58
26.46
31.77
12.29
54.14
54.56
34.80
Table 1: Mathematical reasoning accuracy (%). Bold/underlined: best/second-best available scores per backbone.
ID
OOD
Method
4–8 Roles
9 Roles
10 Roles
11 Roles
Avg.
Vanilla
72.99
67.94
54.44
37.81
58.30
DAPO
73.71
68.81
57.00
40.00
59.88
PPO
74.33
65.59
57.69
39.14
59.19
RLSD
73.54
70.19
57.50
40.12
60.34
VASPO
75.31
69.25
58.06
40.62
60.81
Table 2: Knights and Knaves accuracy (avg@16, %) on Qwen3-4B-Instruct-2507. Avg.: mean across the four role groups. Bold/underlined: best/second-best.
Mathematical Reasoning
Logical Reasoning: Knights and Knaves
Method
AIME2024
AIME2025
AIME2026
Olympiad
Avg.
4–8 Roles
9 Roles
10 Roles
11 Roles
Avg.
VASPO (full)
56.87
43.96
51.15
73.12
56.28
75.31
69.25
58.06
40.62
60.81
− privileged critic
29.58
26.46
31.77
54.56
35.59
47.20
25.19
15.06
5.75
23.30
− self-teacher modulation
55.73
43.44
50.73
73.07
55.74
73.95
69.25
56.13
39.25
59.65
− both reader components
51.67
42.71
48.65
70.96
53.50
74.33
65.59
57.69
39.14
59.19
− token-level mask
55.41
41.56
50.42
72.40
54.95
74.84
67.19
56.94
39.44
59.60
Table 3: Ablation of critic, self-teacher and privileged-information content in VASPO (Qwen3-4B-Instruct-2507; accuracy, %). Rows denote changes relative to the full method. Mathematics uses avg@32 on AIME2024/2025/2026 and avg@8 on OlympiadBench; KK uses avg@16, with 4–8 roles and 9/10/11 roles.
Figure 5: Fusion strength on Qwen3-4B-Instuct-2507. AIME 2024 and AIME 2025 avg@16 accuracy over DeepMath training steps. The probability-ratio clip width ϵw=0 disables teacher modulation.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Critic pretraining dynamics. Logged loss and explained variance for clean, answer-conditioned, and solution-conditioned critics. Panels (a)–(b): mathematics, with 12 recorded checkpoints from step 5 to 555. Panels (c)–(d): KK, with nine recorded checkpoints from step 5 to 85. Points show recorded values, and solid lines connect consecutive observations without additional smoothing.
Table 4: Dataset and model assets. Licenses and question counts describe the linked distributions. “Not specified” means that the linked distribution does not state a license.
Parameter
Value
Training framework
verl
Training hardware
8× H200 GPUs
Parameter updates
Full-parameter
Optimizer
AdamW
Weight decay
0.01
Critic pretraining epochs
2
Appendix
Table 5: Training and decoding configuration for VASPO and matched reader ablations.
Figure 7: Training-time profile of VASPO with Qwen3-4B-Instruct-2507. Mean recorded time in seconds per step over 200 training steps. Bars show individual component timers; the separately recorded end-to-end step time is 412.28 s. The component timers are not an additive decomposition of this total.
Setting
Value or procedure
Critic architecture
Actor backbone with a scalar value head; matched initialization across clean and privileged variants
Critic regression
Monte Carlo terminal return; PPO clipped value loss
Critic update schedule
Pre-update once per rollout batch, then re-score values and recompute GAE before fusion
Fusion
Full clipped probability-ratio reweighting; ϵw=0.2
Token selection
Raw Atz>0 and ∣tanh(dt/τ)∣>δ ; τ=δ=0.02
Student probability source
Behavior policy
Appendix
Table 6: Critic, teacher, and policy-update settings for VASPO. Matched controls retain the applicable settings when a reader is disabled. Optional implementation variants are described in Appendix F .
Figure 8: A pooled trend hides different response groups. Saved-response critic error reduction (left) and teacher KL (right). Shading shows pointwise 95% problem-bootstrap intervals; n counts problems, which can occur in both correctness groups. There are 6,659 correct and 1,629 incorrect responses (8,288 total). Positive critic values favor solution conditioning; teacher KL measures disagreement, not useful guidance.
Figure 9: Prediction accuracy and calibration need not improve together. Three existing critics evaluated on the same 128 problems, 384 prefixes, and 3,072 independent continuations. Markers and bars show estimates and pointwise 95% problem-bootstrap intervals. Suffix labels agree with full-response rescoring for all 3,072 continuations. Arrows indicate the better direction.
Figure 10: Later critic gains depend on the source response. Clean-minus-solution Brier improvement by source correctness (a) and length (b), with paired positional contrasts C in (c,d). Positive values favor the solution critic. Bars are pointwise 95% problem-bootstrap intervals; n counts problems. Horizontal offsets in (a,b) separate overlapping points; the sampled positions are 20%, 50%, and 80%. Each row shares axis limits. Length strata use fit-split tertiles (1,008 and 1,779.33 tokens, upper bounds included). All evaluation labels agree with full-response rescoring.