On-policy self-distillation (OPSD) improves reasoning by training a problem-only student on its own rollouts using dense token-level supervision from a privileged teacher that also sees a reference solution. However, standard OPSD treats the teacher distribution as a fixed target along the student's rollout and updates only the student %, although -- even though privileged conditioning does not guarantee that the teacher always provides the most appropriate target for problem-only reasoning. This one-way supervision can therefore misdirect the student when the teacher distribution is misaligned with valid student reasoning. We therefore introduce Verifier-Informed Student-to-Teacher Adaptation (VISTA), which preserves the standard OPSD student update while using outcome-verified rollouts to adapt the teacher toward the student distribution. Within each verified rollout, VISTA further restricts this adaptation to the top-k positions with the largest teacher--student KL divergence. Notably, VISTA reuses the rollout and loss function from standard OPSD, introducing no additional sampling or separate reward objective. Across AIME24, AIME25, and HMMT25 with Qwen3 models at 1.7B, 4B, and 8B, VISTA achieves the highest Avg@12 at every scale, improving over OPSD by 0.6, 0.7, and 2.1 points, respectively. These results demonstrate the value of student supervision from outcome-verified rollouts and highlight student-to-teacher adaptation as a promising direction for OPSD.
Figures & tables
Figure 1: A schematic of VISTA. Building on on-policy self-distillation (OPSD), VISTA additionally introduces a selective student-to-teacher adaptation on verified trajectories: at token positions selected by their KL divergence, the privileged teacher is revised toward the student’s distribution, so that the teacher in turn provides more benign supervision to the student.
Figure 2: Overview of the VISTA training pipeline. (1) On-policy rollout: A problem-only student samples a solution rollout for a training problem. (2) Outcome gating and position selection: The outcome verifier gates teacher updates based on rollout correctness, while position selection forms the teacher-update mask by selecting the top- k positions with the largest teacher–student KL divergence. (3) Student and teacher updates: Building on standard OPSD, VISTA retains the standard student update at every rollout position and, as explained in (2), updates the teacher toward the student’s distribution at token positions where the teacher-update mask equals one.
Figure 3: A representative case of the teacher-superiority assumption failing. From the same student rollout prefix, the student follows valid reasoning, whereas the teacher emits a reference-specific phrase. This counterexample shows that the privileged distribution is not uniformly a better token-level target along the student’s rollout.
Model
Method
AIME24
AIME25
HMMT25
Average
Qwen3-1.7B
Base
51.5
36.7
23.1
37.1
SFT
48.4
36.3
22.7
35.8
GRPO
51.1
38.3
23.7
37.7
SDPO
52.8
36.9
22.8
37.5
OPSD
57.2
43.9
29.2
43.4
VISTA
57.8
43.3
30.8
44.0
Table 1: Main results on competition-level mathematical reasoning (Avg@12, %). Each benchmark uses its peak checkpoint under the OPSD protocol, and Average is the unweighted mean over the three benchmarks. Base, SFT, GRPO, and OPSD results are from the OPSD paper ( Zhao et al. 2026 ) ; all post-training methods use the same corpus, with baseline budgets retained as reported. The best and second-best results within each model block are shown in boldface and underlined, respectively.
Adaptation rule
Rate
Avg@12 ↑
OPSD
0%
64.8
All rollouts
100%
65.8
Random gate
60%
65.6
Reverse gate
∼40%
64.5
VISTA
∼60%
66.9
Table 2: Outcome-gate ablation on Qwen3-8B. Rate denotes the fraction of rollouts routed to teacher adaptation; the random gate approximately matches VISTA’s rate, while the reverse gate uses only verifier-rejected rollouts.
Figure 4: Rollout-position selection ablation on Qwen3-8B (Avg@12). We compare OPSD with four teacher-adaptation variants: (A) the first k positions, (B) k uniformly random positions, (C) the last k positions, and (D) the top- k KL-divergence positions. VISTA uses (D) and attains the best three-benchmark average.
Figure 5: Avg@12 on AIME24 and AIME25 as the number of teacher-update positions varies.
Figure 6: Teacher-adaptation diagnostics across Qwen3-8B checkpoints: low-probability tokens ( πT(yt)≤0.05 ) on validated student trajectories and explicit reference attribution in teacher responses.
Figure 7: Privileged-teacher validation of the Qwen3-8B teacher during training.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Setting
Model scale
Qwen3-1.7B, Qwen3-4B, or Qwen3-8B.
Data pool
OpenThoughts mathematical-reasoning pool.
Student prompt
Problem only; thinking off.
Teacher prompt
Problem plus reference solution; thinking on.
Temperature
1.1 .
Top- p
0.95 .
Appendix
Table 3: Core VISTA training configuration.
Parameter
Value
Max new tokens
38,912
Thinking mode
Enabled
Temperature
1.0
Top- p
0.95
Top- k
−1 (disabled)
Min- p
0.0
Appendix
Table 4: VISTA evaluation configuration.
Figure 8: Verifier acceptance rate of Qwen3-8B student rollouts during VISTA training, centered near 60% .
Step
AIME24
AIME25
HMMT25
25
74.4
71.9
43.1
50
76.9
69.4
43.9
75
78.6
71.9
45.6
100
76.4
71.1
48.3
Appendix
Table 5: Qwen3-8B VISTA student Avg@12 (%) by checkpoint.
Figure 12: Reference-conditioned over-support: privileged-information leakage at a shared student prefix. At such a prefix, the privileged teacher may favor a continuation that depends on information unavailable to the problem-only student.
Figure 13: Reference-conditioned under-support: suppression of valid problem-only alternatives at a shared student prefix. At such a prefix, the privileged teacher may assign low probability to a locally coherent continuation available to the problem-only student.
On-policy self-distillation (OPSD) uses a privileged teacher to supervise a reasoning model on prefixes sampled from its own rollouts. Yet each rollout also reveals how the student's response unfolds and whether it succeeds, student-specific hindsight that standard OPSD does not use to form the teacher. We introduce Privileged Adaptation from Student Trajectories (PAST), which treats each completed student trajectory as additional privileged information for the OPSD teacher while leaving the student's distillation prefixes unchanged. PAST preserves the student's next-token distribution on correct trajectories and uses failed trajectories to adapt the teacher toward verified success under student-proximity regularization. We characterize what such a trajectory-conditioned teacher can transfer to a prefix-only student. Forward-KL distillation projects the teacher distributions to their conditional arithmetic mean given the prefix. This projection separates trajectory-specific variation that remains privileged from the mean policy shift available to the student. For correct trajectories, the unclipped population objective also has the frozen student as an ideal distributional fixed point. Across three mathematical reasoning benchmarks, PAST improves the Avg@12 macro average over Vanilla OPSD by 5.6 percentage points. A 2×2 factorial study shows gains from both complete-trajectory access and teacher adaptation, while trajectory removal and shuffling confirm that the adapted teacher uses the matching hindsight context.
Yangyang Feng, Zhuoyan Feng, Junlan Chen
1The Hong Kong University of Science and Technology (Guangzhou) · 2Sun Yat-sen University
On-policy self-distillation (OPSD) trains a question-only student with token-level feedback from a teacher given training-only privileged information (PI). OPSD therefore provides dense, on-policy supervision, and is free of a larger external teacher, but its effectiveness rests on how PI is designed and utilized. Our preliminary diagnostics suggest a significant gap between teacher utility and student learnability, where a small fraction of high-disagreement tokens dominate the distillation signal, and short teacher continuations at these positions further expose more explicit PI leakage than transferable correction cues, indicating a strong intent on injecting PI-conditioned shortcuts. We propose Adaptive On-Policy Self-Distillation (AOPSD), which adapts what information the teacher receives and how strongly its feedback influences learning. AOPSD encodes each solution as a reasoning DAG, orders problems by the student's evolving capability, and reveals only the affordable subgraph and its next frontier as PI. For high-disagreement tokens, AOPSD utilizes short teacher continuations as probes to encourage useful guidance while mitigating PI-conditioned shortcuts among teacher supervisions. On HMMT25, AIME24, AIME25, and BRUMo25, AOPSD achieves 72.5% Pass@8, which is 6.7 percentage points above OPSD and 4.2 above the strongest competing baseline while reducing 15 percentage points of training time at lower cost.
Jiacheng Du, Weiwei Xie, Tianyi Du +3
National University of Singapore · Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory
On-policy self-distillation (OPSD) trains mathematical reasoning models using a privileged teacher that sees a reference solution and supervises student-sampled prefixes. Standard OPSD uses one fixed parameter setting at every state, but nearby settings may offer additional supervision. We find that local parameter perturbations reveal complementary reference-aligned corrections under the same reference context. Different experts supply these corrections at different reference positions. Their pool covers more such positions than the unperturbed privileged teacher. We introduce Neighborhood OPSD (N-OPSD) to turn these corrections into supervision at student-visited states. Offline, greedy selection builds a compact pool of frozen experts by rewarding filtered reference-token gains beyond the pool's current best at each position. The highest-peak expert need not provide the best training target. Online routing therefore separates the anchor direction from its level of support. MaxPeak selects the anchor token, and quantile selection chooses among experts whose top token matches it. The student learns from the chosen expert's full next-token distribution through the clipped forward-KL objective inherited from OPSD. We evaluate on AIME 2024, AIME 2025, and HMMT February 2025. Across three independent runs per method, Neighborhood OPSD improves the three-benchmark Average@12 over OPSD by 2.75, 1.67, and 1.94 points on Qwen3-1.7B, 4B, and 8B, respectively. Student-prefix continuations support using the pool beyond the reference trajectories used for selection. Matched ablations support filtered reference-token gains as a selection criterion. Accounting for overlap within the pool and routing by state further improve student accuracy. Inference uses only the distilled student.
Xincheng Wei, Yifan Ding, Yoshua Li +6
The Chinese University of Hong Kong, Shenzhen · Meituan, LongCat Team · University of Toronto +1