On-Policy Self-Distillation (OPSD) has recently attracted interest because it provides dense teacher guidance without a separate, often larger teacher model. However, OPSD conditions its teacher on ground-truth solutions, which limits its use when such solutions are unavailable. We propose On-Policy Self-Distillation with Self-Generated Problem Briefs (B-OPSD). Using a prompt template, the initial model or a saved student checkpoint generates a problem brief from the question alone. The prompt asks for relevant principles, applicability conditions, and likely pitfalls, but not a final answer or a complete solution. A frozen teacher uses the brief as privileged context to guide the current student's on-policy rollouts through token-level distribution matching. B-OPSD uses no ground-truth solutions during training. Results on AIME 2024, AIME 2025, and HMMT 2025 show that both B-OPSD variants achieve higher average accuracy than OPSD at each of three Qwen3 model sizes. Code is available at https://github.com/ZevineXu/B-OPSD
Figures & tables
Figure 1: Overview of B-OPSD. Student and teacher start from the same base model; the teacher stays frozen while the student is updated. The initial model generates static briefs, while a saved student checkpoint generates refreshed briefs. The current student generates its solution trajectory from the question alone. The frozen teacher receives the question and brief, evaluates the same student prefixes, and provides token-level feedback.
Training signal
Training requirements
Method
On-policy data
Dense learning signal
No separate teacher model
No answer labels
SFT
×
✓
✓
×
GRPO
✓
×
✓
×
OPD
✓
✓
×
✓
OPSD
✓
✓
✓
×
B-OPSD
✓
✓
✓
✓
Table 1: B-OPSD combines on-policy student rollouts and token-level teacher feedback without a separate teacher model, reference solutions, or verified answers during training. The SFT and GRPO rows use the reference solutions and verified final answers, respectively, from the compared training setups. The OPD row represents a common setup with a separate teacher model and no answer labels during distillation.
Figure 2: Complete problem brief generation prompt PB . The same prompt is used for static and refreshed briefs. The generator also receives the problem statement x .
Method
AIME 2024
AIME 2025
HMMT 2025
Average
Qwen3-1.7B
Base
51.5
36.7
23.1
37.1
SFT
48.4
36.3
22.7
35.8
GRPO
51.1
38.3
23.7
37.7
OPSD
54.7
43.1
25.0
40.9
B-OPSD
56.4
42.2
27.2
41.9
Table 2: Avg@12 accuracy (%) across three benchmarks. B-OPSD uses static briefs; B-OPSD ⋆ uses refreshed briefs. Average is computed from unrounded scores. Shaded rows show the B-OPSD variants, and bold marks the highest score for each model size and column.
Figure 3: Checkpoint Avg@12 accuracy (%) for Qwen3-1.7B. B-OPSD uses static briefs; B-OPSD ⋆ uses refreshed briefs. The Average panel combines scores from the same checkpoint; Table 2 averages each benchmark’s best score. The vertical axes start near the observed scores and differ across panels.
Method
AIME 2024
HMMT 2025
OPSD
54.7
25.0
B-OPSD(8B)
56.4
27.8
B-OPSD(ref)
56.4
27.5
B-OPSD
56.4
27.2
B-OPSD ⋆
57.5
28.6
Table 3: Avg@12 accuracy (%) for Qwen3-1.7B with different brief generation strategies. B-OPSD uses static briefs generated by the initial 1.7B model, and B-OPSD ⋆ uses refreshed briefs. B-OPSD(8B) uses briefs generated by Qwen3-8B, while B-OPSD(ref) uses briefs generated using the reference solutions supplied to OPSD. Shaded rows show our main variants. Bold marks the best result in each column.
Divergence
OPSD
B-OPSD
B-OPSD ⋆
AIME 2024
Forward KL
54.7
56.4
57.5
Reverse KL
51.1
52.7
54.2
JSD ( β=0.5 )
50.0
50.0
52.9
HMMT 2025
Forward KL
25.0
27.2
28.6
Table 4: Avg@12 accuracy (%) for Qwen3-1.7B with different divergences. B-OPSD uses static briefs; B-OPSD ⋆ uses refreshed briefs. Bold marks the highest score in each row.
B-OPSD
B-OPSD ⋆
Omitted section
AIME 2024
HMMT 2025
AIME 2024
HMMT 2025
None (full brief)
56.4
27.2
57.5
28.6
Targets
57.2
28.6
56.1
27.8
Relevant principles
55.8
26.7
56.7
26.7
Suggested approach
57.5
28.3
57.0
26.9
Pitfalls and checks
56.7
26.7
56.7
28.3
Table 5: Avg@12 accuracy (%) for Qwen3-1.7B when one problem brief section is omitted. B-OPSD uses static briefs, and B-OPSD ⋆ uses refreshed briefs. The shaded row uses the full brief. Bold marks the highest score within each variant and benchmark.
On-policy self-distillation (OPSD) combines student generated trajectories with dense supervision from a privileged teacher of the same model. Its pointwise clipping protects the student from locally incompatible teacher directions, but the fixed teacher receives no feedback about the supervision the student rejects, which limits effective learning from the teacher. We introduce DualOPSD, which alternates updates in a coupled teacher-student system. The student retains OPSD's clipped forward KL objective. The privileged teacher adapts to the updated student after each student step. A residual target based on the frozen initial model preserves the effect of privileged context during teacher adaptation. Across Qwen3-8B/4B/1.7B and three math benchmarks, DualOPSD improves avg@12 accuracy over OPSD by 9.83 percentage points. Further diagnostic analyses show lower bidirectional KL, less persistent mismatch, and retained sensitivity to privileged reference. Together, these results demonstrate that DualOPSD can be a more effective form of privileged self-distillation. Code is available at: https://github.com/on1262/DualOPSD.
Yutong Chen, Guangfu Guo, Zhichao Xu +1
Department of Computer Science, Clemson University, 105 Sikes Hall, Clemson, SC 29634, USA · Department of Computer Science, University of Utah
On-Policy Self-Distillation (OPSD) is commonly interpreted as the transfer of privileged information: a teacher observes the verified solution to the target problem and supervises the student's trajectory. However, this interpretation conflates two effects. The reference solution not only reveals the answer to the current instance but also changes the context under which the teacher provides token-level supervision. We investigate the role of target-specific privilege with OP2SD (On-Policy Self-Distillation from Other Problems), which replaces the paired reference with a problem and solution from a different example, while preserving the student rollout, teacher, and distillation objective. Across three models and three mathematics benchmarks, OP2SD improves over the base model, remains competitive with OPSD. The success of OP2SD implies that OPSD gains do not necessarily come from access to the reference solution, and that the teacher's context-induced behavior is an important factor.
Yuki Ichihara, Naoto Iwase, Mohammad Atif Quamar +1
Mohamed bin Zayed University of Artificial Intelligence · Nagoya University · RIKEN AIP
On-policy self-distillation (OPSD) trains a question-only student with token-level feedback from a teacher given training-only privileged information (PI). OPSD therefore provides dense, on-policy supervision, and is free of a larger external teacher, but its effectiveness rests on how PI is designed and utilized. Our preliminary diagnostics suggest a significant gap between teacher utility and student learnability, where a small fraction of high-disagreement tokens dominate the distillation signal, and short teacher continuations at these positions further expose more explicit PI leakage than transferable correction cues, indicating a strong intent on injecting PI-conditioned shortcuts. We propose Adaptive On-Policy Self-Distillation (AOPSD), which adapts what information the teacher receives and how strongly its feedback influences learning. AOPSD encodes each solution as a reasoning DAG, orders problems by the student's evolving capability, and reveals only the affordable subgraph and its next frontier as PI. For high-disagreement tokens, AOPSD utilizes short teacher continuations as probes to encourage useful guidance while mitigating PI-conditioned shortcuts among teacher supervisions. On HMMT25, AIME24, AIME25, and BRUMo25, AOPSD achieves 72.5% Pass@8, which is 6.7 percentage points above OPSD and 4.2 above the strongest competing baseline while reducing 15 percentage points of training time at lower cost.
Jiacheng Du, Weiwei Xie, Tianyi Du +3
National University of Singapore · Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory