On-policy self-distillation (OPSD) enables large language models (LLMs) to learn from privileged information (PI) without a separate teacher model. However, prior work largely treats PI as a fixed design choice, leaving the effects of its content and construction poorly understood. This challenge is particularly important in multi-turn agentic settings, where trajectories contain actions, observations, and reasoning that can be transformed into PI in many ways. We systematically evaluate 16 PI configurations across three multi-turn agentic benchmarks and three student models that vary in scale and reasoning mode. We find that PI effectiveness depends on its content, generation source, and the student model. Information-dense trajectories and PI produced by more capable generators generally yield larger gains, whereas thinking models benefit less consistently. To assess PI before training, we compare PI utility with two KL-based metrics: sampled-token reverse KL (D) and error-step KL contrast (Δderr−). The KL-based metrics track post-training performance more consistently than PI utility, which can be distorted by information leakage. Empirically, selecting each task's PI by Δderr− also improves post-training performance over random selection, most when the PI candidates differ widely. Finally, effective PI remains beneficial when combined with GRPO, allowing the combined objective to outperform a strong GRPO baseline. Overall, our results provide practical guidance for constructing and evaluating PI in OPSD training.
Figures & tables
Figure 1: Overview of the PI formats and offline metrics studied. Left—PI formats: reference trajectories retain actions ( A ), add observations ( A+O ) or thoughts ( A+T ), or include both ( A+O+T ). Compressed hints include a plain Summary , a Skill with mistake analysis and reusable strategies, and a Checklist of task requirements derived from reference and failed student trajectories. Right—offline metrics: PI utility measures the teacher’s reward gain from PI, but leakage can inflate or deflate it. Sampled-token reverse KL D measures shift magnitude; better PI is generally associated with a larger D , but D does not indicate whether the shift is beneficial. Error-step KL contrast Δderr− measures whether that shift targets the identified errors.
Figure 2: OPSD improvement relative to baseline across PI formats and sources; columns correspond to ALFWorld, AppWorld, and τ2 -retail. (a)–(c) Mean performance across GLM and Qwen sources. (d)–(f) GLM minus Qwen pass@1; positive values favor GLM. (g)–(i) Summary and Skill gains for GLM, Qwen, and Self.
Figure 3: Offline PI metrics versus post-training OPSD performance; columns correspond to ALFWorld, AppWorld, and τ2 -retail. Offline metrics are computed on training or development tasks for all three benchmarks. (a)–(c) PI utility. (d)–(f) Sampled-token reverse KL D . (g)–(i) Error-step KL contrast Δderr− . Lines show per-model linear fits; ρ denotes Spearman correlation.
Figure 4: OPSD+GRPO performance on ALFWorld, AppWorld, and τ2 -retail. The top row shows performance across objective weights, and the bottom row shows training curves at λ=0.9 . Here, λ=0 is vanilla OPSD and λ=1 is GRPO; dashed lines show the matched GRPO baseline.
Figure 5: Each row is one PI built from the same GLM-5.2 trajectories on ALFWorld-Passcode. Utility ranks the PIs by whether they contain the correct key. D and Δderr− rank them by whether they contain the task’s own trajectory, and so does OPSD performance. Utility is the untrained student’s pass@1 with the PI in its prompt, split into completed tasks (solid) and copied keys submitted without completing the task (hatched). Dashed boxes show the same trajectory in standard ALFWorld, and dashed lines mark the student without PI.
Figure 6: Per-task PI selection on ALFWorld with Qwen3-4B-Instruct-2507. Each task trains with a PI drawn from its top 3 candidates by D or by Δderr− , or uniformly from the whole pool; curves show held-out pass@1 at each evaluation step (mean and standard error over three or four seeds). Error-step KL selection helps most when the pool mixes wins and losses, while reverse-KL selection falls below random except on the GLM-5.3-Flash pool.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Qwen3-4B-Instruct
Qwen3-4B-Thinking
Qwen3-30B-A3B-Instruct
All benchmarks
Optimizer
AdamW, β=(0.9,0.999) , weight decay 0.01
Learning-rate schedule
constant, no warm-up
Rollouts per task
8
Training sampling
temperature 1.0, top-p 1.0
Hardware
8× H100 80GB
8× H100 80GB
16× H100 80GB
Appendix
Table 1: OPSD training configuration by student. A value spanning the three columns is shared by all students. Context is the model length, with the input budget for starting a new turn in parentheses. † The GLM-5.2 Action and Action + Obs. arms use 8,192 tokens per turn and a 49,152 (40,960) context.
Student
Action
Action + Obs.
Action + Obs. + Thou.
Qwen3-4B-Instruct-2507
0.0 / 1.0
60.8 / 51.9
82.8 / 71.9
Qwen3-4B-Thinking-2507
0.0 / 1.0
54.8 / 52.0
82.8 / 80.6
Qwen3-30B-A3B-Instruct-2507
0.0 / 1.0
72.6 / 70.1
96.5 / 91.2
Appendix
Table 2: Share of ALFWorld utility rollouts (%) whose first action is the reference trajectory’s final action, for hints from GLM-5.2 / Qwen3.7-Plus. Each cell covers 200 training tasks × 4 samples, the rollouts behind the utility column of Table 4 .
Figure 7: PI ablations on ALFWorld for Qwen3-4B-Instruct-2507: offline metrics against OPSD best-window pass@1. Colored markers are the ablated PIs, hollow markers the GLM-5.2 references: Reference is the task’s own Action + Obs. + Thou. trajectory and Action its action list. Gray dots are the other PIs of Figure 3 for the same student. Action (padded) repeats the action list until it matches the length of the full trajectory.
ALFWorld
AppWorld
τ2 -retail
Source
PI
pass@1
pass@4
pass@1
pass@4
pass@1
pass@4
Qwen3-4B-Thinking-2507
Base
46.98
71.53
4.69
11.61
62.59
88.86
GLM-5.2
Action
78.77
91.36
9.82
21.03
61.92
88.60
Action + Thou.
90.05
95.62
15.48
27.18
58.99
86.55
Action + Obs.
92.12
96.72
13.24
26.19
46.86
76.61
Appendix
Table 3: OPSD performance of each student and PI: best-window pass@1 and pass@4, the highest mean over three consecutive evaluations. Base is the untrained student. Action + Thou. pairs the reference actions with the teacher’s reasoning, without observations. The best value per student and column is in bold.
ALFWorld
AppWorld
τ2 -retail
Source
PI
U
D
Δderr−
U
D
Δderr−
U
D
Δderr−
Qwen3-4B-Thinking-2507
GLM-5.2
Action
0.300
0.101
0.0075
0.733
0.114
0.0010
0.114
0.140
0.0017
Action + Thou.
0.095
0.113
0.0143
0.707
0.122
0.0005
0.032
0.212
− 0.0049
Action + Obs.
0.075
0.130
0.0133
0.821
0.142
0.0019
0.244
0.254
− 0.0072
Action + Obs. + Thou.
0.034
0.118
0.0145
0.762
0.130
0.0011
− 0.095
0.224
− 0.0052
Appendix
Table 4: Offline PI metrics of each student and PI (Appendix C ): utility U ( Δ pass@1), sampled-token reverse KL D , and error-step KL contrast Δderr− .
On-policy self-distillation (OPSD) has become a popular recipe for post-training LLM agents. It supervises the agent model at the token level with a stronger teacher view of the same model, obtained by conditioning on privileged information (PI). In this work, we show that in multi-turn agents, this paradigm teaches the student to act with confidence but without the information behind it. The trained agent behaves as if it had privileged information it never observed, and its performance falls well short of plain RL, in the worst case below the untrained base model. Therefore, we propose Privileged Self-Practice (PSP), which keeps the PI and moves it from the loss to the sampler. When the student's rollouts on a task mostly fail, we inject a short per-task instruction written by an analyzer model, sample the task again with the instruction in context, and train on the result with an unchanged GRPO objective. The privileged information stays in the prompt and never enters the loss. Across AppWorld and SWE-bench Verified, with three different student models, PSP obtains the best average score in every setting and is the only method that consistently outperforms plain GRPO, improving task-goal completion by up to 65% on AppWorld and the resolved rate by up to 61% on SWE-bench Verified.
On-policy self-distillation (OPSD) has emerged as a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access. However, recent studies have found that the teacher's dense token-level supervision, conditioned on privileged information, can lead to overfitting to in-domain patterns, suppress exploration, and hurt cross-domain generalization, while also introducing a more fundamental issue: privileged information leakage, where the student encodes answer-dependent shortcuts that are unavailable at test time. We introduce DemoPSD, a novel framework that resolves such problems through the idea of selective adoption of teacher guidance. Instead of fitting the full teacher distribution, DemoPSD steers the student toward a reverse-KL barycenter target, a weighted geometric combination of the teacher and student distributions, that naturally balances learning from the teacher with preserving the student's own reasoning capacity. We measure the difference between their distributions and use such a discrepancy to adaptively control the blending at each token position. We provably show that DemoPSD achieves (1)leakage attenuation, i.e., effective mitigation of privileged information leakage; and (2)exploration preservation, i.e., preservation of exploration capacity under dense token-level distillation. Extensive experiments on SciKnowEval across four scientific fields show that DemoPSD outperforms both GRPO and SDPO while maintaining higher training entropy and robustly generalizing to out-of-distribution GPQA benchmarks.
Yunhe Li, Hao Shi, Wenhao Liu +5
1City University of Hong Kong · 2Tsinghua University · 3Shenzhen University of Advanced Technology +1
On-policy self-distillation (OPSD) trains a student on its own responses using token-level supervision from the same model conditioned on privileged reference information. We investigate whether performance gains from OPSD show that the student learned the information in the reference or instead reflect recovery of reasoning behavior already present in the base model. We perform OPSD experiments on science and mathematics datasets using Qwen3 models ranging from 1.7B to 8B. Our analysis framework separates the supervision induced by the reference from the supervision provided by the teacher without the reference and measures how each aligns with changes in the student's predictions. The correct reference does not provide a consistent performance benefit across teacher generation modes, model sizes, and training datasets. Students can improve without the correct reference, and a solution from another problem can outperform the correct solution on several mathematical reasoning benchmarks. The student's predictions align more strongly with the base model's thinking behavior than with the supervision induced by the reference, but controls constructed from other problems reproduce much of both alignments. Moreover, stronger alignment attributable to the correct reference does not reliably coincide with a greater performance benefit from the reference. Performance gains and distributional alignment alone therefore cannot determine how privileged reference information contributes to student learning in OPSD.