Predicting Privileged Information Effectiveness for On-Policy Self-Distillation

Kaijian Zou; Emiliano Penaloza; MohammadHossein Rezaei; Anisha Gunjal; Mohit Raghavendra; Lu Wang; Yunzhong He; Anas Mahmoud

cs.LGOct 15, 2026

Predicting Privileged Information Effectiveness for On-Policy Self-Distillation

Authors: Kaijian Zou, Emiliano Penaloza, MohammadHossein Rezaei, Anisha Gunjal, Mohit Raghavendra, Lu Wang, Yunzhong He, Anas Mahmoud

Organizations: University of Michigan – Ann Arbor · Mila · Scale AI

Abstract

On-policy self-distillation (OPSD) enables large language models (LLMs) to learn from privileged information (PI) without a separate teacher model. However, prior work largely treats PI as a fixed design choice, leaving the effects of its content and construction poorly understood. This challenge is particularly important in multi-turn agentic settings, where trajectories contain actions, observations, and reasoning that can be transformed into PI in many ways. We systematically evaluate 16 PI configurations across three multi-turn agentic benchmarks and three student models that vary in scale and reasoning mode. We find that PI effectiveness depends on its content, generation source, and the student model. Information-dense trajectories and PI produced by more capable generators generally yield larger gains, whereas thinking models benefit less consistently. To assess PI before training, we compare PI utility with two KL-based metrics: sampled-token reverse KL ( $D$ ) and error-step KL contrast ( $Δd^{-}_{\mathrm{err}}$ ). The KL-based metrics track post-training performance more consistently than PI utility, which can be distorted by information leakage. Empirically, selecting each task's PI by $Δd^{-}_{\mathrm{err}}$ also improves post-training performance over random selection, most when the PI candidates differ widely. Finally, effective PI remains beneficial when combined with GRPO, allowing the combined objective to outperform a strong GRPO baseline. Overall, our results provide practical guidance for constructing and evaluating PI in OPSD training.

Figures & tables

Figure 1: Overview of the PI formats and offline metrics studied. Left—PI formats: reference trajectories retain actions ( A ), add observations ( A+O ) or thoughts ( A+T ), or include both ( A+O+T ). Compressed hints include a plain Summary , a Skill with mistake analysis and reusable strategies, and a Checklist of task requirements derived from reference and failed student trajectories. Right—offline metrics: PI utility measures the teacher’s reward gain from PI, but leakage can inflate or deflate it. Sampled-token reverse KL

D

measures shift magnitude; better PI is generally associated with a larger

D

, but

D

does not indicate whether the shift is beneficial. Error-step KL contrast

\Delta d^{-}_{\mathrm{err}}

measures whether that shift targets the identified errors.

Panel of Figure 2 — Figure 2: OPSD improvement relative to baseline across PI formats and sources; columns correspond to ALFWorld, AppWorld, and $\tau^{2}$ -retail. (a)–(c) Mean performance across GLM and Qwen sources. (d)–(f) GLM minus Qwen pass@1; positive values favor GLM. (g)–(i) Summary and Skill gains for GLM, Qwen, and Self.

Panel of Figure 3 — Figure 3: Offline PI metrics versus post-training OPSD performance; columns correspond to ALFWorld, AppWorld, and $\tau^{2}$ -retail. Offline metrics are computed on training or development tasks for all three benchmarks. (a)–(c) PI utility. (d)–(f) Sampled-token reverse KL $D$ . (g)–(i) Error-step KL contrast $\Delta d^{-}_{\mathrm{err}}$ . Lines show per-model linear fits; $\rho$ denotes Spearman correlation.

Figure 4: OPSD+GRPO performance on ALFWorld, AppWorld, and

\tau^{2}

-retail. The top row shows performance across objective weights, and the bottom row shows training curves at

\lambda=0.9

. Here,

\lambda=0

is vanilla OPSD and

\lambda=1

is GRPO; dashed lines show the matched GRPO baseline.

Figure 5: Each row is one PI built from the same GLM-5.2 trajectories on ALFWorld-Passcode. Utility ranks the PIs by whether they contain the correct key.

D

and

\Delta d^{-}_{\mathrm{err}}

rank them by whether they contain the task’s own trajectory, and so does OPSD performance. Utility is the untrained student’s pass@1 with the PI in its prompt, split into completed tasks (solid) and copied keys submitted without completing the task (hatched). Dashed boxes show the same trajectory in standard ALFWorld, and dashed lines mark the student without PI.

Figure 6: Per-task PI selection on ALFWorld with Qwen3-4B-Instruct-2507. Each task trains with a PI drawn from its top 3 candidates by

D

or by

\Delta d^{-}_{\mathrm{err}}

, or uniformly from the whole pool; curves show held-out pass@1 at each evaluation step (mean and standard error over three or four seeds). Error-step KL selection helps most when the pool mixes wins and losses, while reverse-KL selection falls below random except on the GLM-5.3-Flash pool.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

All benchmarks
	Qwen3-4B-Instruct	Qwen3-4B-Thinking	Qwen3-30B-A3B-Instruct
Optimizer	AdamW, $\beta=(0.9,0.999)$ , weight decay 0.01
Learning-rate schedule	constant, no warm-up
Rollouts per task	8
Training sampling	temperature 1.0, top-p 1.0
Hardware	$8\times$ H100 80GB	$8\times$ H100 80GB	$16\times$ H100 80GB

Appendix

Table 1: OPSD training configuration by student. A value spanning the three columns is shared by all students. Context is the model length, with the input budget for starting a new turn in parentheses. † The GLM-5.2 Action and Action

+

Obs. arms use 8,192 tokens per turn and a 49,152 (40,960) context.

Student	Action	Action $+$ Obs.	Action $+$ Obs. $+$ Thou.
Qwen3-4B-Instruct-2507	0.0 / 1.0	60.8 / 51.9	82.8 / 71.9
Qwen3-4B-Thinking-2507	0.0 / 1.0	54.8 / 52.0	82.8 / 80.6
Qwen3-30B-A3B-Instruct-2507	0.0 / 1.0	72.6 / 70.1	96.5 / 91.2

Appendix

Table 2: Share of ALFWorld utility rollouts (%) whose first action is the reference trajectory’s final action, for hints from GLM-5.2 / Qwen3.7-Plus. Each cell covers 200 training tasks

\times

4 samples, the rollouts behind the utility column of Table 4 .

Figure 7: PI ablations on ALFWorld for Qwen3-4B-Instruct-2507: offline metrics against OPSD best-window pass@1. Colored markers are the ablated PIs, hollow markers the GLM-5.2 references: Reference is the task’s own Action

+

Obs.

+

Thou. trajectory and Action its action list. Gray dots are the other PIs of Figure 3 for the same student. Action (padded) repeats the action list until it matches the length of the full trajectory.

Qwen3-4B-Thinking-2507
		ALFWorld		AppWorld		$\tau^{2}$ -retail
Source	PI	pass@1	pass@4	pass@1	pass@4	pass@1	pass@4
	Base	46.98	71.53	4.69	11.61	62.59	88.86
GLM-5.2	Action	78.77	91.36	9.82	21.03	61.92	88.60
	Action $+$ Thou.	90.05	95.62	15.48	27.18	58.99	86.55
	Action $+$ Obs.	92.12	96.72	13.24	26.19	46.86	76.61

Appendix

Table 3: OPSD performance of each student and PI: best-window pass@1 and pass@4, the highest mean over three consecutive evaluations. Base is the untrained student. Action

+

Thou. pairs the reference actions with the teacher’s reasoning, without observations. The best value per student and column is in bold.

Qwen3-4B-Thinking-2507
		ALFWorld			AppWorld			$\tau^{2}$ -retail
Source	PI	$U$	$D$	$\Delta d^{-}_{\mathrm{err}}$	$U$	$D$	$\Delta d^{-}_{\mathrm{err}}$	$U$	$D$	$\Delta d^{-}_{\mathrm{err}}$
GLM-5.2	Action	0.300	0.101	0.0075	0.733	0.114	0.0010	0.114	0.140	0.0017
	Action $+$ Thou.	0.095	0.113	0.0143	0.707	0.122	0.0005	0.032	0.212	$-$ 0.0049
	Action $+$ Obs.	0.075	0.130	0.0133	0.821	0.142	0.0019	0.244	0.254	$-$ 0.0072
	Action $+$ Obs. $+$ Thou.	0.034	0.118	0.0145	0.762	0.130	0.0011	$-$ 0.095	0.224	$-$ 0.0052

Appendix

Table 4: Offline PI metrics of each student and PI (Appendix C ): utility

U

(

\Delta

pass@1), sampled-token reverse KL

D

, and error-step KL contrast

\Delta d^{-}_{\mathrm{err}}

On-Policy Self-Distillation

On-policy self-distillation trains a model on its own generated trajectories using a teacher view to provide dense supervision.

See all 162 papers

From Self-Distillation to Self-Practice: Privileged Information for Multi-Turn Agents
Sep 24, 2026Xingyu Su, Abhishek Kumar, Qing Ping +5On-Policy Self-Distillation LLM Agent Training
Texas A&M University · AWS AI, Amazon
DemoPSD: Disagreement-Modulated Policy Self-Distillation
Jul 2, 2026Yunhe Li, Hao Shi, Wenhao Liu +5On-Policy Self-Distillation RL for Language Model Reasoning
1City University of Hong Kong · 2Tsinghua University · 3Shenzhen University of Advanced Technology +1
Rethinking Privileged Information in On-Policy Self-Distillation
Aug 18, 2026Samyak Shrestha, Alexander TessierOn-Policy Self-Distillation Language Model Distillation
FirstPrinciples

Sep 24, 2026J/K move · Enter open · S save

From Self-Distillation to Self-Practice: Privileged Information for Multi-Turn Agents

Xingyu Su, Abhishek Kumar, Qing Ping, Youzhi Luo+4

Texas A&M University · AWS AI, Amazon

On-Policy Self-Distillation LLM Agent Training Learning Using Privileged Information RL for LLM Agents

Open paper AskPDF arXiv

On-policy self-distillation (OPSD) has become a popular recipe for post-training LLM agents. It supervises the agent model at the token level with a stronger teacher view of the same model, obtained by conditioning on privileged information (PI). In this work, we show that in multi-turn agents, this paradigm teaches the student to act with confidence but without the information behind it. The trained agent behaves as if it had privileged information it never observed, and its performance falls well short of plain RL, in the worst case below the untrained base model. Therefore, we propose Privileged Self-Practice (PSP), which keeps the PI and moves it from the loss to the sampler. When the student's rollouts on a task mostly fail, we inject a short per-task instruction written by an analyzer model, sample the task again with the instruction in context, and train on the result with an unchanged GRPO objective. The privileged information stays in the prompt and never enters the loss. Across AppWorld and SWE-bench Verified, with three different student models, PSP obtains the best average score in every setting and is the only method that consistently outperforms plain GRPO, improving task-goal completion by up to 65% on AppWorld and the resolved rate by up to 61% on SWE-bench Verified.

Abstract

On-Policy Self-Distillation

Explore similar work

From Self-Distillation to Self-Practice: Privileged Information for Multi-Turn Agents

DemoPSD: Disagreement-Modulated Policy Self-Distillation

Rethinking Privileged Information in On-Policy Self-Distillation