From Self-Distillation to Self-Practice: Privileged Information for Multi-Turn Agents

Xingyu Su; Abhishek Kumar; Qing Ping; Youzhi Luo; Jonathan Buck; Zach Zhang; Subramanian Chidambaram; Vinayak Arannil

cs.AISep 24, 2026

From Self-Distillation to Self-Practice: Privileged Information for Multi-Turn Agents

Authors: Xingyu Su, Abhishek Kumar, Qing Ping, Youzhi Luo, Jonathan Buck, Zach Zhang, Subramanian Chidambaram, Vinayak Arannil

Organizations: Texas A&M University · AWS AI, Amazon

Abstract

On-policy self-distillation (OPSD) has become a popular recipe for post-training LLM agents. It supervises the agent model at the token level with a stronger teacher view of the same model, obtained by conditioning on privileged information (PI). In this work, we show that in multi-turn agents, this paradigm teaches the student to act with confidence but without the information behind it. The trained agent behaves as if it had privileged information it never observed, and its performance falls well short of plain RL, in the worst case below the untrained base model. Therefore, we propose Privileged Self-Practice (PSP), which keeps the PI and moves it from the loss to the sampler. When the student's rollouts on a task mostly fail, we inject a short per-task instruction written by an analyzer model, sample the task again with the instruction in context, and train on the result with an unchanged GRPO objective. The privileged information stays in the prompt and never enters the loss. Across AppWorld and SWE-bench Verified, with three different student models, PSP obtains the best average score in every setting and is the only method that consistently outperforms plain GRPO, improving task-goal completion by up to 65% on AppWorld and the resolved rate by up to 61% on SWE-bench Verified.

Figures & tables

	test-normal ( $n=168$ )				test-challenge ( $n=417$ )
method	TGC	SGC	TGC@5	SGC@5	TGC	SGC	TGC@5	SGC@5	Avg
Base	$5.4$ $\pm 1.8$	$0.7$ $\pm 0.9$	15.5	5.4	$2.1$ $\pm 0.6$	0	6.2	0	4.4
GRPO	$13.2$ $\pm 1.7$	$4.3$ $\pm 1.4$	28.0	$\underline{16.1}$	$4.6$ $\pm 0.6$	$0.7$ $\pm 0.5$	12.0	$\underline{5.8}$	10.6
OPSD	$3.9$ $\pm 1.0$	$0.4$ $\pm 0.7$	11.9	1.8	$2.0$ $\pm 1.0$	0	7.2	0	3.4
GRPO+OPSD	$7.0$ $\pm 1.5$	$2.1$ $\pm 1.3$	17.3	3.6	$4.0$ $\pm 0.7$	$0.9$ $\pm 0.7$	9.8	3.6	6.0
Skill-SD	$5.8$ $\pm 0.9$	$0.7$ $\pm 1.0$	16.7	5.4	$2.3$ $\pm 0.8$	$0.1$ $\pm 0.3$	6.7	3.6	5.2

Table 1: Qwen3-4B , both test splits. Per column, best in bold and second best underlined. Avg is the unweighted mean of the eight metrics.

	test-normal ( $n=168$ )				test-challenge ( $n=417$ )
method	TGC	SGC	TGC@5	SGC@5	TGC	SGC	TGC@5	SGC@5	Avg
Base	$15.5$ $\pm 1.5$	$3.9$ $\pm 0.7$	33.9	17.9	$6.5$ $\pm 0.2$	$1.7$ $\pm 1.2$	13.7	5.0	12.3
GRPO	$30.8$ $\pm 1.0$	$13.9$ $\pm 2.4$	53.6	$\underline{35.7}$	$14.9$ $\pm 1.1$	$5.2$ $\pm 1.5$	$\underline{30.0}$	$\underline{20.9}$	$\underline{25.6}$
OPSD	$7.7$ $\pm 1.5$	$0.7$ $\pm 0.9$	22.6	3.6	$1.3$ $\pm 1.6$	$0.4$ $\pm 0.6$	4.6	1.4	5.3
GRPO+OPSD	$16.8$ $\pm 2.2$	$3.2$ $\pm 2.9$	36.3	14.3	$7.0$ $\pm 1.0$	$1.7$ $\pm 1.0$	15.4	3.6	12.3
Skill-SD	$16.1$ $\pm 2.9$	$2.5$ $\pm 2.0$	39.9	25.0	$7.7$ $\pm 0.9$	$1.9$ $\pm 0.6$	17.0	8.6	14.8

Table 2: Qwen3-8B , both test splits. Per column, best in bold and second best underlined; RLSD and ours tie for best SGC on test-challenge (5.9), with ours holding the smaller seed spread. Avg is the unweighted mean of the eight metrics.

	test-normal ( $n=168$ )				test-challenge ( $n=417$ )
method	TGC	SGC	TGC@5	SGC@5	TGC	SGC	TGC@5	SGC@5	Avg
Base	$14.6$ $\pm 1.8$	$3.9$ $\pm 1.5$	33.9	23.2	$5.9$ $\pm 1.1$	$1.6$ $\pm 0.9$	12.9	7.9	13.0
GRPO	$33.9$ $\pm 1.3$	$18.2$ $\pm 1.3$	$\underline{54.8}$	$\underline{39.3}$	$15.3$ $\pm 0.8$	$4.3$ $\pm 0.6$	31.9	$\underline{20.1}$	$\underline{27.2}$
OPSD	$0.7$ $\pm 0.5$	0	3.0	0	$0.4$ $\pm 0.2$	0	1.0	0	0.6
GRPO+OPSD	$17.9$ $\pm 2.4$	$6.4$ $\pm 1.8$	35.7	25.0	$6.6$ $\pm 0.8$	$1.6$ $\pm 1.1$	15.6	8.6	14.7
Skill-SD	$14.2$ $\pm 2.6$	$3.2$ $\pm 1.5$	32.7	21.4	$6.1$ $\pm 0.4$	$1.4$ $\pm 0.5$	13.9	6.5	12.4

Table 3: AppWorld results on Gemma4-E4B . The best performance is in bold and second best is underlined. Avg is the mean of the eight metrics.

method	resolved	pass@5	resolved	pass@5	Avg
	Qwen3-4B		Qwen3-8B
Base	$3.56$ $\pm 0.61$	8.00	$4.72$ $\pm 0.69$	10.80	6.77
GRPO	$3.68$ $\pm 0.63$	9.00	$7.08$ $\pm 0.93$	15.00	8.35
OPSD	$2.40$ $\pm 0.89$	7.00	$2.88$ $\pm 0.39$	8.20	5.12
GRPO+OPSD	$2.92$ $\pm 0.37$	7.60	$4.64$ $\pm 0.32$	12.00	6.79
Skill-SD	$1.40$ $\pm 0.44$	3.40	$1.52$ $\pm 0.35$	4.00	2.58

Table 4: SWE-bench Verified results with mini-swe-agent scaffold. The best performance is in bold and second best is underlined.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

	test-normal ( $n=168$ )				test-challenge ( $n=417$ )
analyzer input	TGC	SGC	TGC@5	SGC@5	TGC	SGC	TGC@5	SGC@5	Avg
Qwen3-4B
failures + $r_{x}$	$\mathbf{21.8}$ $\mathbf{\pm 2.4}$	$\mathbf{6.4}$ $\mathbf{\pm 2.4}$	39.9	23.2	$\mathbf{9.4}$ $\mathbf{\pm 0.4}$	$\mathbf{3.2}$ $\mathbf{\pm 0.6}$	19.9	10.1	$\mathbf{16.7}$
failures only	$15.4$ $\pm 2.5$	$5.0$ $\pm 2.3$	31.0	16.1	$3.4$ $\pm 0.8$	$0.1$ $\pm 0.3$	9.4	3.6	10.5
Qwen3-8B
failures + $r_{x}$	$\mathbf{31.6}$ $\mathbf{\pm 2.0}$	$12.1$ $\pm 1.3$	57.7	39.3	$\mathbf{16.8}$ $\mathbf{\pm 0.6}$	$\mathbf{5.9}$ $\mathbf{\pm 0.8}$	32.1	23.7	$\mathbf{27.4}$

Appendix

Table 5: Removing the reference

r_{x}

from the analyzer’s input ( Eq. 6 ). The best results are in bold.

	AppWorld	SWE-bench Verified
group size $n$	8	8
tasks per batch	32	32
PPO mini-batch size (tasks)	32	8
PPO epochs	1	1
training steps	500	70–140 (two-day budget)
checkpoint interval	5	5

Appendix

Table 6: Training hyperparameters. Shared across methods on each benchmark, with baseline coefficients are listed above.

Qwen3-4B
	test-normal ( $n=168$ )				test-challenge ( $n=417$ )
method	TGC	SGC	TGC@5	SGC@5	TGC	SGC	TGC@5	SGC@5	Avg
PSP ( $\gamma=0.1$ )	$21.8$ $\pm 2.4$	$6.4$ $\pm 2.4$	39.9	23.2	$9.4$ $\pm 0.4$	$3.2$ $\pm 0.6$	19.9	10.1	16.7
$\gamma=0.3$	$25.1$ $\pm 1.4$	$11.1$ $\pm 4.4$	47.6	35.7	$8.8$ $\pm 0.4$	$2.9$ $\pm 1.0$	18.9	9.4	19.9
$\gamma=0.5$	$\mathbf{28.0}$ $\mathbf{\pm 4.1}$	$\mathbf{13.6}$ $\mathbf{\pm 3.9}$	50.6	35.7	$\mathbf{13.1}$ $\mathbf{\pm 1.2}$	$\mathbf{4.0}$ $\mathbf{\pm 0.4}$	26.1	18.7	$\mathbf{23.7}$
always	$16.1$ $\pm 2.6$	$3.6$ $\pm 1.3$	36.9	21.4	$8.5$ $\pm 1.1$	$2.4$ $\pm 0.8$	18.0	10.1	14.6

Appendix

Table 7: Effect of the gate threshold

\gamma

. Privileged information is injected into a group when its success rate falls below

\gamma

;

\gamma=0.1

fires only on all-fail groups, while always disables the gate and injects for every sample. GRPO is included for reference.

method	TGC	SGC	TGC@5	SGC@5	TGC	SGC	TGC@5	SGC@5	Avg
	test-normal ( $n=168$ )				test-challenge ( $n=417$ )
SFT	$20.6$ $\pm 2.0$	$8.9$ $\pm 2.2$	33.9	25.0	$7.9$ $\pm 0.5$	$1.6$ $\pm 0.8$	15.8	5.0	14.8
PSP ( $\gamma=0.1$ )	$18.1$ $\pm 1.2$	$7.1$ $\pm 2.2$	36.9	21.4	$7.9$ $\pm 1.3$	$2.6$ $\pm 1.0$	17.5	7.2	14.8
PSP ( $\gamma=0.3$ )	$\mathbf{22.6}$ $\mathbf{\pm 1.3}$	$\mathbf{12.1}$ $\mathbf{\pm 3.7}$	38.7	25.0	$\mathbf{9.6}$ $\mathbf{\pm 0.4}$	$\mathbf{3.7}$ $\mathbf{\pm 0.9}$	20.4	8.6	$\mathbf{17.6}$
PSP ( $\gamma=0.5$ )	$20.0$ $\pm 2.4$	$7.5$ $\pm 2.3$	36.9	23.2	$9.3$ $\pm 0.8$	$3.0$ $\pm 0.9$	18.5	10.1	16.1

Appendix

Table 8: Comparison between supervised fine-tuning and PSP on Qwen3-4B, under an equal budget of Qwen3.6-27B output. The best results are in bold.

method	TGC	SGC	TGC@5	SGC@5	TGC	SGC	TGC@5	SGC@5	Avg
	test-normal ( $n=168$ )				test-challenge ( $n=417$ )
SFT (init)	$20.6$ $\pm 2.0$	$8.9$ $\pm 2.2$	33.9	25.0	$7.9$ $\pm 0.5$	$1.6$ $\pm 0.8$	15.8	5.0	14.8
+ GRPO	$\mathbf{27.4}$ $\mathbf{\pm 2.3}$	$11.1$ $\pm 2.9$	37.5	23.2	$10.7$ $\pm 0.8$	$\mathbf{4.7}$ $\mathbf{\pm 0.4}$	17.7	9.4	17.7
+ PSP ( $\gamma{=}0.1$ )	$24.8$ $\pm 2.3$	$12.1$ $\pm 2.3$	39.3	21.4	$11.1$ $\pm 0.5$	$3.2$ $\pm 0.8$	18.9	10.8	17.7
+ PSP ( $\gamma{=}0.3$ )	$26.3$ $\pm 1.9$	$\mathbf{18.2}$ $\mathbf{\pm 2.0}$	38.1	30.4	$10.1$ $\pm 0.7$	$3.0$ $\pm 0.9$	18.2	8.6	$\mathbf{19.1}$
+ PSP ( $\gamma{=}0.5$ )	$25.0$ $\pm 1.3$	$11.8$ $\pm 2.4$	38.1	19.6	$\mathbf{11.9}$ $\mathbf{\pm 0.5}$	$4.3$ $\pm 0.0$	19.4	8.6	17.3

Appendix

Table 9: Comparison between GRPO and PSP on Qwen3-4B when both start after the SFT warmup. The best results are in bold.

Explore similar work

DemoPSD: Disagreement-Modulated Policy Self-Distillation
Jul 2, 2026Yunhe Li, Hao Shi, Wenhao Liu +5Unsupervised On-Policy Self-Distillation Teacher-Student Distillation
1City University of Hong Kong · 2Tsinghua University · 3Shenzhen University of Advanced Technology +1
Self-Distilled Agentic Reinforcement Learning
May 14, 2026Zhengxi Lu, Zhiyuan Yao, Zhuowen Han +8Unsupervised On-Policy Self-Distillation Agentic Reinforcement Learning
Zhejiang University · Meituan · Tsinghua University
Latent On-Policy Self-Distillation
Aug 13, 2026Guibin Zhang, Jiayang Lyu, Ran Sun +4Unsupervised On-Policy Self-Distillation Self-Teacher
1National University of Singapore · 2Beijing University of Posts and Telecommunications · 3Shanghai Jiao Tong University

Jul 2, 2026, cs.LGJ/K move · Enter open · S save

DemoPSD: Disagreement-Modulated Policy Self-Distillation

Yunhe Li, Hao Shi, Wenhao Liu, Mengzhe Ruan+4

1City University of Hong Kong · 2Tsinghua University · 3Shenzhen University of Advanced Technology+1

Unsupervised On-Policy Self-Distillation Teacher-Student Distillation Data Leakage

Open paper AskPDF arXiv

On-policy self-distillation (OPSD) has emerged as a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access. However, recent studies have found that the teacher's dense token-level supervision, conditioned on privileged information, can lead to overfitting to in-domain patterns, suppress exploration, and hurt cross-domain generalization, while also introducing a more fundamental issue: privileged information leakage, where the student encodes answer-dependent shortcuts that are unavailable at test time. We introduce DemoPSD, a novel framework that resolves such problems through the idea of selective adoption of teacher guidance. Instead of fitting the full teacher distribution, DemoPSD steers the student toward a reverse-KL barycenter target, a weighted geometric combination of the teacher and student distributions, that naturally balances learning from the teacher with preserving the student's own reasoning capacity. We measure the difference between their distributions and use such a discrepancy to adaptively control the blending at each token position. We provably show that DemoPSD achieves (1) leakage attenuation, i.e., effective mitigation of privileged information leakage; and (2) exploration preservation, i.e., preservation of exploration capacity under dense token-level distillation. Extensive experiments on SciKnowEval across four scientific fields show that DemoPSD outperforms both GRPO and SDPO while maintaining higher training entropy and robustly generalizing to out-of-distribution GPQA benchmarks.

Abstract

Explore similar work

DemoPSD: Disagreement-Modulated Policy Self-Distillation

Self-Distilled Agentic Reinforcement Learning

Latent On-Policy Self-Distillation