PR-OPD: Privileged Representation On-policy Self-Distillation for Agentic Reinforcement Learning

Muyang Li; Jie Yang; Zhengyu Fang; Junchao Zhu; Zhengkun Xiao; Ruining Deng; Zhe Jiang; Shigang Chen

cs.LGSep 29, 2026

PR-OPD: Privileged Representation On-policy Self-Distillation for Agentic Reinforcement Learning

Authors: Muyang Li, Jie Yang, Zhengyu Fang, Junchao Zhu, Zhengkun Xiao, Ruining Deng, Zhe Jiang, Shigang Chen

Organizations: University of Florida · University of Illinois at Chicago · Case Western Reserve University · Vanderbilt University ·

Abstract

Language-model agents are usually trained by reinforcement learning from one reward per episode, and privileged self-distillation enriches it by letting the same policy, given a skill, teach its skill-free self through token probabilities. However, we identify two phenomena that question this channel. Invisible Advantage: a skill in context lifts WebShop success from 42.2% to 56.2%, yet changes the probabilities of fewer than a quarter of the sampled tokens. Much to Align: a skill changes the hidden states of over 80% of response tokens, in a way that linear probes can trace back to the specific skill. To exploit this, we propose Privileged Representation On-policy Self-Distillation (PR-OPD). After a GRPO warm start, the policy writes a hindsight skill for each trajectory, re-reads its own responses with that skill as a stop-gradient teacher, and aligns its projected hidden states to the teacher's at every layer alongside the reward objective, with no external skill library, separate teacher, or inference overhead. On ALFWorld and WebShop with two backbones, PR-OPD achieves the best overall results in every setting, improving over GRPO by up to 4.7 points in ALFWorld success and 14.0 points in WebShop accuracy. Code is available at https://github.com/balibata/PR-OPD.

Figures & tables

Figure 1: Two phenomena in privileged self-distillation. (a,b) Fraction of tokens whose hidden states or sampled-token probabilities change when the policy re-reads its own responses with its self-generated skill, on ALFWorld at GRPO@100. (c) In-training WebShop validation success on Qwen2.5-3B-Instruct of the student (no skill), the teacher (skill in context), and a student trained by directly aligning to its teacher’s hidden states after branching from GRPO at step 100.

Figure 2: Overview of PR-OPD. Left: a GRPO warm start first makes the policy able to act and to analyze its own experience. Center: in each later iteration, the policy collects trajectories, writes one hindsight skill per completed trajectory, and updates with the GRPO loss plus a hidden-state alignment loss. Right: the same policy re-reads the student-sampled response twice, without and with the skill. The skill-conditioned pass is the stop-gradient teacher, and every response token in every layer of the student is passed through a lightweight MLP projector and pulled toward it. Skills and teacher states are recomputed as the policy evolves, and neither is needed at inference.

	ALFWorld							WebShop
Method	Pick	Look	Clean	Heat	Cool	Pick2	All	Acc	Score
Qwen2.5-3B-Instruct
Vanilla	44.4	28.6	25.0	7.7	14.3	4.3	23.4	0.8	5.4
Skill-Prompt	44.4	63.6	29.2	5.3	4.8	5.9	25.8	0.8	3.1
GRPO	88.9	72.7	50.0	52.6	61.9	58.8	66.4	55.5	70.4
SDAR	88.9	63.6	66.7	63.2	61.9	47.1	68.8	56.3	74.7

Table 1: Step-150 results on ALFWorld and WebShop. ALFWorld reports success rates (%), and WebShop reports accuracy (Acc, %) and score. Within each backbone, bold is the best and underline the second best.

Figure 3: Ablation and sensitivity study. (a) ALFWorld success of each variant with the 3B backbone, relative to GRPO (dashed line). (b) ALFWorld success without and with the projector for both backbones, with GRPO as dotted lines. (c) WebShop accuracy (filled) and score (open) of PR-OPD started at

N=100

N=120

. (d) Gain of direct alignment over GRPO continued from the same checkpoint on WebShop with 1.7B, averaged over six seeds. Numbers are in Appendix D .

Figure 4: How the skill acts on the teacher’s hidden states across depth. Columns show three analyses at fixed checkpoints on ALFWorld, with 3B on top and 1.7B below in (a–d). (a,b) Mean cosine distance from the skill-free context at each layer at GRPO@100, on a logarithmic axis. (c,d) Probe accuracy for the trajectory’s own versus another trajectory’s skill at each layer, with output features at the right. (e,f) Same-layer

1-\mathrm{CKA}

of the initial 3B policy under linear and debiased CKA. In (a,b,e,f), shaded bands mark the gap between the own skill and unrelated text.

Figure 5: Probing the hindsight context at selected layers. Linear-probe accuracy at GRPO@100 on ALFWorld for distinguishing a trajectory’s own skill from (a) no skill, (b) unrelated text, (c) another trajectory’s skill, and (d) a token-shuffled skill. The first three rows are output features read after the language-model head, and the shaded rows are hidden states.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Variant	Teacher channel	Success (%)	$\Delta$
GRPO	none	66.4	0.0
+ token-level OPD	token probabilities	57.8	$-8.6$
+ hidden-state alignment, w/o projector	hidden states	72.7	$+6.3$
+ hidden-state alignment, w/ projector ( PR-OPD )	hidden states	71.1	$+4.7$
SDAR, with curated skills	token probabilities	68.8	$+2.4$
Projector with the 1.7B backbone

Appendix

Table 2: Ablation on ALFWorld with the 3B backbone, and the projector with the 1.7B backbone at the bottom. All runs end at step 150.

\Delta

is the difference from GRPO in points.

Start $N$	Stage-II updates	Acc (%)	Score
100	50	49.5	72.0
120	30	57.0	73.8
Start $N$	Stage-II updates	$\Delta$ Acc	$\Delta$ Score
100	30	$+3.15$	$+0.70$
120	30	$\mathbf{+9.62}$	$+3.23$
120	10	$+8.22$	$+4.00$

Appendix

Table 3: Sensitivity to the warm-start point on WebShop with the 1.7B backbone. Top: PR-OPD at step 150. Bottom: gain of direct alignment over GRPO continued from the same checkpoint, averaged over six evaluation seeds.

Figure 6: Cross-layer representational structure under hindsight conditioning. Linear CKA on corresponding response positions of the initial Qwen2.5-3B-Instruct policy on ALFWorld. Model weights and sampled response tokens are fixed. (a) Skill-free representations compared across layers: $A_{ij}=\mathrm{CKA}(H_{0}^{(i)},H_{0}^{(j)})$ . (b) Skill-free representations compared with those conditioned on the trajectory’s self-generated skill: $B_{ij}=\mathrm{CKA}(H_{0}^{(i)},H_{\mathrm{own}}^{(j)})$ . (c) The difference $B-A$ , where blue denotes lower CKA and orange denotes higher CKA after changing the column context. Rows index skill-free layers throughout. Panels (a,b) share a common color scale; panel (c) uses a separate, zero-centered scale.

Figure 7: Alignment loss of PR-OPD during Stage II.

\mathcal{L}_{\mathrm{PRA}}

with the projector at every training step, when representation supervision starts after

N=100

(dashed) or

N=120

(solid) GRPO updates, on a logarithmic axis. Values are raw per-step training logs of a single run, without smoothing.

				w/ projector			w/o proj.
Setting	$N$	Init.	${<}0.3$	Upd. 26–30	Steps 146–150	$\cos$	Steps 146–150
3B $\cdot$ ALFWorld	100	1.360	17	0.224	0.170	0.915	0.012
3B $\cdot$ ALFWorld	120	1.391	18	0.239	0.239	0.881	0.012
3B $\cdot$ WebShop	100	1.524	19	0.228	0.175	0.913	0.008
3B $\cdot$ WebShop	120	1.405	18	0.242	0.242	0.879	0.009
1.7B $\cdot$ ALFWorld	100	1.296	11	0.180	0.134	0.933	0.031

Appendix

Table 4: Summary of the alignment curves in Figure 7 . Init. is the loss at the first Stage-II update, and

{<}0.3

is the first update at which the loss falls below 0.3. The next three columns average five consecutive steps: updates 26–30 compare the two starts after the same number of supervised updates, and steps 146–150 compare them at the end of training. The last column is the loss of the matching run without the projector, measured on the student’s raw hidden states.

RL for LLM Agents

Reinforcement learning for large language model agents trains models to plan, use tools, and adapt through feedback from interactive tasks.

See all 616 papers

PCSD: Persistent Consistency for Self-Distillation in Agentic Reinforcement Learning
Aug 3, 2026Chunji Lv, Yangguang Wei, Junlin Liu +6Language Model Distillation On-Policy Distillation
Beijing Institute of Technology · Meituan · Institute of Automation, Chinese Academy of Sciences +1
OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning
Jun 25, 2026Shuo Yang, Jinyang Wu, Zhengxi Lu +8Agentic RL LLM Agent Skill Learning
1Tsinghua University · 2Zhejiang University · 3The Chinese University of Hong Kong +2
Agentic Reinforcement Learning with Observation-Calibrated Self-Distillation
Aug 5, 2026Yi Yang, Cong Qin, Xiaodan Liu +8Agentic RL On-Policy Distillation
Meituan LongCat Interaction · Nanjing University · Peking University +3

Aug 3, 2026J/K move · Enter open · S save

PCSD: Persistent Consistency for Self-Distillation in Agentic Reinforcement Learning

Chunji Lv, Yangguang Wei, Junlin Liu, Yang Gao+5

Beijing Institute of Technology · Meituan · Institute of Automation, Chinese Academy of Sciences+1

Language Model Distillation On-Policy Distillation Sparse-Reward RL RL for LLM Agents

Open paper AskPDF arXiv

Large language model agents have shown strong potential in complex interactive tasks, yet their reinforcement learning (RL) is often hindered by sparse rewards, as a long multi-turn trajectory may receive only a single outcome-level signal. On-policy self-distillation (OPSD) provides dense token-level supervision from a privileged teacher, but the teacher may not be reliable at every position. Existing methods commonly rely on isolated token-level discrepancies, which can be sensitive to noise, or assign a shared step-level weight that may overlook positional variation. We propose Persistent Consistency Self-Distillation (PCSD), which derives token-level distillation weights from the local persistence of teacher-favoring signals. PCSD combines adaptive windows with exponentially decayed aggregation to capture persistent relative teacher support, applies trend-aware modulation to attenuate locally declining support, and produces continuous weights through sigmoid gating. The resulting objective is jointly optimized with GRPO, combining dense teacher guidance with sparse environmental feedback. Without inference-time skills, PCSD achieves the best ALFWorld Overall results among all baselines on both backbones, exceeding GRPO by 15.6 and 13.3 points and SDAR by 6.2 and 5.5 points, while remaining competitive on WebShop and gaining 15.8 points over GRPO on unseen ALFWorld split.

Abstract

RL for LLM Agents

Explore similar work

PCSD: Persistent Consistency for Self-Distillation in Agentic Reinforcement Learning

OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning

Agentic Reinforcement Learning with Observation-Calibrated Self-Distillation