Organizations: University of Florida · University of Illinois at Chicago · Case Western Reserve University · Vanderbilt University · Weill Cornell Medicine
Language-model agents are usually trained by reinforcement learning from one reward per episode, and privileged self-distillation enriches it by letting the same policy, given a skill, teach its skill-free self through token probabilities. However, we identify two phenomena that question this channel. Invisible Advantage: a skill in context lifts WebShop success from 42.2% to 56.2%, yet changes the probabilities of fewer than a quarter of the sampled tokens. Much to Align: a skill changes the hidden states of over 80% of response tokens, in a way that linear probes can trace back to the specific skill. To exploit this, we propose Privileged Representation On-policy Self-Distillation (PR-OPD). After a GRPO warm start, the policy writes a hindsight skill for each trajectory, re-reads its own responses with that skill as a stop-gradient teacher, and aligns its projected hidden states to the teacher's at every layer alongside the reward objective, with no external skill library, separate teacher, or inference overhead. On ALFWorld and WebShop with two backbones, PR-OPD achieves the best overall results in every setting, improving over GRPO by up to 4.7 points in ALFWorld success and 14.0 points in WebShop accuracy. Code is available at https://github.com/balibata/PR-OPD.
Figures & tables
Figure 1: Two phenomena in privileged self-distillation. (a,b) Fraction of tokens whose hidden states or sampled-token probabilities change when the policy re-reads its own responses with its self-generated skill, on ALFWorld at GRPO@100. (c) In-training WebShop validation success on Qwen2.5-3B-Instruct of the student (no skill), the teacher (skill in context), and a student trained by directly aligning to its teacher’s hidden states after branching from GRPO at step 100.
Figure 2: Overview of PR-OPD. Left: a GRPO warm start first makes the policy able to act and to analyze its own experience. Center: in each later iteration, the policy collects trajectories, writes one hindsight skill per completed trajectory, and updates with the GRPO loss plus a hidden-state alignment loss. Right: the same policy re-reads the student-sampled response twice, without and with the skill. The skill-conditioned pass is the stop-gradient teacher, and every response token in every layer of the student is passed through a lightweight MLP projector and pulled toward it. Skills and teacher states are recomputed as the policy evolves, and neither is needed at inference.
ALFWorld
WebShop
Method
Pick
Look
Clean
Heat
Cool
Pick2
All
Acc
Score
Qwen2.5-3B-Instruct
Vanilla
44.4
28.6
25.0
7.7
14.3
4.3
23.4
0.8
5.4
Skill-Prompt
44.4
63.6
29.2
5.3
4.8
5.9
25.8
0.8
3.1
GRPO
88.9
72.7
50.0
52.6
61.9
58.8
66.4
55.5
70.4
SDAR
88.9
63.6
66.7
63.2
61.9
47.1
68.8
56.3
74.7
Table 1: Step-150 results on ALFWorld and WebShop. ALFWorld reports success rates (%), and WebShop reports accuracy (Acc, %) and score. Within each backbone, bold is the best and underline the second best.
Figure 3: Ablation and sensitivity study. (a) ALFWorld success of each variant with the 3B backbone, relative to GRPO (dashed line). (b) ALFWorld success without and with the projector for both backbones, with GRPO as dotted lines. (c) WebShop accuracy (filled) and score (open) of PR-OPD started at N=100 or N=120 . (d) Gain of direct alignment over GRPO continued from the same checkpoint on WebShop with 1.7B, averaged over six seeds. Numbers are in Appendix D .
Figure 4: How the skill acts on the teacher’s hidden states across depth. Columns show three analyses at fixed checkpoints on ALFWorld, with 3B on top and 1.7B below in (a–d). (a,b) Mean cosine distance from the skill-free context at each layer at GRPO@100, on a logarithmic axis. (c,d) Probe accuracy for the trajectory’s own versus another trajectory’s skill at each layer, with output features at the right. (e,f) Same-layer 1−CKA of the initial 3B policy under linear and debiased CKA. In (a,b,e,f), shaded bands mark the gap between the own skill and unrelated text.
Figure 5: Probing the hindsight context at selected layers. Linear-probe accuracy at GRPO@100 on ALFWorld for distinguishing a trajectory’s own skill from (a) no skill, (b) unrelated text, (c) another trajectory’s skill, and (d) a token-shuffled skill. The first three rows are output features read after the language-model head, and the shaded rows are hidden states.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Variant
Teacher channel
Success (%)
Δ
GRPO
none
66.4
0.0
+ token-level OPD
token probabilities
57.8
−8.6
+ hidden-state alignment, w/o projector
hidden states
72.7
+6.3
+ hidden-state alignment, w/ projector ( PR-OPD )
hidden states
71.1
+4.7
SDAR, with curated skills
token probabilities
68.8
+2.4
Projector with the 1.7B backbone
Appendix
Table 2: Ablation on ALFWorld with the 3B backbone, and the projector with the 1.7B backbone at the bottom. All runs end at step 150. Δ is the difference from GRPO in points.
Start N
Stage-II updates
Acc (%)
Score
100
50
49.5
72.0
120
30
57.0
73.8
Start N
Stage-II updates
Δ Acc
Δ Score
100
30
+3.15
+0.70
120
30
+9.62
+3.23
120
10
+8.22
+4.00
Appendix
Table 3: Sensitivity to the warm-start point on WebShop with the 1.7B backbone. Top: PR-OPD at step 150. Bottom: gain of direct alignment over GRPO continued from the same checkpoint, averaged over six evaluation seeds.
Figure 6: Cross-layer representational structure under hindsight conditioning. Linear CKA on corresponding response positions of the initial Qwen2.5-3B-Instruct policy on ALFWorld. Model weights and sampled response tokens are fixed. (a) Skill-free representations compared across layers: Aij=CKA(H0(i),H0(j)) . (b) Skill-free representations compared with those conditioned on the trajectory’s self-generated skill: Bij=CKA(H0(i),Hown(j)) . (c) The difference B−A , where blue denotes lower CKA and orange denotes higher CKA after changing the column context. Rows index skill-free layers throughout. Panels (a,b) share a common color scale; panel (c) uses a separate, zero-centered scale.
Figure 7: Alignment loss of PR-OPD during Stage II. LPRA with the projector at every training step, when representation supervision starts after N=100 (dashed) or N=120 (solid) GRPO updates, on a logarithmic axis. Values are raw per-step training logs of a single run, without smoothing.
w/ projector
w/o proj.
Setting
N
Init.
<0.3
Upd. 26–30
Steps 146–150
cos
Steps 146–150
3B ⋅ ALFWorld
100
1.360
17
0.224
0.170
0.915
0.012
120
1.391
18
0.239
0.239
0.881
0.012
3B ⋅ WebShop
100
1.524
19
0.228
0.175
0.913
0.008
120
1.405
18
0.242
0.242
0.879
0.009
1.7B ⋅ ALFWorld
100
1.296
11
0.180
0.134
0.933
0.031
Appendix
Table 4: Summary of the alignment curves in Figure 7 . Init. is the loss at the first Stage-II update, and <0.3 is the first update at which the loss falls below 0.3. The next three columns average five consecutive steps: updates 26–30 compare the two starts after the same number of supervised updates, and steps 146–150 compare them at the end of training. The last column is the loss of the matching run without the projector, measured on the student’s raw hidden states.
Large language model agents have shown strong potential in complex interactive tasks, yet their reinforcement learning (RL) is often hindered by sparse rewards, as a long multi-turn trajectory may receive only a single outcome-level signal. On-policy self-distillation (OPSD) provides dense token-level supervision from a privileged teacher, but the teacher may not be reliable at every position. Existing methods commonly rely on isolated token-level discrepancies, which can be sensitive to noise, or assign a shared step-level weight that may overlook positional variation. We propose Persistent Consistency Self-Distillation (PCSD), which derives token-level distillation weights from the local persistence of teacher-favoring signals. PCSD combines adaptive windows with exponentially decayed aggregation to capture persistent relative teacher support, applies trend-aware modulation to attenuate locally declining support, and produces continuous weights through sigmoid gating. The resulting objective is jointly optimized with GRPO, combining dense teacher guidance with sparse environmental feedback. Without inference-time skills, PCSD achieves the best ALFWorld Overall results among all baselines on both backbones, exceeding GRPO by 15.6 and 13.3 points and SDAR by 6.2 and 5.5 points, while remaining competitive on WebShop and gaining 15.8 points over GRPO on unseen ALFWorld split.
Chunji Lv, Yangguang Wei, Junlin Liu +6
Beijing Institute of Technology · Meituan · Institute of Automation, Chinese Academy of Sciences +1
Outcome-based reinforcement learning provides a stable optimization backbone for language agents, but its sparse trajectory-level rewards provide little guidance on which intermediate decisions should be reinforced or suppressed. On-policy self-distillation offers dense token-level supervision, yet existing skill-conditioned variants often rely on external skill memories or retrieved privileged context, which are costly to maintain and can be mismatched with the state distribution induced by the current policy in multi-turn interaction. We propose OPID (On-Policy Skill Distillation), a framework that extracts skill supervision directly from completed on-policy trajectories. OPID represents trajectory hindsight as hierarchical skills: episode-level skills capture global workflows or failure-avoidance rules, while step-level skills capture local decision knowledge at critical timesteps. A critical-first routing mechanism uses step-level skills when critical decisions are identified and falls back to episode-level skills as default guidance otherwise. The selected skill is injected into the interaction history, allowing the old policy to re-score the same sampled response under both original and skill-augmented contexts. The resulting log-probability shift yields a token-level self-distillation advantage, which is combined with the outcome advantage for policy optimization. OPID thus preserves RL as the primary training objective while introducing dense, distribution-matched hindsight supervision. Experiments on ALFWorld, WebShop and Search-based QA demonstrate that OPID generally improves agent performance, sample efficiency, and robustness over outcome-only RL and existing skill-distillation baselines. Our code is available at https://github.com/jinyangwu/OPID/tree/main.
Shuo Yang, Jinyang Wu, Zhengxi Lu +8
1Tsinghua University · 2Zhejiang University · 3The Chinese University of Hong Kong +2
Large language model agents are commonly trained through reinforcement learning with sparse trajectory-level rewards, which offer limited guidance on how strongly individual tokens should be updated. On-Policy Self-Distillation (OPSD) addresses this by re-scoring generated tokens under a privileged replay view to obtain dense, token-level supervision. However, we identify a confounding issue: the resulting support may reflect both the privileged information contained in the replay view and score shifts induced by the replay scaffold, making it difficult to attribute the support specifically to that information. This issue is especially pronounced when future environment observations serve as privileged information, since replaying them requires reconstructing an extended scaffold that itself perturbs token scores. To resolve this confounding, we propose Observation-Calibrated Self-Distillation (OCSD), which contrasts two structurally matched replay views, Full and Observation-Ablated, differing only in whether the actual future observation is present, to derive an observation residual that discounts score changes shared by the replay scaffold. OCSD then applies this residual to modulate token-level GRPO updates at high-uncertainty steps, while preserving the trajectory-level update direction. Experiments on ALFWorld, WebShop, and Search-QA across three Qwen3 model scales show that OCSD consistently outperforms strong baselines. Diagnostic analyses further confirm that the calibrated residual aligns better with local environment feedback. Our code is publicly available at https://github.com/yiy1x/OCSD.
Yi Yang, Cong Qin, Xiaodan Liu +8
Meituan LongCat Interaction · Nanjing University · Peking University +3