ECOS: Environment-Calibrated On-Policy Self-Distillation for Language Agents
Authors: Chenhao Zhang, Zhongan Wang, Mingxuan Li, Hehe Fan, Zheng Zhu, Ming Li
Organizations: School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen · College of Artificial Intelligence, Zhejiang University · GigaAI
On-policy self-distillation lets a language agent act as its own Teacher by conditioning on privileged context such as skills written from its own trajectories. Its guidance, however, is weighted by how much this context shifts token likelihoods rather than by whether it improves behavior, so errors in self-written skills return through later updates. We propose environment-calibrated on-policy self-distillation, which uses environmental outcomes to decide how much to trust the Teacher's current guidance and which self-written skills to reinforce. Command-level advantages, estimated by comparing commands executed at similar observations, scale each Teacher preference and reward reflections and skills whose induced preferences favor higher-advantage commands. Without skills at inference, 3B and 7B agents reach 90.5% and 93.3% macro success on seen ALFWorld tasks and 75.5% and 80.5% success on WebShop. On unseen ALFWorld tasks, they outperform GRPO under the same protocol by 11.0 and 7.6 percentage points, respectively. In an ablation study, supervising only executable commands rather than the Actor's full response raises 3B WebShop success from 52.3% to 75.0%.
Figures & tables
Figure 1: Environmental outcomes help check an agent’s own advice. (a) Self-generated advice can carry useful lessons and mistakes into later training. (b) Our method uses observed outcomes to give useful advice more influence and conflicting advice less influence. These outcomes also train the generation of future advice. The comparison concerns how advice is assessed during training.
Figure 2: Environmental feedback connects acting with Teacher construction. Offline annotations initialize reflection and skill generation through SFT. Online, reflections ugj produce a skill zg , which conditions the pre-update policy to score Actor-sampled tokens. Token gaps δ are summed into raw command gaps d ; the mean dˉ=d/L determines a shared modulation multiplier, which scales each command token’s own gap. Environmental advantages calibrate this guidance and supply shared credit cg for reflection and skill generation. Panel (a) omits group and trajectory indices: ot+1 is the observation returned after action at .
Figure 3: An empirical transition graph provides environmental command credit. Recorded observations are grouped into comparable state clusters, and executed commands connect them through observed transitions. Bellman evaluation estimates state values; temporal-difference residuals then enter GAE and group standardization to obtain command advantages. Colors identify observation clusters.
ALFWorld
WebShop
Method
Pick
Look
Clean
Heat
Cool
Pick2
Avg.
Score
Succ.
Qwen2.5-3B-Instruct
Vanilla
44.4
11.1
6.2
15.4
28.6
12.5
21.9
6.7
0.8
Skill-Prompt*
51.7
66.7
48.4
0.0
4.3
10.0
28.9
1.2
0.8
OPSD
48.8
41.7
16.7
0.0
15.8
16.7
28.1
11.3
3.1
GRPO
91.2
62.5
96.2
61.9
65.0
47.4
75.0
79.8
63.3
Table 1: Comparison with published methods. ALFWorld reports category success and Avg.; WebShop reports Score and Success (Succ.), all on a 0–100 scale. Ours reports mean ± standard deviation over three seeds. yellow and blue mark the best and second-best results within each model size, including ties.
Model
Method
Pick
Look
Clean
Heat
Cool
Pick2
Macro
3B
GRPO
86.1 ± 2.4
79.6 ± 8.5
86.0 ± 4.9
62.3 ± 5.0
85.7 ± 4.8
49.0 ± 6.8
74.8 ± 0.9
3B
Ours
80.6 ± 2.4
92.6 ± 3.2
86.0 ± 4.9
91.3 ± 4.3
84.1 ± 9.9
80.4 ± 9.0
85.8 ± 1.8
7B
GRPO
87.5 ± 4.2
88.9 ± 5.6
93.5 ± 3.2
78.3 ± 4.3
66.7 ± 8.2
58.8 ± 10.2
78.9 ± 1.4
7B
Ours
76.4 ± 6.4
94.4
82.8 ± 1.9
85.5 ± 5.0
92.1 ± 2.7
88.2
86.6 ± 1.2
Table 2: Unseen-task generalization on ALFWorld. Category success and six-category macro success (%) on all 134 unseen tasks.
Figure 4: Environmental feedback calibrates Actor supervision and evaluates Teacher construction. (a,b) Reward alignment of Teacher supervision before and after modulation. (c,d) Return concordance of skills grouped by each task’s median R/S credit.
ALFWorld
WebShop
Variant
Macro/Avg.
Score
Success
GRPO
75.0
79.8
63.3
Full method
91.54
85.01
75.00
(a) w/o SFT
82.14
81.89
69.53
(b) Full-response supervision
89.88
75.62
52.34
(c) w/o modulation and R/S loss
85.00
82.10
72.65
Table 3: Ablation study on ALFWorld and WebShop with the 3B backbone. Our ALFWorld branches report six-category macro success (140 tasks); GRPO retains the published Avg. from Table 1 .
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Training tasks / trajectories per task
200 / 8
Supervised targets
1,600 reflections and 200 skills
Batch per optimizer update
1 task group: 8 reflections and 1 skill
Reflection / skill loss weights
0.5 / 0.5
Epochs / optimizer updates
3 / 600
Trainable parameters
Full model
Appendix
Table B.1: Stage 1 SFT settings, shared across the four task/model configurations.
Parameter
Value
Online updates (final checkpoint)
150
Task groups per update
16
Rollouts per group
8
Optimizer
AdamW
Learning rate
10−6
Adam (β1,β2)
(0.9,0.999)
Appendix
Table B.2: Online-training hyperparameters for the 3B and 7B models.
Figure B.1: ALFWorld training trajectory. The Actor locates a mug, heats it, and places it in the coffeemachine. The environment confirms completion after seven actions.
Figure B.2: WebShop training trajectory. The Actor searches for a dress, opens a product, selects large and red, and purchases it. The terminal feedback records the selected options and a score of 1.0.
Split
Pick
Look
Clean
Heat
Cool
Pick2
Total
Seen
35
13
27
16
25
24
140
Unseen
24
18
31
23
21
17
134
Appendix
Table C.1: ALFWorld evaluation tasks by category.
Variant
SFT
Scope
Mod.
R/S loss
Online R/S
Full
Yes
Command
Yes
Yes
Yes
(a)
No
Command
Yes
Yes
Yes
(b)
Yes
Response
Yes
Yes
Yes
(c)
Yes
Command
No
No
Yes
Appendix
Table C.2: Training components used in the ablation study. Scope denotes the tokens covered by direct Teacher supervision. Online R/S denotes reflection and skill generation during training.
Figure D.1: Training dynamics of the 3B agents. Columns show ALFWorld and WebShop; rows show training success, modulated-action coverage, the fraction of skills receiving positive R/S credit, and mean interaction length. Light curves show individual updates; dark curves show trailing ten-update means, using available updates at the start. Success rises and episodes shorten while environmental feedback continues to select and modulate supervision.
Figure D.2: Training dynamics of the 7B agents. Columns and rows follow Figure D.1 , with the same axis ranges and moving-average convention. Both tasks show increasing success and shorter interactions. Modulation coverage remains active on ALFWorld and declines on WebShop.
Language-model agents are usually trained by reinforcement learning from one reward per episode, and privileged self-distillation enriches it by letting the same policy, given a skill, teach its skill-free self through token probabilities. However, we identify two phenomena that question this channel. Invisible Advantage: a skill in context lifts WebShop success from 42.2% to 56.2%, yet changes the probabilities of fewer than a quarter of the sampled tokens. Much to Align: a skill changes the hidden states of over 80% of response tokens, in a way that linear probes can trace back to the specific skill. To exploit this, we propose Privileged Representation On-policy Self-Distillation (PR-OPD). After a GRPO warm start, the policy writes a hindsight skill for each trajectory, re-reads its own responses with that skill as a stop-gradient teacher, and aligns its projected hidden states to the teacher's at every layer alongside the reward objective, with no external skill library, separate teacher, or inference overhead. On ALFWorld and WebShop with two backbones, PR-OPD achieves the best overall results in every setting, improving over GRPO by up to 4.7 points in ALFWorld success and 14.0 points in WebShop accuracy. Code is available at https://github.com/balibata/PR-OPD.
Muyang Li, Jie Yang, Zhengyu Fang +5
University of Florida · University of Illinois at Chicago · Case Western Reserve University +2
Large language model agents are commonly trained through reinforcement learning with sparse trajectory-level rewards, which offer limited guidance on how strongly individual tokens should be updated. On-Policy Self-Distillation (OPSD) addresses this by re-scoring generated tokens under a privileged replay view to obtain dense, token-level supervision. However, we identify a confounding issue: the resulting support may reflect both the privileged information contained in the replay view and score shifts induced by the replay scaffold, making it difficult to attribute the support specifically to that information. This issue is especially pronounced when future environment observations serve as privileged information, since replaying them requires reconstructing an extended scaffold that itself perturbs token scores. To resolve this confounding, we propose Observation-Calibrated Self-Distillation (OCSD), which contrasts two structurally matched replay views, Full and Observation-Ablated, differing only in whether the actual future observation is present, to derive an observation residual that discounts score changes shared by the replay scaffold. OCSD then applies this residual to modulate token-level GRPO updates at high-uncertainty steps, while preserving the trajectory-level update direction. Experiments on ALFWorld, WebShop, and Search-QA across three Qwen3 model scales show that OCSD consistently outperforms strong baselines. Diagnostic analyses further confirm that the calibrated residual aligns better with local environment feedback. Our code is publicly available at https://github.com/yiy1x/OCSD.
Yi Yang, Cong Qin, Xiaodan Liu +8
Meituan LongCat Interaction · Nanjing University · Peking University +3
Outcome-based reinforcement learning provides coarse supervision for long-horizon LLM agents: failed trajectories may contain correct steps, while successful trajectories can contain errors. Trajectory-level skills can also shift teacher preferences at unrelated steps, introducing off-target distillation signals. We introduce SCOPE, a framework for Selective Critical-Step On-Policy Self-Distillation. After a supervised warm-up, the policy analyzes its own trajectories, generates step-specific hindsight skills, and distills their guidance only at selected error steps. A bounded signed teacher-student coefficient stabilizes auxiliary updates while preserving positive reinforcement and negative correction. A success-relative pairwise length reward further encourages efficient task completion. Experiments on ALFWorld and WebShop demonstrate consistent improvements across multiple model backbones. With Qwen2.5-3B, SCOPE achieves 97.7% success on ALFWorld and 85.2% on WebShop, outperforming SEED by 5.9 and 6.3 percentage points, respectively. On unseen ALFWorld tasks, SCOPE reaches 89.6% success, improving over GRPO by 23.2 percentage points with 46.7% fewer interactions. Scaling to Qwen3-32B, SCOPE achieves 99.2% success on seen ALFWorld tasks and 94.0% on unseen tasks.