ECOS: Environment-Calibrated On-Policy Self-Distillation for Language Agents
Authors: Chenhao Zhang, Zhongan Wang, Mingxuan Li, Hehe Fan, Zheng Zhu, Ming Li
Organizations: School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen · College of Artificial Intelligence, Zhejiang University · GigaAI
On-policy self-distillation lets a language agent act as its own Teacher by conditioning on privileged context such as skills written from its own trajectories. Its guidance, however, is weighted by how much this context shifts token likelihoods rather than by whether it improves behavior, so errors in self-written skills return through later updates. We propose environment-calibrated on-policy self-distillation, which uses environmental outcomes to decide how much to trust the Teacher's current guidance and which self-written skills to reinforce. Command-level advantages, estimated by comparing commands executed at similar observations, scale each Teacher preference and reward reflections and skills whose induced preferences favor higher-advantage commands. Without skills at inference, 3B and 7B agents reach 90.5% and 93.3% macro success on seen ALFWorld tasks and 75.5% and 80.5% success on WebShop. On unseen ALFWorld tasks, they outperform GRPO under the same protocol by 11.0 and 7.6 percentage points, respectively. In an ablation study, supervising only executable commands rather than the Actor's full response raises 3B WebShop success from 52.3% to 75.0%.
Figures & tables
Figure 1: Environmental outcomes help check an agent’s own advice. (a) Self-generated advice can carry useful lessons and mistakes into later training. (b) Our method uses observed outcomes to give useful advice more influence and conflicting advice less influence. These outcomes also train the generation of future advice. The comparison concerns how advice is assessed during training.
Figure 2: Environmental feedback connects acting with Teacher construction. Offline annotations initialize reflection and skill generation through SFT. Online, reflections ugj produce a skill zg , which conditions the pre-update policy to score Actor-sampled tokens. Token gaps δ are summed into raw command gaps d ; the mean dˉ=d/L determines a shared modulation multiplier, which scales each command token’s own gap. Environmental advantages calibrate this guidance and supply shared credit cg for reflection and skill generation. Panel (a) omits group and trajectory indices: ot+1 is the observation returned after action at .
Figure 3: An empirical transition graph provides environmental command credit. Recorded observations are grouped into comparable state clusters, and executed commands connect them through observed transitions. Bellman evaluation estimates state values; temporal-difference residuals then enter GAE and group standardization to obtain command advantages. Colors identify observation clusters.
ALFWorld
WebShop
Method
Pick
Look
Clean
Heat
Cool
Pick2
Avg.
Score
Succ.
Qwen2.5-3B-Instruct
Vanilla
44.4
11.1
6.2
15.4
28.6
12.5
21.9
6.7
0.8
Skill-Prompt*
51.7
66.7
48.4
0.0
4.3
10.0
28.9
1.2
0.8
OPSD
48.8
41.7
16.7
0.0
15.8
16.7
28.1
11.3
3.1
GRPO
91.2
62.5
96.2
61.9
65.0
47.4
75.0
79.8
63.3
Table 1: Comparison with published methods. ALFWorld reports category success and Avg.; WebShop reports Score and Success (Succ.), all on a 0–100 scale. Ours reports mean ± standard deviation over three seeds. yellow and blue mark the best and second-best results within each model size, including ties.
Model
Method
Pick
Look
Clean
Heat
Cool
Pick2
Macro
3B
GRPO
86.1 ± 2.4
79.6 ± 8.5
86.0 ± 4.9
62.3 ± 5.0
85.7 ± 4.8
49.0 ± 6.8
74.8 ± 0.9
3B
Ours
80.6 ± 2.4
92.6 ± 3.2
86.0 ± 4.9
91.3 ± 4.3
84.1 ± 9.9
80.4 ± 9.0
85.8 ± 1.8
7B
GRPO
87.5 ± 4.2
88.9 ± 5.6
93.5 ± 3.2
78.3 ± 4.3
66.7 ± 8.2
58.8 ± 10.2
78.9 ± 1.4
7B
Ours
76.4 ± 6.4
94.4
82.8 ± 1.9
85.5 ± 5.0
92.1 ± 2.7
88.2
86.6 ± 1.2
Table 2: Unseen-task generalization on ALFWorld. Category success and six-category macro success (%) on all 134 unseen tasks.
Figure 4: Environmental feedback calibrates Actor supervision and evaluates Teacher construction. (a,b) Reward alignment of Teacher supervision before and after modulation. (c,d) Return concordance of skills grouped by each task’s median R/S credit.
ALFWorld
WebShop
Variant
Macro/Avg.
Score
Success
GRPO
75.0
79.8
63.3
Full method
91.54
85.01
75.00
(a) w/o SFT
82.14
81.89
69.53
(b) Full-response supervision
89.88
75.62
52.34
(c) w/o modulation and R/S loss
85.00
82.10
72.65
Table 3: Ablation study on ALFWorld and WebShop with the 3B backbone. Our ALFWorld branches report six-category macro success (140 tasks); GRPO retains the published Avg. from Table 1 .
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Training tasks / trajectories per task
200 / 8
Supervised targets
1,600 reflections and 200 skills
Batch per optimizer update
1 task group: 8 reflections and 1 skill
Reflection / skill loss weights
0.5 / 0.5
Epochs / optimizer updates
3 / 600
Trainable parameters
Full model
Appendix
Table B.1: Stage 1 SFT settings, shared across the four task/model configurations.
Parameter
Value
Online updates (final checkpoint)
150
Task groups per update
16
Rollouts per group
8
Optimizer
AdamW
Learning rate
10−6
Adam (β1,β2)
(0.9,0.999)
Appendix
Table B.2: Online-training hyperparameters for the 3B and 7B models.
Figure B.1: ALFWorld training trajectory. The Actor locates a mug, heats it, and places it in the coffeemachine. The environment confirms completion after seven actions.
Figure B.2: WebShop training trajectory. The Actor searches for a dress, opens a product, selects large and red, and purchases it. The terminal feedback records the selected options and a score of 1.0.
Split
Pick
Look
Clean
Heat
Cool
Pick2
Total
Seen
35
13
27
16
25
24
140
Unseen
24
18
31
23
21
17
134
Appendix
Table C.1: ALFWorld evaluation tasks by category.
Variant
SFT
Scope
Mod.
R/S loss
Online R/S
Full
Yes
Command
Yes
Yes
Yes
(a)
No
Command
Yes
Yes
Yes
(b)
Yes
Response
Yes
Yes
Yes
(c)
Yes
Command
No
No
Yes
Appendix
Table C.2: Training components used in the ablation study. Scope denotes the tokens covered by direct Teacher supervision. Online R/S denotes reflection and skill generation during training.
Figure D.1: Training dynamics of the 3B agents. Columns show ALFWorld and WebShop; rows show training success, modulated-action coverage, the fraction of skills receiving positive R/S credit, and mean interaction length. Light curves show individual updates; dark curves show trailing ten-update means, using available updates at the start. Success rises and episodes shorten while environmental feedback continues to select and modulate supervision.
Figure D.2: Training dynamics of the 7B agents. Columns and rows follow Figure D.1 , with the same axis ranges and moving-average convention. Both tasks show increasing success and shorter interactions. Modulation coverage remains active on ALFWorld and declines on WebShop.