Training multi-turn LLM agents with reinforcement learning typically relies on trajectory-level rewards, which assign a uniform advantage to every step and cannot identify which decisions led to success or failure. Self-distillation methods can provide finer-grained supervision by augmenting RL with privileged information. However, existing approaches usually apply the same type of privileged information to every step in an indistinguishable manner, ignoring a key asymmetry: routine steps need little additional guidance, while critical error steps require corrective direction that environment feedback alone cannot provide. We propose AHEAD, a step-aware framework that matches different supervision sources to different step types. The teacher receives environment feedback on all steps as a grounded dense signal, and additionally receives LLM-generated corrective hints on error steps to supply the direction that environment feedback lacks. The method introduces minimal changes to the standard GRPO algorithm. Across ALFWorld, WebShop, and Search-based QA, and across three model scales, AHEAD raises task success (+13.3 points on ALFWorld and +11.0 on WebShop at 7B over GRPO), reaches a given success rate in fewer training steps, and solves tasks within tighter interaction budgets than outcome-only RL and prior self-distillation baselines.
Figures & tables
Figure 1: Existing self-distillation methods apply a single source of privileged information (PI) uniformly across all steps (top): on error steps such as t3 and t5 , environment feedback (e.g., “Nothing happens”) only signals failure and provides no corrective direction. Ahead (bottom) uses environment feedback as base PI on all steps, and adaptively provides an LLM-generated corrective hint on error steps, telling the agent what it should have done.
Figure 2: Overview of Ahead . Stage 1 identifies error steps in failed trajectories via an LLM analyzer. Stage 2 constructs step-aware privileged information: environment feedback for routine steps, environment feedback combined with an LLM corrective hint for error steps. Stage 3 computes a token-level self-distillation signal δt,ℓ by comparing the policy’s log-probabilities under original and PI-augmented contexts, and converts it into a bounded reweight of the GRPO advantage. Successful trajectories bypass the PI pipeline entirely and retain the vanilla GRPO advantage.
ALFWorld
Search-based QA
WebShop
Method
Pick
Look
Clean
Heat
Cool
Pick2
Avg
NQ
Triv
Pop
Hotp
2Wk
MuS
Bam
Avg
Score
Succ.
Qwen2.5-3B-Instruct
Vanilla
44.4
11.1
6.2
15.4
28.6
12.5
21.9
24.6
48.1
31.0
26.3
25.3
7.2
59.7
31.7
6.7
0.8
Skill-Prompt*
51.7
66.7
48.4
0.0
4.3
10.0
28.9
23.7
46.2
30.6
24.4
22.1
7.5
12.5
23.9
0.2
0.8
OPSD
48.8
41.7
16.7
0.0
15.8
16.7
28.1
0.1
0.1
0.1
0.0
0.0
0.0
0.0
0.0
11.3
3.1
GRPO
91.2
62.5
96.2
61.9
65.0
47.4
75.0
39.3
60.6
41.1
37.4
34.6
15.4
26.4
36.4
79.8
63.3
Table 1: Performance Comparison on long-horizon benchmarks. We report the success rate (%) on ALFWorld, accuracy on search-based QA, and task-completion score/success rate on WebShop. An asterisk (*) denotes validation with skills. Within each backbone block, the best result in a column is in bold and the second-best is underlined. Baseline numbers are taken from Lu et al. (2026b) .
Figure 4
Components
ALFWorld
WebShop
Configuration
Env. fb.
Hints
Multi
Fail-only
Decay
1.7B
3B
7B
1.7B
3B
7B
Ahead
✓
✓
✓
✓
✓
67.2
87.5
94.5
64.8
73.4
83.6
w/o decay
✓
✓
✓
✓
×
60.2
71.1
85.2
54.7
66.4
83.6
w/o failure-only
✓
✓
✓
×
✓
48.4
82.0
93.0
64.1
72.7
75.8
w/o multi-step
✓
✓
×
✓
✓
61.7
81.2
86.7
57.0
47.7
75.0
w/o hints
✓
×
–
✓
✓
53.1
73.4
87.5
63.3
68.0
74.2
Table 2: Component ablation. Success rate (%) on ALFWorld and WebShop. Ticks give the configuration of each run, so rows differing in a single tick are one-component comparisons. Env. fb. : environment feedback as base PI. Hints : the LLM-generated corrective hint Φllm on error steps. Multi : the analyzer may mark several error steps per trajectory rather than one. Fail-only : PI is applied to failed trajectories only (Sec. 2.3 ). Decay : the linear schedule on λk . The last row removes both the hints and the failure-only filter, leaving environment feedback applied uniformly.
Analyzer
3B
1.7B
Opus 4.7 (default)
87.5
67.2
Sonnet 5
89.1
68.0
Kimi
83.6
68.0
GLM-5
81.2
67.2
GRPO (no analyzer)
75.0
46.1
Table 3: ALFWorld success rate (%) when the error-step analyzer is swapped, holding everything else fixed. GRPO, which uses no analyzer, is repeated from Table 1 as a floor.
Figure 5: Trajectory comparison on an ALFWorld task put a hot plate in cabinet . Left: a GRPO-trained policy overlooks the visible plates, revisits the same countertops and empty cabinets, and never takes, heats, or moves an object before the 50-step limit (failed trajectory). Right: a Ahead -trained policy grounds each action in the current observation and completes the find → pick → heat → place sequence in nine steps. Reasoning and observation excerpts are lightly condensed for space; “ … ” marks omissions and retained text is verbatim.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
#Train
#Test
ALFWorld
2,400
128 (Val-128) / 140 (Seen-140) / 134 (Unseen-134)
WebShop
2,400
128
NQ, TriviaQA, PopQA, HotpotQA,
19,200
51,713
2WikiMultiHopQA, MuSiQue, Bamboogle
Appendix
Table 4: Datasets used in our experiments: ALFWorld (embodied reasoning), WebShop (web navigation), and Search-based QA. For Search-based QA we train only on NQ and HotpotQA; the remaining five datasets are held out and never seen during training.
Table 9Figure 10
Figure 8: Training dynamics on ALFWorld (7B); the success-rate panel is measured on the seen split. AHEAD attains a higher success rate and train reward throughout training, while driving average trajectory length down faster. The agent solves more tasks and does so in fewer steps.
Figure 9: Training dynamics on ALFWorld (Qwen3-1.7B-Instruct); the success-rate panel is measured on the seen split. Companion to Figure 8 .
Figure 10: The reweighting signal targets error steps. Per-step distributions on the reweighted steps Mτ (ALFWorld, frozen step-25 checkpoint). (a) The gap ∣δt,ℓ∣ is larger on error steps t∈Eτ than on routine steps (AUROC 0.87 ). (b) After the clip and mixing λk , the reweight that actually scales the advantage, ∣A~t,ℓ/Aep−1∣ , still favors error steps but with a smaller gap (AUROC 0.70 ), leaving the reward in control of the update direction.
Figure 11: Token-level credit on one failed trajectory (ALFWorld). Top: signed gap δt,ℓ ; red marks tokens the teacher disfavors ( δ<0 ). At the error step ( t∈Eτ ) the tokens for the wrong decision have the largest ∣δ∣ ; the routine step ( t∈/Eτ ) stays near zero. Bottom: the weight wt,ℓ=clip(exp(sgn(Aep)δt,ℓ),1−ϵ,1+ϵ) . Since Aep<0 on failed trajectories, these tokens get the largest amplification ( w>1 ).
Hyperparameter
Value
Training steps
150
Training batch size
16 for ALFWorld and WebShop; 128 for Search-based QA
Rollout group size N
8
Learning rate
1×10−6
Hardware
8 × H100
GRPO clipping range ϵ
0.2
Appendix
Table 7: Training configuration. The block in the middle lists the parameters introduced by Ahead (Sec. 2.4 ); everything else is inherited unchanged from the GRPO baseline, so the comparison in Table 1 isolates the effect of step-aware distillation.
Figure 12: Prompt template for the ALFWorld environment.
Figure 13: Prompt template for the WebShop environment.
Figure 14: Prompt template for the Search-based QA environment.
Figure 15: Prompt template for the LLM error-step analyzer.
Reinforcement learning can train LLM agents from sparse task rewards, but long-horizon credit assignment remains challenging: a single success-or-failure signal must be distributed across many actions. Existing methods rely on trajectory-level rewards or proxy signals, without fully leveraging per-step environmental feedback. Multi-turn agent settings are underexplored, where feedback can include error messages, page changes, observations, or reference trajectories. We systematically study five feedback sources and two insertion granularities and introduce SERL, a selective environment-reweighted learning framework. SERL uses the task reward to determine update direction, while environment feedback adjusts placement and magnitude, focusing on critical actions. On ALFWorld and WebShop, SERL achieves 90.0% and 80.1% success, outperforming strong RL and distillation baselines. Analysis shows that grounded, action-relevant feedback at meaningful points consistently outperforms indiscriminate use of longer or richer context.
Xiaozhe Li, Tianyi Lyu, Yang Li +6
1Tongji University · 2Shanghai AI Laboratory · 5Independent +2
Training long-horizon LLM agents with reinforcement learning is challenging because sparse outcome rewards reveal whether a task succeeds, but not which intermediate actions caused the outcome or how they should be corrected. Recent methods alleviate this issue by generating rewards or textual hints from turn-level action-output signals, or by using feedback-conditioned self-distillation. However, generating feedback at every turn is inefficient when many intermediate turns are already successful or neutral, and applying feedback at a fixed or misaligned turn often fails to supervise the actions that contributed to the failure. To bridge this gap, we propose HINT-SD, a targeted self-distillation framework that uses full-trajectory hindsight to select failure-relevant actions and applies feedback-conditioned distillation only on targeted action spans. Experiments on BFCL v3 and AppWorld show that our method improves over the dense per-turn feedback baseline by up to 18.80 percent while achieving 2.26× lower time per training step, suggesting that selecting where to distill is a key factor for both effective and efficient long-horizon agent training.
Reinforcement learning (RL) has been widely used to train LLM agents for multi-turn interactive tasks, but its sample efficiency is severely limited by sparse rewards and long horizons. On-policy self-distillation (OPSD) alleviates this by providing dense token-level supervision from a privileged teacher that has access to ground-truth answers. However, such fixed privileged information cannot capture the diverse valid strategies in agent tasks, and naively combining OPSD with RL often leads to training collapse. To address these limitations, we introduce Skill-SD, a framework that turns the agent's own trajectories into dynamic training-only supervision. Completed trajectories are summarized into compact natural language skills that describe successful behaviors, mistakes, and workflows. These skills serve as dynamic privileged information conditioning only the teacher, while the student always acts under the plain task prompt and learns to internalize the guidance through distillation. To stabilize the training, we derive an importance-weighted reverse-KL loss to provide gradient-correct token-level distillation, and dynamically synchronize the teacher with the improving student. Experimental results on agentic benchmarks demonstrate that Skill-SD substantially outperforms the standard RL baseline, improving both vanilla GRPO (+14.0%/+10.9% on AppWorld/Sokoban) and vanilla OPD (+42.1%/+40.6%). Project page: https://k1xe.github.io/skill-sd/
Hao Wang, Guozhi Wang, Han Xiao +8
Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences · vivo AI Lab · The Chinese University of Hong Kong +2