Long-horizon LLM agents are typically trained with sparse outcome rewards, making trajectory-level objectives too coarse to distinguish the contribution of individual decisions. Step-level credit assignment provides finer-grained supervision, but its estimates can be unreliable because observed returns also depend on subsequent actions, environment transitions, and trajectory length. We propose AdaStep, an Adaptive Step-credit weighting method that controls how strongly each group-derived local advantage modifies the trajectory-level signal. We formulate this weighting as a mean-squared-error estimation problem for the latent step advantage and, under an explicit conditional sampling assumption, derive an optimal per-state shrinkage coefficient. The coefficient admits a signal-to-total-variance interpretation: it preserves local credit when return variation is attributable to the selected action and suppresses it when variation is dominated by downstream randomness. AdaStep requires only lightweight scalar computation, with no critic, additional rollouts, or extra model inference. Experiments with three model backbones on ALFWorld, WebShop, and ScienceWorld show consistent improvements over baselines at low computational cost.
Figures & tables
Figure 1: The episode-level advantage provides an accurate but coarse signal of trajectory outcomes, while the step-level advantage offers finer-grained but noisy local credit. In GiGPO, combining these signals with equal weights can assign negative advantages to useful actions or excessively large positive advantages to actions unrelated to success. AdaStep retains the episode-level signal and adaptively weights the local correction for each step-level group.
Figure 2: Overview of AdaStep. Rollout trajectories provide both episode-level returns and step-level groups. The episode-level relative advantage AE supplies a global success/failure signal, while the step-level relative advantage AS provides local credit but is affected by return noise. AdaStep estimates the reliability of AS at each anchor state s using the fraction of return variance explained by action-dependent value differences, yielding the per-state weight w∗(s) . The final advantage combines the two signals as A(s,a)=AE(s,a)+w∗(s)AS(s,a) .
Type
Method
ALFWorld
WebShop
ScienceWorld
In-Success
Out-Success
Test Score
Success
Success
Frontier Models
Prompting
DeepSeek-V4.1-Flash (ReAct)
80.60±0.2
88.83±1.9
28.54±1.8
25.13±1.5
58.15±4.3
Prompting
GLM-5.2 (ReAct)
96.36±0.9
95.05±1.2
28.62±2.1
23.47±1.8
82.03±1.4
Model: Qwen3-1.7B
Prompting
Pure model
9.64±1.2
11.46±2.4
53.38±3.1
5.34±0.8
9.38±2.1
Table 1: Main results. Performance across three benchmarks. Results are averaged over 3 random seeds. The best and second-best values within each model block are shown in bold and underlined , respectively. Δ vs. GiGPO reports AdaStep minus GiGPO for the same model.
Std
Method
ALFWorld
WebShop
In-Suc.
Out-Suc.
Score
Suc.
Model: Qwen3-4B
w/
GiGPO
88.02±0.9
83.07±4.6
88.77±0.3
77.17±0.3
w/
AdaStep
93.01±0.9
86.58±1.2
89.90±1.1
79.27±0.5
Δ
+4.99
+3.51
+1.13
+2.10
w/o
GiGPO
88.20±1.2
85.17±0.7
87.34±0.6
78.28±1.8
Table 2: Results with and without std normalization. Δ denotes gains over GiGPO and best results bolded .
Observed group
Handling
One action; identical returns
AS=0 ; no correction.
One action; varying returns
Cannot compute Ea[⋅] .
Multiple actions; one return observation per action
Cannot compute Var(R∣s,a) .
Table 3: Left: groups outside the estimation criteria; right: Qwen2.5-7B-Instruct success rates (%) under different default weights, with the best results in bold .
Figure 3: Coefficient sensitivity and step-credit diagnostics. (a,b) Success rates for fixed weights (blue) and AdaStep (hatched green). (c,d) WebShop within-action variation and held-out reference MSE, each normalized separately to GiGPO =100 ; Fixed uses c=0.5 and Off uses c=0 . Appendix E gives the diagnostic protocol.
Method
Time (s)
Success (%)
GiGPO
0.4968±0.0324
77.60±1.7
HGPO
0.6340±0.0711
78.51±1.4
AdaStep
0.5018±0.0114
82.42±1.2
Table 4: Credit-computation time per iteration and success rates on WebShop.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: The WebShop prompt template with <thought> and </thought> reasoning delimiters.
Figure 5: The ALFWorld prompt template with <thought> and </thought> reasoning delimiters.
Figure 6: The ScienceWorld prompt template with <thought> and </thought> reasoning delimiters.
Long-horizon language-model agents trained with reinforcement learning oftenreceive sparse outcome rewards that do not reveal which decisions along a tra-jectory deserve credit. Episode-level advantages provide coarse trajectory-widecredit, while step-level comparisons offer finer resolution with context-dependentestimation noise. We propose Granularity-Adaptive Credit Assignment (GACA),a critic-free method that adaptively mixes episode- and step-level credit for eachdecision during policy optimization. GACA normalizes the sampled response'smean per-token negative log-likelihood (NLL) within each trajectory and uses theresulting criticality score to determine the step-specific mixture. The computationreuses rollout log-probabilities without additional training rollouts or model eval-uations. Our analysis characterizes optimal score-dependent mixing and derivesconditions linking expected NLL to a lower bound on the preferred step-levelweight. Across ALFWorld and WebShop with 1.5B and 7B backbones, GACAachieves the highest reported mean success rates among the compared methods,while introducing negligible additional computation.
Taoran Liang, Yang Liu, Shang Luo +10
Nankai University · Peking University · Supply Chain Tech Team Y, JD.com +2
Training multi-turn LLM agents with reinforcement learning typically relies on trajectory-level rewards, which assign a uniform advantage to every step and cannot identify which decisions led to success or failure. Self-distillation methods can provide finer-grained supervision by augmenting RL with privileged information. However, existing approaches usually apply the same type of privileged information to every step in an indistinguishable manner, ignoring a key asymmetry: routine steps need little additional guidance, while critical error steps require corrective direction that environment feedback alone cannot provide. We propose AHEAD, a step-aware framework that matches different supervision sources to different step types. The teacher receives environment feedback on all steps as a grounded dense signal, and additionally receives LLM-generated corrective hints on error steps to supply the direction that environment feedback lacks. The method introduces minimal changes to the standard GRPO algorithm. Across ALFWorld, WebShop, and Search-based QA, and across three model scales, AHEAD raises task success (+13.3 points on ALFWorld and +11.0 on WebShop at 7B over GRPO), reaches a given success rate in fewer training steps, and solves tasks within tighter interaction budgets than outcome-only RL and prior self-distillation baselines.
LLM agents are commonly trained with RLVR, but the sparse nature of typical RLVR rewards makes credit assignment challenging, especially for long-horizon tasks. We propose Self-Credit, an end-to-end agentic RL method that creates dense supervision signals from sparse outcome rewards using self-generated credit assignment via intra-group rescaling. Self-Credit builds on group-based RL with three components. First, during its rollouts, the policy model generates summaries with citations to earlier turns containing critical evidence, which serve as self-generated credit. Second, we refine this model-generated credit with intra-group rescaling, which compares cited actions across successful and failed rollouts for the same prompt. This approach uses group-level agreement and disagreement to remove undeserved credit and further emphasize credit assigned to critical actions. Third, advantage reweighting uses the rescaled credit weights to update the advantages of the cited turns. On BrowseComp-Plus, Self-Credit improves over the base models by 37.0 and 36.2 points on Qwen3 8B and 32B respectively, and outperforms the strongest agentic RL baselines by 8.5 and 5.6 points, respectively. Self-Credit also improves over the strongest baselines by 4.9 points on GAIA and 4.7 points on xBench-DeepSearch. Analyses show that self-generated credit provides useful signals for selecting critical actions, and that intra-group rescaling, especially on failed rollouts, is important across model scales.