AdaStep: Adaptive Step Credit Weighting for Agentic Reinforcement Learning

Xin Wang; Wenhao Wu; Menghao Zhang; Zhi Wang; Kun Shao; Jian Luan

cs.LGOct 2, 2026

AdaStep: Adaptive Step Credit Weighting for Agentic Reinforcement Learning

Authors: Xin Wang, Wenhao Wu, Menghao Zhang, Zhi Wang, Kun Shao, Jian Luan

Organizations: Tsinghua University · Xiaomi Inc. · Nanjing University

Abstract

Long-horizon LLM agents are typically trained with sparse outcome rewards, making trajectory-level objectives too coarse to distinguish the contribution of individual decisions. Step-level credit assignment provides finer-grained supervision, but its estimates can be unreliable because observed returns also depend on subsequent actions, environment transitions, and trajectory length. We propose AdaStep, an Adaptive Step-credit weighting method that controls how strongly each group-derived local advantage modifies the trajectory-level signal. We formulate this weighting as a mean-squared-error estimation problem for the latent step advantage and, under an explicit conditional sampling assumption, derive an optimal per-state shrinkage coefficient. The coefficient admits a signal-to-total-variance interpretation: it preserves local credit when return variation is attributable to the selected action and suppresses it when variation is dominated by downstream randomness. AdaStep requires only lightweight scalar computation, with no critic, additional rollouts, or extra model inference. Experiments with three model backbones on ALFWorld, WebShop, and ScienceWorld show consistent improvements over baselines at low computational cost.

Figures & tables

Figure 1: The episode-level advantage provides an accurate but coarse signal of trajectory outcomes, while the step-level advantage offers finer-grained but noisy local credit. In GiGPO, combining these signals with equal weights can assign negative advantages to useful actions or excessively large positive advantages to actions unrelated to success. AdaStep retains the episode-level signal and adaptively weights the local correction for each step-level group.

Figure 2: Overview of AdaStep. Rollout trajectories provide both episode-level returns and step-level groups. The episode-level relative advantage $A_{E}$ supplies a global success/failure signal, while the step-level relative advantage $A_{S}$ provides local credit but is affected by return noise. AdaStep estimates the reliability of $A_{S}$ at each anchor state $\bm{s}$ using the fraction of return variance explained by action-dependent value differences, yielding the per-state weight $w^{*}(\bm{s})$ . The final advantage combines the two signals as $A(\bm{s},\bm{a})=A_{E}(\bm{s},\bm{a})+w^{*}(\bm{s})A_{S}(\bm{s},\bm{a})$ .

Frontier Models
Type	Method	ALFWorld		WebShop		ScienceWorld
Type		In-Success	Out-Success	Test Score	Success	Success
Prompting	DeepSeek-V4.1-Flash (ReAct)	$80.60{\scriptstyle\pm 0.2}$	$88.83{\scriptstyle\pm 1.9}$	$28.54{\scriptstyle\pm 1.8}$	$25.13{\scriptstyle\pm 1.5}$	$58.15{\scriptstyle\pm 4.3}$
Prompting	GLM-5.2 (ReAct)	$96.36{\scriptstyle\pm 0.9}$	$95.05{\scriptstyle\pm 1.2}$	$28.62{\scriptstyle\pm 2.1}$	$23.47{\scriptstyle\pm 1.8}$	$82.03{\scriptstyle\pm 1.4}$
Model: Qwen3-1.7B
Prompting	Pure model	$9.64{\scriptstyle\pm 1.2}$	$11.46{\scriptstyle\pm 2.4}$	$53.38{\scriptstyle\pm 3.1}$	$5.34{\scriptstyle\pm 0.8}$	$9.38{\scriptstyle\pm 2.1}$

Table 1: Main results. Performance across three benchmarks. Results are averaged over

3

random seeds. The best and second-best values within each model block are shown in bold and underlined , respectively.

\Delta

vs. GiGPO reports AdaStep minus GiGPO for the same model.

Model: Qwen3-4B
Std	Method	ALFWorld		WebShop
Std	Method	In-Suc.	Out-Suc.	Score	Suc.
w/	GiGPO	$88.02{\scriptstyle\pm 0.9}$	$83.07{\scriptstyle\pm 4.6}$	$88.77{\scriptstyle\pm 0.3}$	$77.17{\scriptstyle\pm 0.3}$
w/	AdaStep	$\mathbf{93.01}{\scriptstyle\pm 0.9}$	$\mathbf{86.58}{\scriptstyle\pm 1.2}$	$\mathbf{89.90}{\scriptstyle\pm 1.1}$	$\mathbf{79.27}{\scriptstyle\pm 0.5}$
$\Delta$		$+4.99$	$+3.51$	$+1.13$	$+2.10$
w/o	GiGPO	$88.20{\scriptstyle\pm 1.2}$	$85.17{\scriptstyle\pm 0.7}$	$87.34{\scriptstyle\pm 0.6}$	$78.28{\scriptstyle\pm 1.8}$

Table 2: Results with and without std normalization.

\Delta

denotes gains over GiGPO and best results bolded .

Observed group	Handling
One action; identical returns	$A_{S}=0$ ; no correction.
One action; varying returns	Cannot compute $\mathbb{E}_{\bm{a}}[\cdot]$ .
Multiple actions; one return observation per action	Cannot compute $\operatorname{Var}(R\mid\bm{s},\bm{a})$ .

Table 3: Left: groups outside the estimation criteria; right: Qwen2.5-7B-Instruct success rates (%) under different default weights, with the best results in bold .

Figure 3: Coefficient sensitivity and step-credit diagnostics. (a,b) Success rates for fixed weights (blue) and AdaStep (hatched green). (c,d) WebShop within-action variation and held-out reference MSE, each normalized separately to GiGPO

=100

; Fixed uses

c=0.5

and Off uses

c=0

. Appendix E gives the diagnostic protocol.

Method	Time (s)	Success (%)
GiGPO	$0.4968{\scriptstyle\pm 0.0324}$	$77.60{\scriptstyle\pm 1.7}$
HGPO	$0.6340{\scriptstyle\pm 0.0711}$	$78.51{\scriptstyle\pm 1.4}$
AdaStep	$0.5018{\scriptstyle\pm 0.0114}$	$\mathbf{82.42}{\scriptstyle\pm 1.2}$

Table 4: Credit-computation time per iteration and success rates on WebShop.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: The WebShop prompt template with <thought> and </thought> reasoning delimiters.

Panel of Figure 5 — Figure 5: The ALFWorld prompt template with <thought> and </thought> reasoning delimiters.

Panel of Figure 6 — Figure 6: The ScienceWorld prompt template with <thought> and </thought> reasoning delimiters.

Step-Level Credit Assignment in RL

Step-level credit assignment in reinforcement learning links delayed rewards to the actions or reasoning steps that produced them.

See all 165 papers

Granularity-Adaptive Credit Assignment for Long-Horizon LLM Agent Reinforcement Learning
Sep 11, 2026Taoran Liang, Yang Liu, Shang Luo +10Credit Assignment in RL Step-Level Credit Assignment in RL
Nankai University · Peking University · Supply Chain Tech Team Y, JD.com +2
AHEAD: Adaptive Hindsight with Environment-Augmented Distillation for Agentic RL
Aug 25, 2026Xiaolong Jin, Dingmin Wang, Vijay Lingam +1RL for LLM Agents LLM-Guided RL
AWS AI Labs · Purdue University
Self-Credit: Credit Assignment for Long-Horizon Agentic RL via Intra-group Rescaling
Oct 20, 2026Fangcong Yin, Ming Yin, Tanvi Bhandarkar +7Agentic RL Step-Level Credit Assignment in RL
Meta · New York University · Duke University

Sep 11, 2026J/K move · Enter open · S save

Granularity-Adaptive Credit Assignment for Long-Horizon LLM Agent Reinforcement Learning

Taoran Liang, Yang Liu, Shang Luo, Yingguang Yang+9

Nankai University · Peking University · Supply Chain Tech Team Y, JD.com+2

Credit Assignment in RL Step-Level Credit Assignment in RL Critic-Free RL Credit Assignment Policy Learning RL for LLM Agents

Open paper AskPDF arXiv

Long-horizon language-model agents trained with reinforcement learning oftenreceive sparse outcome rewards that do not reveal which decisions along a tra-jectory deserve credit. Episode-level advantages provide coarse trajectory-widecredit, while step-level comparisons offer finer resolution with context-dependentestimation noise. We propose Granularity-Adaptive Credit Assignment (GACA),a critic-free method that adaptively mixes episode- and step-level credit for eachdecision during policy optimization. GACA normalizes the sampled response'smean per-token negative log-likelihood (NLL) within each trajectory and uses theresulting criticality score to determine the step-specific mixture. The computationreuses rollout log-probabilities without additional training rollouts or model eval-uations. Our analysis characterizes optimal score-dependent mixing and derivesconditions linking expected NLL to a lower bound on the preferred step-levelweight. Across ALFWorld and WebShop with 1.5B and 7B backbones, GACAachieves the highest reported mean success rates among the compared methods,while introducing negligible additional computation.

Abstract

Step-Level Credit Assignment in RL

Explore similar work

Granularity-Adaptive Credit Assignment for Long-Horizon LLM Agent Reinforcement Learning

AHEAD: Adaptive Hindsight with Environment-Augmented Distillation for Agentic RL

Self-Credit: Credit Assignment for Long-Horizon Agentic RL via Intra-group Rescaling