SHARPO: Segment-Level Credit Assignment for Agentic Reinforcement Learning

Xinchen Du; Zhengze Zhou; Wenhui Zhu; Han Yu; Sen Na; Rohit Jain; Alborz Geramifard

cs.LGSep 30, 2026

SHARPO: Segment-Level Credit Assignment for Agentic Reinforcement Learning

Authors: Xinchen Du, Zhengze Zhou, Wenhui Zhu, Han Yu, Sen Na, Rohit Jain, Alborz Geramifard

Organizations: LinkedIn Corporation · Georgia Institute of Technology

Abstract

Agentic reinforcement learning (RL) trains a large language model (LLM) to act over long, multi-step interactions. However, a single localized error can cause task failure, while trajectory-level rewards provide limited guidance for assigning credit to individual decisions. To address this limitation, we introduce Segment-level Hindsight Advantage Reweighting for Policy Optimization (SHARPO), a credit-assignment mechanism that refines Group Relative Policy Optimization (GRPO) at the level of environment-facing segments. Inspired by the existing on-policy self-distillation (OPSD) method, SHARPO computes teacher-student log-probability gaps within each segment and uses the resulting signal to compute a bounded multiplier on the GRPO advantage. This multiplier is shared by all tokens within the segment, allowing credit to vary across different segments. With Qwen2.5-7B-Instruct, SHARPO outperforms existing baselines on the ALFWorld and WebShop benchmarks, including GRPO, SDAR, RLSD, and StepOPSD.

Figures & tables

Panel of Figure 1 — Figure 1: Final validation success (%) with Qwen2.5-7B-Instruct. The dashed line marks GRPO. SHARPO improves on GRPO at both mixing weights on both benchmarks, reaching $84.90\%$ on ALFWorld ( $+14.32$ points) and $75.26\%$ on WebShop ( $+9.11$ points). Bars are means over three independent runs at training step 200; the vertical axis starts at $50\%$ . Table 1 reports all baselines with standard deviations.

Figure 2: SHARPO Framework. A successful peer supplies context for a policy-copy teacher. A length-normalized likelihood comparison of each selected segment produces a bounded weight. Length balancing and interpolation with the base advantage then determine the segment’s effective advantage for the policy update.

Method	Success rate (%)	Gain	Success rate (%)	Gain
	ALFWorld		WebShop
GRPO	$70.57\pm 7.83$	–	$66.15\pm 1.19$	–
SDAR	$67.19\pm 9.50$	$-3.39$	$61.72\pm 6.10$	$-4.43$
RLSD	$73.96\pm 5.76$	$+3.39$	$69.01\pm 6.31$	$+2.86$
GRPO+OPSD	$72.92\pm 4.51$	$+2.34$	$69.01\pm 3.16$	$+2.86$
StepOPSD ( $\lambda=0.2$ )	$77.60\pm 2.96$	$+7.03$	$69.27\pm 1.63$	$+3.12$

Table 1: Final validation success rate (%) at step 200 with Qwen2.5-7B-Instruct, mean

\pm

sample standard deviation over three independent runs. Gain is the change in mean success rate relative to GRPO, in percentage points. Red and blue indicate the best and second-best mean results in each column, respectively.

Method	Pick	Look	Clean	Heat	Cool	Pick2	All
GRPO	$89.22$	$73.33$	$81.39$	$61.90$	$58.21$	$60.06$	$70.57$
SDAR	$80.94$	$82.22$	$76.55$	$71.43$	$54.68$	$48.00$	$67.19$
RLSD	$89.65$	$74.44$	$88.49$	$55.24$	$55.04$	$70.71$	$73.96$
GRPO+OPSD	$90.95$	$81.72$	$77.92$	$39.05$	$62.29$	$66.04$	$72.92$
StepOPSD ( $\lambda=0.2$ )	$90.96$	$83.33$	$83.50$	$52.86$	$68.46$	$75.38$	$77.60$
StepOPSD ( $\lambda=0.05$ )	$90.32$	$85.86$	$88.60$	$69.52$	$65.70$	$64.67$	$78.12$

Table 2: ALFWorld success rate (%) by task family at step 200, mean over three independent runs. Family rates are averaged separately; All reports the mean overall validation success rate from Table 1 . Red and blue indicate the best and second-best mean results in each column, respectively.

Method	Task score
GRPO	$0.7781$
StepOPSD ( $\lambda=0.2$ )	$0.8231$
StepOPSD ( $\lambda=0.05$ )	$0.8268$
SHARPO ( $\lambda=0.2$ )	$0.8822$
SHARPO ( $\lambda=0.05$ )	$0.8480$

Table 3: WebShop task score. Higher is better. Red and blue mark the best and second-best results among the listed methods.

Explore similar work

Targeting Pivotal Decisions for Credit Assignment in Agentic Reinforcement Learning
Sep 28, 2026Dongwon Jung, Hemanth Neelgund Ramesh, Yifan Wang +7Credit Assignment Agentic Reinforcement Learning
University of California, Davis · Microsoft · University of Washington
Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learning
May 26, 2026Xin Cheng, Shuo He, Lang Feng +4Group-Based Reinforcement Learning Agentic Reinforcement Learning
Nanyang Technological University, Singapore · Tongyi Lab, Alibaba Group · Southeast University, China
Semantic Consistency Policy Optimization for Reinforcement Learning of LLM Agents
Jun 24, 2026Peng Xu, Sijia Chen, Junzhuo Li +1Large Language Model Agents Step-Level Credit
†The Hong Kong University of Science and Technology (Guangzhou) · ‡The Hong Kong University of Science and Technology

Sep 28, 2026, cs.CLJ/K move · Enter open · S save

Targeting Pivotal Decisions for Credit Assignment in Agentic Reinforcement Learning

Dongwon Jung, Hemanth Neelgund Ramesh, Yifan Wang, Xiaomin Li+6

University of California, Davis · Microsoft · University of Washington

Credit Assignment Agentic Reinforcement Learning Theorem Proving

Open paper AskPDF arXiv

Group Relative Policy Optimization (GRPO) has become a promising approach for training large language model agents. However, its uniform assignment of trajectory-level advantages to all policy tokens fails to distinguish consequential decisions from less relevant ones, obscuring which intermediate decisions contributed to success. We introduce ProVer, a framework that targets potentially pivotal decisions for fine-grained credit assignment in agentic reinforcement learning. Given a rollout group, an agentic judge contrasts successful and failed trajectories to propose a segment potentially responsible for their divergent outcomes. Rather than directly trusting the judge's assessment, ProVer verifies the proposed segment by estimating its advantage from the difference in terminal success rates between current-policy continuations sampled before and after the segment. Positive estimates are then incorporated into the GRPO advantages of policy tokens within the proposed segment. By using model judgment only to select where to verify, ProVer grounds local credit in observed outcomes without exhaustively evaluating every intermediate state. Across ALFWorld, WebShop, and SearchQA, ProVer achieves the strongest average performance at both model scales, with relative improvements over GRPO of 9.91% and 7.12% for Qwen3.5-2B and Qwen3.5-4B, respectively. Further analyses demonstrate that informed segment selection improves policy training with modest additional generation overhead, even without a frontier-scale judge model, highlighting the effectiveness and efficiency of selectively targeting pivotal decisions for fine-grained credit assignment in agentic reinforcement learning.

Abstract

Explore similar work

Targeting Pivotal Decisions for Credit Assignment in Agentic Reinforcement Learning

Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learning

Semantic Consistency Policy Optimization for Reinforcement Learning of LLM Agents