Targeting Pivotal Decisions for Credit Assignment in Agentic Reinforcement Learning

Dongwon Jung; Hemanth Neelgund Ramesh; Yifan Wang; Xiaomin Li; Yuexing Hao; Yu Hu; Muhao Chen; Varun Chandrasekaran

cs.CLSep 28, 2026

Targeting Pivotal Decisions for Credit Assignment in Agentic Reinforcement Learning

Authors: Dongwon Jung, Hemanth Neelgund Ramesh, Yifan Wang, Xiaomin Li, Yuexing Hao, Yu Hu, Muhao Chen, Varun Chandrasekaran, +2 more

Organizations: University of California, Davis · Microsoft · University of Washington

Abstract

Group Relative Policy Optimization (GRPO) has become a promising approach for training large language model agents. However, its uniform assignment of trajectory-level advantages to all policy tokens fails to distinguish consequential decisions from less relevant ones, obscuring which intermediate decisions contributed to success. We introduce ProVer, a framework that targets potentially pivotal decisions for fine-grained credit assignment in agentic reinforcement learning. Given a rollout group, an agentic judge contrasts successful and failed trajectories to propose a segment potentially responsible for their divergent outcomes. Rather than directly trusting the judge's assessment, ProVer verifies the proposed segment by estimating its advantage from the difference in terminal success rates between current-policy continuations sampled before and after the segment. Positive estimates are then incorporated into the GRPO advantages of policy tokens within the proposed segment. By using model judgment only to select where to verify, ProVer grounds local credit in observed outcomes without exhaustively evaluating every intermediate state. Across ALFWorld, WebShop, and SearchQA, ProVer achieves the strongest average performance at both model scales, with relative improvements over GRPO of 9.91% and 7.12% for Qwen3.5-2B and Qwen3.5-4B, respectively. Further analyses demonstrate that informed segment selection improves policy training with modest additional generation overhead, even without a frontier-scale judge model, highlighting the effectiveness and efficiency of selectively targeting pivotal decisions for fine-grained credit assignment in agentic reinforcement learning.

Figures & tables

Qwen3.5-2B
Method	ALFWorld	WebShop	SearchQA	Avg.
GRPO	84.08 $\pm$ 3.02	42.53 $\pm$ 0.31	35.58 $\pm$ 0.38	54.07 $\pm$ 0.78
Budget-Matched GRPO	84.05 $\pm$ 4.13	39.00 $\pm$ 2.62	37.75 $\pm$ 0.25	53.60 $\pm$ 1.32
GiGPO	85.82 $\pm$ 2.24	44.60 $\pm$ 0.35	36.17 $\pm$ 1.66	55.53 $\pm$ 1.25
SPO-tree	56.96 $\pm$ 1.14	38.47 $\pm$ 0.50	31.67 $\pm$ 0.72	42.36 $\pm$ 0.27
SPO-chain	75.87 $\pm$ 1.72	47.80 $\pm$ 2.62	27.67 $\pm$ 0.76	50.45 $\pm$ 1.20

Table 1: Performance of the baselines and ProVer with Qwen3.5-2B and Qwen3.5-4B on three benchmarks. We report the mean and sample standard deviation over three evaluation runs.

Method	Generated Tokens / Step (K)			Judge Cost / Step (¢)			Time / Step (min)
Method	ALF	WS	SQA	ALF	WS	SQA	ALF	WS	SQA
GRPO	237.0	97.1	37.6	–	–	–	3.39	0.82	1.39
SPO-tree	153.2	71.0	63.4	–	–	–	7.18	4.85	4.65
SPO-chain	421.2	172.3	72.3	–	–	–	6.73	5.07	4.77
CriticSearch	227.1	136.3	45.8	17.31	8.54	7.40	4.64	3.15	3.25
ProVer	242.6	108.4	43.9	4.00	10.81	5.69	3.73	2.00	2.25

Table 2: Per-step training cost for Qwen3.5-4B on ALFWorld (ALF), WebShop (WS), and SearchQA (SQA). Generated tokens count policy-rollout outputs. Judge cost is calculated from recorded gpt-5.4-mini API usage using OpenAI’s published pricing as of August 2026. Time is the mean wall-clock time per optimizer step in minutes.

Agentic Judge	Policy Performance (%)			SearchQA Proposal Diagnostics
Agentic Judge	ALFWorld	WebShop	SearchQA	Accept (%)	Mean Len.
Random	94.0	42.4	42.8	19.5	2.38
Qwen3.5-9B	98.5	44.2	45.8	71.2	1.45
gpt-5.4-nano	94.8	46.2	43.5	72.4	1.25
gpt-5.4-mini	98.3	47.9	47.0	62.5	1.18
gpt-5.4	96.3	47.2	43.8	58.5	1.03

Table 3: Effect of segment selection and judge model choice on Qwen3.5-4B policy training. Random replaces judge-guided selection with random segment sampling while retaining the same outcome-verification and credit-assignment procedure. We additionally report proposal diagnostics on SearchQA: acceptance rate (

\widehat{\Delta}_{\mathrm{seg}}>0

) and mean segment length in agent turns.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Model	Benchmark	Phase 1 steps	Rollouts/group	Phase 2 steps	Rollouts/group
Qwen3.5-2B	ALFWorld	1–84	11	85–100	12
Qwen3.5-2B	WebShop	1–94	10	95–100	11
Qwen3.5-2B	SearchQA	1–70	10	71–100	11
Qwen3.5-4B	ALFWorld	1–26	8	27–100	9
Qwen3.5-4B	WebShop	1–35	10	36–100	11
Qwen3.5-4B	SearchQA	1–36	9	37–100	10

Appendix

Table 4: Number of trajectories sampled per group in each phase of Budget-Matched GRPO. Every update contains 16 groups.

Explore similar work

TRIAGE: Role-Typed Credit Assignment for Agentic Reinforcement Learning
Jun 30, 2026Yuanda Xu, Zhengze Zhou, Hejian Sang +6Credit Assignment Agentic Reinforcement Learning
1LinkedIn Corporation · 2Harvard University · 3Johns Hopkins University +1
SHARPO: Segment-Level Credit Assignment for Agentic Reinforcement Learning
Sep 30, 2026Xinchen Du, Zhengze Zhou, Wenhui Zhu +4Agentic Reinforcement Learning Credit Assignment
LinkedIn Corporation · Georgia Institute of Technology
APPO: Agentic Procedural Policy Optimization
Jun 10, 2026Xucong Wang, Ziyu Ma, Yong Wang +5Agentic Reinforcement Learning Frictive Policy Optimization
University of Science and Technology of China · AMAP, Alibaba Group · Southern University of Science and Technology

Jun 30, 2026, cs.LGJ/K move · Enter open · S save

TRIAGE: Role-Typed Credit Assignment for Agentic Reinforcement Learning

Yuanda Xu, Zhengze Zhou, Hejian Sang, Xiaomin Li+5

1LinkedIn Corporation · 2Harvard University · 3Johns Hopkins University+1

Credit Assignment Agentic Reinforcement Learning Step-Level Credit Reward Functions Triage

Open paper AskPDF arXiv

Agentic reinforcement learning requires assigning credit to environment-facing actions such as searches, clicks, edits, navigation commands, and object interactions. Standard GRPO uses the final verifier outcome as a uniform advantage over all action tokens. This outcome signal is useful but structurally incomplete: it punishes useful exploration in failed rollouts and reinforces redundant or regressive actions in successful rollouts. We propose TRIAGE, a role-typed credit assignment framework that adds a semantic role axis to outcome credit. A structured judge classifies each segment as decisive progress, useful exploration, no-progress infrastructure, or regression, and a fixed role-conditioned rule maps these labels to bounded segment-level process rewards. This keeps verifier outcomes as the source of optimization direction while correcting the two main blind spots of outcome-only credit. We further show that role-conditioned credit is the optimal segment-level correction expressible from role labels alone -- a projection of the per-segment advantage residual onto the role variable -- so that the fixed role constants reduce advantage estimation error whenever the judge is reliable, and we connect this to lower-variance policy gradients. Across ALFWorld, Search-QA, and WebShop, TRIAGE improves success rates over GRPO for two policy models and outperforms both a scalar judge-derived process reward and an outcome-supervised shared-backbone value baseline. Ablations show that the gain comes from role typing rather than merely adding dense rewards: reliable detection of regression inside successful trajectories is the dominant contributor, while exploration credit provides a consistent secondary gain; on completed ALFWorld and WebShop rollouts, TRIAGE also reduces environment-facing turns by an additional

10.4\%

and

14.8\%

relative to GRPO.

Abstract

Explore similar work

TRIAGE: Role-Typed Credit Assignment for Agentic Reinforcement Learning

SHARPO: Segment-Level Credit Assignment for Agentic Reinforcement Learning

APPO: Agentic Procedural Policy Optimization