Group-based reinforcement learning enables large language model agents to learn from sparse outcome rewards in long-horizon tasks. Recent methods derive step-level credit by comparing observation-matched experiences across trajectories. However, identical observations can occur in different contexts, and matched peers are not equally informative for a given decision. Moreover, favorable terminal outcomes can obscure redundant actions or detours within successful trajectories. To address these two challenges, we propose Contextual Credit Policy Optimization (CCPO). For the first, CCPO weights observation-matched peers from other trajectories by contextual similarity, computed from frozen LLM representations and accumulated trajectory statistics, allowing peers from similar contexts to contribute more to a step's baseline. For the second, CCPO pairs historical credit, which compares the realized return against the weighted peer baseline, with future credit, which measures the change in contextual outcome value over a short realized continuation. A step is thus credited for the progress it makes, not only for its trajectory's eventual outcome. Both credits are computed in hindsight from rollouts already collected for training, enabling CCPO to address these challenges without an auxiliary value critic. Across two Qwen2.5-Instruct backbones (1.5B and 7B), CCPO achieves up to 97.4% success on ALFWorld and 79.4% on WebShop and outperforms GRPO by up to 21.6 and 18.5 points. Ablations show that contextual weighting outperforms uniform weighting, and that historical and future credit serve as complementary learning signals.
Figures & tables
Figure 1: Overview of CCPO. CCPO combines contextually weighted peer comparisons and short-horizon future progress for critic-free step-level credit assignment.
Figure 2: Contextual credit policy optimization of multi-step tasks. Contextual similarity weights observation-matched peers. A history-conditioned return residual and estimated outcome-value change over a short future window yield step-level credit from collected rollouts without an auxiliary value critic.
Type
Method
ALFWorld
WebShop
All
Pick
Look
Clean
Heat
Cool
Pick2
Succ.
Score
Off-the-Shelf Model
Prompting
Gemini 3.8 Flash *
96.4
100.0
84.6
100.0
100.0
92.0
95.8
49.2
54.0
Prompting
DeepSeek-V4.1-Flash *
75.0
88.6
84.6
66.7
62.5
76.0
66.7
22.0
27.9
Prompting
Qwen3.8-27B *
70.7
85.7
61.5
66.7
56.3
72.0
66.7
29.0
39.5
Qwen2.5-1.5B-Instruct
Table 1: Results on WebShop and ALFWorld. We report the success rate (%) for each sub-task and overall results for ALFWorld, and the success rate (%) and score for WebShop, averaged over 3 seeds with standard deviations. Within each section, the best means are highlighted and the second best are underlined. Higher is better. * marks baselines reproduced or off-the-shelf models evaluated by us. Light-gray rows show results from the original papers and are excluded from ranking highlights. Δ GRPO gives the min–max gain over GRPO across both CCPO variants.
Table 2: Ablations with Qwen2.5-1.5B-Instruct on ALFWorld and WebShop. best and second best are marked. H and F denote active historical and future credit spans, with W=H+F+1 . Zero marks a disabled credit channel. Channel-only rows retain KH=KF=2 . Temporal-window rows also change the actual window lengths (Table S3 ). Markers: current, history, and future.
Change (pp)
>0.25
>0.5
>1
>5
>10
Pairs (%)
62.1
48.7
33.8
8.6
3.6
Table 3: Absolute changes in matched peer weights from context summaries.
Figure 3: ALFWorld evaluation trajectories (Qwen2.5-1.5B-Instruct). Each cell denotes one step. Revisit counts track navigation to previously visited locations. CCPO uses ACC only.
Figure 4: Contextual credit dynamics on ALFWorld (Qwen2.5-1.5B and 7B). (a) ACC versus AEP from the same rollouts. Only ACC is backpropagated. (b) Historical, future, and combined credit. Future-to-history ratios compare iterations 1–30 and 121–150. The heatmap correlates unsmoothed historical and future magnitudes by non-overlapping 10-iteration windows. Curves: mean absolute credit, centered 5-iteration means, and faint raw traces. Bands: 95% trajectory-bootstrap CIs.
Figure 5: Credit preferences on ALFWorld (Qwen2.5-1.5B-Instruct). Mean category-relative credit within matched task–observation groups and iterations, from successful rollouts at iterations 101–150. Asterisks: Bonferroni-adjusted CIs excluding zero. Dashes: fewer than 60 groups. Off : inactive component. Context markers denote current, history, and future.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Definition
Tasks, trajectories, and temporal indexing
x,xi
Task instance; xi is the task associated with trajectory i .
N
Number of trajectories sampled per task in a rollout batch.
τi
Trajectory i , containing observations oi,0:Ti , actions ai,0:Ti−1 , and their outcomes.
i,j
Query-trajectory and peer-trajectory indices.
t,s
Zero-based query-step and peer-step indices: 0≤t<Ti , 0≤s<Tj .
Appendix
Table S1: Notation for CCPO. Subscripts index trajectories and steps; H and F denote historical and future channels. Implementation details appear in Appendix C .
Setting
GRPO
GiGPO
HGPO
G2PO
CCPO
Both benchmarks
Training iterations
150
150
150
150
150
Tasks × rollouts per iteration
16×8
16×8
16×8
16×8
16×8
Total sampled trajectories
19,200
19,200
19,200
19,200
19,200
NVIDIA H200 GPUs per run
8
8
8
8
8
Learning rate
10−6
10−6
10−6
10−6
10−6
Appendix
Table S2: Training configurations shared by 1.5B and 7B models on ALFWorld and WebShop.
Variant
Configuration change
ACC+AEP
Add episode advantage with unit weight after contextual normalization, without renormalizing the sum.
w/o self-trajectory peer exclusion
Allow matching steps from the query trajectory, including the query itself, as peers.
w/o context summary
Set the accumulated-statistics weight from 1 to 0; retain frozen hidden-state features and prompt history.
Cosine-based context similarity
Replace the exponential kernel with cosine-based weighting.
Historical credit only
Turn off future credit by setting its weight from 1 to 0. Retain historical credit with unit weight and the default windows KH=KF=2 ; the active spans are (H,F)=(2,0) .
Future credit only
Turn off historical credit by setting its weight from 1 to 0. Retain future credit with unit weight and horizon KF=2 , together with prompt history KH=2 ; the active spans are (H,F)=(0,2) .
Appendix
Table S3: CCPO variant and ablation configurations. Each ablation changes only the listed settings relative to its corresponding reference configuration. The default uses prompt history KH=2 , future horizon KF=2 , and unit weights for both credit channels. In Table 2 , a disabled channel has active span zero; its underlying window is retained in the channel-only ablations.
Component
Excerpt
Prompt
Role
You are an expert agent operating in the ALFRED Embodied Environment.
Task
Your task is to: put two soapbar in garbagecan.
History
Prior to this step, you have already taken 5 step(s). Below are the most recent 2 observations and the corresponding actions you took:
History excerpt
[Observation 4: ’You arrive at cabinet 1. The cabinet 1 is closed.’, Action 4: ’open cabinet 1’]
History excerpt
[Observation 5: ’You open the cabinet 1. The cabinet 1 is open. In it, you see nothing.’, Action 5: ’go to sinkbasin 1’]
Appendix
Table S4 : ALFWorld prompt and response excerpts at step 6 (Qwen2.5-1.5B-Instruct).
Run
(KH,KF)
Steps
Groups
M
CCPO (full)
(2,2)
39,503
11,003
10
w/o context summary
(2,2)
38,680
9,778
9
Asymmetric (history)
(4,0)
40,972
9,914
10
Asymmetric (future)
(0,4)
49,736
13,235
11
Appendix
Table S5 : Filtered samples for the credit-preference analysis. Groups match training iteration, task instance, and observation. M counts categories with at least 60 usable groups; each contrast requires both the category and other actions in a group.
Figure S1 : Action timing on ALFWorld (Qwen2.5-1.5B-Instruct). First occurrence at checkpoint 150, paired across the same 140 evaluation games against full CCPO. Percentages condition on both policies performing the action. Dashes: fewer than 12 pairs or unavailable results; gray rows are excluded. Context markers denote current, history, and future.
Figure S2: Evaluation trajectories for the navigation task in Table S6 . Reproduced from Figure 3 . Each cell denotes one step; revisit counts track navigation to previously visited locations. CCPO uses ACC only.
Step
Action
Observation excerpt / summary
GiGPO: failed at 50 steps; 47 revisits
3
go to toiletpaperhanger 1
On the toiletpaperhanger 1, you see nothing.
4–49
46 navigation actions among countertop 1, toilet 1, and toiletpaperhanger 1; every returned observation repeats an earlier one.
Table S6: Repeated navigation versus target acquisition. Task: “put a toiletpaper in toiletpaperhanger.” Revisit counts refer to navigation to previously visited locations.
Figure S3: Evaluation trajectories for the two-object task in Table S7 . Each cell denotes one step; revisit counts track navigation to previously visited locations. CCPO uses ACC only.
Step
Action
Observation excerpt / summary
G2PO: failed at 50 steps
10
move spraybottle 2 to toilet 1
You move the spraybottle 2 to the toilet 1.
14
go to toilet 1
On the toilet 1, you see a cloth 3, and a spraybottle 2.
15
take spraybottle 2 from toilet 1
You pick up the spraybottle 2 from the toilet 1.
16
move spraybottle 2 to toilet 1
You move the spraybottle 2 to the toilet 1.
37–39, 48–49
Two further take–replace cycles with spraybottle 2; step 38 is an invalid action.
Appendix
Table S7: Repeated manipulation of one object versus placing two distinct objects. Task: “find two spraybottle and put them in toilet.”
Figure S4 : Average steps among successful trajectories on ALFWorld and WebShop. Panels show Qwen2.5-1.5B/7B-Instruct agents and a shared off-the-shelf Gemini reference. Error bars show sample standard deviations across three evaluation subsets; no deviation is reported for Gemini.
Figure S5 : Exact-match fallback and peer availability during training. Results use Qwen2.5-1.5B-Instruct on ALFWorld and WebShop. (a,b) Percentage of rows falling back from the exact anchor-observation key to a task-only key, comparing CCPO (full) with five ablations. (c,d) No-peer row percentages, computed as the singleton-bucket fraction divided by mean bucket size; CCPO (full) is highlighted and ablations are gray. Faint traces show raw per-step values, while smoothed curves use centered five-step moving averages. Shading marks steps 101–150; annotations report unweighted means of raw per-step values over this window. All ALFWorld arms use ACC only, and all WebShop arms use ACC+AEP . Each arm represents one recorded training run. Vertical-axis ranges differ between benchmarks.
Training long-horizon language model agents requires assigning credit to intermediate decisions from sparse terminal feedback. Within a finite group of sibling rollouts, useful action comparisons depend on their context, evidence for both alternatives, and how that evidence enters learning. We propose Matched-Context Group Policy Optimization (MCGPO), a procedure for constructing weighted action preferences from already collected trajectories. MCGPO defines comparison contexts with pre-action state summaries and observed rubric progress, annotated by task rules. Two-action support selects among ordered exact and coarse matching rules. Fixed folds separate candidate decisions from the terminal outcomes used to orient their preferences, and support-dependent evidence weights scale an auxiliary pairwise objective alongside shared GraphGPO task learning. MCGPO reuses rollout data and each response's original history, preserving critic-free training. Across six environment-backbone settings with three training seeds each, MCGPO improves mean success over matched GraphGPO: by 0.47-1.46 percentage points on ALFWorld and WebShop with Qwen2.5-1.5B and 7B, and by 3.96 and 5.16 points on WebShop with Qwen3-4B and 8B. The ALFWorld gains build on baseline success above 92%, where remaining headroom is limited. Integrated success improves in three of four Qwen2.5 settings. Three-seed factorial controls support opposite-fold evidence and evidence weighting; joint state-rubric matching achieves the highest mean in all four representation-control settings.
Chenxu Zhang, Yaorui Shi, Wenlong Zhang +3
Imperial College London · Meituan · University of Science and Technology of China +1
Long-horizon LLM agents require reinforcement learning methods that can assign credit to intermediate decisions under sparse and delayed rewards. Recent group-based methods such as GiGPO improve over GRPO by constructing step-level advantages at repeated anchor states. However, we show that such dense credit can be statistically unreliable: under limited rollouts, rare but lucky actions may receive overly large advantages, producing divergent anchor bias and late-stage training oscillation. We propose Evidence-Calibrated Policy Optimization (ECPO), a critic-free policy optimization algorithm that calibrates step-level credit before policy updates. ECPO combines Evidence-Calibrated Action Advantage, which groups rollouts by canonical actions and shrinks low-count estimates, with Variance-Gated Credit Weighting, which suppresses anchor states dominated by within-action noise. Experiments on ALFWorld and WebShop with Qwen2.5-1.5B/7B show that ECPO consistently outperforms strong baselines, improving GiGPO by +5.2/+7.3 success points on ALFWorld/WebShop with Qwen2.5-1.5B while adding only 0.1% additional advantage-computation overhead.
Yuanfan Li, Qi Zhou, Wenjing Duan +1
X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Faculty of Electronic and Information Engineering, Xi’an Jiaotong University
Group-based reinforcement learning effectively post-trains LLM agents for long-horizon, sparse-reward tasks by deriving step-level credit from trajectory outcomes. However, this ties a step's credit to its rollout's final outcome: semantically near-identical intermediate steps receive opposite credit depending on whether their trajectory eventually succeeded or failed. Such semantic credit inconsistency sends conflicting gradients to similar actions and wastes the partially-correct progress inside failed rollouts. Motivated by this, we propose Semantic Consistency Policy Optimization (SCPO), a value-free reward-shaping method that mitigates this inconsistency by recovering step-level credit from successful siblings in the same rollout group. Concretely, SCPO scores each failed step against a successful sibling and adds positive step-level credit for new progress along that sibling. On ALFWorld and WebShop, SCPO matches or exceeds strong group-based baselines, reaching 93.7+/-4.1 percent success on ALFWorld and 74.8+/-2.0 percent on WebShop at 1.5B parameters, with gains concentrated on the hardest multi-step tasks.
Peng Xu, Sijia Chen, Junzhuo Li +1
†The Hong Kong University of Science and Technology (Guangzhou) · ‡The Hong Kong University of Science and Technology