Contextual Credit Policy Optimization

Huiwen Han; Ziyao Zeng; Zihao Zhu; Sisong Bei; Hao Wang; Hongyu Xu; Yan Han; Degui Zhi

cs.LGOct 13, 2026

Contextual Credit Policy Optimization

Authors: Huiwen Han, Ziyao Zeng, Zihao Zhu, Sisong Bei, Hao Wang, Hongyu Xu, Yan Han, Degui Zhi, +1 more

Organizations: Stanford University · Texas A&M University · Yale University · Amazon · Meta AI · Microsoft

Abstract

Group-based reinforcement learning enables large language model agents to learn from sparse outcome rewards in long-horizon tasks. Recent methods derive step-level credit by comparing observation-matched experiences across trajectories. However, identical observations can occur in different contexts, and matched peers are not equally informative for a given decision. Moreover, favorable terminal outcomes can obscure redundant actions or detours within successful trajectories. To address these two challenges, we propose Contextual Credit Policy Optimization (CCPO). For the first, CCPO weights observation-matched peers from other trajectories by contextual similarity, computed from frozen LLM representations and accumulated trajectory statistics, allowing peers from similar contexts to contribute more to a step's baseline. For the second, CCPO pairs historical credit, which compares the realized return against the weighted peer baseline, with future credit, which measures the change in contextual outcome value over a short realized continuation. A step is thus credited for the progress it makes, not only for its trajectory's eventual outcome. Both credits are computed in hindsight from rollouts already collected for training, enabling CCPO to address these challenges without an auxiliary value critic. Across two Qwen2.5-Instruct backbones (1.5B and 7B), CCPO achieves up to 97.4% success on ALFWorld and 79.4% on WebShop and outperforms GRPO by up to 21.6 and 18.5 points. Ablations show that contextual weighting outperforms uniform weighting, and that historical and future credit serve as complementary learning signals.

Figures & tables

Figure 1: Overview of CCPO. CCPO combines contextually weighted peer comparisons and short-horizon future progress for critic-free step-level credit assignment.

Figure 2: Contextual credit policy optimization of multi-step tasks. Contextual similarity weights observation-matched peers. A history-conditioned return residual and estimated outcome-value change over a short future window yield step-level credit from collected rollouts without an auxiliary value critic.

Type	Method	ALFWorld							WebShop
		All	Pick	Look	Clean	Heat	Cool	Pick2	Succ.	Score
Off-the-Shelf Model
Prompting	Gemini 3.8 Flash *	$\mathbf{96.4}$	$\mathbf{100.0}$	$\mathbf{84.6}$	$\mathbf{100.0}$	$\mathbf{100.0}$	$\mathbf{92.0}$	$\mathbf{95.8}$	$\mathbf{49.2}$	$\mathbf{54.0}$
Prompting	DeepSeek-V4.1-Flash *	$\underline{75.0}$	$\underline{88.6}$	$\mathbf{84.6}$	$\underline{66.7}$	$\underline{62.5}$	$\underline{76.0}$	$\underline{66.7}$	$22.0$	$27.9$
Prompting	Qwen3.8-27B *	$70.7$	$85.7$	$\underline{61.5}$	$\underline{66.7}$	$56.3$	$72.0$	$\underline{66.7}$	$\underline{29.0}$	$\underline{39.5}$
Qwen2.5-1.5B-Instruct

Table 1: Results on WebShop and ALFWorld. We report the success rate (%) for each sub-task and overall results for ALFWorld, and the success rate (%) and score for WebShop, averaged over 3 seeds with standard deviations. Within each section, the best means are highlighted and the second best are underlined. Higher is better. * marks baselines reproduced or off-the-shelf models evaluated by us. Light-gray rows show results from the original papers and are excluded from ranking highlights.

\Delta

GRPO gives the min–max gain over GRPO across both CCPO variants.

Panel of Table 2 — Table 2: Ablations with Qwen2.5-1.5B-Instruct on ALFWorld and WebShop. best and second best are marked. $H$ and $F$ denote active historical and future credit spans, with $W=H+F+1$ . Zero marks a disabled credit channel. Channel-only rows retain $K_{H}=K_{F}=2$ . Temporal-window rows also change the actual window lengths (Table S3 ). Markers: current, history, and future.

Change (pp)	$>0.25$	$>0.5$	$>1$	$>5$	$>10$
Pairs (%)	62.1	48.7	33.8	8.6	3.6

Table 3: Absolute changes in matched peer weights from context summaries.

Figure 3: ALFWorld evaluation trajectories (Qwen2.5-1.5B-Instruct). Each cell denotes one step. Revisit counts track navigation to previously visited locations. CCPO uses

A_{\mathrm{CC}}

only.

Figure 4: Contextual credit dynamics on ALFWorld (Qwen2.5-1.5B and 7B). (a)

A_{\mathrm{CC}}

versus

A_{\mathrm{EP}}

from the same rollouts. Only

A_{\mathrm{CC}}

is backpropagated. (b) Historical, future, and combined credit. Future-to-history ratios compare iterations 1–30 and 121–150. The heatmap correlates unsmoothed historical and future magnitudes by non-overlapping 10-iteration windows. Curves: mean absolute credit, centered 5-iteration means, and faint raw traces. Bands: 95% trajectory-bootstrap CIs.

Figure 5: Credit preferences on ALFWorld (Qwen2.5-1.5B-Instruct). Mean category-relative credit within matched task–observation groups and iterations, from successful rollouts at iterations 101–150. Asterisks: Bonferroni-adjusted CIs excluding zero. Dashes: fewer than 60 groups. Off : inactive component. Context markers denote current, history, and future.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Definition
Tasks, trajectories, and temporal indexing
$x,\ x_{i}$	Task instance; $x_{i}$ is the task associated with trajectory $i$ .
$N$	Number of trajectories sampled per task in a rollout batch.
$\tau_{i}$	Trajectory $i$ , containing observations $o_{i,0:T_{i}}$ , actions $a_{i,0:T_{i}-1}$ , and their outcomes.
$i,\ j$	Query-trajectory and peer-trajectory indices.
$t,\ s$	Zero-based query-step and peer-step indices: $0\leq t<T_{i}$ , $0\leq s<T_{j}$ .

Appendix

Table S1: Notation for CCPO. Subscripts index trajectories and steps;

\mathrm{H}

and

\mathrm{F}

denote historical and future channels. Implementation details appear in Appendix C .

Setting	GRPO	GiGPO	HGPO	G2PO	CCPO
Both benchmarks
Training iterations	150	150	150	150	150
Tasks $\times$ rollouts per iteration	$16\times 8$	$16\times 8$	$16\times 8$	$16\times 8$	$16\times 8$
Total sampled trajectories	19,200	19,200	19,200	19,200	19,200
NVIDIA H200 GPUs per run	8	8	8	8	8
Learning rate	$10^{-6}$	$10^{-6}$	$10^{-6}$	$10^{-6}$	$10^{-6}$

Appendix

Table S2: Training configurations shared by 1.5B and 7B models on ALFWorld and WebShop.

Variant	Configuration change
$A_{\mathrm{CC}}+A_{\mathrm{EP}}$	Add episode advantage with unit weight after contextual normalization, without renormalizing the sum.
w/o self-trajectory peer exclusion	Allow matching steps from the query trajectory, including the query itself, as peers.
w/o context summary	Set the accumulated-statistics weight from 1 to 0; retain frozen hidden-state features and prompt history.
Cosine-based context similarity	Replace the exponential kernel with cosine-based weighting.
Historical credit only	Turn off future credit by setting its weight from 1 to 0. Retain historical credit with unit weight and the default windows $K_{\mathrm{H}}=K_{\mathrm{F}}=2$ ; the active spans are $(H,F)=(2,0)$ .
Future credit only	Turn off historical credit by setting its weight from 1 to 0. Retain future credit with unit weight and horizon $K_{\mathrm{F}}=2$ , together with prompt history $K_{\mathrm{H}}=2$ ; the active spans are $(H,F)=(0,2)$ .

Appendix

Table S3: CCPO variant and ablation configurations. Each ablation changes only the listed settings relative to its corresponding reference configuration. The default uses prompt history

K_{\mathrm{H}}=2

, future horizon

K_{\mathrm{F}}=2

, and unit weights for both credit channels. In Table 2 , a disabled channel has active span zero; its underlying window is retained in the channel-only ablations.

Component	Excerpt
Prompt
Role	You are an expert agent operating in the ALFRED Embodied Environment.
Task	Your task is to: put two soapbar in garbagecan.
History	Prior to this step, you have already taken 5 step(s). Below are the most recent 2 observations and the corresponding actions you took:
History excerpt	[Observation 4: ’You arrive at cabinet 1. The cabinet 1 is closed.’, Action 4: ’open cabinet 1’]
History excerpt	[Observation 5: ’You open the cabinet 1. The cabinet 1 is open. In it, you see nothing.’, Action 5: ’go to sinkbasin 1’]

Appendix

Table S4 : ALFWorld prompt and response excerpts at step 6 (Qwen2.5-1.5B-Instruct).

Run	$(K_{\mathrm{H}},K_{\mathrm{F}})$	Steps	Groups	$M$
CCPO (full)	$(2,2)$	39,503	11,003	10
w/o context summary	$(2,2)$	38,680	9,778	9
Asymmetric (history)	$(4,0)$	40,972	9,914	10
Asymmetric (future)	$(0,4)$	49,736	13,235	11

Appendix

Table S5 : Filtered samples for the credit-preference analysis. Groups match training iteration, task instance, and observation.

M

counts categories with at least 60 usable groups; each contrast requires both the category and other actions in a group.

Figure S1 : Action timing on ALFWorld (Qwen2.5-1.5B-Instruct). First occurrence at checkpoint 150, paired across the same 140 evaluation games against full CCPO. Percentages condition on both policies performing the action. Dashes: fewer than 12 pairs or unavailable results; gray rows are excluded. Context markers denote current, history, and future.

Figure S2: Evaluation trajectories for the navigation task in Table S6 . Reproduced from Figure 3 . Each cell denotes one step; revisit counts track navigation to previously visited locations. CCPO uses

A_{\mathrm{CC}}

only.

Step	Action	Observation excerpt / summary
GiGPO: failed at 50 steps; 47 revisits
3	go to toiletpaperhanger 1	On the toiletpaperhanger 1, you see nothing.
4–49	46 navigation actions among countertop 1, toilet 1, and toiletpaperhanger 1; every returned observation repeats an earlier one.
G2PO: failed at 50 steps; 42 revisits
5	open cabinet 2	In it, you see a soapbottle 1.
8–15	Two navigation cycles: countertop 1 $\to$ cabinet 2 $\to$ toilet 1 $\to$ toiletpaperhanger 1.

Appendix

Table S6: Repeated navigation versus target acquisition. Task: “put a toiletpaper in toiletpaperhanger.” Revisit counts refer to navigation to previously visited locations.

Figure S3: Evaluation trajectories for the two-object task in Table S7 . Each cell denotes one step; revisit counts track navigation to previously visited locations. CCPO uses

A_{\mathrm{CC}}

only.

Step	Action	Observation excerpt / summary
G2PO: failed at 50 steps
10	move spraybottle 2 to toilet 1	You move the spraybottle 2 to the toilet 1.
14	go to toilet 1	On the toilet 1, you see a cloth 3, and a spraybottle 2.
15	take spraybottle 2 from toilet 1	You pick up the spraybottle 2 from the toilet 1.
16	move spraybottle 2 to toilet 1	You move the spraybottle 2 to the toilet 1.
37–39, 48–49	Two further take–replace cycles with spraybottle 2; step 38 is an invalid action.

Appendix

Table S7: Repeated manipulation of one object versus placing two distinct objects. Task: “find two spraybottle and put them in toilet.”

Figure S4 : Average steps among successful trajectories on ALFWorld and WebShop. Panels show Qwen2.5-1.5B/7B-Instruct agents and a shared off-the-shelf Gemini reference. Error bars show sample standard deviations across three evaluation subsets; no deviation is reported for Gemini.

Figure S5 : Exact-match fallback and peer availability during training. Results use Qwen2.5-1.5B-Instruct on ALFWorld and WebShop. (a,b) Percentage of rows falling back from the exact anchor-observation key to a task-only key, comparing CCPO (full) with five ablations. (c,d) No-peer row percentages, computed as the singleton-bucket fraction divided by mean bucket size; CCPO (full) is highlighted and ablations are gray. Faint traces show raw per-step values, while smoothed curves use centered five-step moving averages. Shading marks steps 101–150; annotations report unweighted means of raw per-step values over this window. All ALFWorld arms use

A_{\mathrm{CC}}

only, and all WebShop arms use

A_{\mathrm{CC}}+A_{\mathrm{EP}}

. Each arm represents one recorded training run. Vertical-axis ranges differ between benchmarks.

Explore similar work

Matched-Context Group Policy Optimization for LLM Agent Training
Oct 9, 2026Chenxu Zhang, Yaorui Shi, Wenlong Zhang +3RL for LLM Agents Preference-Based RL
Imperial College London · Meituan · University of Science and Technology of China +1
When Denser Credit Is Not Enough: Evidence-Calibrated Policy Optimization for Long-Horizon LLM Agent Training
Jun 4, 2026Yuanfan Li, Qi Zhou, Wenjing Duan +1Credit Assignment in RL Step-Level Credit Assignment in RL
X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Faculty of Electronic and Information Engineering, Xi’an Jiaotong University
Semantic Consistency Policy Optimization for Reinforcement Learning of LLM Agents
Jun 24, 2026Peng Xu, Sijia Chen, Junzhuo Li +1Reward Shaping Credit Assignment in RL
†The Hong Kong University of Science and Technology (Guangzhou) · ‡The Hong Kong University of Science and Technology

Oct 9, 2026J/K move · Enter open · S save

Matched-Context Group Policy Optimization for LLM Agent Training

Chenxu Zhang, Yaorui Shi, Wenlong Zhang, Jianchao Tan+2

Imperial College London · Meituan · University of Science and Technology of China+1

RL for LLM Agents Preference-Based RL Group Relative Policy Optimization Credit Assignment in RL

Open paper AskPDF arXiv

Training long-horizon language model agents requires assigning credit to intermediate decisions from sparse terminal feedback. Within a finite group of sibling rollouts, useful action comparisons depend on their context, evidence for both alternatives, and how that evidence enters learning. We propose Matched-Context Group Policy Optimization (MCGPO), a procedure for constructing weighted action preferences from already collected trajectories. MCGPO defines comparison contexts with pre-action state summaries and observed rubric progress, annotated by task rules. Two-action support selects among ordered exact and coarse matching rules. Fixed folds separate candidate decisions from the terminal outcomes used to orient their preferences, and support-dependent evidence weights scale an auxiliary pairwise objective alongside shared GraphGPO task learning. MCGPO reuses rollout data and each response's original history, preserving critic-free training. Across six environment-backbone settings with three training seeds each, MCGPO improves mean success over matched GraphGPO: by 0.47-1.46 percentage points on ALFWorld and WebShop with Qwen2.5-1.5B and 7B, and by 3.96 and 5.16 points on WebShop with Qwen3-4B and 8B. The ALFWorld gains build on baseline success above 92%, where remaining headroom is limited. Integrated success improves in three of four Qwen2.5 settings. Three-seed factorial controls support opposite-fold evidence and evidence weighting; joint state-rubric matching achieves the highest mean in all four representation-control settings.

Abstract

Explore similar work

Matched-Context Group Policy Optimization for LLM Agent Training

When Denser Credit Is Not Enough: Evidence-Calibrated Policy Optimization for Long-Horizon LLM Agent Training

Semantic Consistency Policy Optimization for Reinforcement Learning of LLM Agents