Matched-Context Group Policy Optimization for LLM Agent Training

Chenxu Zhang; Yaorui Shi; Wenlong Zhang; Jianchao Tan; Zihan Wang; Yiming Cheng

cs.LGOct 9, 2026

Matched-Context Group Policy Optimization for LLM Agent Training

Authors: Chenxu Zhang, Yaorui Shi, Wenlong Zhang, Jianchao Tan, Zihan Wang, Yiming Cheng

Organizations: Imperial College London · Meituan · University of Science and Technology of China · Tsinghua University

Abstract

Training long-horizon language model agents requires assigning credit to intermediate decisions from sparse terminal feedback. Within a finite group of sibling rollouts, useful action comparisons depend on their context, evidence for both alternatives, and how that evidence enters learning. We propose Matched-Context Group Policy Optimization (MCGPO), a procedure for constructing weighted action preferences from already collected trajectories. MCGPO defines comparison contexts with pre-action state summaries and observed rubric progress, annotated by task rules. Two-action support selects among ordered exact and coarse matching rules. Fixed folds separate candidate decisions from the terminal outcomes used to orient their preferences, and support-dependent evidence weights scale an auxiliary pairwise objective alongside shared GraphGPO task learning. MCGPO reuses rollout data and each response's original history, preserving critic-free training. Across six environment-backbone settings with three training seeds each, MCGPO improves mean success over matched GraphGPO: by 0.47-1.46 percentage points on ALFWorld and WebShop with Qwen2.5-1.5B and 7B, and by 3.96 and 5.16 points on WebShop with Qwen3-4B and 8B. The ALFWorld gains build on baseline success above 92%, where remaining headroom is limited. Integrated success improves in three of four Qwen2.5 settings. Three-seed factorial controls support opposite-fold evidence and evidence weighting; joint state-rubric matching achieves the highest mean in all four representation-control settings.

Figures & tables

Figure 1: Task context for local action comparisons. The same state summary

h(s)

can occur at different stages of a task. MCGPO groups decisions using the state summary and observed rubric progress

C

, with enabled criteria annotated by the rule-derived frontier

F

Figure 2: Overview of MCGPO. The Verify block labels matched actions using opposite-fold terminal outcomes. The resulting evidence-weighted pairwise loss complements the shared task objective.

Type	Method	ALFWorld							WebShop
Type	Method	Pick	Look	Clean	Heat	Cool	Pick2	All	Score	Succ.
Closed-source models
Prompting	GPT-4o	75.3	60.8	31.2	56.7	21.6	49.8	48.0	31.8	23.7
Prompting	Gemini-2.5-Pro	92.8	63.3	62.1	69.0	26.6	58.7	60.3	42.5	35.9
Base: Qwen2.5-1.5B-Instruct
Prompting	Qwen2.5	5.9	5.5	3.3	9.7	4.2	0.0	4.1	23.1	5.2

Table 1: Task success (%) on ALFWorld and graded score/success (%) on WebShop. All and Succ. are the primary metrics. GraphGPO and MCGPO report mean

\pm

sample SD over three seeds using the evaluation protocol in Section 4.1 . Results for the other methods are from Feng et al. (2025) ; Tan et al. (2026) .

Figure 3: Learning curves and paired gains with Qwen2.5-1.5B. Left axes: validation success for MCGPO and GraphGPO; bands show the three-seed sample SD and faint lines retain raw means. Right axes: the deep-blue curve shows the mean paired difference (MCGPO minus GraphGPO), in percentage points. Each seed is averaged over up to five trailing evaluations, using the available evaluations at the beginning so curves start at update 0; complete five-point windows apply from update 20. The update-100 annotation uses the corresponding smoothed values. Shaded bands apply only to success curves.

(a) Qwen2.5-1.5B: supervision controls (one run)
Component	Configuration	WebShop $\uparrow$	ALFWorld $\uparrow$
MCGPO (Full)		78.13	94.53
Match	w/ Shuffled Rubric	75.78 ( $\downarrow$ 2.34)	93.91 ( $\downarrow$ 0.63)
Label	Candidate-progress labels	76.72 ( $\downarrow$ 1.41)	91.41 ( $\downarrow$ 3.13)
Learn	w/o Pairwise Loss	76.72 ( $\downarrow$ 1.41)	93.75 ( $\downarrow$ 0.78)
Learn	GraphGPO + Rubric Shaping	69.84 ( $\downarrow$ 8.28)	91.88 ( $\downarrow$ 2.66)

Table 2: Component ablations of MCGPO. (a) One Qwen2.5-1.5B run per configuration; parentheses give decreases from Full (pp). (b) Qwen3-4B/WebShop mean

\pm

sample SD over three seeds. Entries are success rates (%) under the common evaluation protocol (Section 4.1 ); differences are computed before rounding. Protocols and paired analyses: Appendix C.3 .

Figure 4: Preference construction on WebShop with Qwen2.5-1.5B. (a) Accepted pairs from exact and coarse matching. (b) Agreement of preference directions with candidate outcomes (ties receive half credit), together with the tie share. (c) Absolute outcome contrast and the resulting evidence weight, averaged over accepted pairs. Each point summarizes 25 updates; lines and bands show the three-seed mean and

\pm 1

sample SD. Ratios and pair-level means pool observations within each seed/window. Appendix C.5 defines the statistics and aggregation.

Backbone	Mean success (%)		Gain (pp)
Backbone	GraphGPO	MCGPO	Gain (pp)
Qwen3-4B	$78.18\pm 2.62$	$\mathbf{82.14}\pm 3.52$	+3.96
Qwen3-8B	$79.69\pm 1.95$	$\mathbf{84.84}\pm 1.96$	+5.16

Table 3: Task success on WebShop with Qwen3 backbones. Entries are mean

\pm

sample SD over three training seeds under the evaluation protocol in Section 4.1 . Gain is the paired mean difference in percentage points, computed before rounding.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Comparison conditions	Evidence and aggregation	Policy interface
GiGPO ( Feng et al., 2025 )	Repeated anchor states	Returns within trajectory and state groups	Trajectory and step advantages
HGPO ( He et al., 2026 )	Current state and historical-context hierarchy	Weighted advantages across context depths	Step advantage
BiPACE ( Wang et al., 2026a )	Behavioral clusters from policy features	Action-conditioned peer returns	Step advantage
GraphGPO ( Cheng et al., 2026 )	Observed transition graph	Changes in distance to observed goals	Graph advantage
G 2 PO ( Wang et al., 2026c )	Identical observations in a global transition graph	Graph state values and globally standardized edge TD errors	Edge and episode advantages
AT-GRPO ( Peng et al., 2026 )	Dialogue trees with adaptive expansion depth	Returns over stage-dependent observation ranges	Group-relative policy loss

Appendix

Table 4: Comparison conditions, evidence, and policy interfaces. Rows summarize method designs; the matched performance evaluation in this paper uses GraphGPO. MCGPO’s frontier is a rule-derived annotation of progress.

Environment	Exact state	Coarse state
ALFWorld	Task type, location, held object, acquired/processed/placed object sets, light status	Task type, location, holding/processing flags, placement count, light status, remaining-budget bin
WebShop	Page, search query, product identity, selected options	Page, searched/product-open flags, binned type/attribute/option match, price satisfaction

Appendix

Table 5: State fields for the ordered matching rules. Both rules retain the rubric signature

(C,F)

Setting	Value
Sibling trajectories / fold sizes	$8$ / $4+4$
Minimum support for each action $m$	$1$
Terminal-score threshold for evidence	$0.999$
Outcome-contrast threshold $\epsilon$	$0.10$
Support shrinkage $\kappa$	$2$
Resolution weights $(q_{\rm exact},q_{\rm coarse})$	$(1,0.70)$

Appendix

Table 6: Fixed preference-construction and learning settings.

	ALFWorld		WebShop
Setting	Small	Large	Small	Large
Qwen2.5 backbone	1.5B	7B	1.5B	7B
Qwen3 backbone	—	—	4B	8B
Task groups / update ( $P$ )	16	16	16	16
Trajectories / group ( $G$ )	8	8	8	8
Admitted trajectories / update	128	128	128	128

Appendix

Table 7: Training and rollout configuration for matched GraphGPO and MCGPO runs. Micro-batch sizes are reported per GPU. Mean–std denotes the shared GraphGPO within-group advantage normalization mode. TP denotes rollout tensor parallelism.

Setting	Seed	MCGPO $L_{5}$	GraphGPO $L_{5}$	MCGPO $E_{250}$	GraphGPO $E_{250}$
ALFWorld / 1.5B	0	92.03	90.47	97.66	92.97
	1	94.53	93.13	94.53	96.88
	2	94.38	94.22	94.53	96.88
ALFWorld / 7B	0	95.31	96.88	96.88	95.31
	1	96.88	95.00	96.88	96.88
	2	96.72	92.66	94.53	92.97

Appendix

Table 8: Per-seed Qwen2.5 policy success (%).

L_{5}

averages the final five scheduled evaluations of the 250-update budget;

E_{250}

reports the final evaluation.

Backbone	Seed	GraphGPO (%)	MCGPO (%)	Gain (pp)
Qwen3-4B	0	79.84	85.63	+5.78
	1	75.16	78.59	+3.43
	2	79.53	82.19	+2.66
Qwen3-8B	0	81.25	85.00	+3.75
	1	77.50	86.72	+9.22
	2	80.31	82.81	+2.50

Appendix

Table 9: All paired Qwen3 WebShop seeds for the primary

L_{5}

readout. Each entry averages updates 230, 235, 240, 245, and 250. Gains are computed before final display rounding; three-seed means appear in Table 3 .

ALFWorld / 1.5B
Setting / readout	Success (%)	Normalized AUC (%)
GraphGPO	$92.60\,\pm\,1.93$	$77.42\,\pm\,1.73$
MCGPO	$93.65\,\pm\,1.40$	$78.68\,\pm\,1.34$
Paired $\Delta$ (pp)	$+1.04$	$+1.25$
95% paired interval	$[-0.87,\,+2.96]$	$[-0.67,\,+3.18]$
ALFWorld / 7B

Appendix

Table 10: Success and learning progress for GraphGPO and MCGPO across three training seeds. Entries give mean

\pm

sample SD; both scores use a 0–100 scale.

\Delta

is MCGPO minus GraphGPO in percentage points. Success follows Eq. 14 ; normalized raw-curve AUC summarizes learning progress. Intervals are 95% paired Student-

t

intervals (

\mathrm{df}=2

); definitions appear in Appendix C .

Figure 5: Qwen2.5-1.5B validation success at nearly matched training-transition budgets. Markers show three-seed means and whiskers sample SD. Each seed’s checkpoint pair differs in budget by less than 1%; displayed budgets average the paired costs. Counts cover retained training interactions; Appendix C.2 specifies matching and restart accounting.

Env.	Seed	Updates	Transitions	Gap (%)	Success (%)	$\Delta$ (pp)
ALFWorld	0	230/250	594,465/592,472	0.336	87.50/97.66	+10.16
	1	250/250	575,983/581,151	0.897	96.88/94.53	-2.34
	2	245/250	590,739/585,967	0.814	94.53/94.53	0.00
WebShop	0	250/215	174,190/173,006	0.684	75.00/80.47	+5.47
	1	250/230	188,293/187,096	0.640	77.34/78.91	+1.56
	2	250/200	172,791/171,773	0.593	75.78/78.91	+3.13

Appendix

Table 11: Selected budget-matched checkpoints for Qwen2.5-1.5B. Slash-separated entries give GraphGPO/MCGPO. Gap is the relative budget difference in Eq. 17 ;

\Delta

is MCGPO minus GraphGPO success in percentage points.

(a) Per-seed task success (%)
Configuration	Seed 0	Seed 1	Seed 2
A: MCGPO (Full)	85.63	78.59	82.19
B: Same-fold terminal	83.28	76.56	79.53
C: Uniform weighting	84.22	76.72	80.63
D: Same-fold + uniform	81.56	74.84	77.97

Appendix

Table 12: Qwen3-4B WebShop factorial controls from Table 2 (b). (a) Per-seed success (%), averaging the final five scheduled evaluations. (b) Paired gains in percentage points, with 95% Student-

t

intervals over three seed differences (

\mathrm{df}=2

). Statistics precede display rounding.

(a) Task success (%)
Method	Qwen3-4B	Qwen3-8B
GraphGPO	$78.18\pm 2.62$	$79.69\pm 1.95$
GraphGPO + shaping	$79.85\pm 3.21$	$82.19\pm 1.44$
MCGPO	$\mathbf{82.14}\pm 3.52$	$\mathbf{84.84}\pm 1.96$

Appendix

Table 13: WebShop comparison with a scorer-matched shaping control. (a) Task success (%), reported as mean

\pm

sample SD over three seeds. (b) MCGPO gains over GraphGPO + shaping, in percentage points. Paired 95% Student-

t

intervals (

\mathrm{df}=2

) use the per-seed differences shown.

Setting	Comparison key	$n$	$L_{5}$	$E_{250}$
ALFWorld / 1.5B	MCGPO	3	$93.65\,\pm\,1.40$	$95.57\,\pm\,1.80$
	State-only	3	$85.57\,\pm\,2.32$	$83.85\,\pm\,1.97$
	Rubric-only	3	$87.81\,\pm\,1.59$	$87.76\,\pm\,1.80$
ALFWorld / 7B	MCGPO	3	$96.30\,\pm\,0.86$	$96.09\,\pm\,1.35$
	State-only	3	$92.34\,\pm\,1.25$	$91.93\,\pm\,1.97$
	Rubric-only	3	$91.15\,\pm\,1.83$	$90.36\,\pm\,1.97$

Appendix

Table 14: Supplementary representation controls at 250 updates. Full, State-only, and Rubric-only use seeds 0, 1, and 2 in every setting. Entries give mean success (%) and sample SD across the three seeds.

L_{5}

averages the final five scheduled evaluations, and

E_{250}

is the terminal evaluation. State-only retains the environment-specific fields of its state abstraction, including WebShop’s match-score bins.

Figure 6: Validation success in all four Qwen2.5 settings. Each seed uses up to five trailing scheduled evaluations, including all available points at the beginning. Lines and bands give the three-seed mean and sample SD; faint lines retain raw means. The fixed

L_{5}

window is defined separately in Eq. 14 .

Figure 7: ALFWorld Qwen2.5-1.5B training behavior: (a) rollout success, (b) environment steps per rollout, and (c) anchor-state group size in the task-credit branch. Dark lines average ten trailing updates within each seed, then report the three-seed mean; bands show sample SD and faint lines show seed means.

Channel (symbol)	Status after the observed transition
target_acquired ( $A$ )	$\min(r,\|\mathcal{A}\|)$ ; acquisition history persists after placement.
target_in_hand ( $H$ )	Whether the worker’s held object matches the target type.
target_processed ( $P$ )	$\mathrm{1}[\mathcal{P}\neq\emptyset]$ ; processing history persists after release.
target_placed ( $L$ )	$\min(r,\|\mathcal{L}\|)$ ; taking or relocating an object updates this set.
light_on ( $B$ )	Tracked target-light state: successful off commands/feedback set false; on, use, or toggle set true unless off is indicated.
goal_progress ( $G$ )	Look: $H+B$ . Clean/Heat/Cool: $\mathrm{1}[\mathcal{P}\neq\emptyset]+\mathrm{1}[\mathcal{L}\neq\emptyset]+\mathrm{1}[\mathcal{P}\cap\mathcal{L}\neq\emptyset]$ . Pick/Pick2: $\min(r,\|\mathcal{L}\|)$ .

Appendix

Table 15: ALFWorld channel definitions. Symbols identify the nodes in Table 17 ;

\mathrm{1}[\cdot]

denotes an indicator. The sets and required object count

r

are defined in the text.

Channel (symbol)	Status after the observed transition
search_issued ( $Q$ )	Whether the session has search keywords or its searched flag is true.
product_opened ( $O$ )	Whether a product is selected or the product-open flag is true.
type_match ( $T$ )	The scorer’s type-match component r_type .
attribute_match ( $A$ )	Requested-attribute match fraction; scorer r_att is stored as r_attr .
option_match ( $U$ )	Requested-option match fraction r_option ; the standard worker retains zero when the scorer omits this component.
price_ok ( $P$ )	The scorer’s Boolean price predicate r_price .

Appendix

Table 16: WebShop channel definitions. Symbols refer to WebShop nodes in Table 17 . Match components come from the environment goal scorer; default handling is specified in the text.

Environment/task	Relation	Directed edges
ALFWorld: all	E	$A\to H$
Pick / Pick2	E	$A\to L$
	C	$L\to G$ , $G\to S$
Clean / Heat / Cool	E	$A\to P$ , $A\to L$
	C	$P\to L$ , $P\to G$ , $L\to G$ , $G\to S$
Look	C	$H\to G$ , $B\to G$ , $H\to S$ , $B\to S$ , $G\to S$

Appendix

Table 17: Complete dependency templates. Symbols use the respective environment’s channel table. E denotes requires_ever and C denotes requires_current . Each ALFWorld family includes the common edge; set notation expands to one edge per element.

Figure 8: First-visit group sizes across ten update-60 captures (1.5B: seeds 1/2; 7B: seeds 0/1/2, in both environments). Exact and coarse rules retain the same rubric coordinates; groups count distinct contributors within a four-trajectory candidate fold. Bars include singletons, and annotations report mean sizes and group counts.

Setting	Exact	Coarse	Exact	Coarse
	Non-singleton records (%)		Supported row pairs
ALFWorld / 1.5B	74.07	68.57	143	133
ALFWorld / 7B	91.43	91.90	86	252
WebShop / 1.5B	29.19	95.76	2	19
WebShop / 7B	36.61	96.89	14	79

Appendix

Table 18: Single-resolution comparison availability in all ten captured update-60 batches. Non-singleton coverage is the fraction of first-visit records in groups with at least two contributors. Supported pairs have distinct actions and at least one opposite-fold observation of each action. These counts precede outcome-contrast filtering and the final quota; exact and coarse counts overlap.

Figure 9: Captured ALFWorld example: put two toiletpaper in drawer . Two trajectories reach countertop 1 with empty hands, zero target placements, and the same budget bin. One previously acquired toiletpaper 2 and placed it at toilet 1. Its changed progress signature retains the later visit as a distinct context under the joint first-visit rule, before preference selection.

Setting	Pairs	$W/T/L$	Agreement (%)	Active	Exact (%)
WebShop / 1.5B	1428	771/568/89	73.88	330/450	8.3
WebShop / 7B	1425	633/693/99	68.74	319/450	20.1

Appendix

Table 19: Archived WebShop training-trace diagnostics for Full, pooled over three seeds per Qwen2.5 backbone.

W/T/L

denotes counts of preferred-candidate wins, ties, and reversed outcomes; agreement is

(W+T/2)/(W+T+L)

. Active gives accepted-pair updates / audited updates; Exact is the fraction of accepted pairs using exact matching. All counts refer to the retained trace sample.

Explore similar work

Jul 22, 2026

Learning from All-Failed Groups via First-Visit Coverage in Long-Horizon Agentic Tasks

Group-based policy optimization has been increasingly used to train large language model (LLM) agents from sparse outcome rewards by comparing trajectories or steps within a group. However, on difficult long-horizon agentic tasks, we found this comparison suffers from a sampling imbalance: repeated or low-effect actions dominate the high-probability region of the policy, leaving useful state-changing actions under-sampled. This imbalance produces many all-failed rollout groups, where the relative advantages within such a group provide no direction for updating the policy. Consequently, the sampling imbalance and the all-failed groups collapse into a self-perpetuating credit trap: failure-dominated sampling yields no learning signal, allowing repeated low-effect actions to persist. To break this trap, we propose Coverage-conditioned Group Policy Optimization (CovGPO), which uses first-visit observation coverage only when all samples in a group receive zero outcome reward. Specifically, within such groups, CovGPO assigns higher relative advantages to state-exploring actions without corrupting the original reward signal, as reaching new observations is a prerequisite for task success. Experiments on two challenging agentic benchmarks, ALFWorld and WebShop, show that CovGPO consistently improves over group-based baselines, with particularly large gains on hard tasks.

Kaibing Yang, Guangfeng Cai, Shengtian Yang +6

Southeast University · Kuaishou Technology · Meituan

Group Relative Policy Optimization RL Exploration Sparse-Reward RL Long-Horizon Agent Tasks Policy Learning RL for LLM Agents

Ask PDF

Jul 5, 2026

Progress- and Reliability-Oriented Group Policy Optimization for Agentic Reinforcement Learning

Group-based reinforcement learning (RL) has become an effective paradigm for improving large language model agents on long-horizon interactive tasks. To obtain finer-grained policy updates than trajectory-level optimization, recent work has moved toward step-level group-based RL, where intermediate steps are grouped and compared within a rollout batch. However, step-level advantage estimation is sensitive to how groups are formed: grouping by broad state keys improves coverage but may compare actions taken under different histories, while enforcing historical consistency yields fairer comparisons at the cost of fragmented groups and missing peer-comparison signal. In this paper, we propose ProGPO (Progress- and Reliability-Oriented Group Policy Optimization), a learned-critic-free method for context-consistent step-level learning. ProGPO keeps exact-prefix action comparison, and complements sparse peer comparisons with transition credit derived from rollout-based state potentials. To estimate these potentials reliably, ProGPO combines semantic expansion with inverse-variance fusion across history depths. We evaluate ProGPO on two challenging agentic tasks, ALFWorld and WebShop, with Qwen2.5-1.5B-Instruct. Results show that ProGPO improves over matched agentic RL baselines under comparable computational overhead, and additional Qwen2.5-3B-Instruct experiments further test the scalability of the proposed method.

Mingxuan Fan, Peiyang Liu

Baidu Inc., China · Peking University, China

Agentic RL Group Relative Policy Optimization Policy Optimization

Ask PDF

Feb 26, 2026

Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks

Group-based reinforcement learning (RL), such as GRPO, has advanced the capabilities of large language models on long-horizon agentic tasks. To enable more fine-grained policy updates, recent research has increasingly shifted toward stepwise group-based policy optimization, which treats each step in a rollout trajectory independently while using a memory module to retain historical context. However, we find a key issue in estimating stepwise relative advantages, namely context inconsistency, where steps within the same group may differ in their historical contexts. Empirically, we reveal that this issue can lead to severely biased advantage estimation, thereby degrading policy optimization significantly. To address the issue, in this paper, we propose Hierarchy-of-Groups Policy Optimization (HGPO) for long-horizon agentic tasks. Specifically, within a group of rollout trajectories, HGPO assigns each step to multiple hierarchical groups according to the consistency of historical contexts. Then, for each step, HGPO computes distinct advantages within each group and aggregates them with an adaptive weighting scheme. In this way, HGPO can achieve a favorable bias-variance trade-off in stepwise advantage estimation, without extra models or rollouts. Evaluations on two challenging agentic tasks, ALFWorld and WebShop with Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct, show that HGPO significantly outperforms existing agentic RL methods under the same computational constraints. Code is available at https://github.com/langfengQ/verl-agent/tree/master/recipe/hgpo.

Shuo He, Lang Feng, Qi Wei +3

Nanyang Technological University · Southeast University

Agentic RL RL for LLM Agents Group Relative Policy Optimization Long-Horizon Agent Tasks

Ask PDF