Matched-Context Group Policy Optimization for LLM Agent Training

Chenxu Zhang; Yaorui Shi; Wenlong Zhang; Jianchao Tan; Zihan Wang; Yiming Cheng

cs.LGOct 9, 2026

Matched-Context Group Policy Optimization for LLM Agent Training

Authors: Chenxu Zhang, Yaorui Shi, Wenlong Zhang, Jianchao Tan, Zihan Wang, Yiming Cheng

Organizations: Imperial College London · Meituan · University of Science and Technology of China · Tsinghua University

Abstract

Training long-horizon language model agents requires assigning credit to intermediate decisions from sparse terminal feedback. Within a finite group of sibling rollouts, useful action comparisons depend on their context, evidence for both alternatives, and how that evidence enters learning. We propose Matched-Context Group Policy Optimization (MCGPO), a procedure for constructing weighted action preferences from already collected trajectories. MCGPO defines comparison contexts with pre-action state summaries and observed rubric progress, annotated by task rules. Two-action support selects among ordered exact and coarse matching rules. Fixed folds separate candidate decisions from the terminal outcomes used to orient their preferences, and support-dependent evidence weights scale an auxiliary pairwise objective alongside shared GraphGPO task learning. MCGPO reuses rollout data and each response's original history, preserving critic-free training. Across six environment-backbone settings with three training seeds each, MCGPO improves mean success over matched GraphGPO: by 0.47-1.46 percentage points on ALFWorld and WebShop with Qwen2.5-1.5B and 7B, and by 3.96 and 5.16 points on WebShop with Qwen3-4B and 8B. The ALFWorld gains build on baseline success above 92%, where remaining headroom is limited. Integrated success improves in three of four Qwen2.5 settings. Three-seed factorial controls support opposite-fold evidence and evidence weighting; joint state-rubric matching achieves the highest mean in all four representation-control settings.

Figures & tables

Figure 1: Task context for local action comparisons. The same state summary

h(s)

can occur at different stages of a task. MCGPO groups decisions using the state summary and observed rubric progress

C

, with enabled criteria annotated by the rule-derived frontier

F

Figure 2: Overview of MCGPO. The Verify block labels matched actions using opposite-fold terminal outcomes. The resulting evidence-weighted pairwise loss complements the shared task objective.

Type	Method	ALFWorld							WebShop
Type	Method	Pick	Look	Clean	Heat	Cool	Pick2	All	Score	Succ.
Closed-source models
Prompting	GPT-4o	75.3	60.8	31.2	56.7	21.6	49.8	48.0	31.8	23.7
Prompting	Gemini-2.5-Pro	92.8	63.3	62.1	69.0	26.6	58.7	60.3	42.5	35.9
Base: Qwen2.5-1.5B-Instruct
Prompting	Qwen2.5	5.9	5.5	3.3	9.7	4.2	0.0	4.1	23.1	5.2

Table 1: Task success (%) on ALFWorld and graded score/success (%) on WebShop. All and Succ. are the primary metrics. GraphGPO and MCGPO report mean

\pm

sample SD over three seeds using the evaluation protocol in Section 4.1 . Results for the other methods are from Feng et al. (2025) ; Tan et al. (2026) .

Figure 3: Learning curves and paired gains with Qwen2.5-1.5B. Left axes: validation success for MCGPO and GraphGPO; bands show the three-seed sample SD and faint lines retain raw means. Right axes: the deep-blue curve shows the mean paired difference (MCGPO minus GraphGPO), in percentage points. Each seed is averaged over up to five trailing evaluations, using the available evaluations at the beginning so curves start at update 0; complete five-point windows apply from update 20. The update-100 annotation uses the corresponding smoothed values. Shaded bands apply only to success curves.

(a) Qwen2.5-1.5B: supervision controls (one run)
Component	Configuration	WebShop $\uparrow$	ALFWorld $\uparrow$
MCGPO (Full)		78.13	94.53
Match	w/ Shuffled Rubric	75.78 ( $\downarrow$ 2.34)	93.91 ( $\downarrow$ 0.63)
Label	Candidate-progress labels	76.72 ( $\downarrow$ 1.41)	91.41 ( $\downarrow$ 3.13)
Learn	w/o Pairwise Loss	76.72 ( $\downarrow$ 1.41)	93.75 ( $\downarrow$ 0.78)
Learn	GraphGPO + Rubric Shaping	69.84 ( $\downarrow$ 8.28)	91.88 ( $\downarrow$ 2.66)

Table 2: Component ablations of MCGPO. (a) One Qwen2.5-1.5B run per configuration; parentheses give decreases from Full (pp). (b) Qwen3-4B/WebShop mean

\pm

sample SD over three seeds. Entries are success rates (%) under the common evaluation protocol (Section 4.1 ); differences are computed before rounding. Protocols and paired analyses: Appendix C.3 .

Figure 4: Preference construction on WebShop with Qwen2.5-1.5B. (a) Accepted pairs from exact and coarse matching. (b) Agreement of preference directions with candidate outcomes (ties receive half credit), together with the tie share. (c) Absolute outcome contrast and the resulting evidence weight, averaged over accepted pairs. Each point summarizes 25 updates; lines and bands show the three-seed mean and

\pm 1

sample SD. Ratios and pair-level means pool observations within each seed/window. Appendix C.5 defines the statistics and aggregation.

Backbone	Mean success (%)		Gain (pp)
Backbone	GraphGPO	MCGPO	Gain (pp)
Qwen3-4B	$78.18\pm 2.62$	$\mathbf{82.14}\pm 3.52$	+3.96
Qwen3-8B	$79.69\pm 1.95$	$\mathbf{84.84}\pm 1.96$	+5.16

Table 3: Task success on WebShop with Qwen3 backbones. Entries are mean

\pm

sample SD over three training seeds under the evaluation protocol in Section 4.1 . Gain is the paired mean difference in percentage points, computed before rounding.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Comparison conditions	Evidence and aggregation	Policy interface
GiGPO ( Feng et al., 2025 )	Repeated anchor states	Returns within trajectory and state groups	Trajectory and step advantages
HGPO ( He et al., 2026 )	Current state and historical-context hierarchy	Weighted advantages across context depths	Step advantage
BiPACE ( Wang et al., 2026a )	Behavioral clusters from policy features	Action-conditioned peer returns	Step advantage
GraphGPO ( Cheng et al., 2026 )	Observed transition graph	Changes in distance to observed goals	Graph advantage
G 2 PO ( Wang et al., 2026c )	Identical observations in a global transition graph	Graph state values and globally standardized edge TD errors	Edge and episode advantages
AT-GRPO ( Peng et al., 2026 )	Dialogue trees with adaptive expansion depth	Returns over stage-dependent observation ranges	Group-relative policy loss

Appendix

Table 4: Comparison conditions, evidence, and policy interfaces. Rows summarize method designs; the matched performance evaluation in this paper uses GraphGPO. MCGPO’s frontier is a rule-derived annotation of progress.

Environment	Exact state	Coarse state
ALFWorld	Task type, location, held object, acquired/processed/placed object sets, light status	Task type, location, holding/processing flags, placement count, light status, remaining-budget bin
WebShop	Page, search query, product identity, selected options	Page, searched/product-open flags, binned type/attribute/option match, price satisfaction

Appendix

Table 5: State fields for the ordered matching rules. Both rules retain the rubric signature

(C,F)

Setting	Value
Sibling trajectories / fold sizes	$8$ / $4+4$
Minimum support for each action $m$	$1$
Terminal-score threshold for evidence	$0.999$
Outcome-contrast threshold $\epsilon$	$0.10$
Support shrinkage $\kappa$	$2$
Resolution weights $(q_{\rm exact},q_{\rm coarse})$	$(1,0.70)$

Appendix

Table 6: Fixed preference-construction and learning settings.

	ALFWorld		WebShop
Setting	Small	Large	Small	Large
Qwen2.5 backbone	1.5B	7B	1.5B	7B
Qwen3 backbone	—	—	4B	8B
Task groups / update ( $P$ )	16	16	16	16
Trajectories / group ( $G$ )	8	8	8	8
Admitted trajectories / update	128	128	128	128

Appendix

Table 7: Training and rollout configuration for matched GraphGPO and MCGPO runs. Micro-batch sizes are reported per GPU. Mean–std denotes the shared GraphGPO within-group advantage normalization mode. TP denotes rollout tensor parallelism.

Setting	Seed	MCGPO $L_{5}$	GraphGPO $L_{5}$	MCGPO $E_{250}$	GraphGPO $E_{250}$
ALFWorld / 1.5B	0	92.03	90.47	97.66	92.97
	1	94.53	93.13	94.53	96.88
	2	94.38	94.22	94.53	96.88
ALFWorld / 7B	0	95.31	96.88	96.88	95.31
	1	96.88	95.00	96.88	96.88
	2	96.72	92.66	94.53	92.97

Appendix

Table 8: Per-seed Qwen2.5 policy success (%).

L_{5}

averages the final five scheduled evaluations of the 250-update budget;

E_{250}

reports the final evaluation.

Backbone	Seed	GraphGPO (%)	MCGPO (%)	Gain (pp)
Qwen3-4B	0	79.84	85.63	+5.78
	1	75.16	78.59	+3.43
	2	79.53	82.19	+2.66
Qwen3-8B	0	81.25	85.00	+3.75
	1	77.50	86.72	+9.22
	2	80.31	82.81	+2.50

Appendix

Table 9: All paired Qwen3 WebShop seeds for the primary

L_{5}

readout. Each entry averages updates 230, 235, 240, 245, and 250. Gains are computed before final display rounding; three-seed means appear in Table 3 .

ALFWorld / 1.5B
Setting / readout	Success (%)	Normalized AUC (%)
GraphGPO	$92.60\,\pm\,1.93$	$77.42\,\pm\,1.73$
MCGPO	$93.65\,\pm\,1.40$	$78.68\,\pm\,1.34$
Paired $\Delta$ (pp)	$+1.04$	$+1.25$
95% paired interval	$[-0.87,\,+2.96]$	$[-0.67,\,+3.18]$
ALFWorld / 7B

Appendix

Table 10: Success and learning progress for GraphGPO and MCGPO across three training seeds. Entries give mean

\pm

sample SD; both scores use a 0–100 scale.

\Delta

is MCGPO minus GraphGPO in percentage points. Success follows Eq. 14 ; normalized raw-curve AUC summarizes learning progress. Intervals are 95% paired Student-

t

intervals (

\mathrm{df}=2

); definitions appear in Appendix C .

Figure 5: Qwen2.5-1.5B validation success at nearly matched training-transition budgets. Markers show three-seed means and whiskers sample SD. Each seed’s checkpoint pair differs in budget by less than 1%; displayed budgets average the paired costs. Counts cover retained training interactions; Appendix C.2 specifies matching and restart accounting.

Env.	Seed	Updates	Transitions	Gap (%)	Success (%)	$\Delta$ (pp)
ALFWorld	0	230/250	594,465/592,472	0.336	87.50/97.66	+10.16
	1	250/250	575,983/581,151	0.897	96.88/94.53	-2.34
	2	245/250	590,739/585,967	0.814	94.53/94.53	0.00
WebShop	0	250/215	174,190/173,006	0.684	75.00/80.47	+5.47
	1	250/230	188,293/187,096	0.640	77.34/78.91	+1.56
	2	250/200	172,791/171,773	0.593	75.78/78.91	+3.13

Appendix

Table 11: Selected budget-matched checkpoints for Qwen2.5-1.5B. Slash-separated entries give GraphGPO/MCGPO. Gap is the relative budget difference in Eq. 17 ;

\Delta

is MCGPO minus GraphGPO success in percentage points.

(a) Per-seed task success (%)
Configuration	Seed 0	Seed 1	Seed 2
A: MCGPO (Full)	85.63	78.59	82.19
B: Same-fold terminal	83.28	76.56	79.53
C: Uniform weighting	84.22	76.72	80.63
D: Same-fold + uniform	81.56	74.84	77.97

Appendix

Table 12: Qwen3-4B WebShop factorial controls from Table 2 (b). (a) Per-seed success (%), averaging the final five scheduled evaluations. (b) Paired gains in percentage points, with 95% Student-

t

intervals over three seed differences (

\mathrm{df}=2

). Statistics precede display rounding.

(a) Task success (%)
Method	Qwen3-4B	Qwen3-8B
GraphGPO	$78.18\pm 2.62$	$79.69\pm 1.95$
GraphGPO + shaping	$79.85\pm 3.21$	$82.19\pm 1.44$
MCGPO	$\mathbf{82.14}\pm 3.52$	$\mathbf{84.84}\pm 1.96$

Appendix

Table 13: WebShop comparison with a scorer-matched shaping control. (a) Task success (%), reported as mean

\pm

sample SD over three seeds. (b) MCGPO gains over GraphGPO + shaping, in percentage points. Paired 95% Student-

t

intervals (

\mathrm{df}=2

) use the per-seed differences shown.

Setting	Comparison key	$n$	$L_{5}$	$E_{250}$
ALFWorld / 1.5B	MCGPO	3	$93.65\,\pm\,1.40$	$95.57\,\pm\,1.80$
	State-only	3	$85.57\,\pm\,2.32$	$83.85\,\pm\,1.97$
	Rubric-only	3	$87.81\,\pm\,1.59$	$87.76\,\pm\,1.80$
ALFWorld / 7B	MCGPO	3	$96.30\,\pm\,0.86$	$96.09\,\pm\,1.35$
	State-only	3	$92.34\,\pm\,1.25$	$91.93\,\pm\,1.97$
	Rubric-only	3	$91.15\,\pm\,1.83$	$90.36\,\pm\,1.97$

Appendix

Table 14: Supplementary representation controls at 250 updates. Full, State-only, and Rubric-only use seeds 0, 1, and 2 in every setting. Entries give mean success (%) and sample SD across the three seeds.

L_{5}

averages the final five scheduled evaluations, and

E_{250}

is the terminal evaluation. State-only retains the environment-specific fields of its state abstraction, including WebShop’s match-score bins.

Figure 6: Validation success in all four Qwen2.5 settings. Each seed uses up to five trailing scheduled evaluations, including all available points at the beginning. Lines and bands give the three-seed mean and sample SD; faint lines retain raw means. The fixed

L_{5}

window is defined separately in Eq. 14 .

Figure 7: ALFWorld Qwen2.5-1.5B training behavior: (a) rollout success, (b) environment steps per rollout, and (c) anchor-state group size in the task-credit branch. Dark lines average ten trailing updates within each seed, then report the three-seed mean; bands show sample SD and faint lines show seed means.

Channel (symbol)	Status after the observed transition
target_acquired ( $A$ )	$\min(r,\|\mathcal{A}\|)$ ; acquisition history persists after placement.
target_in_hand ( $H$ )	Whether the worker’s held object matches the target type.
target_processed ( $P$ )	$\mathrm{1}[\mathcal{P}\neq\emptyset]$ ; processing history persists after release.
target_placed ( $L$ )	$\min(r,\|\mathcal{L}\|)$ ; taking or relocating an object updates this set.
light_on ( $B$ )	Tracked target-light state: successful off commands/feedback set false; on, use, or toggle set true unless off is indicated.
goal_progress ( $G$ )	Look: $H+B$ . Clean/Heat/Cool: $\mathrm{1}[\mathcal{P}\neq\emptyset]+\mathrm{1}[\mathcal{L}\neq\emptyset]+\mathrm{1}[\mathcal{P}\cap\mathcal{L}\neq\emptyset]$ . Pick/Pick2: $\min(r,\|\mathcal{L}\|)$ .

Appendix

Table 15: ALFWorld channel definitions. Symbols identify the nodes in Table 17 ;

\mathrm{1}[\cdot]

denotes an indicator. The sets and required object count

r

are defined in the text.

Channel (symbol)	Status after the observed transition
search_issued ( $Q$ )	Whether the session has search keywords or its searched flag is true.
product_opened ( $O$ )	Whether a product is selected or the product-open flag is true.
type_match ( $T$ )	The scorer’s type-match component r_type .
attribute_match ( $A$ )	Requested-attribute match fraction; scorer r_att is stored as r_attr .
option_match ( $U$ )	Requested-option match fraction r_option ; the standard worker retains zero when the scorer omits this component.
price_ok ( $P$ )	The scorer’s Boolean price predicate r_price .

Appendix

Table 16: WebShop channel definitions. Symbols refer to WebShop nodes in Table 17 . Match components come from the environment goal scorer; default handling is specified in the text.

Environment/task	Relation	Directed edges
ALFWorld: all	E	$A\to H$
Pick / Pick2	E	$A\to L$
	C	$L\to G$ , $G\to S$
Clean / Heat / Cool	E	$A\to P$ , $A\to L$
	C	$P\to L$ , $P\to G$ , $L\to G$ , $G\to S$
Look	C	$H\to G$ , $B\to G$ , $H\to S$ , $B\to S$ , $G\to S$

Appendix

Table 17: Complete dependency templates. Symbols use the respective environment’s channel table. E denotes requires_ever and C denotes requires_current . Each ALFWorld family includes the common edge; set notation expands to one edge per element.

Figure 8: First-visit group sizes across ten update-60 captures (1.5B: seeds 1/2; 7B: seeds 0/1/2, in both environments). Exact and coarse rules retain the same rubric coordinates; groups count distinct contributors within a four-trajectory candidate fold. Bars include singletons, and annotations report mean sizes and group counts.

Setting	Exact	Coarse	Exact	Coarse
	Non-singleton records (%)		Supported row pairs
ALFWorld / 1.5B	74.07	68.57	143	133
ALFWorld / 7B	91.43	91.90	86	252
WebShop / 1.5B	29.19	95.76	2	19
WebShop / 7B	36.61	96.89	14	79

Appendix

Table 18: Single-resolution comparison availability in all ten captured update-60 batches. Non-singleton coverage is the fraction of first-visit records in groups with at least two contributors. Supported pairs have distinct actions and at least one opposite-fold observation of each action. These counts precede outcome-contrast filtering and the final quota; exact and coarse counts overlap.

Figure 9: Captured ALFWorld example: put two toiletpaper in drawer . Two trajectories reach countertop 1 with empty hands, zero target placements, and the same budget bin. One previously acquired toiletpaper 2 and placed it at toilet 1. Its changed progress signature retains the later visit as a distinct context under the joint first-visit rule, before preference selection.

Setting	Pairs	$W/T/L$	Agreement (%)	Active	Exact (%)
WebShop / 1.5B	1428	771/568/89	73.88	330/450	8.3
WebShop / 7B	1425	633/693/99	68.74	319/450	20.1

Appendix

Table 19: Archived WebShop training-trace diagnostics for Full, pooled over three seeds per Qwen2.5 backbone.

W/T/L

denotes counts of preferred-candidate wins, ties, and reversed outcomes; agreement is

(W+T/2)/(W+T+L)

. Active gives accepted-pair updates / audited updates; Exact is the fraction of accepted pairs using exact matching. All counts refer to the retained trace sample.

Explore similar work

Learning from All-Failed Groups via First-Visit Coverage in Long-Horizon Agentic Tasks
Jul 22, 2026Kaibing Yang, Guangfeng Cai, Shengtian Yang +6Group Relative Policy Optimization RL Exploration
Southeast University · Kuaishou Technology · Meituan
Progress- and Reliability-Oriented Group Policy Optimization for Agentic Reinforcement Learning
Jul 5, 2026Mingxuan Fan, Peiyang LiuAgentic RL Group Relative Policy Optimization
Baidu Inc., China · Peking University, China
Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks
Feb 26, 2026Shuo He, Lang Feng, Qi Wei +3Agentic RL RL for LLM Agents
Nanyang Technological University · Southeast University

Jul 22, 2026J/K move · Enter open · S save

Learning from All-Failed Groups via First-Visit Coverage in Long-Horizon Agentic Tasks

Kaibing Yang, Guangfeng Cai, Shengtian Yang, Shuo He+5

Southeast University · Kuaishou Technology · Meituan

Group Relative Policy Optimization RL Exploration Sparse-Reward RL Long-Horizon Agent Tasks Policy Learning RL for LLM Agents

Open paper AskPDF arXiv

Group-based policy optimization has been increasingly used to train large language model (LLM) agents from sparse outcome rewards by comparing trajectories or steps within a group. However, on difficult long-horizon agentic tasks, we found this comparison suffers from a sampling imbalance: repeated or low-effect actions dominate the high-probability region of the policy, leaving useful state-changing actions under-sampled. This imbalance produces many all-failed rollout groups, where the relative advantages within such a group provide no direction for updating the policy. Consequently, the sampling imbalance and the all-failed groups collapse into a self-perpetuating credit trap: failure-dominated sampling yields no learning signal, allowing repeated low-effect actions to persist. To break this trap, we propose Coverage-conditioned Group Policy Optimization (CovGPO), which uses first-visit observation coverage only when all samples in a group receive zero outcome reward. Specifically, within such groups, CovGPO assigns higher relative advantages to state-exploring actions without corrupting the original reward signal, as reaching new observations is a prerequisite for task success. Experiments on two challenging agentic benchmarks, ALFWorld and WebShop, show that CovGPO consistently improves over group-based baselines, with particularly large gains on hard tasks.

Abstract

Explore similar work

Learning from All-Failed Groups via First-Visit Coverage in Long-Horizon Agentic Tasks

Progress- and Reliability-Oriented Group Policy Optimization for Agentic Reinforcement Learning

Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks