Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks

Shuo He; Lang Feng; Qi Wei; Xin Cheng; Lei Feng; Bo An

cs.LGFeb 26, 2026

Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks

Authors: Shuo He, Lang Feng, Qi Wei, Xin Cheng, Lei Feng, Bo An

Organizations: Nanyang Technological University · Southeast University

Abstract

Group-based reinforcement learning (RL), such as GRPO, has advanced the capabilities of large language models on long-horizon agentic tasks. To enable more fine-grained policy updates, recent research has increasingly shifted toward stepwise group-based policy optimization, which treats each step in a rollout trajectory independently while using a memory module to retain historical context. However, we find a key issue in estimating stepwise relative advantages, namely context inconsistency, where steps within the same group may differ in their historical contexts. Empirically, we reveal that this issue can lead to severely biased advantage estimation, thereby degrading policy optimization significantly. To address the issue, in this paper, we propose Hierarchy-of-Groups Policy Optimization (HGPO) for long-horizon agentic tasks. Specifically, within a group of rollout trajectories, HGPO assigns each step to multiple hierarchical groups according to the consistency of historical contexts. Then, for each step, HGPO computes distinct advantages within each group and aggregates them with an adaptive weighting scheme. In this way, HGPO can achieve a favorable bias-variance trade-off in stepwise advantage estimation, without extra models or rollouts. Evaluations on two challenging agentic tasks, ALFWorld and WebShop with Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct, show that HGPO significantly outperforms existing agentic RL methods under the same computational constraints. Code is available at https://github.com/langfengQ/verl-agent/tree/master/recipe/hgpo.

Figures & tables

Figure 1: Figure (a) compares trajectory-wise and stepwise policy optimization frameworks. Given two example group trajectories, Figure (b) illustrates trajectory-level and step-level grouping with their corresponding advantage estimations. Best viewed in color.

Figure 2: Statistics of GRPO and GiGPO. Figures (a) and (b) present the advantage differences relative to Oracle advantages for GRPO and GiGPO, respectively. Figures (c) and (d) report the average group size and the proportion of Oracle steps, respectively.

Figure 3: Overview of HGPO. The LLM-based agent interacts with a set of environments initialized from the same state $\bm{s}_{0}$ , producing four group trajectories (states with the same color are identical). HGPO comprises two key components: context-aware hierarchical grouping and adaptive weighted advantage computation. For illustration, consider the state $\bm{s}_{2}$ (purple). First, HGPO assigns $\bm{s}_{2}$ into three hierarchical groups according to its historical contexts. Then, it computes the final advantage estimate by adaptively aggregating the weighted advantages from these groups.

Model	Type	Method	ALFWorld		WebShop
Model	Type	Method	In-Success	Out-Success	Task Scores	Task Success Rates
Closed	Prompting	GPT-4o	48.0	46.0	31.8	23.7
Closed	Prompting	Gemini-2.5-Pro	60.3	50.5	42.5	35.9
Qwen2.5-1.5B-Instruct	Prompting	Qwen2.5	4.1	-	23.1	5.2
	Prompting	ReAct	12.8	-	40.1	11.3
	Prompting	Reflexion	21.8	-	55.8	21.9

Table 1: Performance comparison on ALFWorld and WebShop. For ALFWorld, we report the overall success rate (

\uparrow

) for both in-distribution (In-Success) and out-of-distribution tasks (Out-Success). For WebShop, we report the average task score (

\uparrow

) and the average task success rate (

\uparrow

). Most results are averaged over 3 random seeds during testing. The best results are highlighted in bold.

Panel of Figure 4 — Figure 4: Distributions of hierarchical group sizes on ALFWorld and WebShop using Qwen2.5-1.5B-Instruct. “0/1/2/3/4-Context” indicates different hierarchical groups. The first two rows correspond to $K=2$ , and the last row corresponds to $K=4$ . The y-axis denotes the proportion.

Table 6

$\alpha$	$K$	ALFWorld		WebShop
$\alpha$	$K$	In-Suc.	Out-Suc.	Task Sco.	Task Suc.
0	2	94.33 ±1.08	90.03 ±2.58	88.40 ±2.20	73.95 ±4.21
0	4	92.12 ±0.40	89.58 ±1.00	87.96 ±2.48	73.82 ±3.26
1	2	92.77 ±1.08	90.16 ±0.78	85.56 ±2.86	71.54 ±4.00
1	4	94.85 ±0.92	92.12 ±1.63	90.64 ±1.05	78.12 ±2.06
2	2	90.36 ±0.96	86.91 ±2.03	88.29 ±2.88	72.00 ±3.03

Table 4: Parameter analysis on the effects of different values of

\alpha

on ALFWorld and WebShop using Qwen2.5-1.5B-Instruct.

Ablation	$K$	ALFWorld		WebShop
Ablation	$K$	In-Suc.	Out-Suc.	Task Sco.	Task Suc.
HGPO	2	92.77 ±1.08	90.16 ±0.78	85.56 ±2.86	71.54 ±4.00
HGPO	4	94.85 ±0.92	92.12 ±1.63	90.64 ±1.05	78.12 ±2.06
w/o $G^{H}_{1:K}$	2	90.88 ±1.30	85.15 ±1.73	87.89 ±0.86	73.24 ±1.47
w/o $G^{H}_{1:K}$	4	91.14 ±1.07	88.80 ±3.21	88.83 ±2.68	74.73 ±3.62
w/o Ada. $w_{k}$	2	94.33 ±1.08	90.03 ±2.58	88.40 ±2.20	73.95 ±4.21

Table 5: Ablation study on ALFWorld and WebShop using Qwen2.5-1.5B-Instruct.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Type	Advantage estimation	Granularity	Bias	Variance
Trajectory-level	$A^{T}(\tau_{i})=\left(R({\tau_{i}})-1/\|G_{\tau}\|\sum\nolimits_{j\in G_{\tau}}R({\tau_{i}})\right)/\sigma_{G_{\tau}}$	Coarse-grained	$B_{T}$	$V_{T}$
Step-level	$A^{S}(\tilde{\bm{s}_{i}})=\left(R(\tilde{\bm{s}_{i}})-1/\|G_{\tilde{\bm{s}_{i}}}\|\sum\nolimits_{j\in G_{{\tilde{\bm{s}_{i}}}}}R(\tilde{\bm{s}_{j}})\right)/\sigma_{G_{{\tilde{\bm{s}_{i}}}}}$	Fine-grained	$b_{0}$	$v_{0}$
Hierarchy-of-Groups	$A^{H}(\bm{s}_{t}^{(i)})=\sum\nolimits_{k=0}^{K}\bm{w}_{k}A^{H}_{k}(\bm{s}_{t}^{(i)})$	Fine-grained	$\sum_{k=0}^{K}w_{k}b_{k}\downarrow$	$\sum_{k=0}^{K}w_{k}^{2}v_{k}\downarrow$

Appendix

Table 6: Overall comparison of three different advantage estimators.

Panel of Figure 5 — Figure 5: The prompt template of ALFWorld agents.

Panel of Figure 6 — Figure 6: The prompt template used for WebShop agents.

Model	Type	Method	ALFWorld		WebShop
Model	Type	Method	In-Success	Out-Success	Task Scores	Task Success Rates
Q2.5-1.5B	RL Training	GiGPO ( $K$ =2)	85.42 ±1.32	80.72 ±1.62	84.52 ±0.98	69.79 ±0.59
	RL Training	HGPO ( $K$ =2)	89.58 ±0.45	80.73 ±2.38	87.53 ±0.77	72.66 ±1.78
	RL Training	GiGPO ( $K$ =4)	85.15 ±2.81	80.98 ±0.45	88.5 ±0.49	74.08 ±0.98
	RL Training	HGPO ( $K$ =4)	92.45 ±0.81	89.06 ±2.34	88.90 ±0.90	75.91 ±1.19
Q2.5-7B	RL Training	GiGPO ( $K$ =2)	89.84 ±2.20	82.81 ±5.46	86.23 ±1.43	75.13 ±1.37

Appendix

Table 7: Performance comparison on ALFWorld and WebShop. Note that we used the old version of Verl-agent and updated the performance of using the new version of Verl-agent (with veRL updating) in Table 1 . For ALFWorld, we report the overall success rate (

\uparrow

) for both in-distribution (In-Success) and out-of-distribution tasks (Out-Success). For WebShop (15 steps), we report the average task score (

\uparrow

) and the average task success rate (

\uparrow

). Most results are averaged over 3 random seeds during testing. The best results are highlighted in bold.

Panel of Figure 7 — Figure 7: Training dynamics of HGPO (Red), GiGPO (Yellow), and GRPO (Purple) on ALFWorld using Qwen2.5-1.5B-Instruct. The details of these metrics are shown in Appendix D.2 .

Panel of Figure 8 — Figure 8: Training dynamics of HGPO (Red), GiGPO (Yellow), and GRPO (Blue) on WebShop using Qwen2.5-1.5B-Instruct. Best viewed in color.

Panel of Figure 9 — Figure 9: The distributions of hierarchical group sizes ( $K=4$ ) on ALFWorld and WebShop using Qwen2.5-1.5B-Instruct.

Explore similar work

Progress- and Reliability-Oriented Group Policy Optimization for Agentic Reinforcement Learning
Jul 5, 2026Mingxuan Fan, Peiyang LiuAgentic RL Group Relative Policy Optimization
Baidu Inc., China · Peking University, China
Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learning
Jun 22, 2026Yunan Wang, Minghui Song, Zihan Zhang +6Agentic RL Reinforcement Learning
1Peking University · 2Microsoft Corporation
Learning from All-Failed Groups via First-Visit Coverage in Long-Horizon Agentic Tasks
Jul 22, 2026Kaibing Yang, Guangfeng Cai, Shengtian Yang +6Group Relative Policy Optimization RL Exploration
Southeast University · Kuaishou Technology · Meituan

Jul 5, 2026J/K move · Enter open · S save

Progress- and Reliability-Oriented Group Policy Optimization for Agentic Reinforcement Learning

Mingxuan Fan, Peiyang Liu

Baidu Inc., China · Peking University, China

Agentic RL Group Relative Policy Optimization Policy Optimization

Open paper AskPDF arXiv

Group-based reinforcement learning (RL) has become an effective paradigm for improving large language model agents on long-horizon interactive tasks. To obtain finer-grained policy updates than trajectory-level optimization, recent work has moved toward step-level group-based RL, where intermediate steps are grouped and compared within a rollout batch. However, step-level advantage estimation is sensitive to how groups are formed: grouping by broad state keys improves coverage but may compare actions taken under different histories, while enforcing historical consistency yields fairer comparisons at the cost of fragmented groups and missing peer-comparison signal. In this paper, we propose ProGPO (Progress- and Reliability-Oriented Group Policy Optimization), a learned-critic-free method for context-consistent step-level learning. ProGPO keeps exact-prefix action comparison, and complements sparse peer comparisons with transition credit derived from rollout-based state potentials. To estimate these potentials reliably, ProGPO combines semantic expansion with inverse-variance fusion across history depths. We evaluate ProGPO on two challenging agentic tasks, ALFWorld and WebShop, with Qwen2.5-1.5B-Instruct. Results show that ProGPO improves over matched agentic RL baselines under comparable computational overhead, and additional Qwen2.5-3B-Instruct experiments further test the scalability of the proposed method.

Abstract

Explore similar work

Progress- and Reliability-Oriented Group Policy Optimization for Agentic Reinforcement Learning

Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learning

Learning from All-Failed Groups via First-Visit Coverage in Long-Horizon Agentic Tasks