Training LLMs as interactive agents for multi-turn decision-making remains challenging, particularly in long-horizon tasks with sparse and delayed rewards, where agents must execute extended sequences of actions before receiving meaningful feedback. Most existing reinforcement learning (RL) approaches model LLM agents as flat policies operating at a single time scale, selecting one action at each turn. In sparse-reward settings, such flat policies must propagate credit across the entire trajectory without explicit temporal abstraction, which often leads to unstable optimization and inefficient credit assignment. We propose HiPER, a novel Hierarchical Plan-Execute RL framework that explicitly separates high-level planning from low-level execution. HiPER factorizes the policy into a high-level planner that proposes subgoals and a low-level executor that carries them out over multiple action steps. To align optimization with this structure, we introduce a key technique called hierarchical advantage estimation (HAE), which carefully assigns credit at both the planning and execution levels. By aggregating returns over the execution of each subgoal and coordinating updates across the two levels, HAE provides an unbiased gradient estimator and provably reduces variance compared to flat generalized advantage estimation. Empirically, HiPER achieves state-of-the-art performance on challenging interactive benchmarks, reaching 97.4% success on ALFWorld and 83.3% on WebShop with Qwen2.5-7B-Instruct (+6.6% and +8.3% over the best prior method), with especially large gains on long-horizon tasks requiring multiple dependent subtasks. These results highlight the importance of explicit hierarchical decomposition for scalable RL training of multi-turn LLM agents.
Figures & tables
Figure 1: Overall performance on agentic benchmarks ALFWorld and WebShop , with Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct as base models. Our method consistently outperforms all evaluated baseline methods, including the best known prior method GiGPO ( Feng et al., 2025 ) , across both benchmarks and model sizes.
Figure 2: Overview of the HiPER framework. The upper panel illustrates standard flat RL for LLM agents, where a single policy operates at one time scale and chooses an action at every turn, often leading to brittle long-horizon behavior. For instance, the agent may prematurely head to the cabinet before picking up, and cleaning the cup. The lower panel presents our HiPER framework, built on two components: the Plan-Execute interface (Sec. 4.1 ), a structured agent interface that explicitly separates high-level planning and low-level execution ; and we propose the hierarchical advantage estimation (Sec. 4.3 ), which aligns credit assignment with this two-level structure by propagating learning signals both within and across subgoal segments.
Method
ALFWorld
WebShop
Pick
Look
Clean
Heat
Cool
Pick2
All
Score
Succ.
Qwen2.5-1.5B-Instruct
Base Model
15.9 ±1.8
13.9 ±6.7
11.2 ±4.1
3.5 ±6.1
0.0 ±0.0
4.2 ±0.7
8.3 ±0.9
25.1 ±8.9
5.5 ±6.3
+PPO
74.0 ±9.0
37.5 ±21.7
67.0 ±5.1
85.6 ±17.1
68.8 ±3.7
56.1 ±12.2
68.2 ±1.8
73.8 ±3.0†
51.5 ±2.9†
+RLOO †
88.3 ±3.0
52.8 ±8.6
71.0 ±5.9
62.8 ±8.7
66.4 ±5.5
56.9 ±4.7
69.7 ±2.5
73.9 ±5.6
52.1 ±6.7
+GRPO
77.4 ±6.7
54.2 ±7.2
75.6 ±8.6
85.6 ±17.1
67.8 ±7.4
56.1 ±11.0
71.1 ±8.2
75.8 ±3.5†
56.8 ±3.8†
Table 1: Performance on ALFWorld and WebShop. We report the success rate (%) of all six task categories, and the overall success rate for ALFWorld. For WebShop, we report the task score and success rate (%). Results are averaged over 3 random seeds. † indicates numbers reported in the GiGPO paper ( Feng et al., 2025 ) . The best values are in bold , and second best values are underlined .
Figure 3: ALFWorld 7B Curves. From the validation curve, HiPER achieves roughly 2.8 × speedup relative to PPO/GRPO. From the training curve, HiPER exhibits more stable training dynamics compared with PPO/GRPO, showing smaller oscillations.
Figure 4: HiPER Switching Behavior on ALFWorld. The switching frequency increases during early training, indicating a high-level exploration phase. After initial exploration, the switching frequency and segment length stabilizes.
Method
ALFWorld
Pick
Look
Clean
Heat
Cool
Pick2
All
Base Model (ReAct)
15.9 ±1.8
13.9 ±6.7
11.2 ±4.1
3.5 ±6.1
0.0 ±0.0
4.2 ±0.7
8.3 ±0.9
+PPO
74.0 ±9.0
37.5 ±21.7
67.0 ±5.1
85.6 ±17.1
68.8 ±3.7
56.1 ±12.2
68.2 ±1.8
+GRPO
77.4 ±6.7
54.2 ±7.2
75.6 ±8.6
85.6 ±17.1
67.8 ±7.4
56.1 ±11.0
71.1 ±8.2
+GiGPO †
94.4 ±5.9
67.5 ±4.6
94.8 ±3.8
94.4 ±7.8
79.8 ±4.7
76.4 ±5.4
86.7 ±1.7
Base Model (Plan-Execute)
3.4 ±3.3
10.4 ±3.6
3.0 ±2.6
0.0 ±0.0
0.0 ±0.0
1.3 ±2.3
2.9 ±1.6
Table 2: Comparison of ReAct and Plan-Execute prompting on ALFWorld. The upper panel reports performance of baselines when trained and evaluated with ReAct prompting; the lower panel reports performance of baselines and HiPER when trained and evaluated with Plan-Execute prompting. The base model used here is Qwen2.5-1.5B-Instruct. We report the success rate (%) of all six task categories, and the overall success rate. Results are averaged over 3 random seeds. The best values are in bold , and second best values are underlined .
Figure 5: ReAct prompt template of ALFWorld agents.
Figure 6: Plan-Execute prompt template of ALFWorld agents.
Figure 7: ReAct prompt template of WebShop agents.
Figure 8: Plan-Execute prompt template of WebShop agents.
Method
ALFWorld
Pick
Look
Clean
Heat
Cool
Pick2
All
HiPER-1.5B-1.5B
98.9 ±1.9
91.7 ±14.4
97.5 ±4.3
90.9 ±4.7
96.7 ±2.8
91.3 ±9.6
95.3 ±1.4
HiPER-1.5B-0.5B
97.4 ±3.7
96.4 ±5.1
98.1 ±2.6
78.6 ±6.4
85.5 ±6.4
39.3 ±5.1
87.5 ±0.6
HiPER-7B-7B
100 ±0.0
84.8 ±2.6
100 ±0.0
96.3 ±6.4
100.0 ±0.0
95.5 ±4.4
97.4 ±1.6
HiPER-7B-1.5B
98.6 ±2.0
95.8 ±5.9
100 ±0.0
100 ±0.0
92.0 ±0.0
88.9 ±7.9
96.1 ±1.1
Appendix
Table 3: Comparison of different critic model sizes on ALFWorld. Rows denote actor-critic size pairs (e.g., HiPER-7B-1.5B uses a 7B actor with a 1.5B critic). We report the success rate (%) of all six task categories, and the overall success rate. Results are averaged over 3 random seeds. The best values are in bold , and second best values are underlined .
cKEEP
0.0
0.1
0.2
0.3
0.4
0.5
Overall Succ. Rate (%)
76.6
90.9
92.9
95.3
92.6
91.8
Appendix
Table 4: Sensitivity analysis on cKEEP for the 1.5B model on ALFWorld task.
Large language model (LLM) agents have demonstrated strong capabilities in complex interactive decision-making tasks. However, existing LLM agents typically rely on increasingly long interaction histories, resulting in high computational cost and limited scalability. In this paper, we propose STEP-HRL, a hierarchical reinforcement learning (HRL) framework that enables step-level learning by conditioning only on single-step transitions rather than full interaction histories. STEP-HRL structures tasks hierarchically, using completed subtasks to represent global progress of overall task. By introducing a local progress module, it also iteratively and selectively summarizes interaction history within each subtask to produce a compact summary of local progress. Together, these components yield augmented step-level transitions for both high-level and low-level policies. Experimental results on ScienceWorld and ALFWorld benchmarks consistently demonstrate that STEP-HRL substantially outperforms baselines in terms of performance and generalization while reducing token usage. Our code is available at https://github.com/TonyStark042/STEP-HRL.
Shuai Zhen, Yanhua Yu, Ruopei Guo +2
Beijing University of Posts and Telecommunications · China Mobile Group Design Institute Co., Ltd · Singapore Management University
Large language model (LLM) agents have recently demonstrated strong capabilities in interactive decision-making, yet they remain fundamentally limited in long-horizon tasks that require structured planning and reliable execution. Existing approaches predominantly rely on flat autoregressive policies, where high-level reasoning and low-level actions are generated within a single token sequence, leading to inefficient exploration and severe error propagation over extended trajectories. In this work, we propose HiMAC, a hierarchical agentic RL framework that explicitly decomposes long-horizon decision-making into macro-level planning and micro-level execution. HiMAC models reasoning as a structured blueprint generation process followed by goal-conditioned action execution, enabling robust long-horizon planning within LLM-based agents. To train this hierarchy efficiently, we introduce a critic-free hierarchical policy optimization paradigm that extends group-based reinforcement learning to bi-level structures through hierarchical relative advantage estimation. Furthermore, we propose an iterative co-evolution training strategy that alternates between planner exploration and executor adaptation, mitigating the non-stationarity inherent in hierarchical learning. Extensive experiments on ALFWorld, WebShop, and Sokoban demonstrate that HiMAC consistently outperforms strong prompting and reinforcement learning baselines, achieving state-of-the-art performance and substantially improved sample efficiency across both text-based and visually grounded environments. Our results show that introducing structured hierarchy, rather than increasing model scale alone, is a key factor for enabling robust long-horizon agentic intelligence.
Hongbo Jin, Rongpeng Zhu, Jiayu Ding +2
School of Electronic and Computer Engineering, Peking University
A central goal of large language model (LLM) research is to build agentic systems that can plan, act, and adapt through sustained interaction with dynamic environments. While recent LLM-based agents exhibit impressive contextual reasoning, their long-horizon decision-making remains fragile, often suffering from objective drift, where goals and plans drift over extended interactions. We introduce Multi2, a hierarchical multi-agent decision-making framework that explicitly decomposes agent behavior into complementary roles. A high-level agent (System 1) focuses on context-aware sub-goal generation using supervised fine-tuning (SFT), while a low-level agent (System 2) executes atomic actions through offline-to-online reinforcement learning (RL) in interactive environments. This separation enables stable long-horizon control, mitigates objective drift, and allows efficient adaptation. Across diverse interactive environments, Multi2 consistently outperforms strong agentic baselines, demonstrating improved robustness and coordination in multi-turn interaction. Beyond performance, we introduce and release three hierarchical benchmark datasets, filling a long-standing gap in training and evaluating hierarchical decision-making for LLM-based agents.
Sangeun Park, Minhae Kwon
Department of Electrical and Computer Engineering, Sungkyunkwan University, Republic of Korea.