Hierarchical Reinforcement Learning with Augmented Step-Level Transitions for LLM Agents

Shuai Zhen; Yanhua Yu; Ruopei Guo; Nan Cheng; Yang Deng

cs.AIApr 7, 2026

Hierarchical Reinforcement Learning with Augmented Step-Level Transitions for LLM Agents

Authors: Shuai Zhen, Yanhua Yu, Ruopei Guo, Nan Cheng, Yang Deng

Organizations: Beijing University of Posts and Telecommunications · China Mobile Group Design Institute Co., Ltd · Singapore Management University

Abstract

Large language model (LLM) agents have demonstrated strong capabilities in complex interactive decision-making tasks. However, existing LLM agents typically rely on increasingly long interaction histories, resulting in high computational cost and limited scalability. In this paper, we propose STEP-HRL, a hierarchical reinforcement learning (HRL) framework that enables step-level learning by conditioning only on single-step transitions rather than full interaction histories. STEP-HRL structures tasks hierarchically, using completed subtasks to represent global progress of overall task. By introducing a local progress module, it also iteratively and selectively summarizes interaction history within each subtask to produce a compact summary of local progress. Together, these components yield augmented step-level transitions for both high-level and low-level policies. Experimental results on ScienceWorld and ALFWorld benchmarks consistently demonstrate that STEP-HRL substantially outperforms baselines in terms of performance and generalization while reducing token usage. Our code is available at https://github.com/TonyStark042/STEP-HRL.

Figures & tables

Figure 1: (a) : The pipeline of STEP-HRL. Local progress policy is responsible for producing a compact summary of local interaction history within each subtask. Specifically, the local progress policy $\pi^{p}_{\theta}$ depends on previous progress $p^{k}_{t-1}$ , current subtask $g_{k}$ , executed action $a^{k}_{t-1}$ and the resulting observation $o^{k}_{t}$ to the generate updated local progress $p^{k}_{t}$ . The low-level policy $\pi^{l}_{\theta}$ combines $p^{k}_{t}$ with observation $o^{k}_{t}$ and subtask $g_{k}$ to generate primitive actions. When current subtask $g_{k}$ terminates, its final local progress $\hat{p}_{k}$ is forwarded to the high-level policy $\pi^{h}_{\theta}$ . Conditioned on the task instruction $c_{i}$ , completed subtasks $G_{k}$ , final local progress $\hat{p}_{k}$ and the initial observation $o^{k+1}_{0}$ of next subtask, $\pi^{h}_{\theta}$ generates the subsequent subtask. (b) : The structure of our model. Three different policies share the same parameters, but equipped with different critic network respectively for offline RL training.

Backbone	Method	ScienceWorld		ALFWorld
Backbone	Method	Seen	Unseen	Seen	Unseen
GPT-3.5-Turbo	ReAct	8.57	5.97	15.41	13.99
GPT-4	ReAct	44.29	38.05	67.32	65.09
Mistral-7B	ReAct	20.72	17.65	7.86	5.22
	Reflexion	21.07	18.11	11.56	6.00
	SwitchSage	48.40	45.25	30.29	26.52

Table 1: Main Results. We report performance comparisons for three backbone models on the ScienceWorld and ALFWorld benchmarks. corresponds to prompt-based methods without updating model parameters, and corresponds to LoRA-based fine-tuning approaches.

\uparrow

indicates the relative improvement of STEP-HRL compared to the best results among the baselines.

Model	ScienceWorld		ALFWorld
Model	Seen	Unseen	Seen	Unseen
Llama-1B	51.88	49.78	89.86	89.60
Llama-3B	65.31	61.79	94.29	94.00
Llama-8B	80.57	77.81	96.43	97.76

Table 2: Performance of STEP-HRL on ScienceWorld and ALFWorld across model scales.

Figure 2: Ablation study of STEP-HRL on unseen ScienceWorld tasks with different backbone models. w/o LP denotes removing the local progress policy, w/o Hier denotes removing the hierarchical structure, and w/o RL denotes removing the offline RL stage, reducing the training procedure to behavior cloning only.

Step	Ours (s)	Normal (s)
1	1.8 $\pm$ 0.3 ⋆	1.3 $\pm$ 0.3
10	1.6 $\pm$ 0.2	1.4 $\pm$ 0.2
20	1.5 $\pm$ 0.2	1.6 $\pm$ 0.2
30	1.5 $\pm$ 0.3	1.7 $\pm$ 0.3
40	1.6 $\pm$ 0.2	1.9 $\pm$ 0.2

Table 3: Per-step inference latency on a single A100 GPU in the ALFWorld Pick&Place.

\star

denotes subtask generation, which introduces an additional forward pass.

Figure 3: Simulated per-step token usage on ALFWorld pick_two_obj_and_place task under identical observation and action sequence across three RL paradigms.

Figure 4: Offline RL sensitivity analysis with respect to advantage temperature

\beta

, expectile parameter

\tau

, data sources, and data amounts across training.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Train	Seen	Unseen
ScienceWorld	1,483	194	211
ALFWorld	3,211	140	134

Appendix

Table 4: Dataset statistics.

Hyperparameter	Value
Optimization
batch size	128
batch size per device	8
gradient accumulation steps	2
optimizer	AdamW
actor learning rate	$1\times 10^{-5}$

Appendix

Table 5: STEP-HRL hyperparameters.

Task ID	Task Name	#Variants	Success Rate (%)	Avg. Steps (Succ.)	Avg. Steps (All)
1	Pick&Place	24	100.0	12.7	12.7
2	Examine in Light	18	100.0	13.3	13.3
3	Clean&Place	31	100.0	10.6	10.6
4	Heat&Place	23	100.0	17.1	17.1
5	Cool&Place	21	100.0	15.7	15.7
6	Pick Two&Place	17	82.4	21.8	26.6

Appendix

Table 6: Evaluation details on ALFWorld unseen task.

Task ID	Task Name	#Variants	Avg Score	Avg. Steps (Succ.)	Avg. Steps (All)
0	boil	9	68.9	45.0	48.3
1	change-the-state-of-matter-of	9	62.2	34.0	46.4
2	chemistry-mix	8	67.8	20.8	25.5
3	chemistry-mix-paint-secondary-color	9	88.9	9.5	9.4
4	chemistry-mix-paint-tertiary-color	9	54.4	16.8	13.7
5	find-animal	10	100.0	11.6	11.6

Appendix

Table 7: Evaluation details on ScienceWorld unseen task.

Panel of Figure 7 — Figure 7: Case study comparing a standard ReAct-style agent and STEP-HRL on a ScienceWorld task. Normal paradigms make decisions based on the accumulated interaction trajectory, which tends to collapse in long-horizon complex tasks. In contrast, STEP-HRL makes decisions based solely on the current observation, augmented with high-level subtask and step-wise local progress, enabling step-level decision making.

RL for LLM Agents

Reinforcement learning for large language model agents trains models to plan, use tools, and adapt through feedback from interactive tasks.

See all 583 papers

HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents
Feb 18, 2026Jiangweizhi Peng, Yuanxin Liu, Ruida Zhou +4RL for LLM Agents Hierarchical RL
University of Minnesota · Northwestern University · Amazon AGI +2
StepPO: Step-Aligned Policy Optimization for Agentic Reinforcement Learning
Apr 20, 2026Daoyu Wang, Qingchuan Li, Mingyue Cheng +4Agentic RL Credit Assignment in RL
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China
Multi $^2$ : Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments
Jun 2, 2026Sangeun Park, Minhae KwonMulti-Agent LLM Systems Hierarchical RL
Department of Electrical and Computer Engineering, Sungkyunkwan University, Republic of Korea.

Feb 18, 2026J/K move · Enter open · S save

HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents

Jiangweizhi Peng, Yuanxin Liu, Ruida Zhou, Charles Fleming+3

University of Minnesota · Northwestern University · Amazon AGI+2

RL for LLM Agents Hierarchical RL Step-Level Credit Assignment in RL Sparse-Reward RL

Open paper AskPDF arXiv

Training LLMs as interactive agents for multi-turn decision-making remains challenging, particularly in long-horizon tasks with sparse and delayed rewards, where agents must execute extended sequences of actions before receiving meaningful feedback. Most existing reinforcement learning (RL) approaches model LLM agents as flat policies operating at a single time scale, selecting one action at each turn. In sparse-reward settings, such flat policies must propagate credit across the entire trajectory without explicit temporal abstraction, which often leads to unstable optimization and inefficient credit assignment. We propose HiPER, a novel Hierarchical Plan-Execute RL framework that explicitly separates high-level planning from low-level execution. HiPER factorizes the policy into a high-level planner that proposes subgoals and a low-level executor that carries them out over multiple action steps. To align optimization with this structure, we introduce a key technique called hierarchical advantage estimation (HAE), which carefully assigns credit at both the planning and execution levels. By aggregating returns over the execution of each subgoal and coordinating updates across the two levels, HAE provides an unbiased gradient estimator and provably reduces variance compared to flat generalized advantage estimation. Empirically, HiPER achieves state-of-the-art performance on challenging interactive benchmarks, reaching 97.4% success on ALFWorld and 83.3% on WebShop with Qwen2.5-7B-Instruct (+6.6% and +8.3% over the best prior method), with especially large gains on long-horizon tasks requiring multiple dependent subtasks. These results highlight the importance of explicit hierarchical decomposition for scalable RL training of multi-turn LLM agents.

Abstract

RL for LLM Agents

Explore similar work

HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents

StepPO: Step-Aligned Policy Optimization for Agentic Reinforcement Learning

Multi2^22: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments

Multi $^2$ : Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments