Hierarchical RL

RL: Reinforcement Learning

Hierarchical reinforcement learning helps agents handle long tasks by grouping primitive actions into skills or planning through intermediate goals. Common approaches include options and manager-worker policies, learned subgoal selection, action chunking, offline skill reuse, and model-based planning. These methods are applied to sparse-reward control, robotics, multi-agent coordination, and language-model agents, where they can improve exploration, credit assignment, and transfer. A central challenge is learning abstractions that are useful and stable without hand-designed boundaries, while keeping high-level plans aligned with what low-level controllers can reliably execute.

Latest papers

160 papers25 papers in the last four weeks, up 32% on the four weeks before. 0.1% of all new papers.

HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents
Feb 18, 2026Jiangweizhi Peng, Yuanxin Liu, Ruida Zhou +4RL for LLM Agents Hierarchical RL
University of Minnesota · Northwestern University · Amazon AGI +2
TWISTED-RL: Hierarchical Skilled Agents for Knot-Tying without Human Demonstrations
Feb 16, 2026Guy Freund, Tom Jurgenson, Matan Sudry +1Robotic Manipulation Robot Skill Learning
Reichman University · Technion – Israel Institute of Technology · The University of Texas at Austin
Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL
Feb 3, 2026Jinwoo Choi, Sang-Hyun Lee, Seung-Woo SeoReinforcement Learning Hierarchical RL
Department of Electrical and Computer Engineering, Seoul National University, Seoul, South Korea · Department of Automotive Engineering, Ajou University, Gyeonggi-do, South Korea
Where to Touch, How to Contact: A Hierarchical RL-MPC Framework for Geometry-Aware Sim-to-Real Manipulation
Jan 16, 2026Zhixian Xie, Yu Xiang, Michael Posa +1Contact-Rich Manipulation Non-Prehensile Manipulation
Arizona State University · University of Texas at Dallas · University of Pennsylvania
Optimization without Future Compromises? Decentralized Coordination via Collective and Reinforcement Learning
Sep 22, 2025Chuhao Qin, Evangelos PournarasMulti-Agent System Optimization Multi-Agent Coordination
School of Computer Science, University of Leeds, UK
Hierarchical Message-Passing Policies for Multi-Agent Reinforcement Learning
Jul 31, 2025Tommaso Marzi, Cesare Alippi, Andrea CiniMulti-Agent Coordination Hierarchical RL
IDSIA USI-SUPSI, Università della Svizzera italiana, Lugano, Switzerland · Politecnico di Milano, Milan, Italy · EPFL, Lausanne, Switzerland
Discovering Temporal Structure: An Overview of Hierarchical Reinforcement Learning
Jun 16, 2025Martin Klissarov, Akhil Bagaria, Ziyan Luo +3Hierarchical RL
Mila, McGill University · Amazon · Amii, University of Alberta Canada CIFAR AI Chair
Action abstractions for amortized sampling
Oct 19, 2024Oussama Boussif, Léna Néhale Ezzine, Joseph D Viviano +6Hierarchical RL Generative Flow Networks
Mila – Qu´ebec AI Institute · Universit´e de Montr´eal · University of Toronto +3
Hierarchical Deep Counterfactual Regret Minimization
May 27, 2023Jiayu Chen, Xudong Wu, Zhekai Wang +1Imperfect-Information Games Hierarchical RL
The University of Hong Kong Hong Kong SAR · Peking University Beijing, China, 100871 · Purdue University2025 West Lafayette, IN 47907
Diversity-Enriched Option-Critic
Nov 4, 2020Anand Kamat, Doina PrecupHierarchical RL Intrinsic Motivation
McGill University · Mila · 3Google deepmind

Feb 18, 2026J/K move · Enter open · S save

HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents

Jiangweizhi Peng, Yuanxin Liu, Ruida Zhou, Charles Fleming+3

University of Minnesota · Northwestern University · Amazon AGI+2

RL for LLM Agents Hierarchical RL Step-Level Credit Assignment in RL Sparse-Reward RL

Open paper AskPDF arXiv

Training LLMs as interactive agents for multi-turn decision-making remains challenging, particularly in long-horizon tasks with sparse and delayed rewards, where agents must execute extended sequences of actions before receiving meaningful feedback. Most existing reinforcement learning (RL) approaches model LLM agents as flat policies operating at a single time scale, selecting one action at each turn. In sparse-reward settings, such flat policies must propagate credit across the entire trajectory without explicit temporal abstraction, which often leads to unstable optimization and inefficient credit assignment. We propose HiPER, a novel Hierarchical Plan-Execute RL framework that explicitly separates high-level planning from low-level execution. HiPER factorizes the policy into a high-level planner that proposes subgoals and a low-level executor that carries them out over multiple action steps. To align optimization with this structure, we introduce a key technique called hierarchical advantage estimation (HAE), which carefully assigns credit at both the planning and execution levels. By aggregating returns over the execution of each subgoal and coordinating updates across the two levels, HAE provides an unbiased gradient estimator and provably reduces variance compared to flat generalized advantage estimation. Empirically, HiPER achieves state-of-the-art performance on challenging interactive benchmarks, reaching 97.4% success on ALFWorld and 83.3% on WebShop with Qwen2.5-7B-Instruct (+6.6% and +8.3% over the best prior method), with especially large gains on long-horizon tasks requiring multiple dependent subtasks. These results highlight the importance of explicit hierarchical decomposition for scalable RL training of multi-turn LLM agents.