A note on goal-based hierarchical RL

Kevin Murphy

cs.AISep 13, 2026

A note on goal-based hierarchical RL

Authors: Kevin Murphy

Abstract

The agent-centric general value function (ACGVF) construction of \citet{tasse2026goal} lets the agent make two decisions that are normally imposed by the environment or agent designer: which goal to pursue and when to declare a goal as finished (in addition to choosing the action). This is a very general framework that subsumes almost all prior work on reinforcement learning, control and planning, as well as more general formalisms proposed in the cognitive sciences. However, it assumes the environment is fully observed, i.e., that the observation is a sufficient statistic. In \citet{murphy2025rl}, a general agent design was proposed where the policy is based on an internal belief state $z_t$ and an internal goal; however, the goals were assumed to be externally provided. In this note, we unify and extend these two approaches using the formalism of hierarchical hidden Markov models (HHMM) \citep{murphy2001hhmm}.

Figures & tables

Panel of Figure 1 — Figure 1: A two-level HHMM drawn as a DBN, adapted from Murphy and Paskin (2001) . (Note: we use superscript $1$ to denote the lower (concrete) level and superscript $2$ the upper (abstract) level, following convention of Theocharous et al. (2004) , which is the reverse of the convention in Murphy and Paskin (2001) .) At time $t$ the system is in state $(Q^{2}_{t},Q^{1}_{t})$ , and the binary variable $F^{1}_{t}$ turns on iff the current lower-level sub-model has just finished; $F^{1}_{t}$ does not affect $Y_{t}$ , so a sub-process boundary is never directly observed and must be inferred. Note the ordering implied by the arcs into slice $t+1$ : the new abstract state $Q^{2}_{t+1}$ is drawn first (from $Q^{2}_{t}$ and $F^{1}_{t}$ ), and only then is the new concrete state $Q^{1}_{t+1}$ drawn, from $Q^{1}_{t}$ , $F^{1}_{t}$ and the freshly chosen parent $Q^{2}_{t+1}$ . When $F^{1}_{t}=0$ both levels persist horizontally; when $F^{1}_{t}=1$ the parent may move and the child is re-entered.

Panel of Figure 2 — Figure 2: A two-slice factor graph representation of a two level conditional HHMM representing a hierarchical agent policy $\pi_{\theta}$ with goal selection function $\mu_{\eta}$ and internal state update function $U_{\phi}$ . Shading marks the observed interface variables $A_{t}$ and $O_{t+1}$ ; all nodes inside the Agent box are hidden internal variables or mechanisms. The Environment box maps the action to the next observation (its internal state and reward are omitted). When $D_{t}=0$ (done node is false), the goal $G_{t}$ is copied forward; when $D_{t}=1$ , the agent selects a new goal using $\mu_{\eta}(G_{t+1}\mid Z_{t+1})$ . The reward generated from the environment is assumed to be part of $O_{t+1}$ .

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,727 papers

Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning
May 27, 2026Kaiqiang Ke, Shenghong He, Chengdong Xu +3Hierarchical RL Offline RL
1Sun Yat-sen University · 2Pengcheng Laboratory
Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL
Feb 3, 2026Jinwoo Choi, Sang-Hyun Lee, Seung-Woo SeoReinforcement Learning Hierarchical RL
Department of Electrical and Computer Engineering, Seoul National University, Seoul, South Korea · Department of Automotive Engineering, Ajou University, Gyeonggi-do, South Korea
SVL: Goal-Conditioned Reinforcement Learning as Survival Learning
Apr 19, 2026Franki Nguimatsia Tiofack, Fabian Schramm, Théotime Le Hellard +1Reinforcement Learning Goal-Conditioned RL
Inria and École Normale Supérieure, PSL Research University, Paris, France.

May 27, 2026J/K move · Enter open · S save

Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning

Kaiqiang Ke, Shenghong He, Chengdong Xu, Yuheng Luo+2

1Sun Yat-sen University · 2Pengcheng Laboratory

Hierarchical RL Offline RL Goal-Conditioned RL

Open paper AskPDF arXiv

Offline goal-conditioned reinforcement learning (GCRL) is challenging in long-horizon tasks, where distant state--goal pairs provide weak supervision and value estimates become vulnerable to accumulated bootstrapping errors. Hierarchical methods mitigate this difficulty by introducing intermediate subgoals, but fixed temporal abstractions or fixed hierarchy depths can be mismatched to state--goal pairs with different reachability horizons. We propose Coarse-to-Fine Hierarchical Goal Reinforcement Learning (CFHRL), a fully offline GCRL framework that adaptively refines distant goals before execution. Starting from the final goal, CFHRL recursively proposes intermediate targets, trained from replay-supported candidates, and stops refinement once the current target is estimated to be locally executable by a learned reachability cost. The key idea is that a subgoal need not be an exact midpoint or globally optimal waypoint; it only needs to provide reliable progress and reduce the remaining reaching difficulty, enabling subsequent refinement over shorter horizons. A stylized analysis further supports the robustness of approximate recursive contraction. Experiments on OGBench show substantial gains on several long-horizon tasks, with ablations validating the proposed refinement and stopping mechanisms

Abstract

Reinforcement Learning

Explore similar work

Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning

Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL

SVL: Goal-Conditioned Reinforcement Learning as Survival Learning