Efficient Hierarchical Implicit Flow Q-learning for Offline Goal-conditioned Reinforcement Learning

Zhiqiang Dong; Teng Pang; Rongjian Xu; Guoqiang Wu

cs.LGApr 10, 2026

Efficient Hierarchical Implicit Flow Q-learning for Offline Goal-conditioned Reinforcement Learning

Authors: Zhiqiang Dong, Teng Pang, Rongjian Xu, Guoqiang Wu

Organizations: School of Software, Shandong University, Jinan, Shandong, China · Luke EI, Jinan, China

Abstract

Offline goal-conditioned reinforcement learning (GCRL) is a practical reinforcement learning paradigm that aims to learn goal-conditioned policies from reward-free offline data. Despite recent advances in hierarchical architectures such as HIQL, long-horizon control in offline GCRL remains challenging due to the limited expressiveness of Gaussian policies and the inability of high-level policies to generate effective subgoals. To address these limitations, we propose the goal-conditioned mean flow policy, which introduces an average velocity field into hierarchical policy modeling for offline GCRL. Specifically, the mean flow policy captures complex target distributions for both high-level and low-level policies through a learned average velocity field, enabling efficient action generation via one-step sampling. Furthermore, considering the insufficiency of goal representation, we introduce a LeJEPA loss that repels goal representation embeddings during training, thereby encouraging more discriminative representations and improving generalization. Experimental results show that our method achieves strong performance across both state-based and pixel-based tasks in the OGBench benchmark.

Figures & tables

Figure 1: Overview of HIFQL. During training (left), a goal-conditioned value function and a shared goal representation encoder are jointly optimized, while high-level and low-level mean flow policies are trained via advantage-weighted regression to generate subgoals and actions, respectively. During inference (right), both policies perform one-step generation from Gaussian noise, enabling efficient hierarchical decision-making without iterative sampling.

Environment	Dataset	GCBC	GCIVL	GCIQL	QRL	CRL	HIQL	HIFQL (ours)
pointmaze	pointmaze-medium-navigate-v0	$9.8\pm 6.7$	$73.8\pm 5.6$	$53.5\pm 3.9$	$86.2\pm 4.2$	$34.7\pm 5.6$	$74.6\pm 3.3$	99.5 $\pm$ 0.9
	pointmaze-large-navigate-v0	$27.3\pm 7.0$	$49.8\pm 8.0$	$31.6\pm 3.2$	$85.8\pm 10.1$	$35.5\pm 13.3$	$55.9\pm 8.1$	$\textbf{80.5}\pm\textbf{2.9}$
	pointmaze-giant-navigate-v0	$4.5\pm 7.0$	$0\pm 0$	$0\pm 0$	$64.7\pm 13.9$	$21.1\pm 10.3$	$54.7\pm 10.0$	$\textbf{74.4}\pm\textbf{5.9}$
	pointmaze-teleport-navigate-v0	$26.9\pm 2.9$	$45.6\pm 2.9$	$23.8\pm 4.1$	$4.2\pm 5.2$	$26.4\pm 6.7$	$12.0\pm 5.0$	$\textbf{39.8}\pm\textbf{6.9}$
	pointmaze-medium-stitch-v0	$27.6\pm 14.5$	$80.6\pm 1.6$	$19.9\pm 11.1$	$83.3\pm 6.7$	$0\pm 1$	$79.0\pm 11.3$	$\textbf{97.0}\pm\textbf{3.2}$
	pointmaze-large-stitch-v0	$4.0\pm 8.0$	$14.0\pm 7.3$	$27.8\pm 2.7$	$76.7\pm 8.9$	$0\pm 0$	$14.3\pm 3.0$	$\textbf{25.4}\pm\textbf{9.9}$

Table 1: Results on state-based tasks. We evaluate all methods on three Maze environment families—PointMaze, AntMaze, and HumanoidMaze—covering navigation, stitching, and teleport variants with increasing state dimensionality and horizon length. A total of 13 sparse-reward tasks are included. We report the average success rate and standard deviation over 5 random seeds for each algorithm. The best result for each task is shown in bold .

Environment	Dataset	GCBC	GCIVL	GCIQL	QRL	CRL	HIQL	HIFQL (ours)
visual-antmaze	visual-antmaze-medium-navigate-v0	$6.5\pm 1.4$	$14.2\pm 3.8$	$5.8\pm 1.8$	$0\pm 0$	$\textbf{94.3}\pm\textbf{1.3}$	$87.9\pm 2.6$	${93.3}\pm{2.8}$
	visual-antmaze-large-navigate-v0	$1.4\pm 1.1$	$3.1\pm 1.4$	$1.8\pm 0.6$	$0\pm 0$	$\textbf{82.6}\pm\textbf{1.6}$	$52.9\pm 5.6$	${74.8}\pm{1.9}$
	visual-antmaze-giant-navigate-v0	$0\pm 0$	$0.5\pm 0.3$	$0.1\pm 0.2$	$0\pm 0$	$\textbf{32.7}\pm\textbf{4.2}$	$3.5\pm 3.0$	${30.2}\pm{4.1}$
	visual-antmaze-teleport-navigate-v0	$8.6\pm 2.3$	$11.9\pm 3.1$	$11.3\pm 1.0$	$9.2\pm 4.0$	$46.7\pm 3.5$	$\textbf{47.0}\pm\textbf{3.8}$	$43.0\pm 2.6$

Table 2: Results on pixel-based tasks. We evaluate all methods on four sparse-reward visual AntMaze tasks, where observations are high-dimensional pixel inputs and long-horizon planning is required to reach the specified goal. All results report the average success rate and standard deviation over 5 random seeds.

Figure 2: Ablation study on the representation regularization coefficient

\lambda

in visual AntMaze tasks.

Offline RL

Offline reinforcement learning learns decision-making policies from fixed datasets without interacting with the environment during training.

See all 298 papers

Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning
May 25, 2026Hyungkyu Kang, Byeongchan Kim, Min-hwan OhReinforcement Learning Offline RL
Seoul National University, Seoul, South Korea.
Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning
May 27, 2026Kaiqiang Ke, Shenghong He, Chengdong Xu +3Hierarchical RL Offline RL
1Sun Yat-sen University · 2Pengcheng Laboratory
Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL
Feb 3, 2026Jinwoo Choi, Sang-Hyun Lee, Seung-Woo SeoReinforcement Learning Hierarchical RL
Department of Electrical and Computer Engineering, Seoul National University, Seoul, South Korea · Department of Automotive Engineering, Ajou University, Gyeonggi-do, South Korea

May 25, 2026J/K move · Enter open · S save

Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning

Hyungkyu Kang, Byeongchan Kim, Min-hwan Oh

Seoul National University, Seoul, South Korea.

Reinforcement Learning Offline RL Representation Learning for RL Goal-Conditioned RL

Open paper AskPDF arXiv

Offline goal-conditioned reinforcement learning (GCRL) provides a practical framework for obtaining goal-reaching policies from fixed datasets. However, learning a reliable goal-conditioned value function in long-horizon tasks remains challenging. In this paper, we identify erroneous generalization in goal-conditioned value functions as a fundamental bottleneck, and demonstrate that appropriate inductive bias in the value function is crucial for addressing the bottleneck. Building on these findings, we propose Latent-Aligned Value Learning (LAVL), an offline GCRL algorithm that integrates latent-representation-based value generalization with hierarchical planning in a unified framework. Extensive experiments on OGBench demonstrate that LAVL consistently outperforms existing offline GCRL methods, achieving the highest performance on 20 out of 22 datasets. Notably, LAVL exhibits strong performance in long-horizon tasks and trajectory stitching datasets, where prior methods suffer significant performance degradation. Our code is available at https://github.com/oh-lab/LAVL.git.

Abstract

Offline RL

Explore similar work

Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning

Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning

Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL