Escaping Path Mirages in Offline Goal-Conditioned Reinforcement Learning

Junhyeon Bae; Jaebak Hwang; Gwanwoo Choi; Minung Kim; Seungyul Han

cs.LGOct 29, 2026

Escaping Path Mirages in Offline Goal-Conditioned Reinforcement Learning

Authors: Junhyeon Bae, Jaebak Hwang, Gwanwoo Choi, Minung Kim, Seungyul Han

Organizations: Graduate School of Artificial Intelligence, UNIST, Ulsan, South Korea

Abstract

Offline goal-conditioned reinforcement learning remains challenging in stochastic long-horizon settings, where compounding value estimation errors hinder reliable goal reaching. While prior methods have sought to address this challenge through various approaches, a key challenge remains path mirage, where agents overcommit to spuriously successful trajectories in offline datasets induced by stochasticity, often leading to failure on difficult tasks. To address this issue, we propose Branch-aware Graph Planning (BGP), which captures stochastic branching structures by identifying high-variability points as graph nodes and designing goal-conditioned edges with segmented learning to avoid unreliable branches and reduce unnecessary stochasticity along paths. As a result, BGP enables reliable goal reaching even in highly stochastic long-horizon environments, and experiments on diverse OGBench tasks show that it substantially outperforms prior state-of-the-art offline goal-conditioned RL methods. Our implementation is available at https://umlic.github.io/papers/BGP/.

Figures & tables

Figure 1: Path mirage in offline GCRL. (a) Offline trajectories can pass through branching regions where future outcomes diverge toward success, failure, or risky areas. (b) Prior methods can be misled by deceptively successful paths, resulting in path mirages with occasional successes but persistent failures. (c) Our branch-aware graph planning identifies branching points near branching regions as landmarks and selects reliable goal-conditioned routes that avoid risky branches.

Figure 2: Visualization of uncertainty scores and the resulting landmarks on two OGBench tasks. High-uncertainty regions correspond to significant trajectory branching, leading to denser landmark selection.

Figure 3: (a) Illustration of downstream branches

\mathcal{R}(v_{j})

. (b)-(c) Edge comparison on a teleport task in OGBench, where

v_{j}

(near the teleport) induces stochastic branching. (b) A prior edge

e(v_{i},v_{j})

favors the transition due to local proximity, whereas (c) our goal-conditioned edge

e_{g}

evaluates all branches

\{v_{k_{1}},v_{k_{2}},v_{k_{3}}\}\in\mathcal{R}(v_{j})

and avoids selecting

v_{j}

since one branch (

v_{k_{2}}

) leads to a dead zone, i.e.,

d(s_{k_{2}},g)=\infty

for

s_{k_{2}}\in\mathcal{C}_{k_{2}}

Figure 5: Visualization of OGBench tasks. (a)–(c) show three maze layouts: large , giant , and teleport . In the teleport layout, two blackholes teleport the agent to one of three whiteholes uniformly at random. (d) and (e) show the pointmaze and antmaze tasks, respectively.

Env	Dataset	HIQL	GCIQL	QRL	CRL	TMD	CGCIVL	GAS	BGP(Ours)
antmaze	large-navigate	91 ( $\pm$ 2)	34 ( $\pm$ 4)	75 ( $\pm$ 6)	83 ( $\pm$ 4)	82 ( $\pm$ 2)	91 ( $\pm$ 2)	93 ( $\pm$ 1)	90 ( $\pm$ 1)
	large-stitch	67 ( $\pm$ 5)	7 ( $\pm$ 2)	18 ( $\pm$ 2)	11 ( $\pm$ 2)	37 ( $\pm$ 3)	79 ( $\pm$ 3)	96 ( $\pm$ 1)	93 ( $\pm$ 1)
	large-explore	4 ( $\pm$ 5)	0 ( $\pm$ 0)	0 ( $\pm$ 0)	0 ( $\pm$ 0)	0 ( $\pm$ 0)	6 ( $\pm$ 5)	94 ( $\pm$ 3)	97 ( $\pm$ 3)
	giant-navigate	65 ( $\pm$ 5)	0 ( $\pm$ 0)	14 ( $\pm$ 3)	16 ( $\pm$ 3)	1 ( $\pm$ 1)	73 ( $\pm$ 5)	78 ( $\pm$ 3)	88 ( $\pm$ 2)
	giant-stitch	2 ( $\pm$ 2)	0 ( $\pm$ 0)	0 ( $\pm$ 0)	0 ( $\pm$ 0)	3 ( $\pm$ 1)	36 ( $\pm$ 7)	88 ( $\pm$ 4)	88 ( $\pm$ 2)
	teleport-navigate	42 ( $\pm$ 3)	35 ( $\pm$ 5)	35 ( $\pm$ 5)	53 ( $\pm$ 2)	27 ( $\pm$ 12)	29 ( $\pm$ 13)	39 ( $\pm$ 3)	90 ( $\pm$ 1)

Table 1: Performance comparison on OGBench. We report the average success rate (%) with standard deviation (

\pm

) over 5 seeds. The best results in each row are highlighted in bold .

Figure 6: Shortest waypoint paths in the teleport layout. (a–b) use the prior edge cost

e_{p}(v,w)=-V_{\theta}(v,w)

and illustrate failure cases, while (c–d) use the proposed goal-conditioned edge cost

e_{g}

. Goals are selected from the evaluation set. (c–d) further illustrate how

e_{g}

accounts for downstream branches and avoids unreliable teleport transitions, yielding more stable paths.

Figure 7: Environments and success rates with respect to the number of teleports $B$ . Panels (a) and (b) show the environments with two and four teleports, respectively. Panel (c) shows the success rates of HIQL, BGP (w/o segmented learning), and BGP on the navigate datasets, and panel (d) shows those on the stitch datasets. Shaded regions denote 95% confidence intervals over 5 seeds.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Autoregressive prediction of the Transformer during evaluation, where predicted transition variables

\hat{y}_{t+l}

are fed back as inputs to estimate multi-step uncertainty from the current state

s_{t}

Environment	Representation for clustering	Dimension
antmaze	PCA projection	2
pointmaze	PCA projection	2
visual-antmaze	Hilbert representation	16
humanoidmaze	Hilbert representation	16
manipulation tasks	Hilbert representation	16

Appendix

Table 3: Representations used for landmark clustering in each environment. Dimension denotes the dimensionality of the representation space

\mathcal{Z}

Figure 10: Illustration of segmented learning. Each offline trajectory is partitioned into sub-trajectories $\tau_{j}^{(k)}$ at cluster transition boundaries, shown in different colors, and $M$ consecutive sub-trajectories are concatenated into a local training window $\tilde{\tau}_{j}^{(k)}$ .

Figure 11: Visualizations of the OGBench environments used in our experiments. Panels (a)-(e) show the agents and the manipulation environment, panels (f)-(i) show the maze layouts, and panels (j)-(l) show example trajectories of the three dataset types.

Transformer (Uncertainty Estimation)
Hyperparameter	Value
Number of layers	4
Number of attention heads	2
Embedding dimension	256
Context length ( $n$ )	5
Transformer learning rate	$3\times 10^{-4}$

Appendix

Table 4: Common hyperparameters.

Domain	Dataset	Percentile ( $\alpha$ )	Num Clusters ( $N$ )
pointmaze	teleport-navigate	1	100
	teleport-stitch	1	100
	giant-navigate	10	100
	giant-stitch	10	100
	large-navigate	1	50
	large-stitch	1	50

Appendix

Table 5: Environment-specific hyperparameters.

Figure 12: antmaze-teleport environments with varying numbers of teleports. Panels (a)-(e) show the environments with $B{=}0$ , $1$ , $2$ , $3$ , and $4$ active blackholes, respectively, and panel (c) is the original OGBench environment.

Env	Dataset	HIQL	GCIQL	QRL	CRL	BGP(Ours)	BGP $(\alpha,N)$
humanoidmaze	medium-navigate	89 ( $\pm$ 2)	27 ( $\pm$ 2)	21 ( $\pm$ 8)	60 ( $\pm$ 4)	95 ( $\pm$ 2)	$(10,50)$
	medium-stitch	88 ( $\pm$ 2)	12 ( $\pm$ 3)	18 ( $\pm$ 2)	36 ( $\pm$ 2)	87 ( $\pm$ 6)	$(10,50)$
	large-navigate	49 ( $\pm$ 4)	2 ( $\pm$ 1)	5 ( $\pm$ 1)	24 ( $\pm$ 4)	70 ( $\pm$ 7)	$(20,50)$
	large-stitch	28 ( $\pm$ 3)	0 ( $\pm$ 0)	3 ( $\pm$ 1)	4 ( $\pm$ 1)	58 ( $\pm$ 8)	$(50,50)$
	giant-navigate	12 ( $\pm$ 4)	0 ( $\pm$ 0)	1 ( $\pm$ 0)	3 ( $\pm$ 2)	75 ( $\pm$ 6)	$(10,150)$
	giant-stitch	3 ( $\pm$ 2)	0 ( $\pm$ 0)	0 ( $\pm$ 0)	0 ( $\pm$ 0)	69 ( $\pm$ 8)	$(10,150)$

Appendix

Table 6: Success rates (%) on humanoidmaze and scene-play-v0 . The highest mean in each row is highlighted in bold . The final column reports the BGP hyperparameter settings

(\alpha,N)

, with – indicating an unreported setting.

Figure 13: Representative trajectory analysis on the teleport environment.

Figure 14: Representative trajectory analysis on the giant environment.

Figure 15: Representative trajectory analysis on the large environment.

Figure 16: Hyperparameter sensitivity on the teleport environment.

Figure 17: Hyperparameter sensitivity on the giant environment.

Backbone	$M=2$	$M=3$	$M=5$	$M=10$ (excl. stitch)
HIQL	$90.0\pm 2.0$	$\mathbf{90.2\pm 1.1}$	$89.0\pm 3.2$	$87.1\pm 4.0$
GCIQL	$73.5\pm 15.6$	$79.6\pm 11.4$	$72.9\pm 16.4$	$48.5\pm 21.7$

Appendix

Table 7: Sensitivity to the segment length (

M

). Results are averaged over five random seeds. The

M=10

case excludes stitch datasets, as their inherent trajectory lengths are too short to support this segment length.

Evaluation episodes	1	5	13	35	100	200
Success rate (%)	65	71	76	80	83	84
95% CI	[12, 88]	[30, 88]	[50, 88]	[62, 88]	[63, 88]	[67, 88]

Appendix

Table 8: Budget-aware evaluation on antmaze-teleport-navigate . We report the expected success rate (%) and 95% confidence interval (CI) as a function of the number of evaluation episodes, following Jackson et al. [55] .

Method	BGP	HIQL	GCIQL	CGCIVL	TMD	GAS
Time (s)	7563	4686	3632	4927	17577	8841

Appendix

Table 9: Simulation time across different methods.

Offline RL

Offline reinforcement learning learns decision-making policies from fixed datasets without interacting with the environment during training.

See all 304 papers

Aug 29, 2026

PathBridger: Subgoal Bridges for Offline Goal-Conditioned Reinforcement Learning

Offline goal-conditioned reinforcement learning (GCRL) aims to learn policies for reaching diverse goals entirely from fixed trajectory data. Long-horizon offline GCRL remains challenging because sparse goal-reaching signals must be propagated over many steps, while execution errors cannot be corrected through additional environment interaction. Existing methods address these challenges by improving long-range value estimation or reducing the effective decision horizon through subgoals, options, and action chunks. In several hierarchical methods, however, a selected subgoal specifies where to go, while the intervening state-space path remains implicit in an endpoint-conditioned low-level policy. To address this interface, we propose PathBridger, a hierarchical offline GCRL method that explicitly connects subgoal selection to short-horizon execution. PathBridger constructs a state-space bridge toward the selected intermediate endpoint and decodes it into a short executable action chunk using an inverse dynamics model. Experiments across the evaluated OGBench tasks demonstrate strong aggregate performance, with particularly large gains on the multi-object Cube manipulation tasks. Code: https://github.com/SChoish/PathBridger

Soohyun Choi, Seonvin Cho, Songnam Hong

Information and Intelligence Systems Laboratory (IISL) Department of Electronic Engineering, Hanyang University Seoul, Republic of Korea

Offline RL Goal-Conditioned RL Hierarchical RL

Ask PDF

Feb 3, 2026

Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL

Offline goal-conditioned reinforcement learning remains challenging for long-horizon tasks. While hierarchical approaches mitigate this issue by decomposing tasks, most existing methods rely on separate high- and low-level networks and generate only a single intermediate subgoal, leaving several structural limitations in long-horizon decision-making. To address this limitation, we draw inspiration from chain-of-thought reasoning and propose the Chain-of-Goals Hierarchical Policy (CoGHP), a novel framework that reformulates hierarchical decision-making as autoregressive sequence modeling within a unified architecture. Given a state and a final goal, CoGHP autoregressively generates a sequence of latent subgoals followed by the primitive action, where each latent subgoal acts as a reasoning step that conditions subsequent predictions. To implement this efficiently, we introduce an MLP-Mixer backbone, which supports cross-token communication and captures structural relationships among state, goal, latent subgoals, and action. Across challenging navigation and manipulation benchmarks, CoGHP consistently outperforms strong offline baselines, demonstrating improved performance on long-horizon tasks. Project page: https://wlsdn9350.github.io/projects/coghp/

Jinwoo Choi, Sang-Hyun Lee, Seung-Woo Seo

Department of Electrical and Computer Engineering, Seoul National University, Seoul, South Korea · Department of Automotive Engineering, Ajou University, Gyeonggi-do, South Korea

Reinforcement Learning Hierarchical RL Offline RL Long-Horizon Agent Tasks Goal-Conditioned RL Long-Horizon RL

Ask PDF

May 25, 2026

Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning

Offline goal-conditioned reinforcement learning (GCRL) provides a practical framework for obtaining goal-reaching policies from fixed datasets. However, learning a reliable goal-conditioned value function in long-horizon tasks remains challenging. In this paper, we identify erroneous generalization in goal-conditioned value functions as a fundamental bottleneck, and demonstrate that appropriate inductive bias in the value function is crucial for addressing the bottleneck. Building on these findings, we propose Latent-Aligned Value Learning (LAVL), an offline GCRL algorithm that integrates latent-representation-based value generalization with hierarchical planning in a unified framework. Extensive experiments on OGBench demonstrate that LAVL consistently outperforms existing offline GCRL methods, achieving the highest performance on 20 out of 22 datasets. Notably, LAVL exhibits strong performance in long-horizon tasks and trajectory stitching datasets, where prior methods suffer significant performance degradation. Our code is available at https://github.com/oh-lab/LAVL.git.

Hyungkyu Kang, Byeongchan Kim, Min-hwan Oh

Seoul National University, Seoul, South Korea.

Reinforcement Learning Offline RL Representation Learning for RL Goal-Conditioned RL

Ask PDF

Abstract

Offline RL

Explore similar work

PathBridger: Subgoal Bridges for Offline Goal-Conditioned Reinforcement Learning

Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL

Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning