PathBridger: Subgoal Bridges for Offline Goal-Conditioned Reinforcement Learning

Soohyun Choi; Seonvin Cho; Songnam Hong

cs.LGAug 29, 2026

PathBridger: Subgoal Bridges for Offline Goal-Conditioned Reinforcement Learning

Authors: Soohyun Choi, Seonvin Cho, Songnam Hong

Organizations: Information and Intelligence Systems Laboratory (IISL) Department of Electronic Engineering, Hanyang University Seoul, Republic of Korea

Abstract

Offline goal-conditioned reinforcement learning (GCRL) aims to learn policies for reaching diverse goals entirely from fixed trajectory data. Long-horizon offline GCRL remains challenging because sparse goal-reaching signals must be propagated over many steps, while execution errors cannot be corrected through additional environment interaction. Existing methods address these challenges by improving long-range value estimation or reducing the effective decision horizon through subgoals, options, and action chunks. In several hierarchical methods, however, a selected subgoal specifies where to go, while the intervening state-space path remains implicit in an endpoint-conditioned low-level policy. To address this interface, we propose PathBridger, a hierarchical offline GCRL method that explicitly connects subgoal selection to short-horizon execution. PathBridger constructs a state-space bridge toward the selected intermediate endpoint and decodes it into a short executable action chunk using an inverse dynamics model. Experiments across the evaluated OGBench tasks demonstrate strong aggregate performance, with particularly large gains on the multi-object Cube manipulation tasks. Code: https://github.com/SChoish/PathBridger

Figures & tables

Figure 1: Overview of PathBridger. Candidate $K$ -step endpoints are ranked using a transitive value score, and the selected endpoint $z^{\star}$ is expanded into an endpoint-pinned state-space bridge. Inverse dynamics decodes an $h_{a}$ -transition bridge prefix into an executable action chunk. PathBridger executes the chunk for up to $h_{a}$ steps and then replans from the resulting state.

Method	GCIVL	CRL	HIQL	OTA	TRL	DQC	PBG	PBF
Time (h)	0.94	0.94	1.40	2.00	1.34	1.88	1.20	1.23

Table 2: Estimated training time for 1M gradient steps on a single NVIDIA H200 GPU.

Environment	Waypoint role	PBG	HIQL
puzzle-3x3	Task goal	40.0	12.8
puzzle-3x3	Selected subgoal	98.7	37.2
puzzle-4x4	Task goal	52.0	7.2
puzzle-4x4	Selected subgoal	98.7	2.4

Table 3: Mean success rates (%) with oracle Puzzle waypoints. A waypoint is supplied either as the current task goal, retaining learned subgoal proposal and selection, or directly as the selected subgoal, evaluating execution alone. All results are averaged over two seeds.

Figure 2: Dataset-size ablation averaged over all eight OGBench tasks. Each point reports the aggregate success rate with 100k, 300k, 500k, or 1M offline transitions while fixing training to 1M gradient steps.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Learning rate	$3\times 10^{-4}$
Optimizer	Adam
Batch size	1024
MLP hidden dimensions	$[512,512,512]$
Activation	GELU
Target EMA rate	0.005

Appendix

Table 4: Shared PathBridger hyperparameters. Goal-sampling probabilities are ordered as

(p_{\rm cur},p_{\rm geom},p_{\rm traj},p_{\rm rand})

Environment	$K$	$\gamma$	$c_{\rm sg}$	$\lambda$	$N$	$T$	$K$	$\gamma$	$c_{\rm sg}$	$\lambda$	$N$	$T$
	PBF						PBG
amm	25	0.99	10	0	2	0.25	25	0.99	10	0	1	0
aml	25	0.995	10	0	16	0.5	25	0.995	10	0	1	0
cs	40	0.99	5	0.7	1	0	25	0.99	10	0.7	1	0
cd	40	0.99	10	1	2	0.25	25	0.99	10	1	1	0
ct	40	0.995	10	1	1	0	25	0.995	10	1	1	0

Appendix

Table 5: Environment-specific PathBridger hyperparameters.

Hyperparameter	Value
Learning rate	$3\times 10^{-4}$
Optimizer	Adam
Batch size	1024 (default) 256 for DQC and OTA $\dagger$
MLP hidden dimensions	$[512,512,512]$ (default) $[512,512,512,512]$ for DQC
Target EMA rate	0.005
Policy goal-sampling probabilities	$(0,0,1,0)$

Appendix

Table 6: Shared hyperparameters for the baseline algorithms.

\dagger

OTA uses a batch size of 1024 for AntMaze tasks and 256 for all other tasks.

Environment	$\beta_{H}$	$\beta_{L}$	$K$	$n$	$\gamma$
amm	3.0	3.0	25	5	0.99
aml	3.0	3.0	25	5	0.99
cs	1.0	3.0	20	4	0.99
cd	0.5	3.0	25	4	0.99
ct	3.0	3.0	20	4	0.99
p3	0.5	3.0	25	4	0.99

Appendix

Table 7: Environment-specific OTA hyperparameters.

Environment	$\alpha$	$\lambda$	$\kappa$	$\gamma$
amm	0.7	0.0	0.7	0.99
aml	0.7	0.0	0.7	0.99
cs	1.0	0.7	0.7	0.99
cd	10.0	1.0	0.7	0.99
ct	1.0	1.0	0.7	0.99
p3	2.0	0.5	0.7	0.99

Appendix

Table 8: Environment-specific TRL hyperparameters.

Environment	$h_{a}$	$h$	$\kappa_{b}$	$\kappa_{d}$	$\gamma$
amm	1	25	0.5	0.8	0.999
aml	1	25	0.5	0.8	0.999
cs	1	5	0.99	0.8	0.999
cd	1	5	0.99	0.8	0.999
ct	5	25	0.7	0.8	0.999
p3	5	10	0.7	0.8	0.999

Appendix

Table 9: Environment-specific DQC hyperparameters.

Environment	$S_{\mathrm{tr}}(s,z,g)$	$V(z,g)$	Uniform
amm	$\mathbf{96.4\pm 0.6}$	$95.8\pm 0.5$	$95.1\pm 1.4$
aml	$\mathbf{84.6\pm 6.4}$	$83.1\pm 5.9$	$71.1\pm 7.0$
cd	$\mathbf{85.1\pm 2.2}$	$83.7\pm 2.4$	$82.4\pm 2.1$
p3	$\mathbf{69.3\pm 3.3}$	$61.4\pm 5.7$	$6.9\pm 1.9$
p4	$\mathbf{78.1\pm 6.0}$	$65.1\pm 9.0$	$4.4\pm 0.5$
scene	$\mathbf{69.1\pm 3.0}$	$68.6\pm 3.8$	$56.0\pm 4.0$

Appendix

Table 10: Endpoint-selection-score ablation using the final trained proposer. Values are mean

\pm

std over four seeds.

Environment	$(1,0)$	$(2,0.25)$	$(16,0.5)$	$(32,1)$
amm	$94.4\pm 0.6$	$\mathbf{96.4\pm 0.6}$	$93.5\pm 0.2$	$88.9\pm 2.0$
aml	$79.7\pm 4.5$	$80.9\pm 3.3$	$\mathbf{84.6\pm 6.4}$	$76.2\pm 9.7$
cs	$\mathbf{99.4\pm 0.5}$	$96.9\pm 0.4$	$92.3\pm 1.0$	$63.7\pm 4.1$
cd	$86.8\pm 7.6$	$\mathbf{85.1\pm 2.2}$	$84.8\pm 8.9$	$71.4\pm 10.2$
ct	$\mathbf{79.8\pm 4.5}$	$78.2\pm 1.6$	$76.2\pm 1.3$	$58.1\pm 0.8$
p3	$17.0\pm 3.3$	$21.1\pm 3.7$	$54.1\pm 3.7$	$\mathbf{69.3\pm 3.3}$

Appendix

Table 11: PBF success rates across the joint

(N,T)

sweep. Values are mean

\pm

one standard deviation over four seeds. Bold indicates the setting used in the main benchmark.

Environment	$(1,0)$	$(2,0.25)$	$(16,0.5)$	$(32,1)$
amm	$\mathbf{95.3\pm 0.6}$	$93.3\pm 2.1$	$91.0\pm 0.7$	$79.5\pm 2.3$
aml	$\mathbf{85.3\pm 3.0}$	$81.8\pm 4.0$	$75.1\pm 4.0$	$60.5\pm 3.1$
cs	$\mathbf{93.0\pm 1.7}$	$79.1\pm 4.1$	$67.1\pm 4.0$	$28.7\pm 3.7$
cd	$\mathbf{75.7\pm 1.9}$	$58.9\pm 4.2$	$43.6\pm 4.9$	$16.6\pm 4.3$
ct	$\mathbf{45.1\pm 5.8}$	$29.7\pm 1.4$	$22.1\pm 1.6$	$11.0\pm 0.8$
p3	$25.8\pm 5.7$	$\mathbf{28.9\pm 3.6}$	$26.3\pm 3.7$	$12.4\pm 0.6$

Appendix

Table 12: PBG success rates across the joint

(N,T)

sweep. Values are mean

\pm

one standard deviation over four seeds. Bold indicates the setting used in the main benchmark.

	PBF		PBG
Environment	$K{=}25$	$K{=}40$	$K{=}25$	$K{=}40$
amm	97.3	91.5	96.0	87.7
aml	93.3	83.5	89.1	82.1
cs	95.7	97.1	94.9	87.2
cd	80.4	82.4	77.3	64.3
ct	55.5	77.3	42.4	6.9

Appendix

Table 13: Success rates across the bridge-horizon sweep. All non-

K

settings are fixed within each environment and variant. Bold indicates the setting used in the main benchmark.

	PBF		PBG
Environment	$c_{\rm sg}=5$	$c_{\rm sg}=10$	$c_{\rm sg}=5$	$c_{\rm sg}=10$
amm	94.9	97.3	91.2	95.7
aml	78.1	78.4	70.1	82.1
cs	99.7	98.9	83.2	92.5
cd	82.9	87.9	55.7	77.1
ct	74.1	77.3	33.1	42.4

Appendix

Table 14: Success rates across the endpoint-weighting sweep. All non-

c_{\rm sg}

settings are fixed within each environment and variant. Bold indicates the setting used in the main benchmark.

Transitions	Method	amm	aml	cs	cd	ct	p3	p4	scene	Average
100k	HIQL	$24.6\pm 0.1$	$12.3\pm 1.4$	$4.9\pm 1.7$	$0.2\pm 0.2$	$0.1\pm 0.1$	$1.7\pm 0.3$	$0.0\pm 0.0$	$4.5\pm 1.4$	$6.0$
	TRL	$9.2\pm 2.8$	$0.0\pm 0.0$	$\mathbf{21.4\pm 0.1}$	$0.1\pm 0.2$	$0.1\pm 0.2$	$1.5\pm 0.5$	$0.1\pm 0.1$	$2.4\pm 3.3$	$4.3$
	DQC	$18.4\pm 0.0$	$5.7\pm 0.0$	$13.9\pm 0.0$	$0.5\pm 0.0$	$0.0\pm 0.0$	$\mathbf{56.4\pm 0.0}$	$0.5\pm 0.0$	$10.8\pm 0.0$	$13.3$
	PBG	$81.2\pm 12.0$	$28.6\pm 6.6$	$10.2\pm 2.1$	$0.7\pm 0.4$	$0.0\pm 0.0$	$5.3\pm 4.7$	$0.0\pm 0.0$	$4.1\pm 0.5$	$16.3$
	PBF	$\mathbf{87.4\pm 2.6}$	$\mathbf{37.5\pm 4.2}$	$\mathbf{22.4\pm 0.4}$	$\mathbf{5.4\pm 0.3}$	$\mathbf{0.3\pm 0.3}$	$38.0\pm 4.0$	$\mathbf{34.8\pm 0.1}$	$\mathbf{14.3\pm 4.5}$	$\mathbf{30.0}$
300k	HIQL	$90.4\pm 1.2$	$\mathbf{80.5\pm 2.1}$	$4.6\pm 0.2$	$0.6\pm 0.4$	$0.0\pm 0.0$	$2.5\pm 0.6$	$0.1\pm 0.2$	$11.6\pm 2.7$	$23.8$

Appendix

Table 15: Per-environment dataset-size results. Values are mean

\pm

one standard deviation over two seeds. All methods are trained for 1M gradient steps at each data budget. Bold indicates mean performance at or above 95% of the best mean in each setting.

Offline RL

Offline reinforcement learning learns decision-making policies from fixed datasets without interacting with the environment during training.

See all 304 papers

Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning
May 27, 2026Kaiqiang Ke, Shenghong He, Chengdong Xu +3Hierarchical RL Offline RL
1Sun Yat-sen University · 2Pengcheng Laboratory
Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning
Sep 28, 2026Hengrui Zhang, Yuhu Cheng, C. L. Philip Chen +1Diffusion-Based Planning Hierarchical RL
School of Information and Control Engineering, China University of Mining and Technology · School of Computer Science and Engineering, South China University of Technology
Escaping Path Mirages in Offline Goal-Conditioned Reinforcement Learning
Oct 29, 2026Junhyeon Bae, Jaebak Hwang, Gwanwoo Choi +2Offline RL Goal-Conditioned RL
Graduate School of Artificial Intelligence, UNIST, Ulsan, South Korea

May 27, 2026J/K move · Enter open · S save

Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning

Kaiqiang Ke, Shenghong He, Chengdong Xu, Yuheng Luo+2

1Sun Yat-sen University · 2Pengcheng Laboratory

Hierarchical RL Offline RL Goal-Conditioned RL

Open paper AskPDF arXiv

Offline goal-conditioned reinforcement learning (GCRL) is challenging in long-horizon tasks, where distant state--goal pairs provide weak supervision and value estimates become vulnerable to accumulated bootstrapping errors. Hierarchical methods mitigate this difficulty by introducing intermediate subgoals, but fixed temporal abstractions or fixed hierarchy depths can be mismatched to state--goal pairs with different reachability horizons. We propose Coarse-to-Fine Hierarchical Goal Reinforcement Learning (CFHRL), a fully offline GCRL framework that adaptively refines distant goals before execution. Starting from the final goal, CFHRL recursively proposes intermediate targets, trained from replay-supported candidates, and stops refinement once the current target is estimated to be locally executable by a learned reachability cost. The key idea is that a subgoal need not be an exact midpoint or globally optimal waypoint; it only needs to provide reliable progress and reduce the remaining reaching difficulty, enabling subsequent refinement over shorter horizons. A stylized analysis further supports the robustness of approximate recursive contraction. Experiments on OGBench show substantial gains on several long-horizon tasks, with ablations validating the proposed refinement and stopping mechanisms

Abstract

Offline RL

Explore similar work

Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning

Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning

Escaping Path Mirages in Offline Goal-Conditioned Reinforcement Learning