Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning

Hengrui Zhang; Yuhu Cheng; C. L. Philip Chen; Xuesong Wang

cs.AISep 28, 2026

Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning

Authors: Hengrui Zhang, Yuhu Cheng, C. L. Philip Chen, Xuesong Wang

Organizations: School of Information and Control Engineering, China University of Mining and Technology · School of Computer Science and Engineering, South China University of Technology

Abstract

Offline goal-conditioned reinforcement learning (GCRL) learns goal-directed policies from reward-free data, but in long-horizon tasks, goal-conditioned value functions often provide unstable guidance due to sparse rewards and discounting. Hierarchical methods partially mitigate this issue via subgoal decomposition; however, high-level decision-making still relies on noise-sensitive value estimates, leading to unstable behavior in complex environments. We address this limitation by proposing \textbf{D}iffusion \textbf{S}ubgoal \textbf{P}lanning (\textbf{DSP}), a diffusion-based framework for high-level subgoal generation. DSP casts high-level planning as guided generative inference over goal-conditioned subgoals and learns both conditional and unconditional flows, enabling classifier-free guidance to introduce a goal-directed bias at inference time. By removing explicit value-based guidance from high-level planning, DSP generates reachable and goal-directed subgoals through a generative model while retaining hierarchical execution. Experiments on offline GCRL benchmarks demonstrate that DSP outperforms prior methods on a range of navigation and manipulation tasks, with particularly strong performance in maze environments that require multi-step subgoal planning.

Figures & tables

Figure 1: 1-D toy environment.

Panel of Figure 2 — Figure 2: Comparative visualization of high-level planning in the antmaze-giant-navigate-v0 environment. The state space is projected onto the 2D $(x,y)$ plane, where the background heatmap illustrates the landscape of the goal-conditioned value function; brighter regions correspond to higher values. HIQL w/o selects subgoals through value-based guidance, making its subgoal trajectories sensitive to value-function noise. In contrast, DSP generates smoother and topologically connected paths without explicit value-based high-level guidance.

Figure 3

Datasets	Flat Policies			Hierarchical Policies
	GCBC	CFGRL	GCIVL	OTA	Pi-HIQL	HIQL	HIQL w/o	DSP
pointmaze-medium-navigate-v0	$4.8\pm 6.9$	$65.2\pm 5.3$	$70.2\pm 5.9$	$85.4\pm 5.0$	$62.2\pm 7.3$	$70.6\pm 7.0$	$74.6\pm 6.1$	$\mathbf{90.6}\pm 3.8$
pointmaze-large-navigate-v0	$25.6\pm 6.3$	$74.6\pm 2.9$	$42.6\pm 5.3$	$87.6\pm 9.2$	$80.2\pm 13.3$	$39.4\pm 2.3$	$50.0\pm 10.4$	$\mathbf{93.6}\pm 2.9$
pointmaze-giant-navigate-v0	$2.2\pm 4.9$	$3.4\pm 2.5$	$1.0\pm 2.2$	$\mathbf{69.4}\pm 11.6$	$3.0\pm 2.9$	$5.0\pm 5.0$	$0.0\pm 0.0$	$43.4\pm 7.2$
pointmaze-teleport-navigate-v0	$23.8\pm 5.7$	$\mathbf{49.2}\pm 5.1$	$43.6\pm 2.5$	$40.2\pm 5.4$	$33.6\pm 9.0$	$8.8\pm 6.4$	$8.2\pm 6.2$	$\mathbf{51.0}\pm 5.8$
antmaze-medium-navigate-v0	$32.6\pm 8.5$	$34.8\pm 5.5$	$74.6\pm 8.0$	$91.6\pm 2.4$	$90.8\pm 2.7$	$93.2\pm 1.3$	$92.8\pm 2.4$	$\mathbf{98.4}\pm 0.5$

Table 1: Complete comparison between DSP and the offline GCRL baselines. The table reports the average binary success rate (%) across five test-time goals for each task, averaged over 5 seeds. Standard deviations are indicated by the

\pm

symbol. Entries within 95% of the best-performing value in each row are highlighted in bold .

Figure 3: Learning curves on challenging OGBench navigation tasks. DSP generally converges more reliably than HIQL and HIQL w/o in these long-horizon settings, suggesting the benefit of guided generative subgoal planning.

Datasets	$\omega$	$D\downarrow$	$V\uparrow$	$R\uparrow$	Succ. $\uparrow$
AntMaze-Giant	1	1.6	100.0	68.8	46.0
	3	1.7	100.0	62.1	69.2
	5	1.8	98.8	51.6	56.4
	10	2.1	96.9	40.6	27.0
Scene-Play	1	1.8	98.0	92.4	35.6
	3	2.0	96.5	90.7	50.4

Table 2: Guidance–support–reachability trade-off.

D

: support distance;

V

: validity;

R

: valid-and-reached rate.

V

R

, and Succ. are percentages.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Learning rate	$0.0003$
Optimizer	Adam
Batch size	$1024$
Total gradient steps	$1000000$
MLP dimensions	$[512,512,512]$
Activation function	GELU

Appendix

Table 3: Common hyperparameters for DSP in our experiments.

Environment	Datasets	$\omega$	$\alpha$	$k$
pointmaze	pointmaze-medium-navigate-v0	$3.0$	$3.0$	$25$
	pointmaze-large-navigate-v0	$3.0$	$3.0$	$25$
	pointmaze-giant-navigate-v0	$3.0$	$3.0$	$25$
	pointmaze-teleport-navigate-v0	$3.0$	$3.0$	$25$
	pointmaze-medium-stitch-v0	$3.0$	$3.0$	$25$
	pointmaze-large-stitch-v0	$3.0$	$3.0$	$25$

Appendix

Table 4: Task specific hyperparameters for DSP in our experiments.

Panel of Figure 4 — Figure 4: Illustration of the OGBench environments, covering diverse locomotion and manipulation tasks used for evaluation.

Panel of Figure 5 — Figure 5: Hyperparameter sensitivity analysis of DSP. (a): sensitivity to the guidance scale $\omega$ under classifier-free guidance, showing a consistent inverted-U trend across representative tasks. (b): sensitivity to the number of diffusion steps $N$ , illustrating the trade-off between performance gains and inference cost. (c): sensitivity to the subgoal steps $k$ , showing a broad performance plateau with degradation for overly large $k$ . Shaded regions indicate one standard deviation over 5 random seeds.

Datasets	MLE	Diffusion $(\omega=1)$	HIQL	DSP
antmaze-medium-navigate-v0	$75.8\pm 2.5$	$77.0\pm 7.5$	$93.2\pm 1.3$	$\mathbf{98.4}\pm 0.5$
antmaze-large-navigate-v0	$58.0\pm 5.1$	$58.2\pm 7.6$	$87.8\pm 1.5$	$\mathbf{92.0}\pm 4.1$
antmaze-giant-navigate-v0	$14.8\pm 4.4$	$43.8\pm 4.0$	$58.2\pm 4.7$	$\mathbf{69.2}\pm 2.9$
cube-single-play-v0	$15.0\pm 3.7$	$19.2\pm 3.8$	$11.8\pm 2.3$	$\mathbf{41.0}\pm 6.1$
cube-double-play-v0	$1.2\pm 1.1$	$4.8\pm 0.8$	$3.6\pm 2.2$	$\mathbf{30.8}\pm 8.4$
scene-play-v0	$6.8\pm 1.8$	$25.6\pm 4.2$	$37.2\pm 4.2$	$\mathbf{62.6}\pm 5.9$

Appendix

Table 5: Ablation on high-level subgoal generation. The table reports average binary success rate (%) over 5 seeds. MLE and diffusion variants are trained on the same

k

-step future-state subgoal targets.

Datasets	Representation	Merlin	HDMI	HD	DSP
PointMaze-Large	Full $=$ XY	$43.2\pm 8.1$	$61.2\pm 4.1$	$73.6\pm 6.8$	$\mathbf{93.6}\pm 2.9$
PointMaze-Giant	Full $=$ XY	$0.4\pm 0.5$	$26.4\pm 5.4$	$12.6\pm 6.9$	$\mathbf{43.4}\pm 7.2$
AntMaze-Large	Full	$22.6\pm 5.2$	$53.4\pm 5.6$	$64.0\pm 8.2$	$\mathbf{92.0}\pm 4.1$
AntMaze-Large	XY	$34.2\pm 5.8$	$65.0\pm 3.7$	$74.4\pm 5.5$	$\mathbf{96.0}\pm 2.0$
AntMaze-Giant	Full	$0.0\pm 0.0$	$0.0\pm 0.0$	$0.2\pm 0.4$	$\mathbf{69.2}\pm 2.9$
AntMaze-Giant	XY	$0.0\pm 0.0$	$22.8\pm 3.5$	$8.6\pm 3.7$	$\mathbf{77.2}\pm 2.7$

Appendix

Table 6: Success rates (%) under the matched OGBench protocol, reported as mean

\pm

standard deviation over five seeds. Bold indicates the highest mean in each setting.

Figure 6: Comparison of success rate and inference time across methods. The left axis shows success rates on antmaze-giant and scene-play , while the right axis shows inference time. DSP with

N=1

corresponds to the closest matched-budget setting, while increasing the number of diffusion steps (e.g.,

N=20

) leads to improved performance at higher inference cost.

Method	GCIVL	OTA	Pi-HIQL	HIQL	DSP
Training Time (min)	24	56	53	42	57

Appendix

Table 7: Training time comparison of different methods measured on the same hardware.

Datasets	Flat Policies			Hierarchical Policies
	GCBC	CFGRL	GCIVL	HIQL	HIQL w/o	DSP ${}^{\mathrm{mix}}_{1}$	DSP 1	DSP mix	DSP
pointmaze-medium-stitch-v0	$23.2\pm 16.7$	$24.6\pm 8.3$	$20.8\pm 4.0$	$62.0\pm 6.8$	$\mathbf{69.4}\pm 13.6$	$2.4\pm 0.9$	$54.6\pm 3.0$	$5.0\pm 2.0$	$\mathbf{65.8}\pm 5.8$
pointmaze-large-stitch-v0	$4.4\pm 8.3$	$13.0\pm 3.9$	$8.0\pm 11.0$	$22.0\pm 7.5$	$8.2\pm 7.7$	$0.0\pm 0.0$	$38.0\pm 1.9$	$3.6\pm 1.1$	$\mathbf{44.2}\pm 4.3$
pointmaze-giant-stitch-v0	$0.0\pm 0.0$	$0.0\pm 0.0$	$0.0\pm 0.0$	$0.0\pm 0.0$	$0.0\pm 0.0$	$0.0\pm 0.0$	$0.0\pm 0.0$	$0.0\pm 0.0$	$\mathbf{1.2}\pm 0.8$
pointmaze-teleport-stitch-v0	$26.8\pm 8.7$	$13.8\pm 8.9$	$28.6\pm 5.0$	$33.2\pm 2.5$	$8.4\pm 4.9$	$19.6\pm 4.2$	$43.4\pm 7.5$	$14.0\pm 5.7$	$\mathbf{51.8}\pm 4.9$
antmaze-medium-stitch-v0	$52.6\pm 5.7$	$44.4\pm 5.3$	$32.6\pm 10.4$	$\mathbf{89.8}\pm 3.8$	$\mathbf{89.6}\pm 3.4$	$66.6\pm 9.2$	$75.8\pm 6.6$	$65.8\pm 5.5$	$\mathbf{85.8}\pm 3.3$

Appendix

Table 8: Complete comparison between DSP and the offline GCRL baselines. The table reports the average binary success rate (%) across five test-time goals for each task, averaged over 5 seeds. Standard deviations are indicated by the

\pm

symbol. Entries within 95% of the best-performing value in each row are highlighted in bold .

Datasets	Fraction	$C_{\mathrm{XY}}$	HIQL	HIQL w/o	DSP
AntMaze-Large	1.00	1.00	$87.8\pm 1.5$	$88.6\pm 2.1$	$\mathbf{92.0}\pm 4.1$
	0.50	0.96	$87.0\pm 4.3$	$81.4\pm 3.2$	$\mathbf{90.2}\pm 1.9$
	0.25	0.91	$79.8\pm 2.7$	$70.2\pm 5.1$	$\mathbf{83.0}\pm 2.1$
	0.10	0.85	$19.4\pm 4.7$	$39.6\pm 5.6$	$\mathbf{52.8}\pm 6.6$
HumanoidMaze-Giant	1.00	1.00	$5.0\pm 1.6$	$0.0\pm 0.0$	$\mathbf{66.0}\pm 3.4$
	0.10	0.94	$1.0\pm 0.7$	$0.0\pm 0.0$	$\mathbf{44.4}\pm 4.2$

Appendix

Table 9: Success rates (%) under nested complete-trajectory subsets. Results are mean

\pm

standard deviation over five seeds.

C_{\mathrm{XY}}

denotes relative XY-bin coverage.

Datasets	HIQL	HIQL w/o	DSP
antmaze-medium-explore-v0	$32.4\pm 10.0$	$33.4\pm 9.7$	$\mathbf{60.4}\pm 9.3$
antmaze-large-explore-v0	$1.6\pm 3.6$	$6.0\pm 6.7$	$\mathbf{16.4}\pm 7.5$
antmaze-teleport-explore-v0	$26.8\pm 16.6$	$\mathbf{30.4}\pm 7.3$	$21.4\pm 4.6$

Appendix

Table 10: Success rates (%) on the official AntMaze Explore datasets, reported as mean

\pm

standard deviation over five seeds.

Datasets	HIQL	DSP
visual-antmaze-medium-navigate	$92.0\pm 2.1$	$\mathbf{96.2}\pm 2.5$
visual-antmaze-large-navigate	$51.8\pm 5.5$	$\mathbf{68.4}\pm 6.2$
visual-antmaze-giant-navigate	$4.4\pm 2.8$	$\mathbf{13.8}\pm 4.4$
visual-scene-play	$43.8\pm 3.9$	$\mathbf{48.6}\pm 1.5$

Appendix

Table 11: Success rates (%) on visual OGBench tasks, reported as mean

\pm

standard deviation over five seeds. Bold indicates the highest mean in each row.

Panel of Figure 7 — Figure 7: Visualization in the medium maze. Top: HIQL w/o , Bottom: DSP. DSP generates smoother trajectories that better follow the topology.

Panel of Figure 8 — Figure 8: Visualization in the large maze. Top: HIQL w/o , Bottom: DSP. As the scale increases, HIQL begins to show significant detours, whereas DSP maintains efficient planning.

Panel of Figure 9 — Figure 9: Visualization in the giant maze. Top: HIQL w/o , Bottom: DSP. In this extreme setting, HIQL often produces less reliable plans, while DSP more consistently reaches the target.

Datasets	HDMI	DTAMP	HD	SIHD	DSP
antmaze-umaze-v2	$86.1\pm 2.4$	$-$	$-$	$-$	$\mathbf{98.6\pm 2.1}$
antmaze-umaze-diverse-v2	$73.7\pm 1.1$	$-$	$94.0\pm 4.9$	$96.5\pm 2.8$	$\mathbf{96.8\pm 2.6}$
antmaze-medium-play-v2	$-$	$\mathbf{89.3\pm 3.9}$	$-$	$-$	$87.0\pm 3.3$
antmaze-medium-diverse-v2	$-$	$76.7\pm 4.5$	$88.7\pm 8.1$	$92.2\pm 5.0$	$\mathbf{92.4\pm 4.4}$
antmaze-large-play-v2	$-$	$62.0\pm 2.9$	$-$	$-$	$\mathbf{88.8\pm 4.1}$
antmaze-large-diverse-v2	$71.5\pm 3.5$	$53.3\pm 9.7$	$83.6\pm 5.8$	$89.4\pm 4.2$	$\mathbf{95.4\pm 2.9}$

Appendix

Table 12: Comparison with diffusion-based hierarchical planning methods on D4RL AntMaze. We report mean

\pm

standard deviation over 5 random seeds, with each seed evaluated on 100 episodes. Baseline numbers are taken from the corresponding original papers when reported under the same D4RL AntMaze setting. The best-performing entry in each row is highlighted in bold .

Datasets	$(d_{s},d_{a},d_{w},d_{g})$	PPO	HIQL	DSP
ShadowHandOver	$(387,40,7,7)$	46.5	$56.6\pm 4.2$	$\mathbf{74.2}\pm 2.9$
ShadowHandCatchUnderarm	$(411,52,7,7)$	37.2	$46.0\pm 4.1$	$\mathbf{54.6}\pm 3.3$
ShadowHandOver	$(387,40,55,7)$	46.5	$54.2\pm 5.9$	$\mathbf{67.6}\pm 2.4$
ShadowHandCatchUnderarm	$(411,52,67,7)$	37.2	$41.2\pm 3.4$	$\mathbf{46.4}\pm 3.9$

Appendix

Table 13: Success rates (%) on Bi-DexHands. PPO denotes the data collector. Dimensions are reported as

(d_{s},d_{a},d_{w},d_{g})

Explore similar work

May 27, 2026cs.LG

Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning

Offline goal-conditioned reinforcement learning (GCRL) is challenging in long-horizon tasks, where distant state--goal pairs provide weak supervision and value estimates become vulnerable to accumulated bootstrapping errors. Hierarchical methods mitigate this difficulty by introducing intermediate subgoals, but fixed temporal abstractions or fixed hierarchy depths can be mismatched to state--goal pairs with different reachability horizons. We propose Coarse-to-Fine Hierarchical Goal Reinforcement Learning (CFHRL), a fully offline GCRL framework that adaptively refines distant goals before execution. Starting from the final goal, CFHRL recursively proposes intermediate targets, trained from replay-supported candidates, and stops refinement once the current target is estimated to be locally executable by a learned reachability cost. The key idea is that a subgoal need not be an exact midpoint or globally optimal waypoint; it only needs to provide reliable progress and reduce the remaining reaching difficulty, enabling subsequent refinement over shorter horizons. A stylized analysis further supports the robustness of approximate recursive contraction. Experiments on OGBench show substantial gains on several long-horizon tasks, with ablations validating the proposed refinement and stopping mechanisms

Kaiqiang Ke, Shenghong He, Chengdong Xu +3

1Sun Yat-sen University · 2Pengcheng Laboratory

Goal-Conditioned Reinforcement Learning Long-Horizon Task Planning

Ask PDF

Feb 3, 2026cs.LG

Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL

Offline goal-conditioned reinforcement learning remains challenging for long-horizon tasks. While hierarchical approaches mitigate this issue by decomposing tasks, most existing methods rely on separate high- and low-level networks and generate only a single intermediate subgoal, leaving several structural limitations in long-horizon decision-making. To address this limitation, we draw inspiration from chain-of-thought reasoning and propose the Chain-of-Goals Hierarchical Policy (CoGHP), a novel framework that reformulates hierarchical decision-making as autoregressive sequence modeling within a unified architecture. Given a state and a final goal, CoGHP autoregressively generates a sequence of latent subgoals followed by the primitive action, where each latent subgoal acts as a reasoning step that conditions subsequent predictions. To implement this efficiently, we introduce an MLP-Mixer backbone, which supports cross-token communication and captures structural relationships among state, goal, latent subgoals, and action. Across challenging navigation and manipulation benchmarks, CoGHP consistently outperforms strong offline baselines, demonstrating improved performance on long-horizon tasks. Project page: https://wlsdn9350.github.io/projects/coghp/

Jinwoo Choi, Sang-Hyun Lee, Seung-Woo Seo

Department of Electrical and Computer Engineering, Seoul National University, Seoul, South Korea · Department of Automotive Engineering, Ajou University, Gyeonggi-do, South Korea

Goal-Conditioned Reinforcement Learning High-Level Subgoal Generation Autoregressive Model Chain-of-Thought Reasoning

Ask PDF

May 25, 2026cs.LG

Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning

Offline goal-conditioned reinforcement learning (GCRL) provides a practical framework for obtaining goal-reaching policies from fixed datasets. However, learning a reliable goal-conditioned value function in long-horizon tasks remains challenging. In this paper, we identify erroneous generalization in goal-conditioned value functions as a fundamental bottleneck, and demonstrate that appropriate inductive bias in the value function is crucial for addressing the bottleneck. Building on these findings, we propose Latent-Aligned Value Learning (LAVL), an offline GCRL algorithm that integrates latent-representation-based value generalization with hierarchical planning in a unified framework. Extensive experiments on OGBench demonstrate that LAVL consistently outperforms existing offline GCRL methods, achieving the highest performance on 20 out of 22 datasets. Notably, LAVL exhibits strong performance in long-horizon tasks and trajectory stitching datasets, where prior methods suffer significant performance degradation. Our code is available at https://github.com/oh-lab/LAVL.git.

Hyungkyu Kang, Byeongchan Kim, Min-hwan Oh

Seoul National University, Seoul, South Korea.

Goal-Conditioned Reinforcement Learning Goal-Conditioned Value Function Latent Representation Alignment Long-Horizon Task Planning

Ask PDF