An Informational Curse of Horizon in Goal-Conditioned Policy Learning

John L. Zhou; Yuxuan Dong; Jonathan C. Kao

cs.LGOct 7, 2026

An Informational Curse of Horizon in Goal-Conditioned Policy Learning

Authors: John L. Zhou, Yuxuan Dong, Jonathan C. Kao

Organizations: University of California, Los Angeles

Abstract

The difficulty of learning goal-reaching policies is often attributed to a "curse of horizon" that manifests as bias accumulation in temporal-difference backups and noisy advantage estimates. In this work, we identify an additional informational curse of horizon in goal-conditioned policy learning, where increasing the goal relabeling horizon can significantly reduce policy generalization and performance. Through a series of controlled experiments with oracle planners, we decouple the goal horizons sampled during training from those that the policy is asked to reach at test time. Even when evaluated only on a sequence of nearby subgoals, goal-conditioned behavioral cloning (BC) policies suffer from severe, training horizon-dependent performance degradation that is mitigated by reinforcement learning (RL) objectives. We explain this phenomenon as a horizon-dependent decrease in the conditional mutual information between actions and hindsight-relabeled goals, and find empirically that both BC and RL policies trained on longer-horizon goals exhibit a shift in sensitivity from goal to state information, as measured by the policy's input Jacobians. Motivated by this observation, we find that distilling the input Jacobians of short-horizon policies into long-horizon policies yields significant performance gains, especially in combinatorial manipulation tasks. Taken together, our results highlight goal relabeling horizon as an important consideration when learning generalist policies from offline data.

Figures & tables

Figure 1: Training on more distant relabeled goals harms nearby goal-reaching capabilities . We record the success rates across policies trained on a geometric distribution of relabeling goal offsets with discount

\gamma

, where

\gamma=1

corresponds to a uniform distribution over future states. At evaluation, policies are given optimal

H_{\mathrm{plan}}

-step subgoals from oracle planners . Success rates are averaged over the last three checkpoints, three seeds, and four environments: cube-double , cube-triple , cube-quadruple , and scene .

Figure 2: Increasing training goal horizon globally decreases policy goal sensitivity . We measure the proportion of the input Jacobian norm corresponding to the goal at fixed state-goal offsets, plotted with

\log_{2}

scaling in cube-triple . Results are averaged over the last three training checkpoints, three seeds, and 1024 randomly sampled state-goal pairs per horizon, with additional environments in Figure 8 .

Environment	Dataset	GCBC	GCIVL	GCIQL	QRL	CRL	HIQL	SAW	S 2 AW
pointmaze	pointmaze-medium-navigate-v0	$9$ $\pm 6$	$63$ $\pm 6$	$53$ $\pm 8$	$82$ $\pm 5$	$29$ $\pm 7$	$79$ $\pm 5$	$\mathbf{97}$ $\pm 2$	$\mathbf{96}$ $\pm 3$
	pointmaze-large-navigate-v0	$29$ $\pm 6$	$45$ $\pm 5$	$34$ $\pm 3$	$\mathbf{86}$ $\pm 9$	$39$ $\pm 7$	$58$ $\pm 5$	$\mathbf{85}$ $\pm 10$	$\mathbf{83}$ $\pm 8$
	pointmaze-giant-navigate-v0	$1$ $\pm 2$	$0$ $\pm 0$	$0$ $\pm 0$	${68}$ $\pm 7$	$27$ $\pm 10$	$46$ $\pm 9$	${68}$ $\pm 8$	$\mathbf{73}$ $\pm 9$
antmaze	antmaze-medium-navigate-v0	$29$ $\pm 4$	$72$ $\pm 8$	$71$ $\pm 4$	$88$ $\pm 3$	$\mathbf{95}$ $\pm 1$	$\mathbf{96}$ $\pm 1$	$\mathbf{97}$ $\pm 1$	$\mathbf{97}$ $\pm 1$
	antmaze-large-navigate-v0	$24$ $\pm 2$	$16$ $\pm 5$	$34$ $\pm 4$	$75$ $\pm 6$	$83$ $\pm 4$	$\mathbf{91}$ $\pm 2$	$\mathbf{90}$ $\pm 3$	$\mathbf{93}$ $\pm 2$
	antmaze-giant-navigate-v0	$0$ $\pm 0$	$0$ $\pm 0$	$0$ $\pm 0$	$14$ $\pm 3$	$16$ $\pm 3$	${65}$ $\pm 5$	${73}$ $\pm 4$	$\mathbf{87}$ $\pm 3$

Table 1: Evaluating Sobolev bootstrapping on offline goal-conditioned RL tasks. We compare our method’s average (binary) success rate (

\%

) against the numbers reported in Park et al. (2024) and Zhou and Kao (2025) across the five test-time goals for each environment, averaged over the last three checkpoints and eight seeds

\pm

standard deviations. Numbers within

5\%

of the best value in the row are in bold .

Figure 3: Sobolev bootstrapping exhibits more interpretable sensitivity structure . We plot the proportion of the total input Jacobian norm corresponding to each cube during a triple pick-and-place on-policy rollout for SAW without (top) and with (bottom) Sobolev bootstrapping. For fairness, we also analyze a scripted optimal trajectory in Figure 11 .

Figure 4: Sobolev bootstrapping recovers policy goal sensitivity . We compare the absolute goal Jacobian norm between S 2 AW and SAW, averaging over batches of goals sampled at fixed intervals from the current state up to the maximum horizon. Results are averaged across four seeds and 1024 state-goal pairs for each horizon, with shaded standard deviations.

Environment	GCIQL	GCIQL ${}_{\ell_{1}}$	SAW	SAW ${}_{\ell_{1}}$	SAW ${}_{\ell_{2}}$	S 2 AW
cube-single	$68$ $\pm 6$	$74$ $\pm 4$	$76$ $\pm 8$	$\mathbf{85}$ $\pm 6$	$\mathbf{85}$ $\pm 4$	$\mathbf{86}$ $\pm 5$
cube-double-task1	$74$ $\pm 8$	$76$ $\pm 8$	$64$ $\pm 9$	$\mathbf{81}$ $\pm 9$	${65}$ $\pm 14$	$\mathbf{78}$ $\pm 12$
cube-triple-task1	$13$ $\pm 3$	$16$ $\pm 10$	$18$ $\pm 11$	$24$ $\pm 9$	$17$ $\pm 12$	$\mathbf{74}$ $\pm 9$
cube-quadruple-task6	$0$ $\pm 1$	$0$ $\pm 1$	$2$ $\pm 2$	$2$ $\pm 2$	$1$ $\pm 1$	$\mathbf{35}$ $\pm 14$

Table 2: Sobolev bootstrapping improves single pick-and-place performance in the presence of distractor objects. We evaluate policies trained in multi-cube environments on single pick-and-place tasks. While naive

\ell_{1}

\ell_{2}

Jacobian regularization [Appendix H.4 ] yields modest improvements on simpler tasks, Sobolev bootstrapping matches or outperforms all baselines. Other settings are identical to Table 1 .

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Training on distant relabeled goals harms nearby goal-reaching capabilities . We record the success rates across learned policies trained on a uniform distribution of relabeling goal offsets

[1,H_{\mathrm{train}}]

, where

H_{\mathrm{train}}=\infty

corresponds to a uniform distribution over all future in-trajectory states. At evaluation, policies are given optimal

H_{\mathrm{plan}}

-step subgoals from oracle planners. Success rates are averaged over the last three checkpoints and three seeds.

Figure 6: Training on a larger proportion of distant relabeled goals harms nearby goal-reaching capabilities . We record the success rates across learned policies trained on a geometric distribution of relabeling goal offsets with discount

\gamma

, where

\gamma=1

corresponds to a uniform distribution over future states. At evaluation, policies are given optimal

H_{\mathrm{plan}}

-step subgoals from oracle planners. Success rates are averaged over the last three checkpoints and three seeds.

Figure 7: Training on more random relabeled goals harms nearby goal-reaching capabilities . We record the success rates across learned policies trained on a mixture of goals either sampled from a uniform distribution over future states or from an arbitrary dataset state with probability

p_{\mathrm{rand}}\in\{0.25,0.50,0.75\}

. At evaluation, policies are given optimal

H_{\mathrm{plan}}

-step subgoals from oracle planners. Success rates are averaged over the last three checkpoints and three seeds.

Figure 8: Increasing training goal horizon globally decreases policy goal sensitivity . We measure the proportion of the input Jacobian norm corresponding to the goal at fixed state-goal offsets, plotted with

\log_{2}

scaling. Results are averaged over the last three training checkpoints, three seeds, and 1024 randomly sampled state-goal pairs per horizon.

Figure 9: Training on distant relabeled goals globally decreases policy goal sensitivity . We measure the proportion of the input Jacobian norm corresponding to the goal at fixed state-goal offsets, plotted with

\log_{2}

scaling. Results are averaged over the last three training checkpoints, three seeds, and 1024 randomly sampled state-goal pairs per horizon.

Figure 10: Random goal sampling globally decreases policy goal sensitivity. We directly reduce the action-goal MI by replacing in-trajectory goals with randomly sampled dataset states with probability

p_{\text{random}}\in\{0.25,0.5,0.75\}

, and measure the goal Jacobian norm proportion at fixed state-goal offsets.

Figure 11: Sobolev bootstrapping pays attention to the correct object during optimal trajectories. To enable fair comparisons across later task stages, we plot the input Jacobian norms with respect to each object over an independent scripted optimal trajectory Ahn et al. (2025) for the cube-triple triple pick-and-place task. S 2 AW (top) allocates a higher proportion of the Jacobian norm to the focal cube both during approach and grasp, whereas SAW’s (bottom) Jacobian norm only changes after the cube is already grasped. The shaded area denotes when the cube is actively grasped.

Dataset	Expectile $\tau$	AWR $\alpha$	KLD $\beta$	Subgoal steps $k$	Sobolev weight $\lambda$
pointmaze-medium-navigate-v0	$0.7$	$3.0$	$3.0$	$25$	$0.03$
pointmaze-large-navigate-v0	$0.7$	$3.0$	$3.0$	$25$	$1.00$
pointmaze-giant-navigate-v0	$0.7$	$3.0$	$3.0$	$25$	$1.00$
antmaze-medium-navigate-v0	$0.7$	$3.0$	$3.0$	$25$	$0.03$
antmaze-large-navigate-v0	$0.7$	$3.0$	$3.0$	$25$	$0.03$
antmaze-giant-navigate-v0	$0.7$	$3.0$	$3.0$	$25$	$0.30$

Appendix

Table 3: Hyperparameters for SAW with Sobolev bootstrapping.

Dataset	$\lambda=\mathbf{0}$	$\mathbf{0.001}$	$\mathbf{0.01}$	$\mathbf{0.1}$
cube-double-play-v0	$40$ $\pm 7$	$40$ $\pm 11$	${55}$ $\pm 7$	$35$ $\pm 9$
cube-triple-play-v0	$4$ $\pm 2$	$9$ $\pm 2$	${30}$ $\pm 7$	$23$ $\pm 10$
scene-play-v0	$63$ $\pm 6$	$54$ $\pm 10$	${71}$ $\pm 5$	$62$ $\pm 5$
antmaze-giant-navigate-v0	$73$ $\pm 4$	$76$ $\pm 4$	$82$ $\pm 4$	${85}$ $\pm 4$

Appendix

Table 4: Sobolev loss weight sensitivity analysis. Sobolev bootstrapping is robust to different weighting factors within a reasonable scale (around 0.01 for cube-double and scene ). Notably, improvements on antmaze-giant and cube-triple are robust across various weighting scales. Experimental results are averaged across four seeds.

hours / 1M iters	GCIQL	SAW	S 2 AW
cube-quadruple-play	0.26	0.46	0.75
scene-play	0.25	0.45	0.74
humanoidmaze-giant-navigate	0.26	0.46	1.75

Appendix

Table 5: Training time comparison. We select the highest-dimensional environments and report the hours required to complete a standard 1M-step training run on an NVIDIA 5090 GPU.

Environment	Dataset	GCBC	GCIVL	GCIQL	QRL	CRL	HIQL	SAW	S 2 AW	S 2 AW ℓ
maze-giant	antmaze-giant-navigate-v0	$0$ $\pm 0$	$0$ $\pm 0$	$0$ $\pm 0$	$14$ $\pm 3$	$16$ $\pm 3$	${65}$ $\pm 5$	${73}$ $\pm 4$	$\mathbf{87}$ $\pm 3$	${80}$ $\pm 5$
maze-giant	humanoidmaze-giant-navigate-v0	$0$ $\pm 0$	$0$ $\pm 0$	$0$ $\pm 0$	$1$ $\pm 0$	$3$ $\pm 2$	${12}$ $\pm 4$	${35}$ $\pm 4$	$\mathbf{38}$ $\pm 6$	${35}$ $\pm 5$
cube	cube-single-play-v0	$6$ $\pm 2$	$53$ $\pm 4$	${68}$ $\pm 6$	$5$ $\pm 1$	$19$ $\pm 2$	$44$ $\pm 9$	${72}$ $\pm 5$	$\mathbf{86}$ $\pm 5$	${79}$ $\pm 8$
	cube-double-play-v0	$1$ $\pm 1$	$36$ $\pm 3$	${40}$ $\pm 5$	$1$ $\pm 0$	$10$ $\pm 2$	$6$ $\pm 2$	${40}$ $\pm 7$	${50}$ $\pm 6$	$\mathbf{55}$ $\pm 8$
	cube-triple-play-v0	$1$ $\pm 1$	$1$ $\pm 0$	$3$ $\pm 1$	$0$ $\pm 0$	$\mathbf{4}$ $\pm 1$	$3$ $\pm 1$	${4}$ $\pm 2$	$\mathbf{30}$ $\pm 5$	$\mathbf{30}$ $\pm 4$
	cube-quadruple-play-v0	$0$ $\pm 0$	$0$ $\pm 0$	$0$ $\pm 0$	$0$ $\pm 0$	$0$ $\pm 0$	$0$ $\pm 0$	$0$ $\pm 0$	$\mathbf{3}$ $\pm 2$	${2}$ $\pm 1$

Appendix

Table 6: Architectural ablations to the Sobolev SAW target actor. We report the performance of Sobolev SAW where the target actor has the same architecture as the full policy, averaged over

4

seeds with standard deviations after the

\pm

sign. Numbers within

5\%

of the best value in the row are in bold .

Dataset	RIS	Sobolev RIS
pointmaze-medium-navigate-v0	$88$ $\pm 6$	$\mathbf{96}$ $\pm 1$
pointmaze-large-navigate-v0	$63$ $\pm 13$	$\mathbf{75}$ $\pm 3$
pointmaze-giant-navigate-v0	$57$ $\pm 12$	$\mathbf{67}$ $\pm 5$
antmaze-medium-navigate-v0	$\mathbf{96}$ $\pm 1$	$\mathbf{97}$ $\pm 1$
antmaze-large-navigate-v0	$\mathbf{89}$ $\pm 3$	$\mathbf{90}$ $\pm 1$
antmaze-giant-navigate-v0	$\mathbf{65}$ $\pm 4$	$\mathbf{69}$ $\pm 0$

Appendix

Table 7: Evaluating RIS with Sobolev bootstrapping. We evaluate the effectiveness of Sobolev bootstrapping on RIS by comparing it against the numbers reported in Zhou and Kao (2025) . Results are averaged over 4 seeds, and other settings are identical to Table 1 .

Environment	OTA n=5	OTA n=10	OTA n=20	SHARSA	DQC	S 2 AW
antmaze-giant-navigate	77 $\pm 4$	-	-	72 $\pm 1$	25 $\pm 6$	87 $\pm 3$
cube-double-play	2 $\pm 0$	3 $\pm 1$	2 $\pm 1$	5 $\pm 1$	24 $\pm 4$	57 $\pm 6$
cube-triple-play	1 $\pm 0$	0 $\pm 0$	1 $\pm 0$	2 $\pm 1$	11 $\pm 4$	30 $\pm 5$

Appendix

Table 8: Additional horizon reduction baselines . The additional baselines do not readily generalize to the standard datasets and perform poorly. All results are averaged across 4 seeds.

	SAW	SAW ${}_{\ell_{1}}$	SAW ${}_{\ell_{2}}$	S 2 AW
antmaze-giant-navigate	73 $\pm 4$	72 $\pm 3$	74 $\pm 2$	87 $\pm 3$
cube-double-play	40 $\pm 7$	47 $\pm 10$	37 $\pm 7$	57 $\pm 6$
cube-triple-play	4 $\pm 2$	5 $\pm 4$	4 $\pm 1$	30 $\pm 5$
scene-play	63 $\pm 6$	69 $\pm 8$	62 $\pm 2$	74 $\pm 6$

Appendix

Table 9: Uniform Jacobian regularization. We apply

\ell_{1}

and

\ell_{2}

norm penalization with coefficient

0.3

to the SAW baseline. We observe that

\ell_{1}

penalization is more effective, leading to notable performance gains in cube-double and scene environments. Results are averaged across four seeds and five evaluation tasks.

Explore similar work

Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning
May 27, 2026Kaiqiang Ke, Shenghong He, Chengdong Xu +3Goal-Conditioned Reinforcement Learning Long-Horizon Task Planning
1Sun Yat-sen University · 2Pengcheng Laboratory
Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL
Feb 3, 2026Jinwoo Choi, Sang-Hyun Lee, Seung-Woo SeoGoal-Conditioned Reinforcement Learning High-Level Subgoal Generation
Department of Electrical and Computer Engineering, Seoul National University, Seoul, South Korea · Department of Automotive Engineering, Ajou University, Gyeonggi-do, South Korea
Learning Multiple Timescales for Goal-Conditioned Reinforcement Learning
Oct 1, 2026Pedro Robles Dutenhefner, Dikshant Shehmar, Wagner Meira +1Goal-Conditioned Reinforcement Learning Timescale
Universidade Federal de Minas Gerais · University of Alberta · Alberta Machine Intelligence Institute (Amii) +1

May 27, 2026, cs.LGJ/K move · Enter open · S save

Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning

Kaiqiang Ke, Shenghong He, Chengdong Xu, Yuheng Luo+2

1Sun Yat-sen University · 2Pengcheng Laboratory

Goal-Conditioned Reinforcement Learning Long-Horizon Task Planning

Open paper AskPDF arXiv

Offline goal-conditioned reinforcement learning (GCRL) is challenging in long-horizon tasks, where distant state--goal pairs provide weak supervision and value estimates become vulnerable to accumulated bootstrapping errors. Hierarchical methods mitigate this difficulty by introducing intermediate subgoals, but fixed temporal abstractions or fixed hierarchy depths can be mismatched to state--goal pairs with different reachability horizons. We propose Coarse-to-Fine Hierarchical Goal Reinforcement Learning (CFHRL), a fully offline GCRL framework that adaptively refines distant goals before execution. Starting from the final goal, CFHRL recursively proposes intermediate targets, trained from replay-supported candidates, and stops refinement once the current target is estimated to be locally executable by a learned reachability cost. The key idea is that a subgoal need not be an exact midpoint or globally optimal waypoint; it only needs to provide reliable progress and reduce the remaining reaching difficulty, enabling subsequent refinement over shorter horizons. A stylized analysis further supports the robustness of approximate recursive contraction. Experiments on OGBench show substantial gains on several long-horizon tasks, with ablations validating the proposed refinement and stopping mechanisms

Abstract

Explore similar work

Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning

Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL

Learning Multiple Timescales for Goal-Conditioned Reinforcement Learning