An Informational Curse of Horizon in Goal-Conditioned Policy Learning

John L. Zhou; Yuxuan Dong; Jonathan C. Kao

cs.LGOct 7, 2026

An Informational Curse of Horizon in Goal-Conditioned Policy Learning

Authors: John L. Zhou, Yuxuan Dong, Jonathan C. Kao

Organizations: University of California, Los Angeles

Abstract

The difficulty of learning goal-reaching policies is often attributed to a "curse of horizon" that manifests as bias accumulation in temporal-difference backups and noisy advantage estimates. In this work, we identify an additional informational curse of horizon in goal-conditioned policy learning, where increasing the goal relabeling horizon can significantly reduce policy generalization and performance. Through a series of controlled experiments with oracle planners, we decouple the goal horizons sampled during training from those that the policy is asked to reach at test time. Even when evaluated only on a sequence of nearby subgoals, goal-conditioned behavioral cloning (BC) policies suffer from severe, training horizon-dependent performance degradation that is mitigated by reinforcement learning (RL) objectives. We explain this phenomenon as a horizon-dependent decrease in the conditional mutual information between actions and hindsight-relabeled goals, and find empirically that both BC and RL policies trained on longer-horizon goals exhibit a shift in sensitivity from goal to state information, as measured by the policy's input Jacobians. Motivated by this observation, we find that distilling the input Jacobians of short-horizon policies into long-horizon policies yields significant performance gains, especially in combinatorial manipulation tasks. Taken together, our results highlight goal relabeling horizon as an important consideration when learning generalist policies from offline data.

Figures & tables

Figure 1: Training on more distant relabeled goals harms nearby goal-reaching capabilities . We record the success rates across policies trained on a geometric distribution of relabeling goal offsets with discount

\gamma

, where

\gamma=1

corresponds to a uniform distribution over future states. At evaluation, policies are given optimal

H_{\mathrm{plan}}

-step subgoals from oracle planners . Success rates are averaged over the last three checkpoints, three seeds, and four environments: cube-double , cube-triple , cube-quadruple , and scene .

Figure 2: Increasing training goal horizon globally decreases policy goal sensitivity . We measure the proportion of the input Jacobian norm corresponding to the goal at fixed state-goal offsets, plotted with

\log_{2}

scaling in cube-triple . Results are averaged over the last three training checkpoints, three seeds, and 1024 randomly sampled state-goal pairs per horizon, with additional environments in Figure 8 .

Environment	Dataset	GCBC	GCIVL	GCIQL	QRL	CRL	HIQL	SAW	S 2 AW
pointmaze	pointmaze-medium-navigate-v0	$9$ $\pm 6$	$63$ $\pm 6$	$53$ $\pm 8$	$82$ $\pm 5$	$29$ $\pm 7$	$79$ $\pm 5$	$\mathbf{97}$ $\pm 2$	$\mathbf{96}$ $\pm 3$
	pointmaze-large-navigate-v0	$29$ $\pm 6$	$45$ $\pm 5$	$34$ $\pm 3$	$\mathbf{86}$ $\pm 9$	$39$ $\pm 7$	$58$ $\pm 5$	$\mathbf{85}$ $\pm 10$	$\mathbf{83}$ $\pm 8$
	pointmaze-giant-navigate-v0	$1$ $\pm 2$	$0$ $\pm 0$	$0$ $\pm 0$	${68}$ $\pm 7$	$27$ $\pm 10$	$46$ $\pm 9$	${68}$ $\pm 8$	$\mathbf{73}$ $\pm 9$
antmaze	antmaze-medium-navigate-v0	$29$ $\pm 4$	$72$ $\pm 8$	$71$ $\pm 4$	$88$ $\pm 3$	$\mathbf{95}$ $\pm 1$	$\mathbf{96}$ $\pm 1$	$\mathbf{97}$ $\pm 1$	$\mathbf{97}$ $\pm 1$
	antmaze-large-navigate-v0	$24$ $\pm 2$	$16$ $\pm 5$	$34$ $\pm 4$	$75$ $\pm 6$	$83$ $\pm 4$	$\mathbf{91}$ $\pm 2$	$\mathbf{90}$ $\pm 3$	$\mathbf{93}$ $\pm 2$
	antmaze-giant-navigate-v0	$0$ $\pm 0$	$0$ $\pm 0$	$0$ $\pm 0$	$14$ $\pm 3$	$16$ $\pm 3$	${65}$ $\pm 5$	${73}$ $\pm 4$	$\mathbf{87}$ $\pm 3$

Table 1: Evaluating Sobolev bootstrapping on offline goal-conditioned RL tasks. We compare our method’s average (binary) success rate (

\%

) against the numbers reported in Park et al. (2024) and Zhou and Kao (2025) across the five test-time goals for each environment, averaged over the last three checkpoints and eight seeds

\pm

standard deviations. Numbers within

5\%

of the best value in the row are in bold .

Figure 3: Sobolev bootstrapping exhibits more interpretable sensitivity structure . We plot the proportion of the total input Jacobian norm corresponding to each cube during a triple pick-and-place on-policy rollout for SAW without (top) and with (bottom) Sobolev bootstrapping. For fairness, we also analyze a scripted optimal trajectory in Figure 11 .

Figure 4: Sobolev bootstrapping recovers policy goal sensitivity . We compare the absolute goal Jacobian norm between S 2 AW and SAW, averaging over batches of goals sampled at fixed intervals from the current state up to the maximum horizon. Results are averaged across four seeds and 1024 state-goal pairs for each horizon, with shaded standard deviations.

Environment	GCIQL	GCIQL ${}_{\ell_{1}}$	SAW	SAW ${}_{\ell_{1}}$	SAW ${}_{\ell_{2}}$	S 2 AW
cube-single	$68$ $\pm 6$	$74$ $\pm 4$	$76$ $\pm 8$	$\mathbf{85}$ $\pm 6$	$\mathbf{85}$ $\pm 4$	$\mathbf{86}$ $\pm 5$
cube-double-task1	$74$ $\pm 8$	$76$ $\pm 8$	$64$ $\pm 9$	$\mathbf{81}$ $\pm 9$	${65}$ $\pm 14$	$\mathbf{78}$ $\pm 12$
cube-triple-task1	$13$ $\pm 3$	$16$ $\pm 10$	$18$ $\pm 11$	$24$ $\pm 9$	$17$ $\pm 12$	$\mathbf{74}$ $\pm 9$
cube-quadruple-task6	$0$ $\pm 1$	$0$ $\pm 1$	$2$ $\pm 2$	$2$ $\pm 2$	$1$ $\pm 1$	$\mathbf{35}$ $\pm 14$

Table 2: Sobolev bootstrapping improves single pick-and-place performance in the presence of distractor objects. We evaluate policies trained in multi-cube environments on single pick-and-place tasks. While naive

\ell_{1}

\ell_{2}

Jacobian regularization [Appendix H.4 ] yields modest improvements on simpler tasks, Sobolev bootstrapping matches or outperforms all baselines. Other settings are identical to Table 1 .

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Training on distant relabeled goals harms nearby goal-reaching capabilities . We record the success rates across learned policies trained on a uniform distribution of relabeling goal offsets

[1,H_{\mathrm{train}}]

, where

H_{\mathrm{train}}=\infty

corresponds to a uniform distribution over all future in-trajectory states. At evaluation, policies are given optimal

H_{\mathrm{plan}}

-step subgoals from oracle planners. Success rates are averaged over the last three checkpoints and three seeds.

Figure 6: Training on a larger proportion of distant relabeled goals harms nearby goal-reaching capabilities . We record the success rates across learned policies trained on a geometric distribution of relabeling goal offsets with discount

\gamma

, where

\gamma=1

corresponds to a uniform distribution over future states. At evaluation, policies are given optimal

H_{\mathrm{plan}}

-step subgoals from oracle planners. Success rates are averaged over the last three checkpoints and three seeds.

Figure 7: Training on more random relabeled goals harms nearby goal-reaching capabilities . We record the success rates across learned policies trained on a mixture of goals either sampled from a uniform distribution over future states or from an arbitrary dataset state with probability

p_{\mathrm{rand}}\in\{0.25,0.50,0.75\}

. At evaluation, policies are given optimal

H_{\mathrm{plan}}

-step subgoals from oracle planners. Success rates are averaged over the last three checkpoints and three seeds.

Figure 8: Increasing training goal horizon globally decreases policy goal sensitivity . We measure the proportion of the input Jacobian norm corresponding to the goal at fixed state-goal offsets, plotted with

\log_{2}

scaling. Results are averaged over the last three training checkpoints, three seeds, and 1024 randomly sampled state-goal pairs per horizon.

Figure 9: Training on distant relabeled goals globally decreases policy goal sensitivity . We measure the proportion of the input Jacobian norm corresponding to the goal at fixed state-goal offsets, plotted with

\log_{2}

scaling. Results are averaged over the last three training checkpoints, three seeds, and 1024 randomly sampled state-goal pairs per horizon.

Figure 10: Random goal sampling globally decreases policy goal sensitivity. We directly reduce the action-goal MI by replacing in-trajectory goals with randomly sampled dataset states with probability

p_{\text{random}}\in\{0.25,0.5,0.75\}

, and measure the goal Jacobian norm proportion at fixed state-goal offsets.

Figure 11: Sobolev bootstrapping pays attention to the correct object during optimal trajectories. To enable fair comparisons across later task stages, we plot the input Jacobian norms with respect to each object over an independent scripted optimal trajectory Ahn et al. (2025) for the cube-triple triple pick-and-place task. S 2 AW (top) allocates a higher proportion of the Jacobian norm to the focal cube both during approach and grasp, whereas SAW’s (bottom) Jacobian norm only changes after the cube is already grasped. The shaded area denotes when the cube is actively grasped.

Dataset	Expectile $\tau$	AWR $\alpha$	KLD $\beta$	Subgoal steps $k$	Sobolev weight $\lambda$
pointmaze-medium-navigate-v0	$0.7$	$3.0$	$3.0$	$25$	$0.03$
pointmaze-large-navigate-v0	$0.7$	$3.0$	$3.0$	$25$	$1.00$
pointmaze-giant-navigate-v0	$0.7$	$3.0$	$3.0$	$25$	$1.00$
antmaze-medium-navigate-v0	$0.7$	$3.0$	$3.0$	$25$	$0.03$
antmaze-large-navigate-v0	$0.7$	$3.0$	$3.0$	$25$	$0.03$
antmaze-giant-navigate-v0	$0.7$	$3.0$	$3.0$	$25$	$0.30$

Appendix

Table 3: Hyperparameters for SAW with Sobolev bootstrapping.

Dataset	$\lambda=\mathbf{0}$	$\mathbf{0.001}$	$\mathbf{0.01}$	$\mathbf{0.1}$
cube-double-play-v0	$40$ $\pm 7$	$40$ $\pm 11$	${55}$ $\pm 7$	$35$ $\pm 9$
cube-triple-play-v0	$4$ $\pm 2$	$9$ $\pm 2$	${30}$ $\pm 7$	$23$ $\pm 10$
scene-play-v0	$63$ $\pm 6$	$54$ $\pm 10$	${71}$ $\pm 5$	$62$ $\pm 5$
antmaze-giant-navigate-v0	$73$ $\pm 4$	$76$ $\pm 4$	$82$ $\pm 4$	${85}$ $\pm 4$

Appendix

Table 4: Sobolev loss weight sensitivity analysis. Sobolev bootstrapping is robust to different weighting factors within a reasonable scale (around 0.01 for cube-double and scene ). Notably, improvements on antmaze-giant and cube-triple are robust across various weighting scales. Experimental results are averaged across four seeds.

hours / 1M iters	GCIQL	SAW	S 2 AW
cube-quadruple-play	0.26	0.46	0.75
scene-play	0.25	0.45	0.74
humanoidmaze-giant-navigate	0.26	0.46	1.75

Appendix

Table 5: Training time comparison. We select the highest-dimensional environments and report the hours required to complete a standard 1M-step training run on an NVIDIA 5090 GPU.

Environment	Dataset	GCBC	GCIVL	GCIQL	QRL	CRL	HIQL	SAW	S 2 AW	S 2 AW ℓ
maze-giant	antmaze-giant-navigate-v0	$0$ $\pm 0$	$0$ $\pm 0$	$0$ $\pm 0$	$14$ $\pm 3$	$16$ $\pm 3$	${65}$ $\pm 5$	${73}$ $\pm 4$	$\mathbf{87}$ $\pm 3$	${80}$ $\pm 5$
maze-giant	humanoidmaze-giant-navigate-v0	$0$ $\pm 0$	$0$ $\pm 0$	$0$ $\pm 0$	$1$ $\pm 0$	$3$ $\pm 2$	${12}$ $\pm 4$	${35}$ $\pm 4$	$\mathbf{38}$ $\pm 6$	${35}$ $\pm 5$
cube	cube-single-play-v0	$6$ $\pm 2$	$53$ $\pm 4$	${68}$ $\pm 6$	$5$ $\pm 1$	$19$ $\pm 2$	$44$ $\pm 9$	${72}$ $\pm 5$	$\mathbf{86}$ $\pm 5$	${79}$ $\pm 8$
	cube-double-play-v0	$1$ $\pm 1$	$36$ $\pm 3$	${40}$ $\pm 5$	$1$ $\pm 0$	$10$ $\pm 2$	$6$ $\pm 2$	${40}$ $\pm 7$	${50}$ $\pm 6$	$\mathbf{55}$ $\pm 8$
	cube-triple-play-v0	$1$ $\pm 1$	$1$ $\pm 0$	$3$ $\pm 1$	$0$ $\pm 0$	$\mathbf{4}$ $\pm 1$	$3$ $\pm 1$	${4}$ $\pm 2$	$\mathbf{30}$ $\pm 5$	$\mathbf{30}$ $\pm 4$
	cube-quadruple-play-v0	$0$ $\pm 0$	$0$ $\pm 0$	$0$ $\pm 0$	$0$ $\pm 0$	$0$ $\pm 0$	$0$ $\pm 0$	$0$ $\pm 0$	$\mathbf{3}$ $\pm 2$	${2}$ $\pm 1$

Appendix

Table 6: Architectural ablations to the Sobolev SAW target actor. We report the performance of Sobolev SAW where the target actor has the same architecture as the full policy, averaged over

4

seeds with standard deviations after the

\pm

sign. Numbers within

5\%

of the best value in the row are in bold .

Dataset	RIS	Sobolev RIS
pointmaze-medium-navigate-v0	$88$ $\pm 6$	$\mathbf{96}$ $\pm 1$
pointmaze-large-navigate-v0	$63$ $\pm 13$	$\mathbf{75}$ $\pm 3$
pointmaze-giant-navigate-v0	$57$ $\pm 12$	$\mathbf{67}$ $\pm 5$
antmaze-medium-navigate-v0	$\mathbf{96}$ $\pm 1$	$\mathbf{97}$ $\pm 1$
antmaze-large-navigate-v0	$\mathbf{89}$ $\pm 3$	$\mathbf{90}$ $\pm 1$
antmaze-giant-navigate-v0	$\mathbf{65}$ $\pm 4$	$\mathbf{69}$ $\pm 0$

Appendix

Table 7: Evaluating RIS with Sobolev bootstrapping. We evaluate the effectiveness of Sobolev bootstrapping on RIS by comparing it against the numbers reported in Zhou and Kao (2025) . Results are averaged over 4 seeds, and other settings are identical to Table 1 .

Environment	OTA n=5	OTA n=10	OTA n=20	SHARSA	DQC	S 2 AW
antmaze-giant-navigate	77 $\pm 4$	-	-	72 $\pm 1$	25 $\pm 6$	87 $\pm 3$
cube-double-play	2 $\pm 0$	3 $\pm 1$	2 $\pm 1$	5 $\pm 1$	24 $\pm 4$	57 $\pm 6$
cube-triple-play	1 $\pm 0$	0 $\pm 0$	1 $\pm 0$	2 $\pm 1$	11 $\pm 4$	30 $\pm 5$

Appendix

Table 8: Additional horizon reduction baselines . The additional baselines do not readily generalize to the standard datasets and perform poorly. All results are averaged across 4 seeds.

	SAW	SAW ${}_{\ell_{1}}$	SAW ${}_{\ell_{2}}$	S 2 AW
antmaze-giant-navigate	73 $\pm 4$	72 $\pm 3$	74 $\pm 2$	87 $\pm 3$
cube-double-play	40 $\pm 7$	47 $\pm 10$	37 $\pm 7$	57 $\pm 6$
cube-triple-play	4 $\pm 2$	5 $\pm 4$	4 $\pm 1$	30 $\pm 5$
scene-play	63 $\pm 6$	69 $\pm 8$	62 $\pm 2$	74 $\pm 6$

Appendix

Table 9: Uniform Jacobian regularization. We apply

\ell_{1}

and

\ell_{2}

norm penalization with coefficient

0.3

to the SAW baseline. We observe that

\ell_{1}

penalization is more effective, leading to notable performance gains in cube-double and scene environments. Results are averaged across four seeds and five evaluation tasks.

Explore similar work

May 27, 2026cs.LG

Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning

Offline goal-conditioned reinforcement learning (GCRL) is challenging in long-horizon tasks, where distant state--goal pairs provide weak supervision and value estimates become vulnerable to accumulated bootstrapping errors. Hierarchical methods mitigate this difficulty by introducing intermediate subgoals, but fixed temporal abstractions or fixed hierarchy depths can be mismatched to state--goal pairs with different reachability horizons. We propose Coarse-to-Fine Hierarchical Goal Reinforcement Learning (CFHRL), a fully offline GCRL framework that adaptively refines distant goals before execution. Starting from the final goal, CFHRL recursively proposes intermediate targets, trained from replay-supported candidates, and stops refinement once the current target is estimated to be locally executable by a learned reachability cost. The key idea is that a subgoal need not be an exact midpoint or globally optimal waypoint; it only needs to provide reliable progress and reduce the remaining reaching difficulty, enabling subsequent refinement over shorter horizons. A stylized analysis further supports the robustness of approximate recursive contraction. Experiments on OGBench show substantial gains on several long-horizon tasks, with ablations validating the proposed refinement and stopping mechanisms

Kaiqiang Ke, Shenghong He, Chengdong Xu +3

1Sun Yat-sen University · 2Pengcheng Laboratory

Goal-Conditioned Reinforcement Learning Long-Horizon Task Planning

Ask PDF

Feb 3, 2026cs.LG

Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL

Offline goal-conditioned reinforcement learning remains challenging for long-horizon tasks. While hierarchical approaches mitigate this issue by decomposing tasks, most existing methods rely on separate high- and low-level networks and generate only a single intermediate subgoal, leaving several structural limitations in long-horizon decision-making. To address this limitation, we draw inspiration from chain-of-thought reasoning and propose the Chain-of-Goals Hierarchical Policy (CoGHP), a novel framework that reformulates hierarchical decision-making as autoregressive sequence modeling within a unified architecture. Given a state and a final goal, CoGHP autoregressively generates a sequence of latent subgoals followed by the primitive action, where each latent subgoal acts as a reasoning step that conditions subsequent predictions. To implement this efficiently, we introduce an MLP-Mixer backbone, which supports cross-token communication and captures structural relationships among state, goal, latent subgoals, and action. Across challenging navigation and manipulation benchmarks, CoGHP consistently outperforms strong offline baselines, demonstrating improved performance on long-horizon tasks. Project page: https://wlsdn9350.github.io/projects/coghp/

Jinwoo Choi, Sang-Hyun Lee, Seung-Woo Seo

Department of Electrical and Computer Engineering, Seoul National University, Seoul, South Korea · Department of Automotive Engineering, Ajou University, Gyeonggi-do, South Korea

Goal-Conditioned Reinforcement Learning High-Level Subgoal Generation Autoregressive Model Chain-of-Thought Reasoning

Ask PDF

Oct 1, 2026cs.AI

Learning Multiple Timescales for Goal-Conditioned Reinforcement Learning

Existing approaches to offline goal-conditioned reinforcement learning (GCRL) struggle with long-horizon tasks. Discounting shrinks value differences between distant states until they fall below the function approximation error, leaving the agent with no signal for ranking states. Temporal abstraction, which treats k environment steps as a single transition, restores this signal at long range, but no single fixed k suits all state-goal distances: large k preserves value differences across long temporal distances while collapsing distinctions between nearby states, and small k does the reverse. We make this trade-off explicit and introduce Generalized Implicit Temporal Abstraction (GITA), which conditions a single value function on k. GITA trains one policy by aggregating advantage-weighted supervision across multiple k values, so scales assigning larger positive advantages to a state-goal pair contribute more strongly to its update. GITA does not need to choose between local resolution and long-range signal; it retains both without committing to a single k. On OGBench, GITA outperforms a broad range of offline GCRL baselines, raising average success rate across all tasks by 25 percentage points (73% relative improvement) over HIQL. It also improves over the strongest fixed-k method, OTA, by 7 percentage points (14% relative).

Pedro Robles Dutenhefner, Dikshant Shehmar, Wagner Meira +1

Universidade Federal de Minas Gerais · University of Alberta · Alberta Machine Intelligence Institute (Amii) +1

Goal-Conditioned Reinforcement Learning Timescale Long-Horizon Task Planning Temporal Reasoning

Ask PDF