Learning Multiple Timescales for Goal-Conditioned Reinforcement Learning

Pedro Robles Dutenhefner; Dikshant Shehmar; Wagner Meira; Marlos C. Machado

cs.AIOct 1, 2026

Learning Multiple Timescales for Goal-Conditioned Reinforcement Learning

Authors: Pedro Robles Dutenhefner, Dikshant Shehmar, Wagner Meira, Marlos C. Machado

Organizations: Universidade Federal de Minas Gerais · University of Alberta · Alberta Machine Intelligence Institute (Amii) · Canada CIFAR AI Chair

Abstract

Existing approaches to offline goal-conditioned reinforcement learning (GCRL) struggle with long-horizon tasks. Discounting shrinks value differences between distant states until they fall below the function approximation error, leaving the agent with no signal for ranking states. Temporal abstraction, which treats k environment steps as a single transition, restores this signal at long range, but no single fixed k suits all state-goal distances: large k preserves value differences across long temporal distances while collapsing distinctions between nearby states, and small k does the reverse. We make this trade-off explicit and introduce Generalized Implicit Temporal Abstraction (GITA), which conditions a single value function on k. GITA trains one policy by aggregating advantage-weighted supervision across multiple k values, so scales assigning larger positive advantages to a state-goal pair contribute more strongly to its update. GITA does not need to choose between local resolution and long-range signal; it retains both without committing to a single k. On OGBench, GITA outperforms a broad range of offline GCRL baselines, raising average success rate across all tasks by 25 percentage points (73% relative improvement) over HIQL. It also improves over the strongest fixed-k method, OTA, by 7 percentage points (14% relative).

Figures & tables

Figure 1: Overview of the proposed multi-scale temporal abstraction framework. Left : a shared value function conditioned on $k$ learns multiple abstraction scales. Right : scale-specific advantages set the relative contribution of each $k$ value, inducing an implicit soft assignment when training the shared high-level policy.

Figure 2: No single abstraction factor is best at all distances. APC for different abstraction factors

k

as the goal becomes more distant (mean

\pm

std over four seeds). The strip beneath each panel indicates the factor with the highest correlation in each distance regime; the best factor consistently grows with distance, but its range differs across environments. Inset shows the learned value

V^{h}_{k}(s,g)

as a function of geodesic distance.

Figure 3: Varying $k$ directly controls the temporal abstraction scale. Normalized value function $V(s,g;k)$ on pointmaze-large-navigate , evaluated at every navigable cell for $k\in\{1,2,3,5,13,21\}$ . The star marks the fixed reference state. As $k$ increases, the value-sensitive region expands over longer distances while local resolution progressively degrades.

Figure 4: The abstraction scale receiving the strongest policy weight changes with state-goal distance. Clipped scale-specific AWR weight $\min\{\exp(\beta_{h}A^{h}(s_{t},s_{t+m},g;k)),100\}$ across geodesic state-goal distance bins in antmaze-giant-navigate , for each $k$ .

			Non-Hierarchical					Hierarchical
Environment	Type	Size	GCBC	GCIVL	GCIQL	QRL	CRL	HIQL	OTA	GITA
PointMaze	navigate	large	$29$ $\pm 6$	$45$ $\pm 5$	$34$ $\pm 3$	$\mathbf{86}$ $\mathbf{\pm 9}$	$39$ $\pm 7$	$58$ $\pm 5$	$85$ $\pm 5$	$\mathbf{93}$ $\mathbf{\pm 2}$
	navigate	giant	$1$ $\pm 2$	$0$ $\pm 0$	$0$ $\pm 0$	$68$ $\pm 7$	$27$ $\pm 10$	$46$ $\pm 9$	$72$ $\pm 6$	$\mathbf{80}$ $\mathbf{\pm 3}$
	stitch	large	$7$ $\pm 5$	$12$ $\pm 6$	$31$ $\pm 2$	$\mathbf{84}$ $\mathbf{\pm 15}$	$0$ $\pm 0$	$13$ $\pm 6$	$46$ $\pm 7$	$\mathbf{72}$ $\mathbf{\pm 13}$
	stitch	giant	$0$ $\pm 0$	$0$ $\pm 0$	$0$ $\pm 0$	$50$ $\pm 8$	$0$ $\pm 0$	$0$ $\pm 0$	$44$ $\pm 8$	$\mathbf{60}$ $\mathbf{\pm 7}$
AntMaze	navigate	large	$24$ $\pm 2$	$16$ $\pm 5$	$34$ $\pm 4$	$75$ $\pm 6$	$83$ $\pm 4$	$\mathbf{91}$ $\mathbf{\pm 2}$	$\mathbf{91}$ $\mathbf{\pm 1}$	$\mathbf{91}$ $\mathbf{\pm 1}$

Table 1: GITA outperforms fixed-scale and non-abstracted baselines across OGBench tasks. Success rate (%), mean

\pm

standard deviation over 8 seeds (4 for pixel-based tasks). The best mean, and any result within one standard deviation of it, are in bold .

Environment	Type	Size	Unconditioned Value	Highest Advantage Factor	Mean Advantage Before Exponentiation	GITA with Geometric $\bm{k}$	GITA
PointMaze	navigate	large	$69$ $\pm 10$	$92$ $\pm 4$	$94$ $\pm 3$	$95$ $\pm 2$	$92$ $\pm 6$
	navigate	giant	$40$ $\pm 13$	$77$ $\pm 6$	$84$ $\pm 5$	$81$ $\pm 5$	$82$ $\pm 5$
	stitch	large	$37$ $\pm 16$	$67$ $\pm 16$	$80$ $\pm 9$	$65$ $\pm 15$	$73$ $\pm 17$
	stitch	giant	$58$ $\pm 9$	$54$ $\pm 15$	$55$ $\pm 16$	$45$ $\pm 10$	$56$ $\pm 13$
AntMaze	navigate	large	$89$ $\pm 2$	$92$ $\pm 3$	$91$ $\pm 2$	$90$ $\pm 2$	$91$ $\pm 2$
	navigate	giant	$62$ $\pm 4$	$78$ $\pm 4$	$77$ $\pm 3$	$80$ $\pm 4$	$79$ $\pm 3$

Table 2: Every variant degrades sharply somewhere; GITA does not. Success rate (%) on the maze tasks, mean

\pm

standard deviation over 8 seeds.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Maze layouts. OGBench provides the medium , large , and giant layouts considered in our experiments, with progressively longer state-goal distances.

Figure 6: OGBench datasets exhibit distinct trajectory structures. Representative trajectories from the OGBench offline dataset variants in the large maze. The navigate dataset contains long goal-directed trajectories, stitch consists of shorter local trajectory segments that must be combined to solve long-horizon tasks, and explore contains high-coverage but highly suboptimal behavior.

Environment	Variant	Observation	Action Dim.	Max. Episode Length
PointMaze	large	state (2)	2	1000
PointMaze	giant	state (2)	2	1000
AntMaze	large	state (29) / pixels ( $64\times 64\times 3$ )	8	1000
AntMaze	giant	state (29) / pixels ( $64\times 64\times 3$ )	8	1000
HumanoidMaze	large	state (69)	21	2000
HumanoidMaze	giant	state (69)	21	4000

Appendix

Table 3: Environment specifications. Specifications for the OGBench domains considered in our experiments.

Environment	Dataset	Variant	# Transitions	# Episodes	Data Episode Length
PointMaze	navigate	large	1M	1000	1000
	navigate	giant	1M	500	2000
	stitch	large	1M	5000	200
	stitch	giant	1M	5000	200
AntMaze	navigate	large	1M	1000	1000
	navigate	giant	1M	500	2000

Appendix

Table 4: Offline dataset specifications. Specifications of the OGBench offline datasets considered in our experiments. Visual-AntMaze uses the same trajectory datasets and statistics as the corresponding AntMaze configuration.

– Optimization and Architecture –
Hyperparameter	Value
Learning rate	$3\times 10^{-4}$
Optimizer	Adam
Gradient steps	$10^{6}$ (state), $5\times 10^{5}$ (pixels)
Minibatch size	$1024$ (state), $256$ (pixels)
Actor hidden dimensions	$(512,512,512)$

Appendix

Table 5: Hyperparameters used for GITA. Unless marked as GITA-specific, the values follow the HIQL reference implementation in OGBench.

Comparison	$W$	Raw $p$	$p_{\mathrm{adj}}$
GITA vs. GCBC	$0.0$	$2.69\times 10^{-5}$	$1.88\times 10^{-4}$
GITA vs. GCIVL	$14.0$	$1.62\times 10^{-4}$	$8.08\times 10^{-4}$
GITA vs. GCIQL	$18.5$	$2.78\times 10^{-4}$	$8.08\times 10^{-4}$
GITA vs. QRL	$6.5$	$6.31\times 10^{-5}$	$3.79\times 10^{-4}$
GITA vs. CRL	$17.0$	$2.32\times 10^{-4}$	$8.08\times 10^{-4}$
GITA vs. HIQL	$7.0$	$1.62\times 10^{-4}$	$8.08\times 10^{-4}$

Appendix

Table 6: Two-sided paired Wilcoxon signed-rank tests comparing GITA with the baselines in Table 1 . Each test is performed over the

23

tasks reported in the main results table.

p_{\mathrm{adj}}

denotes the Holm–Bonferroni-adjusted

p

-value across the seven comparisons.

Comparison	$W$	Raw $p$	$p_{\mathrm{adj}}$
GITA vs. Unconditioned Value	$10.0$	$0.0100$	$0.0300$
GITA vs. Highest Advantage Factor	$5.0$	$0.00537$	$0.0215$
GITA vs. Mean Advantage Before Exponentiation	$30.5$	$0.873$	$0.873$
GITA vs. Geometric $k$	$21.5$	$0.186$	$0.371$

Appendix

Table 7: Two-sided paired Wilcoxon signed-rank tests comparing GITA with the variants in Table 2 .

p_{\mathrm{adj}}

denotes the Holm–Bonferroni-adjusted

p

-value across the four comparisons.

Figure 7: Scale-specific AWR weights across navigation and stitching tasks. Mean clipped exponentiated high-level advantage,

\min\{\exp(\beta_{h}A^{h}),100\}

, as a function of geodesic state–goal distance for different abstraction factors

k

. From top to bottom: large-navigate , giant-navigate , large-stitch , and giant-stitch . Curves show the mean across seeds and shaded regions indicate one standard deviation. Each plot contains the corresponding maze morphologies.

Environment	Type	Size	$\bm{\gamma=0.98}$	$\bm{\gamma=0.99}$	$\bm{\gamma=0.995}$	Best $\bm{\gamma}$ per Task	Best Global $\bm{\gamma=0.99}$
AntMaze	navigate	large	$91.0$ $\pm 1.6$	$89.0$ $\pm 1.9$	$0.7$ $\pm 0.8$	$91.0$ $\pm 1.6$ $(0.98)$	$89.0$ $\pm 1.9$
	navigate	giant	$78.8$ $\pm 2.7$	$65.8$ $\pm 2.6$	$2.2$ $\pm 3.0$	$78.8$ $\pm 2.7$ $(0.98)$	$65.8$ $\pm 2.6$
	stitch	large	$85.6$ $\pm 2.9$	$60.2$ $\pm 27.8$	$0.1$ $\pm 0.4$	$85.6$ $\pm 2.9$ $(0.98)$	$60.2$ $\pm 27.8$
	stitch	giant	$41.5$ $\pm 5.8$	$16.6$ $\pm 13.2$	$0.0$ $\pm 0.0$	$41.5$ $\pm 5.8$ $(0.98)$	$16.6$ $\pm 13.2$
	explore	large	$65.9$ $\pm 12.5$	$87.0$ $\pm 8.8$	$50.8$ $\pm 21.9$	$87.0$ $\pm 8.8$ $(0.99)$	$87.0$ $\pm 8.8$
HumanoidMaze	navigate	large	$50.8$ $\pm 7.1$	$72.2$ $\pm 4.6$	$68.2$ $\pm 2.8$	$72.2$ $\pm 4.6$ $(0.99)$	$72.2$ $\pm 4.6$

Appendix

Table 8: Success rate (%) of GITA for each discount factor considered in our sweep. Results use only the final checkpoint after

10^{6}

gradient steps and are reported as mean

\pm

standard deviation over 8 seeds. The Best

\gamma

per Task column reports the best task-level result together with the corresponding discount factor in parentheses. The best global discount factor is

\gamma=0.99

, selected by the average performance across all tasks; the last column reports the corresponding task-level performance.

Explore similar work

Sep 27, 2026cs.LG

GTRL: Grounding Divide-and-Conquer Value Learning with Temporal Differences

In offline goal-conditioned reinforcement learning (GCRL), divide-and-conquer scales to long horizons by joining two shorter segments at a subgoal. However, under stochastic dynamics, the base case of this rule values the luckiest trajectories through the data. The subgoal must also lie on a shared trajectory, so a state-goal pair that no trajectory connects gets no value update at all. To address both, we present Grounded Transitive RL (GTRL), an offline GCRL value learning algorithm that grounds the divide-and-conquer update with a one-step TD target. Over a single step, TD is correct, as its target averages over the successors and needs no subgoal. GTRL adds this target to the composition rather than replacing it, so every pair receives an update, and the composition still carries the long horizon. GTRL also corrects the bias from hindsight relabeling by reweighting each goal against how reachable it was from other successors. We evaluate our algorithm on nineteen OGBench tasks spanning stochastic, deterministic, and stitching environments, where it achieves the highest average success rate. Code will be released soon.

Abdul Monaf Chowdhury, MD Sameer Iqbal Chowdhury, Shifat E Arman +1

Department of Robotics and Mechatronics Engineering, University of Dhaka, Bangladesh · Department of Computer Science, Texas State University, USA · Department of Computer Science, University of Oxford, UK

Goal-Conditioned Reinforcement Learning Temporal Difference State-Tracking Value

Ask PDF

May 27, 2026cs.LG

Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning

Offline goal-conditioned reinforcement learning (GCRL) is challenging in long-horizon tasks, where distant state--goal pairs provide weak supervision and value estimates become vulnerable to accumulated bootstrapping errors. Hierarchical methods mitigate this difficulty by introducing intermediate subgoals, but fixed temporal abstractions or fixed hierarchy depths can be mismatched to state--goal pairs with different reachability horizons. We propose Coarse-to-Fine Hierarchical Goal Reinforcement Learning (CFHRL), a fully offline GCRL framework that adaptively refines distant goals before execution. Starting from the final goal, CFHRL recursively proposes intermediate targets, trained from replay-supported candidates, and stops refinement once the current target is estimated to be locally executable by a learned reachability cost. The key idea is that a subgoal need not be an exact midpoint or globally optimal waypoint; it only needs to provide reliable progress and reduce the remaining reaching difficulty, enabling subsequent refinement over shorter horizons. A stylized analysis further supports the robustness of approximate recursive contraction. Experiments on OGBench show substantial gains on several long-horizon tasks, with ablations validating the proposed refinement and stopping mechanisms

Kaiqiang Ke, Shenghong He, Chengdong Xu +3

1Sun Yat-sen University · 2Pengcheng Laboratory

Goal-Conditioned Reinforcement Learning Long-Horizon Task Planning

Ask PDF

May 25, 2026cs.LG

Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning

Offline goal-conditioned reinforcement learning (GCRL) provides a practical framework for obtaining goal-reaching policies from fixed datasets. However, learning a reliable goal-conditioned value function in long-horizon tasks remains challenging. In this paper, we identify erroneous generalization in goal-conditioned value functions as a fundamental bottleneck, and demonstrate that appropriate inductive bias in the value function is crucial for addressing the bottleneck. Building on these findings, we propose Latent-Aligned Value Learning (LAVL), an offline GCRL algorithm that integrates latent-representation-based value generalization with hierarchical planning in a unified framework. Extensive experiments on OGBench demonstrate that LAVL consistently outperforms existing offline GCRL methods, achieving the highest performance on 20 out of 22 datasets. Notably, LAVL exhibits strong performance in long-horizon tasks and trajectory stitching datasets, where prior methods suffer significant performance degradation. Our code is available at https://github.com/oh-lab/LAVL.git.

Hyungkyu Kang, Byeongchan Kim, Min-hwan Oh

Seoul National University, Seoul, South Korea.

Goal-Conditioned Reinforcement Learning Goal-Conditioned Value Function Latent Representation Alignment Long-Horizon Task Planning

Ask PDF