Learning Multiple Timescales for Goal-Conditioned Reinforcement Learning

Pedro Robles Dutenhefner; Dikshant Shehmar; Wagner Meira; Marlos C. Machado

cs.AIOct 1, 2026

Learning Multiple Timescales for Goal-Conditioned Reinforcement Learning

Authors: Pedro Robles Dutenhefner, Dikshant Shehmar, Wagner Meira, Marlos C. Machado

Organizations: Universidade Federal de Minas Gerais · University of Alberta · Alberta Machine Intelligence Institute (Amii) · Canada CIFAR AI Chair

Abstract

Existing approaches to offline goal-conditioned reinforcement learning (GCRL) struggle with long-horizon tasks. Discounting shrinks value differences between distant states until they fall below the function approximation error, leaving the agent with no signal for ranking states. Temporal abstraction, which treats k environment steps as a single transition, restores this signal at long range, but no single fixed k suits all state-goal distances: large k preserves value differences across long temporal distances while collapsing distinctions between nearby states, and small k does the reverse. We make this trade-off explicit and introduce Generalized Implicit Temporal Abstraction (GITA), which conditions a single value function on k. GITA trains one policy by aggregating advantage-weighted supervision across multiple k values, so scales assigning larger positive advantages to a state-goal pair contribute more strongly to its update. GITA does not need to choose between local resolution and long-range signal; it retains both without committing to a single k. On OGBench, GITA outperforms a broad range of offline GCRL baselines, raising average success rate across all tasks by 25 percentage points (73% relative improvement) over HIQL. It also improves over the strongest fixed-k method, OTA, by 7 percentage points (14% relative).

Figures & tables

Figure 1: Overview of the proposed multi-scale temporal abstraction framework. Left : a shared value function conditioned on $k$ learns multiple abstraction scales. Right : scale-specific advantages set the relative contribution of each $k$ value, inducing an implicit soft assignment when training the shared high-level policy.

Figure 2: No single abstraction factor is best at all distances. APC for different abstraction factors

k

as the goal becomes more distant (mean

\pm

std over four seeds). The strip beneath each panel indicates the factor with the highest correlation in each distance regime; the best factor consistently grows with distance, but its range differs across environments. Inset shows the learned value

V^{h}_{k}(s,g)

as a function of geodesic distance.

Figure 3: Varying $k$ directly controls the temporal abstraction scale. Normalized value function $V(s,g;k)$ on pointmaze-large-navigate , evaluated at every navigable cell for $k\in\{1,2,3,5,13,21\}$ . The star marks the fixed reference state. As $k$ increases, the value-sensitive region expands over longer distances while local resolution progressively degrades.

Figure 4: The abstraction scale receiving the strongest policy weight changes with state-goal distance. Clipped scale-specific AWR weight $\min\{\exp(\beta_{h}A^{h}(s_{t},s_{t+m},g;k)),100\}$ across geodesic state-goal distance bins in antmaze-giant-navigate , for each $k$ .

			Non-Hierarchical					Hierarchical
Environment	Type	Size	GCBC	GCIVL	GCIQL	QRL	CRL	HIQL	OTA	GITA
PointMaze	navigate	large	$29$ $\pm 6$	$45$ $\pm 5$	$34$ $\pm 3$	$\mathbf{86}$ $\mathbf{\pm 9}$	$39$ $\pm 7$	$58$ $\pm 5$	$85$ $\pm 5$	$\mathbf{93}$ $\mathbf{\pm 2}$
	navigate	giant	$1$ $\pm 2$	$0$ $\pm 0$	$0$ $\pm 0$	$68$ $\pm 7$	$27$ $\pm 10$	$46$ $\pm 9$	$72$ $\pm 6$	$\mathbf{80}$ $\mathbf{\pm 3}$
	stitch	large	$7$ $\pm 5$	$12$ $\pm 6$	$31$ $\pm 2$	$\mathbf{84}$ $\mathbf{\pm 15}$	$0$ $\pm 0$	$13$ $\pm 6$	$46$ $\pm 7$	$\mathbf{72}$ $\mathbf{\pm 13}$
	stitch	giant	$0$ $\pm 0$	$0$ $\pm 0$	$0$ $\pm 0$	$50$ $\pm 8$	$0$ $\pm 0$	$0$ $\pm 0$	$44$ $\pm 8$	$\mathbf{60}$ $\mathbf{\pm 7}$
AntMaze	navigate	large	$24$ $\pm 2$	$16$ $\pm 5$	$34$ $\pm 4$	$75$ $\pm 6$	$83$ $\pm 4$	$\mathbf{91}$ $\mathbf{\pm 2}$	$\mathbf{91}$ $\mathbf{\pm 1}$	$\mathbf{91}$ $\mathbf{\pm 1}$

Table 1: GITA outperforms fixed-scale and non-abstracted baselines across OGBench tasks. Success rate (%), mean

\pm

standard deviation over 8 seeds (4 for pixel-based tasks). The best mean, and any result within one standard deviation of it, are in bold .

Environment	Type	Size	Unconditioned Value	Highest Advantage Factor	Mean Advantage Before Exponentiation	GITA with Geometric $\bm{k}$	GITA
PointMaze	navigate	large	$69$ $\pm 10$	$92$ $\pm 4$	$94$ $\pm 3$	$95$ $\pm 2$	$92$ $\pm 6$
	navigate	giant	$40$ $\pm 13$	$77$ $\pm 6$	$84$ $\pm 5$	$81$ $\pm 5$	$82$ $\pm 5$
	stitch	large	$37$ $\pm 16$	$67$ $\pm 16$	$80$ $\pm 9$	$65$ $\pm 15$	$73$ $\pm 17$
	stitch	giant	$58$ $\pm 9$	$54$ $\pm 15$	$55$ $\pm 16$	$45$ $\pm 10$	$56$ $\pm 13$
AntMaze	navigate	large	$89$ $\pm 2$	$92$ $\pm 3$	$91$ $\pm 2$	$90$ $\pm 2$	$91$ $\pm 2$
	navigate	giant	$62$ $\pm 4$	$78$ $\pm 4$	$77$ $\pm 3$	$80$ $\pm 4$	$79$ $\pm 3$

Table 2: Every variant degrades sharply somewhere; GITA does not. Success rate (%) on the maze tasks, mean

\pm

standard deviation over 8 seeds.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Maze layouts. OGBench provides the medium , large , and giant layouts considered in our experiments, with progressively longer state-goal distances.

Figure 6: OGBench datasets exhibit distinct trajectory structures. Representative trajectories from the OGBench offline dataset variants in the large maze. The navigate dataset contains long goal-directed trajectories, stitch consists of shorter local trajectory segments that must be combined to solve long-horizon tasks, and explore contains high-coverage but highly suboptimal behavior.

Environment	Variant	Observation	Action Dim.	Max. Episode Length
PointMaze	large	state (2)	2	1000
PointMaze	giant	state (2)	2	1000
AntMaze	large	state (29) / pixels ( $64\times 64\times 3$ )	8	1000
AntMaze	giant	state (29) / pixels ( $64\times 64\times 3$ )	8	1000
HumanoidMaze	large	state (69)	21	2000
HumanoidMaze	giant	state (69)	21	4000

Appendix

Table 3: Environment specifications. Specifications for the OGBench domains considered in our experiments.

Environment	Dataset	Variant	# Transitions	# Episodes	Data Episode Length
PointMaze	navigate	large	1M	1000	1000
	navigate	giant	1M	500	2000
	stitch	large	1M	5000	200
	stitch	giant	1M	5000	200
AntMaze	navigate	large	1M	1000	1000
	navigate	giant	1M	500	2000

Appendix

Table 4: Offline dataset specifications. Specifications of the OGBench offline datasets considered in our experiments. Visual-AntMaze uses the same trajectory datasets and statistics as the corresponding AntMaze configuration.

– Optimization and Architecture –
Hyperparameter	Value
Learning rate	$3\times 10^{-4}$
Optimizer	Adam
Gradient steps	$10^{6}$ (state), $5\times 10^{5}$ (pixels)
Minibatch size	$1024$ (state), $256$ (pixels)
Actor hidden dimensions	$(512,512,512)$

Appendix

Table 5: Hyperparameters used for GITA. Unless marked as GITA-specific, the values follow the HIQL reference implementation in OGBench.

Comparison	$W$	Raw $p$	$p_{\mathrm{adj}}$
GITA vs. GCBC	$0.0$	$2.69\times 10^{-5}$	$1.88\times 10^{-4}$
GITA vs. GCIVL	$14.0$	$1.62\times 10^{-4}$	$8.08\times 10^{-4}$
GITA vs. GCIQL	$18.5$	$2.78\times 10^{-4}$	$8.08\times 10^{-4}$
GITA vs. QRL	$6.5$	$6.31\times 10^{-5}$	$3.79\times 10^{-4}$
GITA vs. CRL	$17.0$	$2.32\times 10^{-4}$	$8.08\times 10^{-4}$
GITA vs. HIQL	$7.0$	$1.62\times 10^{-4}$	$8.08\times 10^{-4}$

Appendix

Table 6: Two-sided paired Wilcoxon signed-rank tests comparing GITA with the baselines in Table 1 . Each test is performed over the

23

tasks reported in the main results table.

p_{\mathrm{adj}}

denotes the Holm–Bonferroni-adjusted

p

-value across the seven comparisons.

Comparison	$W$	Raw $p$	$p_{\mathrm{adj}}$
GITA vs. Unconditioned Value	$10.0$	$0.0100$	$0.0300$
GITA vs. Highest Advantage Factor	$5.0$	$0.00537$	$0.0215$
GITA vs. Mean Advantage Before Exponentiation	$30.5$	$0.873$	$0.873$
GITA vs. Geometric $k$	$21.5$	$0.186$	$0.371$

Appendix

Table 7: Two-sided paired Wilcoxon signed-rank tests comparing GITA with the variants in Table 2 .

p_{\mathrm{adj}}

denotes the Holm–Bonferroni-adjusted

p

-value across the four comparisons.

Figure 7: Scale-specific AWR weights across navigation and stitching tasks. Mean clipped exponentiated high-level advantage,

\min\{\exp(\beta_{h}A^{h}),100\}

, as a function of geodesic state–goal distance for different abstraction factors

k

. From top to bottom: large-navigate , giant-navigate , large-stitch , and giant-stitch . Curves show the mean across seeds and shaded regions indicate one standard deviation. Each plot contains the corresponding maze morphologies.

Environment	Type	Size	$\bm{\gamma=0.98}$	$\bm{\gamma=0.99}$	$\bm{\gamma=0.995}$	Best $\bm{\gamma}$ per Task	Best Global $\bm{\gamma=0.99}$
AntMaze	navigate	large	$91.0$ $\pm 1.6$	$89.0$ $\pm 1.9$	$0.7$ $\pm 0.8$	$91.0$ $\pm 1.6$ $(0.98)$	$89.0$ $\pm 1.9$
	navigate	giant	$78.8$ $\pm 2.7$	$65.8$ $\pm 2.6$	$2.2$ $\pm 3.0$	$78.8$ $\pm 2.7$ $(0.98)$	$65.8$ $\pm 2.6$
	stitch	large	$85.6$ $\pm 2.9$	$60.2$ $\pm 27.8$	$0.1$ $\pm 0.4$	$85.6$ $\pm 2.9$ $(0.98)$	$60.2$ $\pm 27.8$
	stitch	giant	$41.5$ $\pm 5.8$	$16.6$ $\pm 13.2$	$0.0$ $\pm 0.0$	$41.5$ $\pm 5.8$ $(0.98)$	$16.6$ $\pm 13.2$
	explore	large	$65.9$ $\pm 12.5$	$87.0$ $\pm 8.8$	$50.8$ $\pm 21.9$	$87.0$ $\pm 8.8$ $(0.99)$	$87.0$ $\pm 8.8$
HumanoidMaze	navigate	large	$50.8$ $\pm 7.1$	$72.2$ $\pm 4.6$	$68.2$ $\pm 2.8$	$72.2$ $\pm 4.6$ $(0.99)$	$72.2$ $\pm 4.6$

Appendix

Table 8: Success rate (%) of GITA for each discount factor considered in our sweep. Results use only the final checkpoint after

10^{6}

gradient steps and are reported as mean

\pm

standard deviation over 8 seeds. The Best

\gamma

per Task column reports the best task-level result together with the corresponding discount factor in parentheses. The best global discount factor is

\gamma=0.99

, selected by the average performance across all tasks; the last column reports the corresponding task-level performance.

Explore similar work

GTRL: Grounding Divide-and-Conquer Value Learning with Temporal Differences
Sep 27, 2026Abdul Monaf Chowdhury, MD Sameer Iqbal Chowdhury, Shifat E Arman +1Goal-Conditioned Reinforcement Learning Temporal Difference
Department of Robotics and Mechatronics Engineering, University of Dhaka, Bangladesh · Department of Computer Science, Texas State University, USA · Department of Computer Science, University of Oxford, UK
Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning
May 27, 2026Kaiqiang Ke, Shenghong He, Chengdong Xu +3Goal-Conditioned Reinforcement Learning Long-Horizon Task Planning
1Sun Yat-sen University · 2Pengcheng Laboratory
Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning
May 25, 2026Hyungkyu Kang, Byeongchan Kim, Min-hwan OhGoal-Conditioned Reinforcement Learning Goal-Conditioned Value Function
Seoul National University, Seoul, South Korea.

Sep 27, 2026, cs.LGJ/K move · Enter open · S save

GTRL: Grounding Divide-and-Conquer Value Learning with Temporal Differences

Abdul Monaf Chowdhury, MD Sameer Iqbal Chowdhury, Shifat E Arman, Md Mehedi Hasan

Department of Robotics and Mechatronics Engineering, University of Dhaka, Bangladesh · Department of Computer Science, Texas State University, USA · Department of Computer Science, University of Oxford, UK

Goal-Conditioned Reinforcement Learning Temporal Difference State-Tracking Value

Open paper AskPDF arXiv

In offline goal-conditioned reinforcement learning (GCRL), divide-and-conquer scales to long horizons by joining two shorter segments at a subgoal. However, under stochastic dynamics, the base case of this rule values the luckiest trajectories through the data. The subgoal must also lie on a shared trajectory, so a state-goal pair that no trajectory connects gets no value update at all. To address both, we present Grounded Transitive RL (GTRL), an offline GCRL value learning algorithm that grounds the divide-and-conquer update with a one-step TD target. Over a single step, TD is correct, as its target averages over the successors and needs no subgoal. GTRL adds this target to the composition rather than replacing it, so every pair receives an update, and the composition still carries the long horizon. GTRL also corrects the bias from hindsight relabeling by reweighting each goal against how reachable it was from other successors. We evaluate our algorithm on nineteen OGBench tasks spanning stochastic, deterministic, and stitching environments, where it achieves the highest average success rate. Code will be released soon.

Abstract

Explore similar work

GTRL: Grounding Divide-and-Conquer Value Learning with Temporal Differences

Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning

Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning