Learning accurate off-policy state-action value functions (Q-functions) from offline datasets becomes harder as the task horizon grows. One-step temporal-difference (TD) learning bootstraps after every transition, so small per-step errors accumulate along the horizon, and early states can be badly misvalued even when the TD loss is small. Multi-step TD shortens this chain by following the dataset trajectory for several steps; however, since the behavior policy is often suboptimal, every followed action adds a bias, and no fixed number of steps suits a whole trajectory. In this paper, we propose Horizon-Scalable Q-Learning (HQL), a simple method that enables accurate long-horizon value learning by automatically balancing between one-step and multi-step returns. HQL extends a target through a dataset action only when the critic values it at least as highly as the best-of-N actions sampled from the BC policy, and otherwise falls back to a one-step target. We show that this rule is a principled off-policy correction that needs no action densities, which makes HQL compatible with expressive flow and diffusion policies. Experiments show that HQL avoids the failures of both one-step and multi-step TD on a combination-lock task across horizons, and outperforms prior horizon-reduction methods on long-horizon OGBench and OCBench tasks.
Figures & tables
Figure 1: Horizon-Scalable Q-Learning (HQL). Purple arcs show the reach of one training target. One-step TD accumulates residuals, and multi-step TD inherits the bias of the suboptimal action it spans (yellow). HQL compares each logged action with the best-of- N policy samples, continuing the trace where it passes ( c=1 ) and cutting it where it fails ( c=0 , dashed), so reward travels far without inheriting off-policy bias.
Figure 2: HQL and Chunked-HQL. H sets how many decisions one target chains, and L how many environment steps each decision covers.
Figure 3: Combination lock. HQL remains accurate under data corruption as the horizon grows. The figure illustrates the RMSE between the learned Qθ and the true Q∗ , averaged uniformly over all 2H state–action pairs of the corrupted combination lock, after 1M updates (log scale). Panels vary the corruption fraction of the dataset. n-step TD ( n=64 ) diverges as soon as any corruption is present; 1-step TD is unaffected by corruption but its error grows sharply with the horizon H .
Figure 4: HQL versus baseline TD methods at matched critic update budget. Success rate averaged over the five tasks of each configuration and over 5 seeds, with standard deviation shown as the band; gray marks the offline phase. HQL outperforms all baselines on all six configurations, especially on the challenging long-horizon ones.
HQL
Chunked-HQL
LQL
QC
antmaze-giant
1 → 87 ± 2
0 → 86 ± 1
0 → 44 ± 4
0 → 46 ± 1
cube-double
67 → 87 ± 1
73 → 100 ± 0
45 → 83 ± 2
28 → 100 ± 0
cube-triple
12 → 54 ± 1
7 → 84 ± 1
5 → 35 ± 3
1 → 26 ± 3
hmaze-giant
10 → 28 ± 1
0 → 29 ± 2
12 → 26 ± 1
0 → 0 ± 0
hmaze-medium
56 → 86 ± 1
68 → 88 ± 2
48 → 86 ± 1
0 → 0 ± 0
scene
93 → 99 ± 0
98 → 100 ± 0
96 → 97 ± 1
97 → 100 ± 0
Table 1: HQL and Chunked-HQL against LQL and QC. Success rates (%) are averaged over five tasks and reported as offline → online with standard error over five seeds; we bold the best offline and the best online score at integer level. HQL and Chunked-HQL achieve the best results, especially on the long-horizon tasks.
Figure 6
cube-
cube-
antmz-
hmz-
double
triple
giant
giant
QAM
29 → 50
1 → 1
14 → 83
1 → 0
+ HQL
38 → 99
0 → 80
0 → 98
0 → 63
VGF
69 → 74
3 → 25
1 → 28
0 → 0
+ HQL
80 → 89
4 → 37
0 → 90
0 → 4
Table 2: HQL as a drop-in critic for different policy extraction methods. Success rates (%) on task 2, offline → online; we bold the better score of each pair. HQL improves both QAM and VGF on all four tasks.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: The combination lock dataset with linearly arranged H states.
Method
Seq. length
Seq. per update
Targets per seq.
Target
HQL
16
32
16
gated, every position
16-window TD
16
32
16
n -step, every position
8-step TD
8
512
1
8-step, first state
1-step TD
1
512
1
1-step
Appendix
Table 3: How each method forms its 512 critic targets per update.
Parameter
Value
Window length H
16
Windows per batch
32
Critic targets per update
512 ( =32×16 )
Gate width N (Eq. 7 )
4
Gate critic
target network Q
Discount factor
0.99 (0.995 for the two giant mazes)
Appendix
Table 4: Hyperparameters for the OGBench benchmark experiment (Figures 5 and 9 ).
Figure 9: Per-task view of the OGBench benchmark (Figure 5 ). Reading down a column shows how one task index behaves across environments, and a row shows how one environment varies over tasks.
Task group
Dataset size (transitions)
Episode length
Action dimension
Observation dimension
Default task
Locomotion ( locomaze )
pointmaze-medium
1M
1000
2
2
1
pointmaze-large
1M
1000
2
2
1
pointmaze-giant
1M
1000
2
2
1
pointmaze-teleport
1M
1000
2
2
1
antmaze-medium
1M
1000
8
29
1
Appendix
Table 5: OGBench state-based task characteristics. Episode length is the maximum number of environment steps per episode. Rows marked † are the groups used in this work.
Parameter
Value
Action chunk size L
25 (cut at episode boundaries)
Window length H (Chunked-HQL)
8 chunks ( 200 environment steps)
Windows per batch (Chunked-HQL)
128
Critic targets per update
1024 ( =128×8 )
Batch size of n -step TD
1024 (one target per sequence)
n of n -step TD
100, 200
Appendix
Table 6: Hyperparameters for the OCBench benchmark experiment (Figure 5 ).
Figure 10: Soft gate at different temperatures versus HQL’s hard gate. Success rate averaged over the five tasks of each configuration, one seed; gray marks the offline phase. The soft gate improves as τ grows toward the hard gate, and the hard gate is the best on average, especially on the long-horizon mazes.
soft gate
hard gate
τ=0
τ=0.5
τ=1
τ=3
τ=30
(HQL)
cube-double
75
81
87
84
80
81
cube-triple
16
37
39
46
44
48
scene
59
60
59
60
60
60
puzzle-3x3
21
20
20
20
20
20
antmaze-giant
24
11
24
36
58
65
Appendix
Table 7: Soft gate versus hard gate. Success rate (%) at the end of online training, mean of the last three evaluations over five tasks, one seed. We bold the best score in each row at integer level.