Learning accurate off-policy state-action value functions (Q-functions) from offline datasets becomes harder as the task horizon grows. One-step temporal-difference (TD) learning bootstraps after every transition, so small per-step errors accumulate along the horizon, and early states can be badly misvalued even when the TD loss is small. Multi-step TD shortens this chain by following the dataset trajectory for several steps; however, since the behavior policy is often suboptimal, every followed action adds a bias, and no fixed number of steps suits a whole trajectory. In this paper, we propose Horizon-Scalable Q-Learning (HQL), a simple method that enables accurate long-horizon value learning by automatically balancing between one-step and multi-step returns. HQL extends a target through a dataset action only when the critic values it at least as highly as the best-of-N actions sampled from the BC policy, and otherwise falls back to a one-step target. We show that this rule is a principled off-policy correction that needs no action densities, which makes HQL compatible with expressive flow and diffusion policies. Experiments show that HQL avoids the failures of both one-step and multi-step TD on a combination-lock task across horizons, and outperforms prior horizon-reduction methods on long-horizon OGBench and OCBench tasks.
Figures & tables
Figure 1: Horizon-Scalable Q-Learning (HQL). Purple arcs show the reach of one training target. One-step TD accumulates residuals, and multi-step TD inherits the bias of the suboptimal action it spans (yellow). HQL compares each logged action with the best-of- N policy samples, continuing the trace where it passes ( c=1 ) and cutting it where it fails ( c=0 , dashed), so reward travels far without inheriting off-policy bias.
Figure 2: HQL and Chunked-HQL. H sets how many decisions one target chains, and L how many environment steps each decision covers.
Figure 3: Combination lock. HQL remains accurate under data corruption as the horizon grows. The figure illustrates the RMSE between the learned Qθ and the true Q∗ , averaged uniformly over all 2H state–action pairs of the corrupted combination lock, after 1M updates (log scale). Panels vary the corruption fraction of the dataset. n-step TD ( n=64 ) diverges as soon as any corruption is present; 1-step TD is unaffected by corruption but its error grows sharply with the horizon H .
Figure 4: HQL versus baseline TD methods at matched critic update budget. Success rate averaged over the five tasks of each configuration and over 5 seeds, with standard deviation shown as the band; gray marks the offline phase. HQL outperforms all baselines on all six configurations, especially on the challenging long-horizon ones.
HQL
Chunked-HQL
LQL
QC
antmaze-giant
1 → 87 ± 2
0 → 86 ± 1
0 → 44 ± 4
0 → 46 ± 1
cube-double
67 → 87 ± 1
73 → 100 ± 0
45 → 83 ± 2
28 → 100 ± 0
cube-triple
12 → 54 ± 1
7 → 84 ± 1
5 → 35 ± 3
1 → 26 ± 3
hmaze-giant
10 → 28 ± 1
0 → 29 ± 2
12 → 26 ± 1
0 → 0 ± 0
hmaze-medium
56 → 86 ± 1
68 → 88 ± 2
48 → 86 ± 1
0 → 0 ± 0
scene
93 → 99 ± 0
98 → 100 ± 0
96 → 97 ± 1
97 → 100 ± 0
Table 1: HQL and Chunked-HQL against LQL and QC. Success rates (%) are averaged over five tasks and reported as offline → online with standard error over five seeds; we bold the best offline and the best online score at integer level. HQL and Chunked-HQL achieve the best results, especially on the long-horizon tasks.
Figure 6
cube-
cube-
antmz-
hmz-
double
triple
giant
giant
QAM
29 → 50
1 → 1
14 → 83
1 → 0
+ HQL
38 → 99
0 → 80
0 → 98
0 → 63
VGF
69 → 74
3 → 25
1 → 28
0 → 0
+ HQL
80 → 89
4 → 37
0 → 90
0 → 4
Table 2: HQL as a drop-in critic for different policy extraction methods. Success rates (%) on task 2, offline → online; we bold the better score of each pair. HQL improves both QAM and VGF on all four tasks.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: The combination lock dataset with linearly arranged H states.
Method
Seq. length
Seq. per update
Targets per seq.
Target
HQL
16
32
16
gated, every position
16-window TD
16
32
16
n -step, every position
8-step TD
8
512
1
8-step, first state
1-step TD
1
512
1
1-step
Appendix
Table 3: How each method forms its 512 critic targets per update.
Parameter
Value
Window length H
16
Windows per batch
32
Critic targets per update
512 ( =32×16 )
Gate width N (Eq. 7 )
4
Gate critic
target network Q
Discount factor
0.99 (0.995 for the two giant mazes)
Appendix
Table 4: Hyperparameters for the OGBench benchmark experiment (Figures 5 and 9 ).
Figure 9: Per-task view of the OGBench benchmark (Figure 5 ). Reading down a column shows how one task index behaves across environments, and a row shows how one environment varies over tasks.
Task group
Dataset size (transitions)
Episode length
Action dimension
Observation dimension
Default task
Locomotion ( locomaze )
pointmaze-medium
1M
1000
2
2
1
pointmaze-large
1M
1000
2
2
1
pointmaze-giant
1M
1000
2
2
1
pointmaze-teleport
1M
1000
2
2
1
antmaze-medium
1M
1000
8
29
1
Appendix
Table 5: OGBench state-based task characteristics. Episode length is the maximum number of environment steps per episode. Rows marked † are the groups used in this work.
Parameter
Value
Action chunk size L
25 (cut at episode boundaries)
Window length H (Chunked-HQL)
8 chunks ( 200 environment steps)
Windows per batch (Chunked-HQL)
128
Critic targets per update
1024 ( =128×8 )
Batch size of n -step TD
1024 (one target per sequence)
n of n -step TD
100, 200
Appendix
Table 6: Hyperparameters for the OCBench benchmark experiment (Figure 5 ).
Figure 10: Soft gate at different temperatures versus HQL’s hard gate. Success rate averaged over the five tasks of each configuration, one seed; gray marks the offline phase. The soft gate improves as τ grows toward the hard gate, and the hard gate is the best on average, especially on the long-horizon mazes.
soft gate
hard gate
τ=0
τ=0.5
τ=1
τ=3
τ=30
(HQL)
cube-double
75
81
87
84
80
81
cube-triple
16
37
39
46
44
48
scene
59
60
59
60
60
60
puzzle-3x3
21
20
20
20
20
20
antmaze-giant
24
11
24
36
58
65
Appendix
Table 7: Soft gate versus hard gate. Success rate (%) at the end of online training, mean of the last three evaluations over five tasks, one seed. We bold the best score in each row at integer level.
Off-policy, value-based reinforcement learning methods such as Q-learning are appealing because they can learn from arbitrary experience, including data collected by older policies or other agents. In practice, however, bootstrapping makes long-horizon learning brittle: estimation errors at later states propagate backward through temporal-difference (TD) updates and can compound over time. We propose long-horizon Q-learning (LQL), which introduces a principled backstop against compounding error when learning the optimal action-value function. LQL builds on a prior optimality tightening observation: any realized action sequence lower-bounds what the optimal policy can achieve in expectation, so acting optimally earlier should not be worse than following the observed actions for several steps before switching to optimal behavior. Our contribution is to turn this inequality into a practical stabilization mechanism for Q-learning by using a hinge loss to penalize violations of these bounds. Importantly, LQL computes these penalties using network outputs already produced for the TD error, requiring no auxiliary networks and no additional forward passes relative to Q-learning. When combined with multiple state-of-the-art methods on a range of online and offline-to-online benchmarks, LQL consistently outperforms both 1-step TD and n-step TD learning at similar runtime.
Armaan A. Abraham, Lucy Xiaoyang Shi, Chelsea Finn
Learning accurate value functions plays a decisive role for reinforcement learning (RL) agents to solve long-horizon, complex tasks. Conventional temporal-difference (TD) learning objectives suffer from value-estimation bias that accumulates over the horizon, while extended-horizon modeling methods, such as n-step TD backups and Q-chunking, adopt a rigid, fixed-horizon value-modeling recipe that is often not flexible enough to capture complex value structures in long-horizon, multi-stage tasks. In this paper, we show that enabling value updates with dynamic horizon composition can yield a strong offline policy learning scheme. Our method, Horizon Adaptive Offline Policy Learning via VAlue STitching (VAST), replaces fixed-horizon backups with recursive, horizon-adaptive value composition. Its key ingredient is to couple value optimization with a future state- and horizon-length-conditioned auxiliary value function that is learned through direct data supervision, and a stitching policy that optimally selects the reward-maximizing horizon length and future sub-goal to achieve horizon-adaptive value stitching. This design enables direct estimation and compositional "stitching" of variable-length returns grounded in actionable sub-goal states, providing an accurate and greedily exploitable value-supervision signal for offline policy optimization. Across 50 tasks on OGBench, VAST outperforms fixed-step, extended-horizon methods, and generative-value offline RL baselines, achieving strong performance particularly in high-complexity, long-horizon decision-making tasks.
Offline goal-conditioned reinforcement learning (RL) holds the promise of learning general-purpose policies from static datasets. However, scaling these methods to long-horizon tasks remains a challenge due to the curse of horizon, where value estimation errors can compound through long chains of bootstrapped Bellman backups. Existing hierarchical approaches mitigate this by decomposing tasks into subgoals, yet they often rely on low-level controllers that suffer from myopic execution and biased value estimates. In this work, we propose Hierarchical Implicit Q-Chunking (HiQC), an offline goal-conditioned RL algorithm that combines high-level latent planning with low-level action chunking. By conditioning the low-level critic on temporally extended action sequences, HiQC enables unbiased k-step value backups, compressing the horizon at both the planning and execution levels. We theoretically demonstrate that this dual decomposition results in a tighter bound on value error under a bounded per-backup error model compared to standard hierarchy or flat chunking alone. Empirically, HiQC achieves the highest aggregate performance among the compared methods on the OGBench suite, with its largest gains on long-horizon navigation tasks such as humanoid-giant.