Hierarchical Time-aware Bootstrapping for Off-Policy Subgoal Value Learning

Bingyun Liu; Yuheng Jing

cs.LGOct 4, 2026

Hierarchical Time-aware Bootstrapping for Off-Policy Subgoal Value Learning

Authors: Bingyun Liu, Yuheng Jing

Organizations: Institute of Automation Chinese Academy of Sciences

Abstract

Off-policy hierarchical reinforcement learning must estimate the values of high-level decisions while the low-level policy changes. HIRO adapts replay data through subgoal relabeling, but after a label change, the value update targets the relabeled subgoal instead of the subgoal the high-level policy originally needed to update. We propose Hierarchical Time-aware Bootstrapping (HTB), which evaluates specified subgoals under the current low-level policy while retaining accumulated task rewards. Remaining execution time distinguishes subgoal continuation from a new high-level decision. Together with primitive-action conditioning, it enables off-policy Bellman updates based on the stationary environment transition law. HTB combines these one-step updates with multi-step suffix returns and truncated relabeling, reducing dependence on intermediate value estimates. A shared value component supports learning across actions, while nonnegative residuals constrain upward corrections relative to that component. At a fixed mixture weight of 0.95, HTB achieves 32.8% AntFall success versus 9.6% for matched local HIRO over five paired seeds at 10M environment steps. Ablations identify contributions from recursive continuation and mixed supervision; fixed-policy tests show more accurate predictions for actions whose returns were excluded from fitting.

Figures & tables

Figure 1: Subgoal values can be learned at every primitive step. The high-level policy acts every

H

steps. HTB learns

V_{H-i}(s_{i},G,g_{i})

throughout each segment using a mixture of its remaining return (blue) and a one-step backup (orange). The current subgoal continues until the next high-level decision. Arrows run from target dependencies to the value being updated; displayed continuations assume no termination.

Figure 2: Multi-step and one-step updates can supervise different subgoals. (a) Each suffix is relabeled using only its recorded actions. (b) The multi-step target trains the relabeled subgoal; the one-step estimator fits the recorded action and evaluates the current-policy action for the specified subgoal. (c) Updating

V

changes the shared value offset in every primitive-action value, including actions without direct supervision; for fixed

A

\Delta\widehat{Q}(a)=\Delta V

. Repeated task-goal inputs and action clipping are omitted; the diagram shows a nonterminal segment.

Variant	One-step subgoal	Next residual	Output / target
HTB	Replay	Included	Nonnegative / clipped
No next residual	Replay	Omitted	Nonnegative / clipped
Signed residual	Replay	Omitted	Signed / signed
Policy subgoal	Current proposal	Omitted	Nonnegative / clipped

Table 1: Controlled variants of the HTB update. All variants use remaining-time values and the same

V-A

architecture. The last two variants use the nonrecursive residual estimator.

Figure 3: HTB improves success on Fall at a fixed configuration. Success over training, averaged across five paired seeds. Both HTB conditions use

\lambda=.95

; HIRO uses its original off-policy correction. Shading denotes pointwise

95\%

empirical bootstrap intervals. No temporal smoothing is applied.

Figure 4: Residual recursion and mixed backups contribute to task success. (a) Full HTB minus the variant without the next-step residual, both at

\lambda=.95

. (b) Mixed backups at

.95

minus multi-step-only learning at

1

. Points are paired mean success differences; bars are nominal

95\%

bootstrap intervals.

Figure 5: Direct subgoal supervision and loss compatibility are distinct. (a) Circles measure retention of the original subgoal after multi-step correction; hollow squares mark the designated one-step input rule. (b) Each line connects correction/no-correction measurements from one paired repeat, grouped by task, variant and

\lambda

. Each point is the multi-step–one-step gradient cosine within that model (Equation ( 13 )), averaged over two minibatches. Signed and Policy are the nonrecursive signed-residual and current-policy-subgoal variants.

	Separate test trajectories		Actions excluded from fitting
Task	Direct- $Q$	Shared $V-A$	Direct- $Q$	Shared $V-A$
Reacher-v0 dense	$70.60{}\!\pm\!48.07{}$	$60.22{}\!\pm\!36.09{}$	$50.45{}\!\pm\!35.64{}$	$19.29{}\!\pm\!18.79{}$
Reacher-v1 sparse	$9.49{}\!\pm\!0.36{}$	$8.86{}\!\pm\!0.91{}$	$8.50{}\!\pm\!0.35{}$	$1.13{}\!\pm\!0.03{}$

Table 2: Shared values improve estimates for actions without direct labels. MAE against complete rollout returns (mean

\pm

SD across 3 policy seeds; lower is better). Left: all three action types on trajectories excluded from fitting and calibration. Right: uniformly sampled actions at training inputs

(s,G,g,t)

, whose return labels enter no loss.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Environment steps	$10{,}000{,}000$
High-level period / update interval	$10$ / $10$ primitive steps
Batch size (both levels)	$128$
Replay capacity (both levels)	$200{,}000$ transitions each
Actor learning rate (both levels)	$10^{-4}$
Critic learning rate (both levels)	$10^{-3}$

Appendix

Table 3: Push/Fall training settings. Values are shared by HIRO and the remaining-time variants unless the component exists only in the latter.

Analysis set	Settings	Wins	Mean gain
Interior mixing weights	64	60	+26.82
Full grid, deduplicated	84	68	+20.88
Full logical grid	88	71	+20.49
Single-setting exclusion, logical grid	87	71	+20.74
Single-setting exclusion, deduplicated	83	68	+21.14

Appendix

Table 4: Reacher conclusions persist across analysis sets. Gains are mean success-rate differences in percentage points; wins count positive configuration means.

Setting	Seed	Correction	Steps (millions)	Original	Corrected	Current
Rv0/s/Signed	781	yes	0.95–1.00	0.316	0.315	0.322
Rv0/s/Signed	784	yes	0.70–0.75	0.333	0.327	0.336
Rv0/s/Signed	781	no	0.90–1.00	0.214	0.205	0.199
Rv0/s/Signed	784	no	0.90–1.00	0.950	0.945	0.989
Rv1/d/HTB	783	yes	0.95–1.00	3.366	3.324	3.394
Rv1/s/HTB	779	yes	0.95–1.00	2.667	0.789	1.080

Appendix

Table 5: Value changes at fixed subgoal inputs. Mean absolute critic change across earlier and later models. R0/R1 denote Reacher-v0/v1; d/s denote dense/sparse. Values use each run’s reward scale.

Env.	Rule	$\lambda$	$\Delta$ return	$\Delta$ success
Fall	No next residual	0	-971.5	+0.0
Fall	No next residual	0.6	+5938.8	+0.0
Fall	No next residual	0.9	-767.3	-9.2
Fall	No next residual	0.95	+2850.9	+14.4
Fall	No next residual	0.99	+4225.6	+13.2
Fall	No next residual	1	-309.2	-8.8

Appendix

Table 6: Complete Push/Fall correction ablation. Differences are no correction minus correction. Success differences are in percentage points. Entries average the paired seed differences. Repeated

\lambda=1

rows preserve the full grid and share the same seed outcomes.

Task	Correction	No correction	Difference (pp)
Reacher-v0 / dense	60.00	88.13	+28.12
Reacher-v1 / sparse	26.00	81.63	+55.62

Appendix

Table 7: Current-policy goal queries can also benefit from disabling correction. Eight-seed mean final-window success percentages for two current-policy-subgoal/.99 examples. These are illustrative configurations within the full grid.

Figure 6: Late-phase signed-residual/.6 loss scales differ between correction modes. Each curve is the median across seed-level phase-mean losses; the vertical axis is logarithmic. Panel counts give the number of correction/no-correction seeds. The late phase is the final third of each run’s high-level updates. The profiles measure scalar loss magnitude; gradient alignment is evaluated in Figure 5 .

Environment	Reward	HIRO	Multi-step only	HTB/.95
Reacher-v0	dense	$94.3{}\!\pm\!2.2{}$	$92.6{}\!\pm\!4.0{}$	$95.4{}\!\pm\!3.2{}$
Reacher-v0	sparse	$32.1{}\!\pm\!32.1{}$	$54.3{}\!\pm\!37.8{}$	$48.6{}\!\pm\!41.8{}$
Reacher-v1	dense	$96.5{}\!\pm\!2.2{}$	$97.1{}\!\pm\!1.7{}$	$97.3{}\!\pm\!2.3{}$
Reacher-v1	sparse	$76.0{}\!\pm\!30.3{}$	$95.4{}\!\pm\!7.9{}$	$96.1{}\!\pm\!3.4{}$
Maze-v1	dense	—	$60.6{}\!\pm\!18.8{}$	$73.8{}\!\pm\!9.5{}$
Maze-v1	sparse	—	$35.0{}\!\pm\!39.9{}$	$19.4{}\!\pm\!31.8{}$

Appendix

Table 8: Fixed-configuration success across eight goal-reaching tasks. Success (%), mean

\pm

seed SD over eight seeds, with correction disabled. Multi-step only is the remaining-time baseline at

\lambda=1

; HTB uses

\lambda=.95

. HIRO uses the local high-level-policy implementation. Dashes mark conditions without an eight-seed HIRO comparison, not zero success.

Figure 7: Component effects depend on the task and query distribution. Positive differences favor the left-hand condition. Thick bars resample eight shared seed blocks on the fixed grid; thin bars also resample configurations. Residual constraints, recursion, and query contrasts average all sixteen task–weight settings per panel; the short-path contrast averages four tasks. Nonnegative and query contrasts use the nonrecursive base. Intervals are nominal

95\%

Contrast	Tasks	Corr.	$K$	Mean difference
Recursive $-$ nonrecursive	Maze	off	16	-0.1
Recursive $-$ nonrecursive	Push/Fall	off	8	-151.3
Recursive $-$ nonrecursive	Push/Fall	on	8	1136.0
Recursive $-$ nonrecursive	Reacher	off	16	-0.1
Recursive $-$ nonrecursive	Reacher	on	16	2.8
Nonnegative $-$ signed	Maze	off	16	-2.0

Appendix

Table 9: Interior-grid component averages. Differences are left minus right. Reacher/Maze use success percentage points; Push/Fall use return. Correction is on/off.

K

counts configurations. Each entry is the mean paired difference across the evaluated configurations. No family or metric is pooled with another.

Environment	Reward	Correction	Metric	Mean difference
Maze-v1	dense	off	Success	13.1
Maze-v1	sparse	off	Success	-15.6
MazeW-v2	dense	off	Success	2.4
MazeW-v2	sparse	off	Success	-4.4
Fall	dense	off	Return	344.7
Fall	dense	off	Success	19.6

Appendix

Table 10: The short branch has task-dependent effects. Fixed HTB/

\lambda=.95

minus Multi-step only at

\lambda=1

. Success differences use percentage points; returns use the environment’s unscaled reward. All entries pair the same seeds within a task and mode.

Figure 8: Final return and early learning can favor different settings. Actual returns at all 200 evaluations from

0.05

M to

10

M steps. Both HTB conditions use

\lambda=.95

. Lines average five matched seeds; bands are pointwise

95\%

empirical bootstrap intervals. No temporal smoothing is applied.

Task	HTB correction	Normalized AUC difference	Final-return difference
Push	on	-84.1	508.3
Push	off	-82.2	75.5
Fall	on	69.4	875.4
Fall	off	226.6	560.4

Appendix

Table 11: Integrated return and final return at the fixed configuration. HTB/.95 minus HIRO, five paired seeds. AUC is the trapezoidal return integral divided by the duration of its observed step window. Each cell gives the mean paired difference.

Environment	Setting	Return	Success (%)
Push	HIRO	-5553.6	0.0
Push	HTB, suffix correction	-5045.3	14.0
Push	HTB, no correction	-5478.1	0.8
Fall	HIRO	-8137.8	9.6
Fall	HTB, suffix correction	-7262.4	32.8
Fall	HTB, no correction	-7577.4	24.4

Appendix

Table 12: Fixed HTB settings against the Push/Fall HIRO reference. Five-seed means at ten million steps; both HTB modes use

\lambda=.95

. The HIRO rows use paper-style correction.

Task	Reward	Independent	Shared critic	Shared actor/critic
Reacher-v0	dense	$87.7{}\!\pm\!4.5{}$	$81.6{}\!\pm\!8.2{}$	$86.7{}\!\pm\!5.5{}$
Reacher-v0	sparse	$12.7{}\!\pm\!10.6{}$	$24.3{}\!\pm\!27.3{}$	$9.8{}\!\pm\!11.2{}$
Maze-v1	dense	$33.0{}\!\pm\!22.3{}$	$17.0{}\!\pm\!21.6{}$	$39.0{}\!\pm\!27.6{}$
Maze-v1	sparse	$27.5{}\!\pm\!32.6{}$	$9.0{}\!\pm\!22.3{}$	$7.5{}\!\pm\!22.0{}$
MazeW-v2	dense	$21.1{}\!\pm\!7.2{}$	$14.8{}\!\pm\!7.0{}$	$22.6{}\!\pm\!6.7{}$
MazeW-v2	sparse	$6.8{}\!\pm\!9.2{}$	$7.3{}\!\pm\!7.7{}$	$9.5{}\!\pm\!10.6{}$

Appendix

Table 13: Horizon-sharing effects vary across goal-reaching tasks. Multi-step only algorithms with correction, final-window success (%), mean

\pm

SD over ten seeds. The architecture intervention differs from separating

V

and

A

within an action-conditioned value.

Figure 9: The benefit of correction reverses with the mixture. Positive final-return differences favor disabling correction. Points average 5 paired seeds; bars enumerate 3125 empirical bootstrap resamples. Symbols identify methods; identical

\lambda=1

results are shown once per environment.

Figure 10: The correction effect depends on the task. Mean final-window success differences on the interior-

\lambda

grid. Intervals resample configurations and their 8 paired seeds;

n

counts configurations. Each configuration has equal weight. This correction-effect bootstrap differs from the shared-seed-block component sensitivity defined in this appendix.

Task	Reward	Corr.	$\lambda$	Mean difference
Reacher-v0	dense	off	$0.6$	0.1
Reacher-v0	dense	off	$0.9$	-2.5
Reacher-v0	dense	off	$0.95$	2.9
Reacher-v0	dense	off	$0.99$	-0.9
Reacher-v0	dense	on	$0.6$	18.5
Reacher-v0	dense	on	$0.9$	34.5

Appendix

Table 14: Complete Reacher nonnegative-residual contrasts. No next residual minus Signed residual; success difference in percentage points, eight paired seeds. Both conditions use the nonrecursive base. Correction is on/off. All interior weights and both effect directions appear.

Task	Reward	Corr.	$\lambda$	Mean difference
Maze-v1	dense	off	$0.6$	-63.1
Maze-v1	dense	off	$0.9$	21.3
Maze-v1	dense	off	$0.95$	13.8
Maze-v1	dense	off	$0.99$	8.1
Maze-v1	sparse	off	$0.6$	-8.1
Maze-v1	sparse	off	$0.9$	-0.6

Appendix

Table 15: Complete Maze nonnegative-residual contrasts. No next residual minus Signed residual on the nonrecursive base; success difference in percentage points, eight paired seeds.

Task	Reward	Corr.	$\lambda$	Mean difference
Fall	dense	off	$0.6$	112.2
Fall	dense	off	$0.9$	174.9
Fall	dense	off	$0.95$	276.3
Fall	dense	off	$0.99$	421.1
Fall	dense	on	$0.6$	-345.2
Fall	dense	on	$0.9$	47.4

Appendix

Table 16: Complete Push/Fall nonnegative-residual contrasts. No next residual minus Signed residual on the nonrecursive base; final-return difference, five paired seeds.

Task	Reward	Corr.	$\lambda$	Mean difference
Reacher-v0	dense	off	$0.6$	2.0
Reacher-v0	dense	off	$0.9$	0.3
Reacher-v0	dense	off	$0.95$	0.9
Reacher-v0	dense	off	$0.99$	2.1
Reacher-v0	dense	on	$0.6$	-5.0
Reacher-v0	dense	on	$0.9$	10.1

Appendix

Table 17: Complete Reacher residual-recursion contrasts. HTB minus No next residual; success difference in percentage points, eight paired seeds.

Task	Reward	Corr.	$\lambda$	Mean difference
Maze-v1	dense	off	$0.6$	15.6
Maze-v1	dense	off	$0.9$	-8.8
Maze-v1	dense	off	$0.95$	4.4
Maze-v1	dense	off	$0.99$	-2.5
Maze-v1	sparse	off	$0.6$	0.0
Maze-v1	sparse	off	$0.9$	5.0

Appendix

Table 18: Complete Maze residual-recursion contrasts. HTB minus No next residual; success difference in percentage points, eight paired seeds.

Task	Reward	Corr.	$\lambda$	Mean difference
Fall	dense	off	$0.6$	-1226.5
Fall	dense	off	$0.9$	-1053.1
Fall	dense	off	$0.95$	3.5
Fall	dense	off	$0.99$	-332.5
Fall	dense	on	$0.6$	2189.9
Fall	dense	on	$0.9$	268.7

Appendix

Table 19: Complete Push/Fall residual-recursion contrasts. HTB minus No next residual; final-return difference, five paired seeds.

Task	Reward	Corr.	$\lambda$	Mean difference
Reacher-v0	dense	off	$0.6$	-68.4
Reacher-v0	dense	off	$0.9$	-13.1
Reacher-v0	dense	off	$0.95$	-12.1
Reacher-v0	dense	off	$0.99$	-4.5
Reacher-v0	dense	on	$0.6$	7.9
Reacher-v0	dense	on	$0.9$	-39.6

Appendix

Table 20: Complete Reacher query-distribution contrasts. Current-policy subgoal minus No next residual on the nonrecursive base; success difference in percentage points, eight paired seeds. Negative values favor replay goals over current high-level policy queries.

Task	Reward	Corr.	$\lambda$	Mean difference
Maze-v1	dense	off	$0.6$	-6.3
Maze-v1	dense	off	$0.9$	-65.6
Maze-v1	dense	off	$0.95$	-42.5
Maze-v1	dense	off	$0.99$	-21.9
Maze-v1	sparse	off	$0.6$	0.0
Maze-v1	sparse	off	$0.9$	-18.8

Appendix

Table 21: Complete Maze query-distribution contrasts. Current-policy subgoal minus No next residual on the nonrecursive base; success difference in percentage points, eight paired seeds.

Figure 11: Value decomposition reduces error under matched supervision. MAE on separate test trajectories during fixed-policy fitting, averaged over 3 policy seeds per task. Shading is one standard deviation across policy seeds. The horizontal axis counts critic-fitting updates under frozen policies.

Task	Estimator	MAE	Action MAE	Signed error	Regret
Reacher-v0 dense	Direct-Q	$70.60{}\!\pm\!48.07{}$	$66.30{}\!\pm\!38.61{}$	$-14.21{}\!\pm\!30.58{}$	$22.87{}\!\pm\!21.86{}$
Reacher-v0 dense	Shared V-A	$60.22{}\!\pm\!36.09{}$	$60.38{}\!\pm\!36.12{}$	$-11.30{}\!\pm\!17.37{}$	$23.54{}\!\pm\!23.28{}$
Reacher-v0 dense	Output+target	$72.75{}\!\pm\!48.04{}$	$73.00{}\!\pm\!48.09{}$	$30.72{}\!\pm\!12.15{}$	$19.67{}\!\pm\!17.42{}$
Reacher-v0 dense	Output only	$73.25{}\!\pm\!47.82{}$	$73.51{}\!\pm\!47.88{}$	$32.75{}\!\pm\!12.38{}$	$20.81{}\!\pm\!19.25{}$
Reacher-v0 dense	Target only	$72.86{}\!\pm\!48.11{}$	$73.10{}\!\pm\!48.17{}$	$30.48{}\!\pm\!12.30{}$	$21.23{}\!\pm\!19.42{}$
Reacher-v0 dense	Signed residual	$73.35{}\!\pm\!48.08{}$	$73.60{}\!\pm\!48.15{}$	$32.17{}\!\pm\!12.25{}$	$21.59{}\!\pm\!20.04{}$

Appendix

Table 22: Complete neural diagnostic results. Mean

\pm

SD across 3 policy seeds, using candidate-slot weighting. MAE and signed error use separate test trajectories; Action MAE evaluates their uniformly sampled primitive actions, whose labels are excluded in the representation comparison. Signed error is prediction minus rollout return. Regret is measured among the evaluated subgoal candidates.

Task	Policy seed	Slot-weighted difference	Deduplicated difference
Reacher-v0 dense	777	-3.15	-3.13
Reacher-v0 dense	778	-24.92	-25.01
Reacher-v0 dense	779	-3.08	-3.00
Reacher-v1 sparse	777	-0.77	-0.77
Reacher-v1 sparse	778	-0.03	-0.03
Reacher-v1 sparse	779	-1.11	-1.12

Appendix

Table 23: Exact-input deduplication preserves the MAE comparison. Shared

V-A

minus direct-

Q

test-episode MAE; negative values favor the shared representation.

		Selected signed error		MAE
Candidates	Noise	Signed	Projected	Signed	Projected
2	0.0	$0.002{}\!\pm\!0.017{}$	$-0.268{}\!\pm\!0.019{}$	$0.019{}\!\pm\!0.006{}$	$0.270{}\!\pm\!0.016{}$
2	0.3	$0.006{}\!\pm\!0.028{}$	$-0.466{}\!\pm\!0.032{}$	$0.032{}\!\pm\!0.010{}$	$0.464{}\!\pm\!0.029{}$
2	1.0	$0.059{}\!\pm\!0.071{}$	$-1.241{}\!\pm\!0.083{}$	$0.096{}\!\pm\!0.034{}$	$1.245{}\!\pm\!0.086{}$
4	0.0	$0.004{}\!\pm\!0.015{}$	$-0.247{}\!\pm\!0.017{}$	$0.017{}\!\pm\!0.005{}$	$0.246{}\!\pm\!0.014{}$
4	0.3	$0.010{}\!\pm\!0.022{}$	$-0.448{}\!\pm\!0.027{}$	$0.031{}\!\pm\!0.007{}$	$0.446{}\!\pm\!0.027{}$

Appendix

Table 24: Nonnegative projection trades optimism for underestimation. Final mean

\pm

SD across 30 noise seeds with repeated boundary maximization. Selected error is the critic-selected subgoal’s prediction minus its exact optimal-policy value; MAE averages all evaluated values. The complete candidate-count and reward-noise grid is shown.

Explore similar work

Jun 19, 2026cs.LG

Horizon Adaptive Offline Policy Learning via Value Stitching

Learning accurate value functions plays a decisive role for reinforcement learning (RL) agents to solve long-horizon, complex tasks. Conventional temporal-difference (TD) learning objectives suffer from value-estimation bias that accumulates over the horizon, while extended-horizon modeling methods, such as n-step TD backups and Q-chunking, adopt a rigid, fixed-horizon value-modeling recipe that is often not flexible enough to capture complex value structures in long-horizon, multi-stage tasks. In this paper, we show that enabling value updates with dynamic horizon composition can yield a strong offline policy learning scheme. Our method, Horizon Adaptive Offline Policy Learning via VAlue STitching (VAST), replaces fixed-horizon backups with recursive, horizon-adaptive value composition. Its key ingredient is to couple value optimization with a future state- and horizon-length-conditioned auxiliary value function that is learned through direct data supervision, and a stitching policy that optimally selects the reward-maximizing horizon length and future sub-goal to achieve horizon-adaptive value stitching. This design enables direct estimation and compositional "stitching" of variable-length returns grounded in actionable sub-goal states, providing an accurate and greedily exploitable value-supervision signal for offline policy optimization. Across 50 tasks on OGBench, VAST outperforms fixed-step, extended-horizon methods, and generative-value offline RL baselines, achieving strong performance particularly in high-complexity, long-horizon decision-making tasks.

Kexin Zheng, Xianyuan Zhan, Xintao Yan

The University of Hong Kong · Tsinghua University

Value Functions Horizon Sewing

Ask PDF

Feb 3, 2026cs.LG

Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL

Offline goal-conditioned reinforcement learning remains challenging for long-horizon tasks. While hierarchical approaches mitigate this issue by decomposing tasks, most existing methods rely on separate high- and low-level networks and generate only a single intermediate subgoal, leaving several structural limitations in long-horizon decision-making. To address this limitation, we draw inspiration from chain-of-thought reasoning and propose the Chain-of-Goals Hierarchical Policy (CoGHP), a novel framework that reformulates hierarchical decision-making as autoregressive sequence modeling within a unified architecture. Given a state and a final goal, CoGHP autoregressively generates a sequence of latent subgoals followed by the primitive action, where each latent subgoal acts as a reasoning step that conditions subsequent predictions. To implement this efficiently, we introduce an MLP-Mixer backbone, which supports cross-token communication and captures structural relationships among state, goal, latent subgoals, and action. Across challenging navigation and manipulation benchmarks, CoGHP consistently outperforms strong offline baselines, demonstrating improved performance on long-horizon tasks. Project page: https://wlsdn9350.github.io/projects/coghp/

Jinwoo Choi, Sang-Hyun Lee, Seung-Woo Seo

Department of Electrical and Computer Engineering, Seoul National University, Seoul, South Korea · Department of Automotive Engineering, Ajou University, Gyeonggi-do, South Korea

Goal-Conditioned Reinforcement Learning High-Level Subgoal Generation Autoregressive Model Chain-of-Thought Reasoning

Ask PDF

Jul 23, 2026cs.LG

Offline RL with Hierarchical Action Chunking

Offline goal-conditioned reinforcement learning (RL) holds the promise of learning general-purpose policies from static datasets. However, scaling these methods to long-horizon tasks remains a challenge due to the curse of horizon, where value estimation errors can compound through long chains of bootstrapped Bellman backups. Existing hierarchical approaches mitigate this by decomposing tasks into subgoals, yet they often rely on low-level controllers that suffer from myopic execution and biased value estimates. In this work, we propose Hierarchical Implicit Q-Chunking (HiQC), an offline goal-conditioned RL algorithm that combines high-level latent planning with low-level action chunking. By conditioning the low-level critic on temporally extended action sequences, HiQC enables unbiased k-step value backups, compressing the horizon at both the planning and execution levels. We theoretically demonstrate that this dual decomposition results in a tighter bound on value error under a bounded per-backup error model compared to standard hierarchy or flat chunking alone. Empirically, HiQC achieves the highest aggregate performance among the compared methods on the OGBench suite, with its largest gains on long-horizon navigation tasks such as humanoid-giant.

Ahad Jawaid

The University of Texas at Dallas

Goal-Conditioned Reinforcement Learning Offline Reinforcement Learning Action Chunks Long-Horizon Task Planning Classical Planning Chunk

Ask PDF