SEAR: Sample Efficient Action Chunking Reinforcement Learning

C. F. Maximilian Nagy; Onur Celik; Emiliyan Gospodinov; Florian Seligmann; Weiran Liao; Aryan Kaushik; Gerhard Neumann

cs.LGMar 2, 2026

SEAR: Sample Efficient Action Chunking Reinforcement Learning

Authors: C. F. Maximilian Nagy, Onur Celik, Emiliyan Gospodinov, Florian Seligmann, Weiran Liao, Aryan Kaushik, Gerhard Neumann

Organizations: Autonomous Learning Robots, Karlsruhe Institute of Technology · FZI Forschungszentrum Informatik, Karlsruhe

Abstract

Action chunking improves exploration and accelerates value propagation in long-horizon reinforcement learning, but naively applying off-policy methods to the temporally extended action space at reduced decision frequency offsets these gains, leading to poor sample efficiency. Existing online action chunking methods address these issues through computationally expensive critic-only approaches or by relying on offline data. We introduce SEAR, a sample-efficient off-policy algorithm that enables online reinforcement learning with action chunks by addressing both challenges. To handle high-dimensional action sequences, SEAR employs a causal transformer critic trained with multi-horizon targets that provide a training signal for every prefix of an action chunk, effectively increasing the useful gradients per sample. To restore decision frequency, SEAR operates with a receding horizon and random replanning, ensuring uniform state coverage while combining the fast value propagation of large chunks with the reactivity of small ones. SEAR outperforms state-of-the-art online reinforcement learning methods including SimbaV2 on challenging Metaworld manipulation tasks. Beyond online RL, applying SEAR to the offline-to-online method QC improves its performance on the demanding OGBench cube-triple tasks.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of SEAR’s (a) critic design and (b) how it affects performance. The critic function expects the state $s_{t}$ and an action chunk $a_{t:t+N}$ of chunk size $N$ as argument. During the critic update (a) , these training points are sampled from the replay buffer and the causal transformer critic predicts Q-values $Q^{(1)}(s_{t},a_{t}),Q^{(2)}(s_{t},a_{t:t+1}),\dots,Q^{(N)}(s_{t},a_{t+N-1})$ for all action chunk prefixes , thereby generating multi-horizon predictions that increase the amount of training data for the subsequent critic update. Only fitting the critic on the whole-chunk target $Q^{(N)}$ , as is common in offline RL [ 17 , 32 ] . does not work well in online RL (b) . Furthermore, bootstrapping an action chunking critic with a single-step policy, comparable to T-SAC (“No Action Chunking”) lowers success, as does replacing the transformer backbone with an MLP.

Panel of Figure 2 — Figure 2: Uniform Action Chunking Limits Actor State Coverage. In methods without action chunking like T-SAC and SimbaV2, the actor is evaluated in each state of the trajectory (a) . Prior action chunking methods evaluate the actor in every N-th state (b) . This creates areas of the state space where the actor is rarely evaluated during training (white dots). When evaluating such a policy with receding horizons, the actor is queried in these out-of-distribution states and the resulting performance drop offsets most gains from the higher decision frequency. SEAR introduces random replanning which breaks this pattern, restoring uniform state coverage (c) . As a result, SEAR benefits from receding horizon control to a much higher degree than methods without random replanning (d) .

Panel of Figure 3 — Figure 3: SEAR works well across Chunk Sizes and benefits consistently from Receding Horizons. We compare IQM Success on Metaworld-Hard for different chunk sizes. (a) All SEAR policies with action chunking achieve higher performance than SimbaV2, with only the single-action version SEAR-1 having marginally lower performance. The best performance is achieved by SEAR-10, indicating a favourable trade-off between value propagation and decision frequency. (b) For all tested chunk sizes, including one (i.e., no action chunking), training a policy on a larger chunk size and using it with a receding horizon yields higher performance than using a policy with a smaller chunk size and no receding horizon. All tested policies degrade in performance when increasing the receding horizon towards the chunk size, highlighting the challenge of reduced decision frequencies. Similarly, all action chunking policies drop in performance when reducing the receding horizon towards one. Across all tested chunks sizes (and including CQN-AS), the optimal receding horizon length lies between three and five.

Panel of Figure 4 — Figure 4: Comparison to State-of-the-Art Baselines. (a) In the online setting, SEAR achieves over 80% IQM success, significantly above the 60% success of CQN-AS and SimbaV2, which we consider to be the state-of-the art in online RL with and without action chunking respectively. As further baselines we include T-SAC which shares a similar critic with SEAR, SAC and a naive action chunking extention of SimbaV2. They achieve around 20% IQM success. (b) In the offline-to-online setting, we achieve higher IQM success rates than QC, especially early in training.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Comparison between our SimbaV2 implementation and the reference implementation.

Component	SEAR	CQN-AS	SimbaV2
Actor
architecture	SimbaV2-MLP	N/A	SimbaV2-MLP
hidden dimension	512	N/A	512
number of blocks	1	N/A	2
Critic
architecture	Transformer	MLP + GRU	SimbaV2-MLP

Appendix

Table 1: Hyperparameters used in our experiments. CQN-AS has no actor component. SimbaV2-MLP denotes the MLP head used in SimbaV2, including its spherical normalization.

\dim(A)

denotes the dimensionality of a single action, and

N

is the chunk size.

Figure 6: All tasks in the Metaworld benchmark [ 31 ]

Figure 7: Learning Curves for the critic ablations discussed in Figure 2

Panel of Figure 8 — Figure 8: The experiments of 3 , but over all 50 Metaworld environments, not just the 20 hardest that were used in the main text. SEAR still reaches a higher success rate than SimbaV2. The effect of a receding horizon is very similar to the result of the main text: $k=1$ is suboptimal, as is $k=N$ . The optimal value is around $k=4$ independent of chunk size.

	SEAR-1	SEAR-5	SEAR-10	SEAR-20
Average	$0.57$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.07\\ -0.08\end{smallmatrix}}$	$\pagecolor{overlapcolor}{0.84}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.03\\ -0.04\end{smallmatrix}}$	$\pagecolor{bestcolor}{0.84}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.04\\ -0.05\end{smallmatrix}}$	$0.70$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.06\\ -0.07\end{smallmatrix}}$
pick-place-wall-v3	$\pagecolor{overlapcolor}{0.00}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.00\\ -0.00\end{smallmatrix}}$	$\pagecolor{overlapcolor}{0.00}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.02\\ -0.00\end{smallmatrix}}$	$\pagecolor{overlapcolor}{0.00}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.01\\ -0.00\end{smallmatrix}}$	$\pagecolor{overlapcolor}{0.00}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.00\\ -0.00\end{smallmatrix}}$
pick-place-v3	$0.27$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.28\\ -0.21\end{smallmatrix}}$	$0.63$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.16\\ -0.12\end{smallmatrix}}$	$\pagecolor{bestcolor}{0.90}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.07\\ -0.10\end{smallmatrix}}$	$\pagecolor{overlapcolor}{0.74}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.18\\ -0.37\end{smallmatrix}}$
shelf-place-v3	$\pagecolor{overlapcolor}{0.00}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.17\\ -0.00\end{smallmatrix}}$	$\pagecolor{bestcolor}{0.38}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.45\\ -0.38\end{smallmatrix}}$	$\pagecolor{overlapcolor}{0.22}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.45\\ -0.22\end{smallmatrix}}$	$\pagecolor{overlapcolor}{0.00}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.12\\ -0.00\end{smallmatrix}}$
basketball-v3	$0.01$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.15\\ -0.01\end{smallmatrix}}$	$0.69$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.17\\ -0.20\end{smallmatrix}}$	$\pagecolor{overlapcolor}{0.97}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.03\\ -0.06\end{smallmatrix}}$	$\pagecolor{bestcolor}{0.97}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.03\\ -0.11\end{smallmatrix}}$
coffee-push-v3	$\pagecolor{overlapcolor}{0.67}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.18\\ -0.33\end{smallmatrix}}$	$\pagecolor{overlapcolor}{0.70}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.09\\ -0.10\end{smallmatrix}}$	$\pagecolor{bestcolor}{0.80}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.08\\ -0.05\end{smallmatrix}}$	$\pagecolor{overlapcolor}{0.76}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.12\\ -0.09\end{smallmatrix}}$

Appendix

Table 2: IQM success with 95% bootstrapped confidence interval across the 20 hardest Metaworld tasks, comparing SEAR with various chunk sizes. The best result is highlighted, results that are non-significantly worse that the best are lightly highlighted. All policies except for SEAR-1 use a fixed receding horizon of five actions during evaluation. On average, SEAR-5 and SEAR-10 perform best, with SEAR-10 outperforming SEAR-5 on individual environments.

	SEAR-10	CQN-AS-5	CQN-AS-10	SimbaV2
Average	$\pagecolor{bestcolor}{0.84}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.04\\ -0.05\end{smallmatrix}}$	$0.61$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.04\\ -0.05\end{smallmatrix}}$	$0.47$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.06\\ -0.06\end{smallmatrix}}$	$0.62$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.05\\ -0.06\end{smallmatrix}}$
pick-place-wall-v3	$\pagecolor{overlapcolor}{0.00}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.01\\ -0.00\end{smallmatrix}}$	$\pagecolor{overlapcolor}{0.12}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.30\\ -0.12\end{smallmatrix}}$	$\pagecolor{bestcolor}{0.18}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.24\\ -0.18\end{smallmatrix}}$	$\pagecolor{overlapcolor}{0.00}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.33\\ -0.00\end{smallmatrix}}$
pick-place-v3	$\pagecolor{bestcolor}{0.90}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.07\\ -0.10\end{smallmatrix}}$	$0.58$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.18\\ -0.17\end{smallmatrix}}$	$0.48$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.22\\ -0.40\end{smallmatrix}}$	$0.00$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.17\\ -0.00\end{smallmatrix}}$
shelf-place-v3	$\pagecolor{overlapcolor}{0.22}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.43\\ -0.22\end{smallmatrix}}$	$\pagecolor{bestcolor}{0.43}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.40\\ -0.33\end{smallmatrix}}$	$0.00$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.00\\ -0.00\end{smallmatrix}}$	$\pagecolor{overlapcolor}{0.30}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.37\\ -0.30\end{smallmatrix}}$
basketball-v3	$\pagecolor{bestcolor}{0.97}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.03\\ -0.06\end{smallmatrix}}$	$0.25$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.03\\ -0.10\end{smallmatrix}}$	$0.15$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.03\\ -0.07\end{smallmatrix}}$	$0.27$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.36\\ -0.17\end{smallmatrix}}$
coffee-push-v3	$\pagecolor{bestcolor}{0.80}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.08\\ -0.05\end{smallmatrix}}$	$0.43$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.15\\ -0.08\end{smallmatrix}}$	$0.37$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.18\\ -0.13\end{smallmatrix}}$	$\pagecolor{overlapcolor}{0.38}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.37\\ -0.22\end{smallmatrix}}$

Appendix

Table 3: IQM success rate with 95% bootstrapped confidence interval across the 20 hardest Metaworld tasks, comparing SEAR-10, the best performing SEAR variant, with baselines. The best result is highlighted, results that are non-significantly worse that the best are lightly highlighted. All policies except for SimbaV2 use a fixed receding horizon of five actions during evaluation. CQN-AS performs better with chunk size five (CQN-AS-5) than with chunk size ten (CQN-AS-10). Overall, SEAR-10 performs best.

	SEAR-1	SEAR-5	SEAR-10	SEAR-20	SimbaV2
Average	$\pagecolor{bestcolor}{1.00}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.00\\ -0.00\end{smallmatrix}}$	$\pagecolor{overlapcolor}{1.00}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.00\\ -0.00\end{smallmatrix}}$	$\pagecolor{overlapcolor}{1.00}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.00\\ -0.00\end{smallmatrix}}$	$\pagecolor{overlapcolor}{1.00}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.00\\ -0.00\end{smallmatrix}}$	$\pagecolor{overlapcolor}{1.00}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.00\\ -0.00\end{smallmatrix}}$
reach-wall-v3	$\pagecolor{bestcolor}{0.98}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.02\\ -0.03\end{smallmatrix}}$	$\pagecolor{overlapcolor}{0.98}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.02\\ -0.04\end{smallmatrix}}$	$\pagecolor{overlapcolor}{0.98}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.02\\ -0.08\end{smallmatrix}}$	$\pagecolor{overlapcolor}{0.96}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.03\\ -0.16\end{smallmatrix}}$	$\pagecolor{overlapcolor}{0.97}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.03\\ -0.05\end{smallmatrix}}$
button-press-wall-v3	$\pagecolor{bestcolor}{1.00}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.00\\ -0.00\end{smallmatrix}}$	$\pagecolor{overlapcolor}{1.00}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.00\\ -0.00\end{smallmatrix}}$	$\pagecolor{overlapcolor}{1.00}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.00\\ -0.18\end{smallmatrix}}$	$\pagecolor{overlapcolor}{1.00}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.00\\ -0.01\end{smallmatrix}}$	$\pagecolor{overlapcolor}{0.98}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.02\\ -0.09\end{smallmatrix}}$
handle-press-v3	$\pagecolor{bestcolor}{1.00}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.00\\ -0.03\end{smallmatrix}}$	$\pagecolor{overlapcolor}{1.00}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.00\\ -0.00\end{smallmatrix}}$	$\pagecolor{overlapcolor}{1.00}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.00\\ -0.00\end{smallmatrix}}$	$\pagecolor{overlapcolor}{1.00}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.00\\ -0.01\end{smallmatrix}}$	$\pagecolor{overlapcolor}{1.00}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.00\\ -0.03\end{smallmatrix}}$
handle-pull-side-v3	$\pagecolor{bestcolor}{1.00}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.00\\ -0.03\end{smallmatrix}}$	$\pagecolor{overlapcolor}{0.92}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.07\\ -0.12\end{smallmatrix}}$	$\pagecolor{overlapcolor}{1.00}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.00\\ -0.17\end{smallmatrix}}$	$\pagecolor{overlapcolor}{1.00}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.00\\ -0.03\end{smallmatrix}}$	$\pagecolor{overlapcolor}{0.98}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.02\\ -0.03\end{smallmatrix}}$
plate-slide-v3	$\pagecolor{bestcolor}{1.00}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.00\\ -0.00\end{smallmatrix}}$	$\pagecolor{overlapcolor}{1.00}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.00\\ -0.01\end{smallmatrix}}$	$\pagecolor{overlapcolor}{1.00}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.00\\ -0.00\end{smallmatrix}}$	$\pagecolor{overlapcolor}{1.00}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.00\\ -0.00\end{smallmatrix}}$	$\pagecolor{overlapcolor}{1.00}$ ${\color[rgb]{0.6016,0.6016,0.6016}\begin{smallmatrix}+0.00\\ -0.00\end{smallmatrix}}$

Appendix

Table 4: IQM success rate with 95% bootstrapped confidence interval across the 30 easiest Metaworld tasks, comparing SEAR variants with a non-chunking SimbaV2 baseline. On nearly all tasks, the policies achieve close to a perfect success rate. We therefore exclude these environments from our other experiments.

Panel of Figure 9 — Figure 9: Learning curves for all 50 Metaworld Environments.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

May 7, 2026

Adaptive Q-Chunking for Offline-to-Online Reinforcement Learning

Offline-to-online reinforcement learning with action chunking eliminates multi-step off-policy bias and enables temporally coherent exploration, but all existing methods use a fixed chunk size across every state. This is suboptimal: near contact events the agent needs short chunks for reactive control, while during free-space motion long chunks provide better credit assignment. The natural solution is to train critics for several chunk sizes and select the best one at each state, but naive comparison of learned critic values systematically collapses to the shortest chunk due to discount-scale mismatch, and degrades to noise in low-value states. We propose Adaptive Q-Chunking (AQC), which resolves both failures by comparing the advantage of each chunk size relative to a per-horizon baseline, normalized by the discount factor. This criterion converts biased wrong answers into unbiased near-random choices when no genuine signal exists, and becomes discriminative when a particular scale enables better planning. We prove theoretical bounds on the advantage selector's noise immunity and on the value dominance of adaptive chunking over any fixed chunk size. We demonstrate that AQC achieves state-of-the-art offline and online success rates on OGBench and Robomimic, and can be applied to enhance the performance of large-scale VLA models that predict action sequences, significantly boosting performance on RoboCasa-GR1 tasks.

Nandiraju Gireesh, Yuanliang Ju, He Wang

Peking University · Galbot · University of Toronto

Q-Learning Offline RL Action Chunking Policy Learning Offline-To-Online RL

Ask PDF

Aug 31, 2026

Three Steps at a Time: Learning Representations from Action Sequences in Contrastive RL

While self-supervised approaches to reinforcement learning have achieved strong results by learning representations of states and actions, a key open question is the time scale over which actions should be modeled. Departing from the standard formulation relying on single-step actions, we extend contrastive reinforcement learning (CRL), a prototypical self-supervised method, to operate over action chunks, and find that this results in large, pervasive gains across established offline and online benchmarks: +31.7% and +93.1% across 18 and 11 environments respectively. While action-chunking-driven gains are generally explained through the ability to model non-Markovian, temporally extended policies, and to propagate unbiased multi-step returns, interestingly, we find that these arguments only partially apply to CRL. Our empirical studies suggest that, in the context of CRL, an action chunk carries more information about the goal than a single action, measurably improving the critic's representations, and rendering the algorithm significantly more effective.

Michal Korniak, Kamil Dybek, Benjamin Eysenbach +2

ETH Zurich · University of Warsaw · Princeton University +3

Reinforcement Learning Offline RL Representation Learning for RL Contrastive RL Self-Supervised RL

Ask PDF

Jul 23, 2026

Offline RL with Hierarchical Action Chunking

Offline goal-conditioned reinforcement learning (RL) holds the promise of learning general-purpose policies from static datasets. However, scaling these methods to long-horizon tasks remains a challenge due to the curse of horizon, where value estimation errors can compound through long chains of bootstrapped Bellman backups. Existing hierarchical approaches mitigate this by decomposing tasks into subgoals, yet they often rely on low-level controllers that suffer from myopic execution and biased value estimates. In this work, we propose Hierarchical Implicit Q-Chunking (HiQC), an offline goal-conditioned RL algorithm that combines high-level latent planning with low-level action chunking. By conditioning the low-level critic on temporally extended action sequences, HiQC enables unbiased k-step value backups, compressing the horizon at both the planning and execution levels. We theoretically demonstrate that this dual decomposition results in a tighter bound on value error under a bounded per-backup error model compared to standard hierarchy or flat chunking alone. Empirically, HiQC achieves the highest aggregate performance among the compared methods on the OGBench suite, with its largest gains on long-horizon navigation tasks such as humanoid-giant.

Ahad Jawaid

The University of Texas at Dallas

Hierarchical RL Offline RL Action Chunking Long-Horizon Agent Tasks Goal-Conditioned RL

Ask PDF

Abstract

Reinforcement Learning

Explore similar work

Adaptive Q-Chunking for Offline-to-Online Reinforcement Learning

Three Steps at a Time: Learning Representations from Action Sequences in Contrastive RL

Offline RL with Hierarchical Action Chunking