Predicting long action chunks, often spanning dozens of timesteps, has become a key ingredient in increasingly capable robot policies. In this paper, we find that adapting these long-chunk policies with existing reinforcement learning (RL) methods can require many online samples due to the high dimensionality of the action chunks being optimized, making such adaptation impractical in real-world scenarios where interaction data are costly to collect. Our key insight is to shift adaptation from optimizing actions to optimizing how long to execute each predicted action chunk. This keeps the adaptation space low-dimensional while its expressivity grows with the chunk size. We thus introduce Execution at the Right Length (ExRL), a simple approach which augments the base policy with a lightweight adapter that predicts the execution length of each chunk and is trained using off-policy RL algorithms. Across simulation benchmarks and real-world robotic tasks, ExRL enables sample-efficient adaptation of a diverse range of long-chunk policies with minimal task-specific hyperparameter tuning.
Figures & tables
Fig. 2: ExRL learns execution lengths with RL, and we optionally use Online BC to distill successful online rollouts back into the base policy during RL.
Fig. 3: ExRL can also leverage offline data by treating each H -step trajectory segment as a virtual action chunk, whose prefixes provide replay experience for different execution lengths.
Fig. 4: ExRL enables sample-efficient online adaptation of DP and ACT of chunk size 50 on Robomimic .
Fig. 6: ExRL enables sample-efficient online adaptation of different classes of generalist robot policies on Libero , including the flow-based VLA models π0 and π0.5 , the autoregressive VLA model π0 -FAST, and the world action model Cosmos Policy. All base policies are of chunk size 50 .
Fig. 9: How ExRL adapts execution length. (a) Mean execution lengths during free motion and near interaction, with 95% trajectory-bootstrap intervals. (b) Planned vertical motion over rejected and accepted full chunks on Square ; gray shows the median and 10–90% range of rejected proposals. (c) In another Square rollout, short executions progressively align the nut with the peg before ExRL selects a long execution.
Stack cube
Clothing folding
Power insertion
Parts organization
k=1
0/20
0/20
7/25
0/125
k=10
9/20
12/20
10/25
69/125
k=50
1/20
11/20
0/25
46/125
ExRL-Q
17/20
17/20
22/25
104/125
TABLE I: Real-world evaluation of π0.5 . Entries show successful trials / total trials for stacking, folding, and insertion, and completed stages / total stages for organization.
Fig. 7: Human demonstrations accelerate ExRL-Q on Can ( H=50 ).
Algorithm 1 Execution-Length Control via Q-Learning (ExRL-Q)
Algorithm 2 Execution-Length Control via Discrete Soft Actor-Critic (ExRL-SAC)
TABLE II: Training epochs of the Robomimic base policies used to initialize online adaptation. Fractional ACT epochs correspond to checkpoints saved within a data pass.
Fig. 11: DSRL on Can. Three-seed means and 95% CIs. Prefixes 2 – 32 share an H=32 checkpoint; prefix 50 uses an H=50 checkpoint.
Parameter
ExRL-Q
ExRL-SAC
Optimizer
Adam
Adam
Critic learning rate
3×10−4
3×10−4
Critic hidden layers
512×3
512×3
Activation
ReLU
ReLU
Critic ensemble size
10
10
Target critic subsample size
2
2
Appendix
TABLE III: Shared ExRL hyperparameters.
Setting
Changes to the shared defaults
Transport, all base policies
Critic and SAC actor hidden layers: 2048×3 .
Transport, DP; ACT with ExRL-SAC
γ=0.999 .
Tool Hang, all base policies
γ=0.999 .
LIBERO, all base policies
Replay capacity: 105 ; ExRL-Q UTD: 1 .
Appendix
TABLE IV: Main task-specific hyperparameter changes.
Parameter
DP
ACT
BC learning rate
10−5
10−5
BC batch size
256
128
BC updates / RL update
1/2
1/32
Success-buffer capacity (windows)
25,000
25,000
Teacher weight
1
1
Appendix
TABLE V: Online BC hyperparameters. DP settings apply to both Robomimic and Adroit.
Baseline
Hyperparameter selection
DSRL
Task-specific searches over the DSRL-NA/SAC variant, latent-noise range, learning rate, update frequency, discount, and network width.
Policy Decorator
Task-specific searches over residual scale, entropy coefficient, learning rates, progressive exploration, and actor/critic update frequencies.
PA-RL
Searches over critic capacity and update frequency, action-optimization steps and step size, and the learning rate, start time, and frequency of policy distillation.
ResFiT
Search on DP Square , then reuse the selected optimizer, residual, exploration, and update settings across tasks and backbones.
OGPO
Adapt the released learning objectives to DP and adjust update frequency for our collection setup and longer chunks.
DICE-RL
Use the paper’s Can recipe for the main learning hyperparameters, with adaptations to our policies, replay, and demonstration datasets.
Appendix
TABLE VI: Hyperparameter selection for the RL baselines.
Setting
Stack cube
Clothing folding
Power insertion
Parts organization
BC demonstrations
13
50
128
21
BC frames
2,995
38,412
22,865
19,288
Warmup rollouts
25
14
30
29
Offline critic updates
10,000
10,000
5,000
5,000
Online updates
UTD 16
UTD 16
400 /rollout
400 /rollout
Critic hidden dimensions
(512,512,512)
(512,512,512)
(1024,1024,1024)
(1024,1024,1024)
Appendix
TABLE VII: Real-world ExRL-Q settings. Rollout totals include warmup. Only the execution-length critic is updated during RL.
Fig. 12: Example setups for power insertion and parts organization.
Fig. 13: Online adaptation of DP and ACT on Robomimic with chunk size H=32 .
Fig. 14: PPO versus ExRL-Q on Can. Mean online success over 200 episodes, with 95% CIs.
Fig. 15: Evolution of execution lengths and policy entropy learned by ExRL-SAC. Orange solid curves show mean execution length; blue dashed curves show the categorical policy entropy on training minibatches. Shaded bands indicate 95% confidence intervals over three training seeds.
Fig. 16: Fixed- k sweep across base policies and tasks. We evaluate each base policy under different fixed execution lengths on Robomimic , LIBERO , and Adroit . Performance varies substantially with k , and the optimal fixed execution length differs across tasks and policies, highlighting the importance of adaptive execution-length control. Shaded bands indicate 95% confidence intervals over three seeds.
Fig. 17: ExRL varies execution length within a rollout to balance local adjustments and sustained motion. Representative frames and execution-length decisions across four real-world tasks. For example, stacking and folding use short executions near grasping, followed by longer motions to lift and transport the object or fabric; insertion concentrates short executions near contact, whereas parts organization sustains longer executions across much of the pick-and-place sequence.
Base policy
ExRL-Q
Permuted lengths
Gain (95% CI)
DP
95.8% (23/24)
37.5% (9/24)
58.3[37.5,79.2]
ACT
70.8% (17/24)
33.3% (8/24)
37.5[16.7,58.3]
Appendix
TABLE VIII: Conditional assignment matters beyond the length distribution. Square success rates (counts), and gains over the permuted control in percentage points with 95% cluster-bootstrap intervals.
Fig. 18: Frequent replanning can repeat the start of a useful action sequence. (a) Feature changes correlate with changes in the newly predicted first action across DP and ACT ( 95% CIs). (b) From the same state and initial chunk, replanning produces nearly unchanged commands, while longer execution preserves the changing actions within the chunk.
Base policy / task
Always k=1
k=32 , then ExRL-Q
ExRL-Q
ACT / Square
1/3
1/3
2/3
Appendix
TABLE IX: Completion from states with repeated first actions. Success counts from the same three selected ACT Square states; these are diagnostic cases, not benchmark success rates.
Action chunking emerged as a pivotal technique in imitation learning, enabling policies to predict cohesive action sequences rather than single actions. Recently, this approach has expanded to reinforcement learning (RL), enhancing behavioral consistency and reducing bootstrapping errors in value function estimation. However, existing methods rely on a fixed chunk length, creating a performance bottleneck as the optimal length varies across states and tasks. In this paper, we propose Adaptive Action CHunking (ACH), a novel offline-to-online RL algorithm that dynamically modulates chunk length during both training and inference. To find the optimal chunk length for a dynamically varying current state, we simultaneously estimate action-values for all candidate chunk lengths in a single forward pass, using a Transformer-based architecture. Our mechanism allows the agent to select the most effective chunk length adaptively based on the current state. Evaluated on 34 challenging tasks, ACH consistently outperforms fixed-length baselines, demonstrating superior generalization and learning efficiency in complex environments.
Long-horizon manipulation requires actions whose validity depends on earlier interactions, yet visual progress can conceal missing task prerequisites. We propose RoboLongR, an offline-to-online framework that learns history-dependent rewards to guide policy adaptation to new tasks. Its central objective is causal consistency: aligning rewarded progress with prerequisites supported by execution history. Stage-shuffle supervision encourages execution-order sensitivity, recurrent memory preserves evidence of earlier interactions, and two-round training refines trajectory segments using predicted progress. The resulting rewards guide stage-conditioned policy refinement from behavior cloning to online reinforcement learning. Sparse terminal feedback connects policy experience to reward learning by triggering correction for repeated high-scoring failures. Pretrained on demonstrations from 88 RLBench tasks and Open-X data, RoboLongR achieves 31.8% average success on 18 long-horizon tasks excluded from reward pretraining, exceeding the strongest evaluated baseline by 9.2 percentage points. Across four real-robot tasks, it achieves 57.5% success, a 32.5-percentage-point improvement over the comparison method. Reward-level controls probe sensitivity to synthetic order perturbations, while ablations examine the contributions of individual components to policy improvement.
Zixuan Chen, Lin Cao, Fenggang Zhang +5
Nanjing University · Southeast University · Fudan University
Offline-to-online reinforcement learning with action chunking eliminates multi-step off-policy bias and enables temporally coherent exploration, but all existing methods use a fixed chunk size across every state. This is suboptimal: near contact events the agent needs short chunks for reactive control, while during free-space motion long chunks provide better credit assignment. The natural solution is to train critics for several chunk sizes and select the best one at each state, but naive comparison of learned critic values systematically collapses to the shortest chunk due to discount-scale mismatch, and degrades to noise in low-value states. We propose Adaptive Q-Chunking (AQC), which resolves both failures by comparing the advantage of each chunk size relative to a per-horizon baseline, normalized by the discount factor. This criterion converts biased wrong answers into unbiased near-random choices when no genuine signal exists, and becomes discriminative when a particular scale enables better planning. We prove theoretical bounds on the advantage selector's noise immunity and on the value dominance of adaptive chunking over any fixed chunk size. We demonstrate that AQC achieves state-of-the-art offline and online success rates on OGBench and Robomimic, and can be applied to enhance the performance of large-scale VLA models that predict action sequences, significantly boosting performance on RoboCasa-GR1 tasks.
Nandiraju Gireesh, Yuanliang Ju, He Wang
Peking University · Galbot · University of Toronto