Predicting long action chunks, often spanning dozens of timesteps, has become a key ingredient in increasingly capable robot policies. In this paper, we find that adapting these long-chunk policies with existing reinforcement learning (RL) methods can require many online samples due to the high dimensionality of the action chunks being optimized, making such adaptation impractical in real-world scenarios where interaction data are costly to collect. Our key insight is to shift adaptation from optimizing actions to optimizing how long to execute each predicted action chunk. This keeps the adaptation space low-dimensional while its expressivity grows with the chunk size. We thus introduce Execution at the Right Length (ExRL), a simple approach which augments the base policy with a lightweight adapter that predicts the execution length of each chunk and is trained using off-policy RL algorithms. Across simulation benchmarks and real-world robotic tasks, ExRL enables sample-efficient adaptation of a diverse range of long-chunk policies with minimal task-specific hyperparameter tuning.
Figures & tables
Fig. 2: ExRL learns execution lengths with RL, and we optionally use Online BC to distill successful online rollouts back into the base policy during RL.
Fig. 3: ExRL can also leverage offline data by treating each H -step trajectory segment as a virtual action chunk, whose prefixes provide replay experience for different execution lengths.
Fig. 4: ExRL enables sample-efficient online adaptation of DP and ACT of chunk size 50 on Robomimic .
Fig. 6: ExRL enables sample-efficient online adaptation of different classes of generalist robot policies on Libero , including the flow-based VLA models π0 and π0.5 , the autoregressive VLA model π0 -FAST, and the world action model Cosmos Policy. All base policies are of chunk size 50 .
Fig. 9: How ExRL adapts execution length. (a) Mean execution lengths during free motion and near interaction, with 95% trajectory-bootstrap intervals. (b) Planned vertical motion over rejected and accepted full chunks on Square ; gray shows the median and 10–90% range of rejected proposals. (c) In another Square rollout, short executions progressively align the nut with the peg before ExRL selects a long execution.
Stack cube
Clothing folding
Power insertion
Parts organization
k=1
0/20
0/20
7/25
0/125
k=10
9/20
12/20
10/25
69/125
k=50
1/20
11/20
0/25
46/125
ExRL-Q
17/20
17/20
22/25
104/125
TABLE I: Real-world evaluation of π0.5 . Entries show successful trials / total trials for stacking, folding, and insertion, and completed stages / total stages for organization.
Fig. 7: Human demonstrations accelerate ExRL-Q on Can ( H=50 ).
Algorithm 1 Execution-Length Control via Q-Learning (ExRL-Q)
Algorithm 2 Execution-Length Control via Discrete Soft Actor-Critic (ExRL-SAC)
TABLE II: Training epochs of the Robomimic base policies used to initialize online adaptation. Fractional ACT epochs correspond to checkpoints saved within a data pass.
Fig. 11: DSRL on Can. Three-seed means and 95% CIs. Prefixes 2 – 32 share an H=32 checkpoint; prefix 50 uses an H=50 checkpoint.
Parameter
ExRL-Q
ExRL-SAC
Optimizer
Adam
Adam
Critic learning rate
3×10−4
3×10−4
Critic hidden layers
512×3
512×3
Activation
ReLU
ReLU
Critic ensemble size
10
10
Target critic subsample size
2
2
Appendix
TABLE III: Shared ExRL hyperparameters.
Setting
Changes to the shared defaults
Transport, all base policies
Critic and SAC actor hidden layers: 2048×3 .
Transport, DP; ACT with ExRL-SAC
γ=0.999 .
Tool Hang, all base policies
γ=0.999 .
LIBERO, all base policies
Replay capacity: 105 ; ExRL-Q UTD: 1 .
Appendix
TABLE IV: Main task-specific hyperparameter changes.
Parameter
DP
ACT
BC learning rate
10−5
10−5
BC batch size
256
128
BC updates / RL update
1/2
1/32
Success-buffer capacity (windows)
25,000
25,000
Teacher weight
1
1
Appendix
TABLE V: Online BC hyperparameters. DP settings apply to both Robomimic and Adroit.
Baseline
Hyperparameter selection
DSRL
Task-specific searches over the DSRL-NA/SAC variant, latent-noise range, learning rate, update frequency, discount, and network width.
Policy Decorator
Task-specific searches over residual scale, entropy coefficient, learning rates, progressive exploration, and actor/critic update frequencies.
PA-RL
Searches over critic capacity and update frequency, action-optimization steps and step size, and the learning rate, start time, and frequency of policy distillation.
ResFiT
Search on DP Square , then reuse the selected optimizer, residual, exploration, and update settings across tasks and backbones.
OGPO
Adapt the released learning objectives to DP and adjust update frequency for our collection setup and longer chunks.
DICE-RL
Use the paper’s Can recipe for the main learning hyperparameters, with adaptations to our policies, replay, and demonstration datasets.
Appendix
TABLE VI: Hyperparameter selection for the RL baselines.
Setting
Stack cube
Clothing folding
Power insertion
Parts organization
BC demonstrations
13
50
128
21
BC frames
2,995
38,412
22,865
19,288
Warmup rollouts
25
14
30
29
Offline critic updates
10,000
10,000
5,000
5,000
Online updates
UTD 16
UTD 16
400 /rollout
400 /rollout
Critic hidden dimensions
(512,512,512)
(512,512,512)
(1024,1024,1024)
(1024,1024,1024)
Appendix
TABLE VII: Real-world ExRL-Q settings. Rollout totals include warmup. Only the execution-length critic is updated during RL.
Fig. 12: Example setups for power insertion and parts organization.
Fig. 13: Online adaptation of DP and ACT on Robomimic with chunk size H=32 .
Fig. 14: PPO versus ExRL-Q on Can. Mean online success over 200 episodes, with 95% CIs.
Fig. 15: Evolution of execution lengths and policy entropy learned by ExRL-SAC. Orange solid curves show mean execution length; blue dashed curves show the categorical policy entropy on training minibatches. Shaded bands indicate 95% confidence intervals over three training seeds.
Fig. 16: Fixed- k sweep across base policies and tasks. We evaluate each base policy under different fixed execution lengths on Robomimic , LIBERO , and Adroit . Performance varies substantially with k , and the optimal fixed execution length differs across tasks and policies, highlighting the importance of adaptive execution-length control. Shaded bands indicate 95% confidence intervals over three seeds.
Fig. 17: ExRL varies execution length within a rollout to balance local adjustments and sustained motion. Representative frames and execution-length decisions across four real-world tasks. For example, stacking and folding use short executions near grasping, followed by longer motions to lift and transport the object or fabric; insertion concentrates short executions near contact, whereas parts organization sustains longer executions across much of the pick-and-place sequence.
Base policy
ExRL-Q
Permuted lengths
Gain (95% CI)
DP
95.8% (23/24)
37.5% (9/24)
58.3[37.5,79.2]
ACT
70.8% (17/24)
33.3% (8/24)
37.5[16.7,58.3]
Appendix
TABLE VIII: Conditional assignment matters beyond the length distribution. Square success rates (counts), and gains over the permuted control in percentage points with 95% cluster-bootstrap intervals.
Fig. 18: Frequent replanning can repeat the start of a useful action sequence. (a) Feature changes correlate with changes in the newly predicted first action across DP and ACT ( 95% CIs). (b) From the same state and initial chunk, replanning produces nearly unchanged commands, while longer execution preserves the changing actions within the chunk.
Base policy / task
Always k=1
k=32 , then ExRL-Q
ExRL-Q
ACT / Square
1/3
1/3
2/3
Appendix
TABLE IX: Completion from states with repeated first actions. Success counts from the same three selected ACT Square states; these are diagnostic cases, not benchmark success rates.