Learning to Act While Waiting: RL Finetuning of Generalist Robot Policies Under Inference Latency
Authors: Brian Zhu, Momen Khalil, E Harrison, Emanuele Poggi, Philipp Schmitt, Bernd Kast, Philine Meister, Pranav Atreya, Qiyang Li, Finn Ferchau, Cesar Colmenero, Yash Shahapurkar, Gokul Narayanan, Melih Erdogan, Kai Wurm, Georg von Wichert, Oier Mees, Eugen Solowjow, Andrew Wagenmaker, Sergey Levine
Organizations: Siemens · UC Berkeley · Microsoft · ETH Zurich
While reinforcement learning (RL) allows generalist robot policies to continually improve during deployment, the large model size of modern generalist policies, such as VLAs, poses a fundamental obstacle to effective RL improvement. In particular, their severe inference latency---which can lead to pauses or jerky movements---can alter the effective environment dynamics and, if not correctly accounted for, break the Markov assumption that RL relies on, causing standard RL algorithms to fail completely. In this work, we introduce a latency-aware framework, Asynchronous RL with Intermediate Information (ARLI), that enables RL-based improvement of generalist policies under inference delays. Our framework builds on asynchronous inference approaches, which interleave action generation with execution to hide latency, and addresses its incompatibility with RL by providing a low-latency RL policy design that maximizes reactivity within the inference window through two contributions: state augmentations that restore near-Markovian structure by incorporating committed actions and a mid-inference observation. We evaluate our approach across simulated and real-world manipulation tasks, and find that it enables effective finetuning under inference delays where standard RL fails entirely, even matching or exceeding the performance of standard RL in idealized no-latency settings.
Figures & tables
Figure 1: Enabling RL improvement of generalist robot policies under inference latency: Generalist policies incur significant action inference latency delays, either forcing pauses during inference or relying on stale observations, making RL improvement brittle. Arli uses intermediate actions and updated asynchronous observations to steer the next action chunk, enabling effective real-world improvement under asynchronous inference, and real-world VLA improvement.
Figure 2: An overview of our approach, Arli . Inference begins tdelay steps before the next action chunk is required, in which the VLM backbone πptvlm takes state st−tdelay as input. At step t−tdelayrl , RL policy πrl must be run to ensure that the action expert πptae is provided a noise input w once VLM inference finishes. πrl is conditioned on (a) the initial state st−tdelay , (b) intermediate actions a−9:0 , and (c) intermediate state st−tdelayrl . Once the action expert’s inference is complete, the next action played, a0 , starts at step ndelay within the newly-generated chunk At .
Figure 3: The benefit of intermediate state conditioning: In naive asynchronous inference, the initial state does not capture stochastic events (e.g., moving obstacles). By having the ability to condition on an intermediate state, our policy is capable of reacting to disturbances.
Figure 4: Comparison of Arli and Dsrl on simulation tasks with and without Rtc .
Figure 5: Comparison of Arli against other RL finetuning methods on real-world tasks. Left : task images. Middle : training success curves corresponding to average success rate in the last 10 episodes. Right : estimated throughput in successes/hour computed based on the duration of each success in the last 20 episodes and assuming failure is declared after 2x average success duration.
Figure 6
Figure 8: Arli performance at varying RL inference delays on mjc_swimmer .
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Kinetix
AlohaTransferCube
Real-World
Observation space
Symbolic
Pixels & State
Pixels & State
Image observation size
N/A
64 × 64
64 × 64
Actor and critic architecture
MLP
CNN+MLP
CNN+MLP
Number of actor and critic MLP layers
3
3
3
MLP hidden size
256
128
128
CNN features
N/A
(32,32,32,32)
(32,32,32,32)
Appendix
Table 1: Observation space and architecture used for each environment
Assembly
Shoe-in-Bag
Bag-Placement
Number of episodes
60
123
51
Total dataset steps
84821
474301
50041
Batch size
32
32
32
Approx. number of epochs
3
2
4
Warm-up steps
265
1000
1000
Decay steps
8000
30000
30000
Appendix
Table 2: Real-world experiments base policy training data details
Hyperparameter
Kinetix
AlohaTransferCube
Real-World
Batch size
256
256
256
Actor learning rate
0.0001
0.0001
0.0001
Critic learning rate
0.0003
0.0003
0.0003
Temperature learning rate
0.0003
0.0003
0.0003
Activation
ReLU
ReLU
ReLU
Discount factor
0.999
0.999
0.999
Appendix
Table 3: Hyperparameters used for Dsrl
Hyperparameter
Kinetix
AlohaTransferCube
Real-World
Denoising steps
5
10
10
Inference Delay ( tdelay )
[1,2,3,4]
[10,20]
10
Soft Masking
True
True
True
Max. Guidance Weight
5
10
10
Appendix
Table 4: Hyperparameters used for Rtc
Hyperparameter
Kinetix
AlohaTransferCube
Batch size
256
256
Actor learning rate
0.000001
0.000001
Critic learning rate
0.0001
0.0001
Activation
ReLU
ReLU
Discount factor
0.99
0.99
Target update rate ( τ )
0.005
0.005
Appendix
Table 5: Hyperparameters used for Residual RL
Hyperparameter
Kinetix
AlohaTransferCube
Real-World
Number of offline transitions
500
1000
1200
Offline training steps
0
0
24000
Offline noise distribution
N(0,1)
N(0,0.3)
N(0,1)
Clip noise
False
False
True (0.5)
Appendix
Table 6: Hyperparameters used for initial offline data collection
Figure 9: Variants of Arli (no Rtc ) with one of intermediate state or actions on mjc_swimmer , mjc_walker , car_launch and AlohaTransferCube .
Figure 10: Variants of Arli (with Rtc ) with one of intermediate state or actions on mjc_swimmer , mjc_walker , car_launch and AlohaTransferCube .
Figure 11: Arli performance at varying RL inference delays on mjc_swimmer . Left : without Rtc . Right : with Rtc .
Figure 12: Checkpoint-wise evolution of the Arli (actions only) noise distribution on four held-out Shoe-in-Bag episodes. Left : a representative frame from the middle of each trajectory. Middle : predicted noise mean (upper plot) and variance (lower plot) by inference step and checkpoint, averaged over latent noise dimensions. Right : across-checkpoint standard deviation of the predicted noise mean (upper plot) and variance (lower plot) by inference step and latent noise dimension.
Deploying robot policies in the physical world requires satisfying two fundamental desiderata: reliability and smooth real-time execution. However, deploying state-of-the-art generalist models presents challenges on both fronts. Achieving the precision and robustness required for real-world deployment necessitates sample-efficient online reinforcement learning (RL) to adapt pretrained models. Meanwhile, the increasing scale of robot foundation models has led to higher inference latency. To satisfy real-time constraints under high latency, modern systems adopt asynchronous inference with action chunking, overlapping policy computation with chunk execution to hide latency and enable smooth control. Despite their complementary roles, integrating asynchronous execution with gradient-based online RL remains underexplored. We present SmoothRL, an online RL framework that fine-tunes a pretrained policy within an asynchronous inference loop. SmoothRL follows a value-gradient paradigm, directly updating policy parameters using gradients of the action-value function with respect to policy actions. To enable correct optimization under asynchronous execution, SmoothRL explicitly models the asynchronous inference process during training. Specifically, each generated action chunk is partitioned by frame index into three regions: a committed region, consisting of actions committed by the previous inference cycle; an execution region, containing newly generated actions executed by the robot; and a discarded region, containing actions superseded by the next inference cycle. Gradients are propagated only through the execution region, ensuring policy optimization aligns with the trajectory distribution induced by asynchronous execution. We evaluate SmoothRL on real-world robotic tasks requiring high precision, as well as highly dynamic tasks that necessitate asynchronous execution.
Vision-Language-Action (VLA) models offer a promising path to generalist robot control, but their inference latency causes observation staleness when generated actions are executed asynchronously. Several methods have been proposed concurrently to mitigate this problem: inference-time inpainting (IT-RTC), training-time delay simulation (TT-RTC), future-state-aware conditioning (VLASH), and lightweight residual correction (A2C2). Each takes a fundamentally different approach, but they have so far been evaluated independently with different codebases, base policies, and protocols. We present a systematic comparison of these four methods under controlled conditions. We develop two unified codebases that integrate all methods with harmonized library and dataset versions, and we benchmark them on the Kinetix suite with MLPMixer policies and on the LIBERO manipulation benchmark with SmolVLA, sweeping inference delays up to d=20 control steps. A2C2's per-step residual correction is the most effective method on Kinetix, holding above 90% solve rate up to d=8, and also leads on LIBERO from d=4 onwards. IT-RTC is competitive at low delays but degrades sharply under long chunks (H=30) and high delays. TT-RTC is the most robust training-based method: stable across d_\max choices, generalizes beyond its training delay distribution, and adds zero inference overhead. VLASH exhibits a clear low-delay vs. high-delay trade-off governed by the fine-tuning delay range [0,d_\max]. Code is available at https://github.com/TheAyos/async-vla-inference
Flow-based vision-language-action (VLA) models are highly effective for generalist robot manipulation, yet their reliance on computationally expensive VLM encoding and multi-step iterative action generation imposes a significant latency bottleneck. The resulting inference latency makes it difficult for robots to respond quickly, especially in dynamic environments. We address this limitation with RAVEL (Rolling Asynchronous VLA Enabling Low-Latency Control), an asynchronous inference framework that addresses the computational bottlenecks of both the VLM backbone and the action expert. To reduce the delay from multi-step action denoising, RAVEL allows near-term actions to be executed after a single denoising step by carrying partially denoised future actions forward in a rolling buffer. To avoid blocking on slow VLM encoding, RAVEL decouples VLM encoding from rolling action generation, allowing the action expert to operate continuously using the latest available VLM context, while a lightweight Fast Observation Pathway (FOP) directly conditions the action expert on current observations. Across simulated and real-world manipulation tasks, RAVEL consistently achieves substantially lower response latency while maintaining the task capability of the underlying VLA, enabling high-frequency and responsive closed-loop control.
Yuhan Chen, Ke Yu, Pengfei Liu +3
Zhejiang University · Nanyang Technological University