Learning to Act While Waiting: RL Finetuning of Generalist Robot Policies Under Inference Latency

Brian Zhu; Momen Khalil; E Harrison; Emanuele Poggi; Philipp Schmitt; Bernd Kast; Philine Meister; Pranav Atreya

cs.ROAug 24, 2026

Learning to Act While Waiting: RL Finetuning of Generalist Robot Policies Under Inference Latency

Authors: Brian Zhu, Momen Khalil, E Harrison, Emanuele Poggi, Philipp Schmitt, Bernd Kast, Philine Meister, Pranav Atreya,

Organizations: Siemens · UC Berkeley · Microsoft · ETH Zurich

Abstract

While reinforcement learning (RL) allows generalist robot policies to continually improve during deployment, the large model size of modern generalist policies, such as VLAs, poses a fundamental obstacle to effective RL improvement. In particular, their severe inference latency---which can lead to pauses or jerky movements---can alter the effective environment dynamics and, if not correctly accounted for, break the Markov assumption that RL relies on, causing standard RL algorithms to fail completely. In this work, we introduce a latency-aware framework, Asynchronous RL with Intermediate Information (ARLI), that enables RL-based improvement of generalist policies under inference delays. Our framework builds on asynchronous inference approaches, which interleave action generation with execution to hide latency, and addresses its incompatibility with RL by providing a low-latency RL policy design that maximizes reactivity within the inference window through two contributions: state augmentations that restore near-Markovian structure by incorporating committed actions and a mid-inference observation. We evaluate our approach across simulated and real-world manipulation tasks, and find that it enables effective finetuning under inference delays where standard RL fails entirely, even matching or exceeding the performance of standard RL in idealized no-latency settings.

Figures & tables

Figure 1: Enabling RL improvement of generalist robot policies under inference latency: Generalist policies incur significant action inference latency delays, either forcing pauses during inference or relying on stale observations, making RL improvement brittle. Arli uses intermediate actions and updated asynchronous observations to steer the next action chunk, enabling effective real-world improvement under asynchronous inference, and real-world VLA improvement.

Figure 2: An overview of our approach, Arli . Inference begins $t_{\mathrm{delay}}$ steps before the next action chunk is required, in which the VLM backbone $\pi_{\mathrm{pt}}^{\mathrm{vlm}}$ takes state $s_{t-t_{\mathrm{delay}}}$ as input. At step $t-t_{\mathrm{delay}}^{\mathrm{rl}}$ , RL policy $\pi_{\mathrm{rl}}$ must be run to ensure that the action expert $\pi_{\mathrm{pt}}^{\mathrm{ae}}$ is provided a noise input $w$ once VLM inference finishes. $\pi_{\mathrm{rl}}$ is conditioned on (a) the initial state $s_{t-t_{\mathrm{delay}}}$ , (b) intermediate actions $a_{-9:0}$ , and (c) intermediate state $s_{t-t_{\mathrm{delay}}^{\mathrm{rl}}}$ . Once the action expert’s inference is complete, the next action played, $a_{0}$ , starts at step $n_{\text{delay}}$ within the newly-generated chunk $A_{t}$ .

Figure 3: The benefit of intermediate state conditioning: In naive asynchronous inference, the initial state does not capture stochastic events (e.g., moving obstacles). By having the ability to condition on an intermediate state, our policy is capable of reacting to disturbances.

Figure 4: Comparison of Arli and Dsrl on simulation tasks with and without Rtc .

Figure 5: Comparison of Arli against other RL finetuning methods on real-world tasks. Left : task images. Middle : training success curves corresponding to average success rate in the last 10 episodes. Right : estimated throughput in successes/hour computed based on the duration of each success in the last 20 episodes and assuming failure is declared after 2x average success duration.

Figure 8: Arli performance at varying RL inference delays on mjc_swimmer .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Kinetix	AlohaTransferCube	Real-World
Observation space	Symbolic	Pixels & State	Pixels & State
Image observation size	N/A	64 $\times$ 64	64 $\times$ 64
Actor and critic architecture	MLP	CNN+MLP	CNN+MLP
Number of actor and critic MLP layers	3	3	3
MLP hidden size	256	128	128
CNN features	N/A	$(32,32,32,32)$	$(32,32,32,32)$

Appendix

Table 1: Observation space and architecture used for each environment

	Assembly	Shoe-in-Bag	Bag-Placement
Number of episodes	60	123	51
Total dataset steps	84821	474301	50041
Batch size	32	32	32
Approx. number of epochs	3	2	4
Warm-up steps	265	1000	1000
Decay steps	8000	30000	30000

Appendix

Table 2: Real-world experiments base policy training data details

Hyperparameter	Kinetix	AlohaTransferCube	Real-World
Batch size	256	256	256
Actor learning rate	0.0001	0.0001	0.0001
Critic learning rate	0.0003	0.0003	0.0003
Temperature learning rate	0.0003	0.0003	0.0003
Activation	ReLU	ReLU	ReLU
Discount factor	0.999	0.999	0.999

Appendix

Table 3: Hyperparameters used for Dsrl

Hyperparameter	Kinetix	AlohaTransferCube	Real-World
Denoising steps	5	10	10
Inference Delay ( $t_{\mathrm{delay}}$ )	$[1,2,3,4]$	$[10,20]$	10
Soft Masking	True	True	True
Max. Guidance Weight	5	10	10

Appendix

Table 4: Hyperparameters used for Rtc

Hyperparameter	Kinetix	AlohaTransferCube
Batch size	256	256
Actor learning rate	0.000001	0.000001
Critic learning rate	0.0001	0.0001
Activation	ReLU	ReLU
Discount factor	0.99	0.99
Target update rate ( $\tau$ )	0.005	0.005

Appendix

Table 5: Hyperparameters used for Residual RL

Hyperparameter	Kinetix	AlohaTransferCube	Real-World
Number of offline transitions	500	1000	1200
Offline training steps	0	0	24000
Offline noise distribution	$\mathcal{N}(0,1)$	$\mathcal{N}(0,0.3)$	$\mathcal{N}(0,1)$
Clip noise	False	False	True (0.5)

Appendix

Table 6: Hyperparameters used for initial offline data collection

Figure 9: Variants of Arli (no Rtc ) with one of intermediate state or actions on mjc_swimmer , mjc_walker , car_launch and AlohaTransferCube .

Figure 10: Variants of Arli (with Rtc ) with one of intermediate state or actions on mjc_swimmer , mjc_walker , car_launch and AlohaTransferCube .

Figure 11: Arli performance at varying RL inference delays on mjc_swimmer . Left : without Rtc . Right : with Rtc .

Figure 12: Checkpoint-wise evolution of the Arli (actions only) noise distribution on four held-out Shoe-in-Bag episodes. Left : a representative frame from the middle of each trajectory. Middle : predicted noise mean (upper plot) and variance (lower plot) by inference step and checkpoint, averaged over latent noise dimensions. Right : across-checkpoint standard deviation of the predicted noise mean (upper plot) and variance (lower plot) by inference step and latent noise dimension.

RL for Robotics

Reinforcement learning for robotics trains robots to act through interaction, from locomotion and navigation to dexterous manipulation.

See all 253 papers

SmoothRL: Online Reinforcement Learning During Asynchronous Execution
Aug 30, 2026Guang Gao, Yuxuan Nong, Baifu Huang +1Robot Policy Learning Asynchronous RL
Astribot Team
Understanding Asynchronous Inference Methods for Vision-Language-Action Models
May 4, 2026Ayoub AgouzoulRobotic Control Efficient VLM Inference
École polytechnique · Systems Group, ETH Zürich
RAVEL: Asynchronous Rolling Inference for Flow-Based Vision-Language-Action Models
Sep 28, 2026Yuhan Chen, Ke Yu, Pengfei Liu +3Efficient VLA Model Inference Vision-Language-Action Models
Zhejiang University · Nanyang Technological University

Aug 30, 2026J/K move · Enter open · S save

SmoothRL: Online Reinforcement Learning During Asynchronous Execution

Guang Gao, Yuxuan Nong, Baifu Huang, Jianan Wang

Astribot Team

Robot Policy Learning Asynchronous RL Reinforcement Learning Robotic RL

Open paper AskPDF arXiv

Deploying robot policies in the physical world requires satisfying two fundamental desiderata: reliability and smooth real-time execution. However, deploying state-of-the-art generalist models presents challenges on both fronts. Achieving the precision and robustness required for real-world deployment necessitates sample-efficient online reinforcement learning (RL) to adapt pretrained models. Meanwhile, the increasing scale of robot foundation models has led to higher inference latency. To satisfy real-time constraints under high latency, modern systems adopt asynchronous inference with action chunking, overlapping policy computation with chunk execution to hide latency and enable smooth control. Despite their complementary roles, integrating asynchronous execution with gradient-based online RL remains underexplored. We present SmoothRL, an online RL framework that fine-tunes a pretrained policy within an asynchronous inference loop. SmoothRL follows a value-gradient paradigm, directly updating policy parameters using gradients of the action-value function with respect to policy actions. To enable correct optimization under asynchronous execution, SmoothRL explicitly models the asynchronous inference process during training. Specifically, each generated action chunk is partitioned by frame index into three regions: a committed region, consisting of actions committed by the previous inference cycle; an execution region, containing newly generated actions executed by the robot; and a discarded region, containing actions superseded by the next inference cycle. Gradients are propagated only through the execution region, ensuring policy optimization aligns with the trajectory distribution induced by asynchronous execution. We evaluate SmoothRL on real-world robotic tasks requiring high precision, as well as highly dynamic tasks that necessitate asynchronous execution.

Abstract

RL for Robotics

Explore similar work

SmoothRL: Online Reinforcement Learning During Asynchronous Execution

Understanding Asynchronous Inference Methods for Vision-Language-Action Models

RAVEL: Asynchronous Rolling Inference for Flow-Based Vision-Language-Action Models