VLALagBench: Understanding When and Why VLA Inference Latency Matters
Organizations: National University of Singapore · Nanyang Technological University · Stanford University · NVIDIA
Abstract
Vision-language-action (VLA) models offer a promising path toward general-purpose robot manipulation. While inference latency can affect task success in a constantly evolving physical world, the relationship between latency and task success rate remains poorly understood. In this paper, we introduce VLALagBench, a comprehensive benchmark for studying VLA inference latency sensitivity across task types, object speeds, execution modes (synchronous and asynchronous), and VLA policies. Using VLALagBench, we find that even a modest inference latency of 100 ms can nearly halve task success on moving targets, with sensitivity varying substantially across tasks and policies. These differences reflect task-specific requirements for precision and continuous motion. By overlapping inference with action execution, asynchronous execution can recover some of the lost success at the cost of more frequent inference requests. Building on these findings, we develop an empirical model that predicts task success from limited calibration data, achieving root-mean-square prediction errors of only 5.7-7.8% across various VLA policies. Given an inference system's latency and per-robot throughput, these predictions guide the selection of system-specific VLA algorithm configurations to maximize task success. We believe VLALagBench and these findings provide a solid foundation for future robotics algorithm-system co-design.
Figures & tables
| Task | (%) | GR00T N1.7 (%) | MolmoAct2 (%) |
|---|---|---|---|
| Release | 93.0 | 52.0 | 83.0 |
| Set-down | 93.5 | 56.0 | 74.0 |
| Grasping | 98.5 | 98.0 | 99.5 |
| Articulated | 94.5 | 69.5 | 81.0 |
| MolmoAct2 | GR00T N1.7 † | |||||
|---|---|---|---|---|---|---|
| Task | ||||||
| Release | 0.04 | 0.59 | 0.80 | 2.48 | 1.31 | |
| Set-down | 0.33 | 0.32 | 0.31 | 0.09 | 3.65 | |
| Grasping | 0.23 | 0.07 | 0.23 | 0.26 | ||
| Articulated | 1.41 | 0.09 | 1.37 | 1.19 | ||
| System | GPU / network | Latency (ms) | Rate limit (req/s) | Predicted Mode , | Success (%) Pred./Meas. |
|---|---|---|---|---|---|
| A | Jetson AGX Thor | 72.8 | Uncapped | Cont., 2 | |
| B | RTX 4090, WiFi 7 | 37.8 | 20.0 | Cont., 1 | |
| C | H100, 5G | 42.9 | 5.0 | Async., 4 | |
| D | H100, slow cloud | 217.9 | Uncapped | Lazy, 5 |
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| RMS error (points) | Release | Set-down | Grasping | Articulated | All |
|---|---|---|---|---|---|
| Synchronous | 7.1 | 2.8 | 5.5 | 3.8 | 5.1 |
| Lazy | 6.3 | 5.3 | 5.7 | 5.2 | 5.6 |
| Continuous | 6.5 | 5.7 | 6.7 | 6.1 | 6.3 |
| Every mode | 6.7 | 4.8 | 6.0 | 5.1 | 5.7 |
| Point sampling noise | 2.7 | 2.6 | 2.8 | 2.2 | 2.6 |
| RMS error (points) | Release | Set-down | Grasping | Articulated | All |
|---|---|---|---|---|---|
| : one complete curve and two points, no zero-latency data | |||||
| Median (80% of calibrations) | 8.5 (5.4–18.4) | 6.8 (4.2–15.0) | 9.1 (6.2–13.0) | 5.4 (4.2–10.9) | 7.7 (4.5–14.4) |
| Its zero-latency curve, median | 5.9 | 7.2 | 8.0 | 5.8 | 6.8 |
| Fit at 0, 100, 200 ms; held-out latencies | 6.7 | 4.8 | 6.0 | 5.1 | 5.7 |
| MolmoAct2 : one complete curve and two points, no zero-latency data | |||||
| Median (80% of calibrations) | 12.3 (8.6–19.0) | 12.2 (8.1–24.3) | 10.8 (8.2–14.0) | 7.4 (4.8–19.2) | 11.1 (6.5–19.4) |
| GPU | openpi | FlashRT |
|---|---|---|
| RTX 3090 | 131.6 | 66.3 |
| RTX 4090 | 60.6 | 32.8 |
| RTX 5090 | 56.8 | 27.4 |
| H100 | 66.3 | 17.9 |
| Mode | Points | RMS error | Mean error |
|---|---|---|---|
| Synchronous | 53 | 3.4 | |
| Lazy | 45 | 4.5 | |
| Continuous | 35 | 5.0 | |
| All | 133 | 4.2 |
| Synchronous | Lazy | Continuous | ||
| Task / stage | 0 ms | 200 ms | 200 ms | 200 ms |
| Release | ||||
| Pickup | 100 | 100 | 100 | 100 |
| Release | 92 | 96 | 98 | 96 |
| Success after release | 80 | 58 | 69 | 72 |
| Set-down | ||||
VLMs for Robotics
Vision-language models for robotics connect visual and linguistic understanding to grounded navigation, manipulation, and robot control.
See all 700 papers