Manipulating dynamic objects remains an open challenge for Vision-Language-Action (VLA) models. Although recent VLAs generalize well in static manipulation, dynamic scenes introduce a latency-induced perception-execution mismatch: object states continue to evolve during inference, making actions predicted from past observations stale at execution time. We present DynamicVLA, a latency-aware VLA model for dynamic object manipulation. It combines a compact 0.4B architecture and convolutional vision encoder for efficient multimodal inference with a continuous inference schedule that overlaps reasoning and execution for non-blocking control. Latent-aware Action Streaming then discards latency-invalid action prefixes and executes only the temporally valid suffix of each predicted chunk, preserving action-time alignment under dynamic object motion. To fill the missing foundation of dynamic manipulation data, we introduce the Dynamic Object Manipulation (DOM) benchmark, built with an automated collection pipeline that gathers 200K synthetic episodes across 2.8K scenes and 206 objects, and enables fast collection of 2K real-world episodes without teleoperation. Extensive evaluations in simulation and on real robots show that DynamicVLA improves dynamic manipulation success under changing object motion, perception-heavy instructions, and unseen motion patterns.
Figures & tables
Figure 2: The DynamicVLA architecture. Multimodal features condition a diffusion-based action expert for action-chunk prediction.
Figure 3: Automatic Simulation and Real-world Data Collection. Environment Setup: simulation and real-world settings share objects, scenes, and synchronized cameras. Object State Acquisition: simulation provides ground-truth 6D object states, while real-world RGB observations are converted into a real-world “simulator” interface that enables automatic dynamic-manipulation data collection without teleoperation.
Methods
Interaction
Perception
Generalization
Average
CR
DA
LS
VU
SR
MP
VG
MG
DR
Success ↑
Time ↓
Diffusion Policy [ 29 ]
0.50
0.50
0.00
1.00
0.00
0.00
1.00
0.50
0.00
0.38
10.89
ACT [ 16 ]
2.50
1.50
0.00
1.00
1.50
0.50
2.50
2.50
0.00
1.33
10.80
OpenVLA-OFT [ 54 ]
3.50
0.50
0.50
0.00
1.50
0.50
3.50
2.00
0.00
1.33
10.83
π0.5 [ 55 ]
9.50
17.50
3.50
5.00
12.50
9.00
5.00
19.50
18.00
11.06
10.62
RTC [ 17 ]
12.00
15.00
4.00
7.50
13.50
8.50
6.50
14.00
20.00
11.22
10.63
Table 1: Dynamic Object Manipulation Simulation Benchmark Results. Average success rates (%) are reported across nine evaluation sub-dimensions, organized under three categories: Interaction, Perception, and Generalization. In addition, overall average success rate (Success, %) and task completion time (Time, seconds) are reported. Each method is evaluated over 1,800 trials (10 scenes × 9 dimensions × 20 trials). All baseline models are fine-tuned on the DOM dataset using their official implementations and released pretrained weights. Best results are highlighted in bold.
Figure 4: Real-world Evaluation. We compare VLA models on 16 real-world dynamic manipulation tasks using Franka and PiPER as the manipulation robot. Each method–task pair is evaluated over 60 trials, comprising 20 trials for each of three paired motion–position configurations. An additional robot arm initializes object motion, providing comparable motion conditions across trials.
Table 5
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
t−3
t−2
t−1
t
SR ↑
T.Time ↓
I.Time ↓
✗
✗
✗
✓
38.22
9.52
0.225
✗
✗
✓
✓
43.39
8.77
0.226
✗
✓
✗
✓
47.06
8.53
0.226
✓
✗
✗
✓
46.89
8.51
0.226
✗
✓
✓
✓
47.11
8.46
0.228
✓
✓
✓
✓
47.06
8.53
0.229
Appendix
Table 4: Ablation on Temporal Visual Context. The temporal observation window is varied by enabling different visual frames at time steps {t−3,t−2,t−1,t} , while keeping the model architecture, inference frequency, and execution pipeline fixed. Note that SR, T.Time, and I.Time represent the success rate (in %), task completion time (in seconds), and inference time (in seconds, measured on an NVIDIA RTX A6000 GPU), respectively.
#Layers
SR ↑
T.Time ↓
I.Time ↓
#Param ↓
8
44.17
8.92
0.127
303
16
47.06
8.53
0.226
430
24
48.44
8.43
0.317
558
32
42.11
8.39
0.373
685
Appendix
Table 5: Ablation on LLM Depth. Different LLM depths are evaluated by retaining the first l transformer layers. Note that SR, T.Time, I.Time, and #Param denote success rate (%), task completion time (seconds), inference time (seconds, measured on an NVIDIA RTX A6000 GPU), and parameter count (in millions), respectively.
While Vision-Language-Action (VLA) models excel in static tasks, they struggle in dynamic environments where objects are in motion (e.g., conveyor belt manipulation). We identify three fundamental limitations hindering current VLAs in these scenarios: the \textbf{perception gap}, where static visual inputs lack temporal motion cues; the \textbf{latency gap}, where inference delays render actions obsolete; and the \textbf{control gap}, caused by the open-loop action chunk execution without real-time adjustment. In this work, we propose \textbf{DSDyn-VLA}, a Slow-Fast \textbf{D}ual-\textbf{S}tream \textbf{Dyn}amic manipulation framework that integrates motion-aware foresighted planning with real-time residual correction. The slow \textbf{Flow-Planner} serves as a macro-planner. By enhancing the VLA with optical flow for temporal perception and a future state awareness mechanism to preemptively offset inference latency, it produces globally consistent, motion-aware action chunks. Complementing this, the fast \textbf{Res-Refiner} employs a lightweight RL policy to inject high-frequency, closed-loop corrections into the planned action chunks based on real-time observations. In addition, we introduce \textbf{DynBench}, a MuJoCo-based benchmark for dynamic object manipulation that comprises nine tasks. Extensive experiments demonstrate that DSDyn-VLA reduces the failure rate by over 76% compared to current SOTA method in high-latency setting on the Kinetix dynamic benchmark, while achieving about 6× the success rate of PI0.5 in real-world dynamic settings and about 5× on DynBench. We will open-source all the code and weights.
Wenhao Li, Xiu Su, Yu Han +3
University of Sydney · Central South University · University of California, San Diego +1
Vision-Language-Action (VLA) models excel in static manipulation but struggle in dynamic environments with moving targets. This performance gap primarily stems from a scarcity of dynamic manipulation datasets and the reliance of mainstream VLAs on single-frame observations, restricting their spatiotemporal reasoning capabilities. To address this, we introduce DOMINO, a large-scale dataset and benchmark for generalizable dynamic manipulation, featuring 35 tasks with hierarchical complexities, over 110K expert trajectories, and a multi-dimensional evaluation suite. Through comprehensive experiments, we systematically evaluate existing VLAs on dynamic tasks, explore effective training strategies for dynamic awareness, and validate the generalizability of dynamic data. Furthermore, we propose PUMA, a dynamics-aware VLA architecture. By integrating scene-centric historical optical flow and specialized world queries to implicitly forecast object-centric future states, PUMA couples history-aware perception with short-horizon prediction. Results demonstrate that PUMA achieves state-of-the-art performance, yielding a 6.3% absolute improvement in success rate over baselines. Moreover, we show that training on dynamic data fosters robust spatiotemporal representations that transfer to static tasks. All code and data are available at https://github.com/H-EmbodVis/DOMINO.
Heng Fang, Shangru Li, Shuhan Wang +3
Huazhong University of Science and Technology, China · Huawei Technologies Co. Ltd, China
Vision-Language-Action (VLA) models generalize across static manipulation but fail when objects move during task execution. They map the current observation to an action and assume the scene is stationary between observation and execution, so at any non-trivial object speed the resulting latency exceeds the time available to grasp. We close this gap with AHEAD (Anticipatory Horizon Extrapolation with Adaptive Dynamics), a predict-then-act wrapper that augments a frozen VLA with a motion-aware latent world model. A small world model trained on manipulation video forecasts future patch tokens in the VLA's feature space, conditioned on per-token velocity and acceleration from optical flow. A language-and-motion saliency mask concentrates prediction on task-relevant patches, and the model rolls forward for an adaptive horizon, halting when prediction uncertainty crosses a threshold. The frozen action decoder then receives the predicted future tokens in place of the current ones. AHEAD adds 4.9M parameters to a frozen 7B OpenVLA and reaches 79 to 97% success across 20 dynamic simulation scenarios where the strongest baseline reaches 31 to 58%. On a physical UFactory xArm 7, AHEAD succeeds on 29/30 to 30/30 on three conveyor and rolling-ball tasks, 23/30 on paddle interception, and 19/30 on projectile catching where every baseline scores 0/30.
Shahram Najam Syed, Arthur Jakobsson, Haoran Hao +1
Robotics Institute, Carnegie Mellon University, Pittsburgh, USA