Vision-language-action (VLA) models and world action models (WAMs) have advanced robotic policy learning by building on pretrained vision-language models (VLMs) and video generative models (VGMs), respectively. Yet VLAs learn without explicit supervision from future world states, while WAMs that condition actions on predicted futures incur extra inference cost and let visual prediction errors propagate into actions. We present Forward Dynamics Models (FDM), a family of robotic action models that couples action and future world-state prediction through an action-to-future paradigm. Future prediction is conditioned on action features, never the reverse: end-to-end training lets ground-truth future states supervise the policy through these features, while prediction errors cannot propagate into actions. At inference, FDM skips future generation entirely, avoiding its computational cost. The paradigm is backbone-agnostic, and we instantiate it with both VLM and VGM backbones. Experiments in simulation and on real robots demonstrate the effectiveness of FDM across both backbone families, achieving a state-of-the-art 99.4% average success rate on LIBERO and absolute gains of up to 13% over VLA and WAM methods on real-world tasks.
Figures & tables
Figure 1: Architectural comparison of action prediction and future supervision. (a) VLA predicts actions directly from the current observation and instruction. (b) A future-conditioned WAM conditions action prediction on predicted future video, requiring future generation at inference. (c) FDM conditions future prediction on context and action features through action-to-future learning. During training, action imitation and future prediction are jointly optimized; the future loss backpropagates through action features to supervise the policy. At inference, FDM denoises action chunks using only the current observation and instruction, skipping the future-generation branch marked as training only while retaining context processing.
Figure 2: Real-world success rate vs. latency. Success rates are averaged over five manipulation tasks on a dual-arm ARX-X5 robot: drawer packing, box packing and closing, shuttlecock insertion, paper-cup stacking, and block stacking, with 20 trials per method per task. Inference latency is measured on one NVIDIA H100 GPU.
Model family
Training
Inference
VLA
pθ(A∣C)
pθ(A∣C)
WAM
pθ(V∣C)pθ(A∣C,V)
pθ(V∣C)pθ(A∣C,V)
FDM
pθ(A∣C)pθ(V∣C,A)
pθ(A∣C)
Table 1: Formulation comparison. C=(Ot,L) : input context comprising the current observation Ot and task instruction L ; A : action chunk; V : future world state target. WAM here denotes the future-conditioned formulation. For FDM, the formulation expresses action-to-future dependencies realized through action features; inference requires only action prediction.
Figure 3: Overview of our Forward Dynamics Models. The context expert encodes visual observations and the instruction to guide the action expert. During training, the generation expert predicts future states from context and action features; its loss backpropagates through these action features to complement action imitation. This asymmetric connection lets future supervision shape the policy while keeping action prediction independent of predicted futures. At inference, the action expert generates actions using the required context features, skipping future generation.
Table 5
Figure 4: Real-world task examples. Each column shows two frames from one task, ordered chronologically from top to bottom: drawer packing, box packing and closing, shuttlecock insertion, cup stacking, and block stacking.
Figure 5: Main results on the five real-robot tasks. Per-task and average success rates, 20 trials per method per task. Both FDM variants outperform the corresponding comparison methods on average.
Action-to-future pathway
Success rate (%) ↑
Model
Variant
Future
Reads Ha
Backprop
Drawer
Box
Shuttlecock
Cup
Block
Average
VLM
A. Action-only
✗
✗
✗
75
80
75
95
90
83
B. Context-only future
✓
✗
✗
85
80
80
95
95
87
C. Stop-gradient
✓
✓
✗
80
85
85
95
100
89
D. FDM (Ours)
✓
✓
✓
95
95
90
100
100
96
VGM
A. Action-only
✗
✗
✗
50
60
55
85
80
66
Table 4: Action-to-future ablation on five real-robot tasks. Future: future supervision; Reads Ha : action-feature conditioning; Backprop: future-loss gradients through Ha .
Figure 6: Action sensitivity to future perturbations. Solid curves show FastWAM-Joint’s percentage increase in action error relative to α=0 .
Figure 7: Alignment between future and action gradients. Cosine similarity between action-imitation gradients and future-loss gradients propagated through the action-to-future interface.
Method
Step time ↓
Throughput ↑
Memory ↓
Overhead ↓
(s/step)
(samples/s)
(GiB/GPU, peak)
(%)
π0.5
0.4436
288.58
67.08
—
FDM w/ VLM
0.6305
203.01
72.04
+42.14
FastWAM-Joint
0.9754
131.23
66.80
—
FDM w/ VGM
0.9729
131.56
66.80
−0.26
Table 5: Training cost and inference efficiency. Comparisons use matched settings within each backbone family. (a) π0.5 (VLA) uses action supervision; FastWAM-Joint uses joint action–future supervision. (b) Only FastWAM-Joint requires future generation for action prediction. FastWAM-Joint denotes the same future-conditioned method evaluated in Figure 5 . Relative metrics use π0.5 for FDM w/ VLM and FastWAM-Joint for FDM w/ VGM.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Asymmetric attention mask in FDM. Rows denote queries; columns denote keys/values. Colored blocks allow attention; white blocks mask it. Context/Future denote context/generation tokens in FDM w/ VLM and current-observation/future-video tokens in FDM w/ VGM.
Model
Query group
Key/value group
Context
Action
Generation
VLM
Context
✓
✗
✗
Action
✓
✓
✗
Generation
✓
✓
✓
Current observation
Action
Future video
VGM
Current observation
✓
✗
✗
Appendix
Table 6: Attention permissions between token groups. Rows denote queries and columns denote keys/values; a checkmark permits attention. VLM generation tokens originate from historical and current image latents and are used to predict the future target. VGM language conditioning is separate from the groups shown here.
FDM w/ VLM
FDM w/ VGM
Setting
LIBERO
RoboTwin
LIBERO
RoboTwin
Training episodes
1,659
26,950
1,659
27,225
Validation episodes
34
550
34
275
Validation fraction
2%
2%
2%
1%
Training steps
30,000
60,000
22,000
29,375
Warmup steps
1,000
1,000
1,100
1,468
Appendix
Table 7: Simulation training configurations. Counts and final optimizer steps follow the training logs. Validation intervals refer to offline validation.
Setting
LIBERO
RoboTwin
Number of tasks
40
50
Episodes per task per condition
50
100
Evaluation seed
42
42
Action denoising steps
10
10
FDM w/ VLM action prediction horizon
15
50
FDM w/ VGM action prediction horizon
32
32
Appendix
Table 8: Current default simulation evaluation settings, shared by both model families except for the prediction horizon. Historical evaluation runs require verification against these defaults.
Hyperparameter
Value
Peak learning rate
2.5×10−5
Minimum learning rate
0
Weight decay
10−10
Training steps / warmup steps
30,000 / 1,000
Per-device / effective batch size
8 / 64
Camera views
3
Appendix
Table 9: Training hyperparameters for FDM w/ VLM in the real-world experiments.
Hyperparameter
Value
Peak learning rate
10−4
Minimum learning rate
10−6
Weight decay
10−2
Training steps / warmup steps
30,000 / 1,000
Per-device / effective batch size
8 / 64
Gradient accumulation steps
1
Appendix
Table 10: Training hyperparameters for FDM w/ VGM in the real-world experiments.
Figure 9: Real-world experimental setup. Two ARX-X5 robot arms are equipped with wrist-mounted D405 cameras, while a D435 camera provides a third-person view of the workspace.
Figure 10: Drawer packing. The task requires opening the drawer, placing both the charger and block inside, and closing the drawer. The six frames are ordered chronologically from left to right.
Figure 11: Box packing and closing. The robot places the scarf in the box, adds the flower, and closes the lid. The six frames are ordered chronologically from left to right.
Figure 12: Shuttlecock insertion. One arm holds the tube while the other inserts the two shuttlecocks sequentially. The six frames are ordered chronologically from left to right.
Figure 13: Paper-cup stacking. The robot arranges three cups into a base and places two cups above them to form a two-tier stack. The six frames are ordered chronologically from left to right.
Figure 14: Block stacking. The robot places the small red block on the larger blue base block and then adds the small yellow block. The six frames are ordered chronologically from left to right.
FDM w/ VLM
FDM w/ VGM
Task
Clean
Rand.
Clean
Rand.
adjust_bottle
100
100
100
100
beat_block_hammer
99
98
97
96
blocks_ranking_rgb
91
90
100
99
blocks_ranking_size
75
67
93
90
click_alarmclock
48
46
100
100
Appendix
Table 11: Per-task success on RoboTwin 2.0. Success rates are in percent. Each condition mean gives equal weight to all 50 tasks; the overall mean also weights clean and randomized conditions equally.
World Action Models (WAMs) connect visual prediction with robot control, but supplying predictive context often requires expensive future-video generation. Direct policies avoid this cost but lack an explicit interface for accessing future-indexed predictive information. We introduce ForeWAM, a World Action Model that separates forecasting from rendering to expose and shape latent predictive context for efficient control. Its core mechanism, Future-KV, performs a single Video DiT prefill over the current visual latent and noise-initialized future slots, then reuses the resulting key-value states throughout action denoising. To make this context relevant to control, we introduce dynamics registers supervised by latent actions from a frozen teacher during training, encouraging representations of interaction-induced transitions. This reusable context supports a lightweight, single-layer action decoder. We evaluate ForeWAM on LIBERO, LIBERO-Plus, RoboCasa, and real-world manipulation tasks. Without additional policy-level embodied pretraining, ForeWAM improves RoboCasa success by 9.7 percentage points over Fast-WAM at the same budget of 50 demonstrations per task, reaching 59.2%. With a single-layer decoder, it achieves 77.6% success on LIBERO-Plus and reduces policy-query latency to 88.7 ms on an NVIDIA A800, delivering a 6.27-fold speedup over Fast-WAM. These results show that latent predictive computation provides useful foresight for robust, efficient control without explicit future-video generation.
Jiakai Huang, Zhongbo Wu, Siyu Xu +5
Shanghai Jiao Tong University · ACE Robotics · Nanyang Technological University
Vision-Language-Action models (VLAs) leverage large-scale vision-language pretraining for semantic robot control, but often lack explicit foresight into how robot actions change the scene. World-Action Models (WAMs) address this limitation by conditioning policies on predicted futures, yet existing approaches typically rely on computationally expensive video generation with substantial pixel-level redundancy. We present LaWAM, a Latent World Action Model that exposes predictive dynamics to robot policies through compact latent visual subgoals instead of reconstructed future video. At the core of LaWAM is a latent-action-conditioned Latent World Model (LaWM). We obtain LaWM by training a latent action model in the latent space of a pretrained vision foundation model and repurposing its forward decoder to predict future observation features for scene evolution. LaWAM then conditions action generation on these predicted latent visual subgoals to enable dynamics-aware robot control. LaWAM achieves state-of-the-art or competitive success rates (SRs) across LIBERO (98.6% SR), RoboTwin (91.22% SR), and real-world manipulation tasks while retaining low-latency inference. LaWAM runs in 187 ms per action-chunk prediction and achieves up to 24x lower wall-clock latency than pixel-space WAMs.
Jialei Chen, Kai Wang, Kang Chen +9
Jilin University · Zhongguancun Academy · Nankai University +4
Vision-Language-Action (VLA) models have recently incorporated world models to provide richer dynamic supervision beyond sparse action labels. However, explicitly predicting future images or videos may include control-irrelevant appearance, while guidance derived from holistic future visual representations and shared global action features may fail to establish timestep-specific correspondence between actions and local visual changes. To address this issue, we propose MotionWeave, a motion-centric future-dynamics framework for action-chunk prediction with two modules: the Action-Induced Motion Grounder (AIMG) and the Horizon Residual Composer (HRC). Specifically, AIMG conditions on action and proprioceptive representations to construct horizon-specific queries that localize interaction regions associated with each future action timestep from current visual tokens. HRC extracts differences between interaction representations at adjacent horizons, encodes them as temporal motion cues, and injects them into action tokens through a gated residual. During training, robot-arm masks rendered from future frames are used to construct KL-based motion-grounding supervision, while inference uses only the current observation. On six MetaWorld tasks, MotionWeave achieves a 75.3% average success rate, an absolute gain of 8.6% over π0 (66.7%), especially on sustained-interaction tasks. Our code is available at https://github.com/autu-mn/MotionWeave.
Jingqiu Wang, Yan Wang
School of Data Science and Engineering East China Normal University, Shanghai, China