Forward Dynamics Models

Zipei Ma; Junzhe Jiang; Jiahui Zhang; Juntong Zhang; Chun Gu; Bozhou Zhang; Jiankang Deng; Xiatian Zhu

cs.ROOct 9, 2026

Forward Dynamics Models

Authors: Zipei Ma, Junzhe Jiang, Jiahui Zhang, Juntong Zhang, Chun Gu, Bozhou Zhang, Jiankang Deng, Xiatian Zhu, +1 more

Organizations: School of Data Science, Fudan University · Shanghai Innovation Institute · Imperial College London · University of Surrey

Abstract

Vision-language-action (VLA) models and world action models (WAMs) have advanced robotic policy learning by building on pretrained vision-language models (VLMs) and video generative models (VGMs), respectively. Yet VLAs learn without explicit supervision from future world states, while WAMs that condition actions on predicted futures incur extra inference cost and let visual prediction errors propagate into actions. We present Forward Dynamics Models (FDM), a family of robotic action models that couples action and future world-state prediction through an action-to-future paradigm. Future prediction is conditioned on action features, never the reverse: end-to-end training lets ground-truth future states supervise the policy through these features, while prediction errors cannot propagate into actions. At inference, FDM skips future generation entirely, avoiding its computational cost. The paradigm is backbone-agnostic, and we instantiate it with both VLM and VGM backbones. Experiments in simulation and on real robots demonstrate the effectiveness of FDM across both backbone families, achieving a state-of-the-art 99.4% average success rate on LIBERO and absolute gains of up to 13% over VLA and WAM methods on real-world tasks.

Figures & tables

Figure 1: Architectural comparison of action prediction and future supervision. (a) VLA predicts actions directly from the current observation and instruction. (b) A future-conditioned WAM conditions action prediction on predicted future video, requiring future generation at inference. (c) FDM conditions future prediction on context and action features through action-to-future learning. During training, action imitation and future prediction are jointly optimized; the future loss backpropagates through action features to supervise the policy. At inference, FDM denoises action chunks using only the current observation and instruction, skipping the future-generation branch marked as training only while retaining context processing.

Figure 2: Real-world success rate vs. latency. Success rates are averaged over five manipulation tasks on a dual-arm ARX-X5 robot: drawer packing, box packing and closing, shuttlecock insertion, paper-cup stacking, and block stacking, with 20 trials per method per task. Inference latency is measured on one NVIDIA H100 GPU.

Model family	Training	Inference
VLA	$p_{\theta}(A\mid C)$	$p_{\theta}(A\mid C)$
WAM	$p_{\theta}(V\mid C)\,p_{\theta}(A\mid C,V)$	$p_{\theta}(V\mid C)\,p_{\theta}(A\mid C,V)$
FDM	$p_{\theta}(A\mid C)\,p_{\theta}(V\mid C,A)$	$p_{\theta}(A\mid C)$

Table 1: Formulation comparison.

C=(O_{t},L)

: input context comprising the current observation

O_{t}

and task instruction

L

;

A

: action chunk;

V

: future world state target. WAM here denotes the future-conditioned formulation. For FDM, the formulation expresses action-to-future dependencies realized through action features; inference requires only action prediction.

Figure 3: Overview of our Forward Dynamics Models. The context expert encodes visual observations and the instruction to guide the action expert. During training, the generation expert predicts future states from context and action features; its loss backpropagates through these action features to complement action imitation. This asymmetric connection lets future supervision shape the policy while keeping action prediction independent of predicted futures. At inference, the action expert generates actions using the required context features, skipping future generation.

Table 5

Panel of Figure 4 — Figure 4: Real-world task examples. Each column shows two frames from one task, ordered chronologically from top to bottom: drawer packing, box packing and closing, shuttlecock insertion, cup stacking, and block stacking.

Figure 5: Main results on the five real-robot tasks. Per-task and average success rates, 20 trials per method per task. Both FDM variants outperform the corresponding comparison methods on average.

		Action-to-future pathway			Success rate (%) $\uparrow$
Model	Variant	Future	Reads $H_{a}$	Backprop	Drawer	Box	Shuttlecock	Cup	Block	Average
VLM	A. Action-only	✗	✗	✗	75	80	75	95	90	83
	B. Context-only future	✓	✗	✗	85	80	80	95	95	87
	C. Stop-gradient	✓	✓	✗	80	85	85	95	100	89
	D. FDM (Ours)	✓	✓	✓	95	95	90	100	100	96
VGM	A. Action-only	✗	✗	✗	50	60	55	85	80	66

Table 4: Action-to-future ablation on five real-robot tasks. Future: future supervision; Reads

H_{a}

: action-feature conditioning; Backprop: future-loss gradients through

H_{a}

Figure 6: Action sensitivity to future perturbations. Solid curves show FastWAM-Joint’s percentage increase in action error relative to

\alpha=0

Figure 7: Alignment between future and action gradients. Cosine similarity between action-imitation gradients and future-loss gradients propagated through the action-to-future interface.

Method	Step time $\downarrow$	Throughput $\uparrow$	Memory $\downarrow$	Overhead $\downarrow$
	(s/step)	(samples/s)	(GiB/GPU, peak)	(%)
$\pi_{0.5}$	0.4436	288.58	67.08	—
FDM w/ VLM	0.6305	203.01	72.04	$+42.14$
FastWAM-Joint	0.9754	131.23	66.80	—
FDM w/ VGM	0.9729	131.56	66.80	$-0.26$

Table 5: Training cost and inference efficiency. Comparisons use matched settings within each backbone family. (a)

\pi_{0.5}

(VLA) uses action supervision; FastWAM-Joint uses joint action–future supervision. (b) Only FastWAM-Joint requires future generation for action prediction. FastWAM-Joint denotes the same future-conditioned method evaluated in Figure 5 . Relative metrics use

\pi_{0.5}

for FDM w/ VLM and FastWAM-Joint for FDM w/ VGM.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Asymmetric attention mask in FDM. Rows denote queries; columns denote keys/values. Colored blocks allow attention; white blocks mask it. Context/Future denote context/generation tokens in FDM w/ VLM and current-observation/future-video tokens in FDM w/ VGM.

Model	Query group	Key/value group
		Context	Action	Generation
VLM	Context	✓	✗	✗
	Action	✓	✓	✗
	Generation	✓	✓	✓
		Current observation	Action	Future video
VGM	Current observation	✓	✗	✗

Appendix

Table 6: Attention permissions between token groups. Rows denote queries and columns denote keys/values; a checkmark permits attention. VLM generation tokens originate from historical and current image latents and are used to predict the future target. VGM language conditioning is separate from the groups shown here.

	FDM w/ VLM		FDM w/ VGM
Setting	LIBERO	RoboTwin	LIBERO	RoboTwin
Training episodes	1,659	26,950	1,659	27,225
Validation episodes	34	550	34	275
Validation fraction	2%	2%	2%	1%
Training steps	30,000	60,000	22,000	29,375
Warmup steps	1,000	1,000	1,100	1,468

Appendix

Table 7: Simulation training configurations. Counts and final optimizer steps follow the training logs. Validation intervals refer to offline validation.

Setting	LIBERO	RoboTwin
Number of tasks	40	50
Episodes per task per condition	50	100
Evaluation seed	42	42
Action denoising steps	10	10
FDM w/ VLM action prediction horizon	15	50
FDM w/ VGM action prediction horizon	32	32

Appendix

Table 8: Current default simulation evaluation settings, shared by both model families except for the prediction horizon. Historical evaluation runs require verification against these defaults.

Hyperparameter	Value
Peak learning rate	$2.5\times 10^{-5}$
Minimum learning rate	$0$
Weight decay	$10^{-10}$
Training steps / warmup steps	30,000 / 1,000
Per-device / effective batch size	8 / 64
Camera views	3

Appendix

Table 9: Training hyperparameters for FDM w/ VLM in the real-world experiments.

Hyperparameter	Value
Peak learning rate	$10^{-4}$
Minimum learning rate	$10^{-6}$
Weight decay	$10^{-2}$
Training steps / warmup steps	30,000 / 1,000
Per-device / effective batch size	8 / 64
Gradient accumulation steps	1

Appendix

Table 10: Training hyperparameters for FDM w/ VGM in the real-world experiments.

Figure 9: Real-world experimental setup. Two ARX-X5 robot arms are equipped with wrist-mounted D405 cameras, while a D435 camera provides a third-person view of the workspace.

Panel of Figure 10 — Figure 10: Drawer packing. The task requires opening the drawer, placing both the charger and block inside, and closing the drawer. The six frames are ordered chronologically from left to right.

Panel of Figure 11 — Figure 11: Box packing and closing. The robot places the scarf in the box, adds the flower, and closes the lid. The six frames are ordered chronologically from left to right.

Panel of Figure 12 — Figure 12: Shuttlecock insertion. One arm holds the tube while the other inserts the two shuttlecocks sequentially. The six frames are ordered chronologically from left to right.

Panel of Figure 13 — Figure 13: Paper-cup stacking. The robot arranges three cups into a base and places two cups above them to form a two-tier stack. The six frames are ordered chronologically from left to right.

Panel of Figure 14 — Figure 14: Block stacking. The robot places the small red block on the larger blue base block and then adds the small yellow block. The six frames are ordered chronologically from left to right.

	FDM w/ VLM		FDM w/ VGM
Task	Clean	Rand.	Clean	Rand.
adjust_bottle	100	100	100	100
beat_block_hammer	99	98	97	96
blocks_ranking_rgb	91	90	100	99
blocks_ranking_size	75	67	93	90
click_alarmclock	48	46	100	100

Appendix

Table 11: Per-task success on RoboTwin 2.0. Success rates are in percent. Each condition mean gives equal weight to all 50 tasks; the overall mean also weights clean and randomized conditions equally.

Explore similar work

Foresight Without Seeing: Latent Futures for World Action Models
Aug 12, 2026Jiakai Huang, Zhongbo Wu, Siyu Xu +5World Models for Robotics Visuomotor Policy Learning
Shanghai Jiao Tong University · ACE Robotics · Nanyang Technological University
LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies
Jun 14, 2026Jialei Chen, Kai Wang, Kang Chen +9Latent Action Learning Action-Conditioned World Models
Jilin University · Zhongguancun Academy · Nankai University +4
MotionWeave: Learning Motion-Centered Future Dynamics for Vision-Language-Action Policies
Sep 30, 2026Jingqiu Wang, Yan WangVisuomotor Policy Learning Robotic Manipulation
School of Data Science and Engineering East China Normal University, Shanghai, China

Aug 12, 2026J/K move · Enter open · S save

Foresight Without Seeing: Latent Futures for World Action Models

Jiakai Huang, Zhongbo Wu, Siyu Xu, Zheng Zhang+4

Shanghai Jiao Tong University · ACE Robotics · Nanyang Technological University

World Models for Robotics Visuomotor Policy Learning World Action Models Latent World Models

Open paper AskPDF arXiv

World Action Models (WAMs) connect visual prediction with robot control, but supplying predictive context often requires expensive future-video generation. Direct policies avoid this cost but lack an explicit interface for accessing future-indexed predictive information. We introduce ForeWAM, a World Action Model that separates forecasting from rendering to expose and shape latent predictive context for efficient control. Its core mechanism, Future-KV, performs a single Video DiT prefill over the current visual latent and noise-initialized future slots, then reuses the resulting key-value states throughout action denoising. To make this context relevant to control, we introduce dynamics registers supervised by latent actions from a frozen teacher during training, encouraging representations of interaction-induced transitions. This reusable context supports a lightweight, single-layer action decoder. We evaluate ForeWAM on LIBERO, LIBERO-Plus, RoboCasa, and real-world manipulation tasks. Without additional policy-level embodied pretraining, ForeWAM improves RoboCasa success by 9.7 percentage points over Fast-WAM at the same budget of 50 demonstrations per task, reaching 59.2%. With a single-layer decoder, it achieves 77.6% success on LIBERO-Plus and reduces policy-query latency to 88.7 ms on an NVIDIA A800, delivering a 6.27-fold speedup over Fast-WAM. These results show that latent predictive computation provides useful foresight for robust, efficient control without explicit future-video generation.

Abstract

Explore similar work

Foresight Without Seeing: Latent Futures for World Action Models

LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies

MotionWeave: Learning Motion-Centered Future Dynamics for Vision-Language-Action Policies