World Motion Models: Flexible Sequence Modeling of SE(3) Trajectories

Jiahui Lei; Qianqian Wang; Trevor Darrell; Angjoo Kanazawa

cs.ROOct 1, 2026

World Motion Models: Flexible Sequence Modeling of SE(3) Trajectories

Authors: Jiahui Lei, Qianqian Wang, Trevor Darrell, Angjoo Kanazawa

Organizations: UC Berkeley · Harvard University

Abstract

Equipping artificial agents with spatial intelligence requires a comprehensive generative prior over the dynamic 3D world. We propose World Motion Models (WMMs) that capture "what was, is, and will be where across time" via sparse SE(3) pose trajectories. WMMs are built on the observation that elements of dynamic scenes can be well approximated by a set of rigid SE(3) trajectories, a minimal yet expressive primitive for 4D modeling. This representation unifies articulated objects, human bodies, hand-object interactions, piecewise-rigid scene dynamics, camera motion, and even robot states and actions into a single shared space. Given this representation, we cast the joint distribution of these entities as a flexible sequence modeling problem, utilizing flow-matching with per-token noise levels. Coupled with a context token mechanism for non-sequential conditioning, this formulation supports any-to-any marginal conditioning across an arbitrary number of entities and time steps. Tasks such as future prediction, motion infilling, model-predictive control, inverse kinematics, cross-embodiment retargeting, and policy learning all reduce to the application of different masks over the same network. Experiments on 6 diverse applications of 3D vision and robotics demonstrate the versatility and flexibility of WMMs with strong performance.

Figures & tables

Figure 1 : World Motion Models represent the 4D world across diverse entities and prediction modes in a unified manner, and can be flexibly conditioned for various downstream tasks.

Figure 2 : World Motion Models : (A) Dynamic scenes can be approximated by pose sequences. (B) During training, tokens corrupted at per-token noise levels are denoised to clean tokens. (C) During inference, various conditioning masks can be applied to one trained model.

Figure 3 : Network : The WMM architecture contains five factored attention types over a token grid padded with three register blocks, enabling efficient modeling (Sec. ).

Panel of Figure 4 — Figure 4 : Language Table. Top: inference pattern; history is the input, while future and goal are predicted by WMMs. Bottom: policy rollout results.

Panel of Figure 7 — Figure 7 : TraceGen. Qualitative results on the TraceGen benchmark [ 46 ] .

Panel of Figure 9 — Figure 9 : OMOMO. Qualitative results on OMOMO [ 51 ] . Top left: inference pattern. Top right: comparison with CHOIS [ 50 ] . Bottom: additional results from WMMs.

Panel of Figure 11 — Figure 11 : Human-to-Humanoid Retargeting. Top left: inference pattern. Top right: retargeting from real-world video. Bottom: comparison against baselines.

Figure 13 : More Applications. Qualitative results of WMMs on additional tasks.

Variant	SR $\uparrow$	Variant	MAE $\downarrow$	MSE $\downarrow$	Endp. $\downarrow$
WMM	0.860	WMM	2.167	0.329	0.580
no AdaLN	0.800	no eff.	2.356	0.344	0.602
no per-tk	0.000	plain DiT	32.05	16.97	17.64

Table 1 : Ablations. Language Table B2B (left), TraceGen EpicKitchen (right).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	SR-seq $\uparrow$	gerr $\downarrow$	lerr $\downarrow$
Paper ( $\delta=0.2$ )	0.8690	0.5938	0.0420
$\delta=0.6$	0.8645	0.6440	0.0449
$\delta=1.0$	0.8030	0.7709	0.0504

Appendix

Table 2 : Rotation-scale ablation on humanoid retargeting.

	Paper	noFsem	noFgeo	noBoth
Avg Succ Rate $\uparrow$	0.8720	0.8560	0.1960	0.1680

Appendix

Table 3 : Feature ablations on Language Table. noFsem and noFgeo zero the respective features; noBoth zeros both.

	EPIC			DROID
Setting	MAE $\downarrow$	MSE $\downarrow$	Endpoint $\downarrow$	MAE $\downarrow$	MSE $\downarrow$	Endpoint $\downarrow$
Paper	2.2564	0.3458	0.6146	1.1718	0.1665	0.2253
noFsem	2.4032	0.3627	0.6148	1.2894	0.1876	0.2539

Appendix

Table 4 : Semantic-feature ablation on TraceGen.

Bone length error	Bone direction error	Raw vs. projected MPJPE
$7.21$ mm	$2.99^{\circ}$	$19.15$ mm

Appendix

Table 5 : Self-consistency of SMPL predictions on the OMOMO test set.

Off-hinge-axis rotation	Off-hinge translation	Raw vs. projected MPJPE
$3.57^{\circ}$	$13.37$ mm	$31.38$ mm

Appendix

Table 6 : Self-consistency of predictions on the G1 humanoid test set.

Steps	Time, VRAM
Calibration : resize and run GeoCalib on key frames to undistort intrinsics and estimate the gravity/vertical direction.	4.3 s / 5,856 MB
Geometry : run chunked slice-and-align DA3 (to keep VRAM bounded on long sequences) to estimate depth and camera parameters; also run DA3-metric on key frames to compute the metric scale.	34.6 s / 14,137 MB
Tracking : run 2D tracking using TAPNext and multiple rounds of motion-aware resampling (to keep VRAM bounded and focus more on moving areas), then lift the 2D tracks to 3D at visible time steps using the camera parameters and depth, with depth-boundary and anti-jitter filtering.	64.6 s / 8,151 MB
[OPT] Full VOS (not used in the paper) : our pipeline also runs DEVA in auto-mode to segment and track anything in the full video.	27.2 s / 14,079 MB
Total cost	131 s / 14.1 GB

Appendix

Table 7 : RGB-video-to-3D-track processing cost.

Explore similar work

4D-WAM: Infusing Spatiotemporal Awareness into World Action Models through Trajectory Fields
Aug 8, 2026Lishan Yang, Wenxuan Song, Xi Wang +14Efficient World-Action Model Faster-Wam
Adelaide University · HKUST(GZ) · Tsinghua University +3
MoWAM: Explicit Future Motion Prediction for Efficient World Action Models
Sep 17, 2026Jiayu Wang, Bin Zhu, Yue Yu +1Efficient World-Action Model World Models
College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China · Singapore Management University, Singapore · Institute of Trustworthy Embodied AI, Fudan University, Shanghai, China
Physical Object Understanding with a Physically Controllable World Model
May 30, 2026Rahul Venkatesh, Klemen Kotar, Lilian Naing Chen +9Video World Models World Models
Stanford University · OpenAI · Noetik Inc. +1

Aug 8, 2026, cs.ROJ/K move · Enter open · S save

4D-WAM: Infusing Spatiotemporal Awareness into World Action Models through Trajectory Fields

Lishan Yang, Wenxuan Song, Xi Wang, Pingyue Sheng+13

Adelaide University · HKUST(GZ) · Tsinghua University+3

Efficient World-Action Model Faster-Wam World Models Future Video Prediction Spatio-Temporal Reasoning Trajectory Generation

Open paper AskPDF arXiv

Building on recent advances in world models, World Action Models (WAMs) jointly model video prediction and action generation. However, they typically represent videos in 2D pixel space, creating a representation gap with 3D space in which robotic actions are executed. Recent 3D approaches introduce 3D information, but fail to fully exploit the dynamics of 3D structures. In this work, we propose 4D-WAM, a model-agnostic training strategy that injects spatiotemporal knowledge from 3D trajectory fields into WAMs through representation alignment. To this end, we introduce two complementary objectives: 1) motion alignment, which aligns temporal feature variations across adjacent frames and encourages the model to build local 4D awareness during training, and 2) destination alignment, which guides the model to infer the final destination from the source frame by minimizing the gap between their attention-like similarity distributions. Together, these objectives provide both local motion supervision and long-horizon goal guidance, enabling WAMs to learn trajectory-level spatiotemporal representations. Extensive in-distribution and out-of-distribution experiments across different base models demonstrate the model's improvements in spatial understanding, execution precision, robustness, generalization, and versatility.

Abstract

Explore similar work

4D-WAM: Infusing Spatiotemporal Awareness into World Action Models through Trajectory Fields

MoWAM: Explicit Future Motion Prediction for Efficient World Action Models

Physical Object Understanding with a Physically Controllable World Model