Current world action models (WAMs) typically operate on 2D visual data. These models can achieve exceptional visual quality, but they lack explicit spatial structure for individual objects and repeatedly process redundant background content. Although point clouds can represent the world in 3D space, they can be difficult to align and accumulate across viewpoints. In this paper, we leverage an explicit 4D Gaussian Splatting (4DGS) representation that separately models dynamic objects and the static background of a scene. For dynamic objects, we use a policy model to predict future actor actions and a world model to predict transformations of their observed Gaussian splats. The static background need not be regenerated for future states, as much of it has already been observed in past frames. This forms an object-centric world action model, which we name 4DGS-WAM. It lifts 2D observations into a persistent 4D representation so that previously observed static content can be reused during future prediction. Future-state extrapolation can then focus on modeling the evolution of dynamic objects. Experiments on KITTI-MOT evaluate short-horizon prediction and past reconstruction.
Figures & tables
Figure 1: Architecture of 4DGS-WAM. Past RGB frames I0:t are processed by a 4DGS reconstruction module and vision foundation models for segmentation, depth, camera pose, and optical flow. Policy π predicts target-horizon SE(3) actions at→t′ , which condition W to produce the future 4DGS state for rendering.
Figure 2: World action model comparison. (a) Representative autoregressive WAMs. Image and action tokens are predicted jointly in a unified model. (b) 4DGS-WAM. Our model maintains a decomposed, renderable 4D Gaussian state: the static bank persists while policy-conditioned world dynamics transport only dynamic object splats.
Figure 3: Architectures of the policy network and the world model. (a) Policy network π . Each actor trajectory is augmented with a learned memory token m and a horizon embedding of the queried h , and the resulting sequences are processed by stacked temporal-attention layers followed by actor-level k NN attention. The action head reads m and predicts a target-horizon SE(3) action. (b) World model W . Conditioned on the same h and the predicted action, splat embeddings and category-specific part tokens are processed by stacked geometric tensor layers. Residual SE(3) twists predicted at the part nodes are blended onto splats, and the resulting transformations are composed with the object action; source appearance attributes are updated through residual mixing.
Method
Camera
PSNR ↑
SSIM ↑
LPIPS ↓
Epona
given
17.46±1.21
0.482±0.054
0.203±0.021
DriveDreamer-2
given †
13.73±2.67
0.360±0.082
0.426±0.064
Envision4D
predicted
17.19±0.65
0.514±0.072
0.307±0.069
4DGS-WAM (policy)
given
18.80±1.21
0.596±0.091
0.161±0.042
4DGS-WAM (policy)
predicted
16.47±3.05
0.482±0.033
0.245±0.107
Table 1: Future prediction on KITTI-MOT. Mean ± std over the three sequence-level means at horizons h=1,3 ( 6 frames per method). Every method is scored on the same prefixes and target frames. The best given-camera result in each column is bold; predicted-camera rows are not eligible for this mark. † DriveDreamer-2 uses KITTI-adapted conditioning in place of its native nuScenes inputs.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
dyn-PSNR ↑
dyn-LPIPS ↓
Point Cloud Reconstruction
VGGT- Ω
6.66±0.17
0.046±0.017
0.861±0.038
—
—
VGGT-SLAM
6.64±0.39
0.105±0.051
0.746±0.070
—
—
3DGS Reconstruction
3DGS
14.54±2.14
0.477±0.087
0.525±0.156
12.83±3.72
0.504±0.142
MonoGS
20.42±5.55
0.652±0.225
0.236±0.177
15.51±5.12
0.224±0.159
Table 2: Past reconstruction of observed frames on KITTI-MOT. Mean ± std over three sequences. The 4DGS-WAM full-frame scores use a frozen NeoVerse fusion underlay; other methods are scored on their native outputs. Its dynamic-region scores come from the corresponding no-fusion evaluation.
Figure 4: Future prediction on KITTI-MOT scenes. The three rows use horizons h=1 , h=2 , and h=1 , respectively. Left to right: ground truth, Epona, DriveDreamer-2, Envision4D, and policy-fed 4DGS-WAM. Epona and DriveDreamer-2 use the given future camera; Envision4D and 4DGS-WAM use their predicted cameras. The 4DGS-WAM panels include the frozen fusion underlay described in the text.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Symbol
Meaning
t,t′,h=t′−t
Current time, target time, and prediction horizon
In,It′
RGB observation at time n and rendered future prediction
G0:t,Gn,Gt′
Reconstructed 4DGS, active snapshot, and predicted future state
Gn(o),Gstatic
Object-specific Gaussian subset and static Gaussian bank
st
Reconstructed 4DGS, object associations, and center trajectories
Ot,At
Dynamic scene objects and policy actors, where At=Ot∪{ego}
Appendix
Table 3: Core notation. The symbols follow the conventions of the main text.
Building on recent advances in world models, World Action Models (WAMs) jointly model video prediction and action generation. However, they typically represent videos in 2D pixel space, creating a representation gap with 3D space in which robotic actions are executed. Recent 3D approaches introduce 3D information, but fail to fully exploit the dynamics of 3D structures. In this work, we propose 4D-WAM, a model-agnostic training strategy that injects spatiotemporal knowledge from 3D trajectory fields into WAMs through representation alignment. To this end, we introduce two complementary objectives: 1) motion alignment, which aligns temporal feature variations across adjacent frames and encourages the model to build local 4D awareness during training, and 2) destination alignment, which guides the model to infer the final destination from the source frame by minimizing the gap between their attention-like similarity distributions. Together, these objectives provide both local motion supervision and long-horizon goal guidance, enabling WAMs to learn trajectory-level spatiotemporal representations. Extensive in-distribution and out-of-distribution experiments across different base models demonstrate the model's improvements in spatial understanding, execution precision, robustness, generalization, and versatility.
Lishan Yang, Wenxuan Song, Xi Wang +14
Adelaide University · HKUST(GZ) · Tsinghua University +3
World action models (WAMs) have recently shown promise in jointly modeling future observations and executable robot actions. However, most existing WAMs still operate in 2D video or latent spaces, where visually plausible rollouts miss the 3D spatial constraints and occluded contact geometry required for precise manipulation. While geometric foundation models offer strong priors for recovering dense 3D structure and motion from visual observations, forcing WAMs to predict the dense 4D representation introduces costly geometric decoding and slows down causal action generation. To address the trade-off, we present WAM4D, a fast 4D world action model that uses lightweight spatial register tokens as training-time future-depth readouts to transfer pretrained geometric priors into a causal video-action transformer, then removes the register branch for lightweight action inference. To prevent non-causal shortcuts, we further design causal mixture attention for the Mixture-of-Transformers (MoT) WAM backbone, defining modality-specific visibility among video, action, and geometry tokens. Comprehensive experiments on RoboTwin 2.0 and challenging real-world manipulation tasks show that WAM4D improves spatial consistency and achieves competitive action prediction while maintaining efficient inference.
Ying Li, Xiaobao Wei, Jiajun Cao +10
1Peking University · 2The Hong Kong University of Science and Technology · 3Beijing Innovation Center of Humanoid Robotics
World models enable intelligent agents to predict the consequences of their actions on the environment. In this paper, we propose Multi Rigid Object Gaussian World Model (MRO-GWM), a novel model that learns action-conditional dynamics of rigid objects in 3D. Using object-centric Gaussians, we can represent arbitrary object shapes and multi-object scenes. We develop a novel temporal-spatial transformer architecture that predicts future rigid body motion from a history of object Gaussians and future actions. Objects are represented by Gaussians in an object-centric frame, which allows to describe object motion as a rigid body transformation. Our model is trained on reconstructions from multiple viewpoints, which requires the model to also handle partial observations of objects due to occlusions. We analyze prediction performance of our approach on synthetic datasets composed of typical household objects with multi-object dynamics and interactions by a robot end effector. We also evaluate our model in model-predictive control for non-prehensile manipulation in simulation and for image-based tracking and prediction of object motion in simulated and real-world scenes.
Jens U. Kreber, Lukas Mack, Joerg Stueckler
Intelligent Perception in Technical Systems Group, University of Augsburg, Augsburg, Germany