4DGS-WAM: Bridging Past and Future with an Object-Centric World Action Model based on 4D Gaussian Splatting

Yueen Ma; Zenglin Xu; Irwin King

cs.CVAug 26, 2026

4DGS-WAM: Bridging Past and Future with an Object-Centric World Action Model based on 4D Gaussian Splatting

Authors: Yueen Ma, Zenglin Xu, Irwin King

Organizations: The Chinese University of Hong Kong · Shanghai Academy of AI for Science · Fudan University

Abstract

Current world action models (WAMs) typically operate on 2D visual data. These models can achieve exceptional visual quality, but they lack explicit spatial structure for individual objects and repeatedly process redundant background content. Although point clouds can represent the world in 3D space, they can be difficult to align and accumulate across viewpoints. In this paper, we leverage an explicit 4D Gaussian Splatting (4DGS) representation that separately models dynamic objects and the static background of a scene. For dynamic objects, we use a policy model to predict future actor actions and a world model to predict transformations of their observed Gaussian splats. The static background need not be regenerated for future states, as much of it has already been observed in past frames. This forms an object-centric world action model, which we name 4DGS-WAM. It lifts 2D observations into a persistent 4D representation so that previously observed static content can be reused during future prediction. Future-state extrapolation can then focus on modeling the evolution of dynamic objects. Experiments on KITTI-MOT evaluate short-horizon prediction and past reconstruction.

Figures & tables

Panel of Figure 1 — Figure 1: Architecture of 4DGS-WAM. Past RGB frames $\mathcal{I}_{0:t}$ are processed by a 4DGS reconstruction module and vision foundation models for segmentation, depth, camera pose, and optical flow. Policy $\pi$ predicts target-horizon $\mathrm{SE}(3)$ actions $\widehat{\mathbf{a}}_{t\rightarrow t^{\prime}}$ , which condition $W$ to produce the future 4DGS state for rendering.

Panel of Figure 2 — Figure 2: World action model comparison. (a) Representative autoregressive WAMs. Image and action tokens are predicted jointly in a unified model. (b) 4DGS-WAM. Our model maintains a decomposed, renderable 4D Gaussian state: the static bank persists while policy-conditioned world dynamics transport only dynamic object splats.

Panel of Figure 3 — Figure 3: Architectures of the policy network and the world model. (a) Policy network $\pi$ . Each actor trajectory is augmented with a learned memory token $\mathbf{m}$ and a horizon embedding of the queried $h$ , and the resulting sequences are processed by stacked temporal-attention layers followed by actor-level $k$ NN attention. The action head reads $\mathbf{m}$ and predicts a target-horizon $\mathrm{SE}(3)$ action. (b) World model $W$ . Conditioned on the same $h$ and the predicted action, splat embeddings and category-specific part tokens are processed by stacked geometric tensor layers. Residual $\mathrm{SE}(3)$ twists predicted at the part nodes are blended onto splats, and the resulting transformations are composed with the object action; source appearance attributes are updated through residual mixing.

Method	Camera	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Epona	given	$17.46{\pm}1.21$	$0.482{\pm}0.054$	$0.203{\pm}0.021$
DriveDreamer-2	given †	$13.73{\pm}2.67$	$0.360{\pm}0.082$	$0.426{\pm}0.064$
Envision4D	predicted	$17.19{\pm}0.65$	$0.514{\pm}0.072$	$0.307{\pm}0.069$
4DGS-WAM (policy)	given	$\mathbf{18.80{\pm}1.21}$	$\mathbf{0.596{\pm}0.091}$	$\mathbf{0.161{\pm}0.042}$
4DGS-WAM (policy)	predicted	$16.47{\pm}3.05$	$0.482{\pm}0.033$	$0.245{\pm}0.107$

Table 1: Future prediction on KITTI-MOT. Mean

{\pm}

std over the three sequence-level means at horizons

h{=}1,3

(

6

frames per method). Every method is scored on the same prefixes and target frames. The best given-camera result in each column is bold; predicted-camera rows are not eligible for this mark. † DriveDreamer-2 uses KITTI-adapted conditioning in place of its native nuScenes inputs.

Point Cloud Reconstruction
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	dyn-PSNR $\uparrow$	dyn-LPIPS $\downarrow$
VGGT- $\Omega$	$6.66{\pm}0.17$	$0.046{\pm}0.017$	$0.861{\pm}0.038$	—	—
VGGT-SLAM	$6.64{\pm}0.39$	$0.105{\pm}0.051$	$0.746{\pm}0.070$	—	—
3DGS Reconstruction
3DGS	$14.54{\pm}2.14$	$0.477{\pm}0.087$	$0.525{\pm}0.156$	$12.83{\pm}3.72$	$0.504{\pm}0.142$
MonoGS	$20.42{\pm}5.55$	$0.652{\pm}0.225$	$0.236{\pm}0.177$	$15.51{\pm}5.12$	$0.224{\pm}0.159$

Table 2: Past reconstruction of observed frames on KITTI-MOT. Mean

{\pm}

std over three sequences. The 4DGS-WAM full-frame scores use a frozen NeoVerse fusion underlay; other methods are scored on their native outputs. Its dynamic-region scores come from the corresponding no-fusion evaluation.

Panel of Figure 4 — Figure 4: Future prediction on KITTI-MOT scenes. The three rows use horizons $h{=}1$ , $h{=}2$ , and $h{=}1$ , respectively. Left to right: ground truth, Epona, DriveDreamer-2, Envision4D, and policy-fed 4DGS-WAM. Epona and DriveDreamer-2 use the given future camera; Envision4D and 4DGS-WAM use their predicted cameras. The 4DGS-WAM panels include the frozen fusion underlay described in the text.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$t,t^{\prime},h=t^{\prime}-t$	Current time, target time, and prediction horizon
$\mathcal{I}_{n},\widehat{\mathcal{I}}_{t^{\prime}}$	RGB observation at time $n$ and rendered future prediction
$\mathcal{G}_{0:t},\mathcal{G}_{n},\widehat{\mathcal{G}}_{t^{\prime}}$	Reconstructed 4DGS, active snapshot, and predicted future state
$\mathcal{G}^{(o)}_{n},\mathcal{G}_{\mathrm{static}}$	Object-specific Gaussian subset and static Gaussian bank
$\mathbf{s}_{t}$	Reconstructed 4DGS, object associations, and center trajectories
$\mathcal{O}_{t},\mathcal{A}_{t}$	Dynamic scene objects and policy actors, where $\mathcal{A}_{t}=\mathcal{O}_{t}\cup\{\mathrm{ego}\}$

Appendix

Table 3: Core notation. The symbols follow the conventions of the main text.

World Action Models

World Action Models connect predictions of how scenes change under action to executable policies for robots, vehicles, and other embodied agents.

See all 221 papers

4D-WAM: Infusing Spatiotemporal Awareness into World Action Models through Trajectory Fields
Aug 8, 2026Lishan Yang, Wenxuan Song, Xi Wang +14World Models for Robotics Action-Conditioned World Models
Adelaide University · HKUST(GZ) · Tsinghua University +3
WAM4D: Fast 4D World Action Model via Spatial Register Tokens
Jun 12, 2026Ying Li, Xiaobao Wei, Jiajun Cao +10World Models for Robotics Robotic Manipulation
1Peking University · 2The Hong Kong University of Science and Technology · 3Beijing Innovation Center of Humanoid Robotics
Learning Action-Conditional and Object-Centric Gaussian Splatting World Models for Rigid Objects
Jun 1, 2026Jens U. Kreber, Lukas Mack, Joerg Stueckler3D Gaussian Splatting Dynamical Systems
Intelligent Perception in Technical Systems Group, University of Augsburg, Augsburg, Germany

Aug 8, 2026J/K move · Enter open · S save

4D-WAM: Infusing Spatiotemporal Awareness into World Action Models through Trajectory Fields

Lishan Yang, Wenxuan Song, Xi Wang, Pingyue Sheng+13

Adelaide University · HKUST(GZ) · Tsinghua University+3

World Models for Robotics Action-Conditioned World Models World Action Models

Open paper AskPDF arXiv

Building on recent advances in world models, World Action Models (WAMs) jointly model video prediction and action generation. However, they typically represent videos in 2D pixel space, creating a representation gap with 3D space in which robotic actions are executed. Recent 3D approaches introduce 3D information, but fail to fully exploit the dynamics of 3D structures. In this work, we propose 4D-WAM, a model-agnostic training strategy that injects spatiotemporal knowledge from 3D trajectory fields into WAMs through representation alignment. To this end, we introduce two complementary objectives: 1) motion alignment, which aligns temporal feature variations across adjacent frames and encourages the model to build local 4D awareness during training, and 2) destination alignment, which guides the model to infer the final destination from the source frame by minimizing the gap between their attention-like similarity distributions. Together, these objectives provide both local motion supervision and long-horizon goal guidance, enabling WAMs to learn trajectory-level spatiotemporal representations. Extensive in-distribution and out-of-distribution experiments across different base models demonstrate the model's improvements in spatial understanding, execution precision, robustness, generalization, and versatility.

Abstract

World Action Models

Explore similar work

4D-WAM: Infusing Spatiotemporal Awareness into World Action Models through Trajectory Fields

WAM4D: Fast 4D World Action Model via Spatial Register Tokens

Learning Action-Conditional and Object-Centric Gaussian Splatting World Models for Rigid Objects