Organizations: South China University of Technology · Tsinghua University · KlingAI Research · Nanjing University · Beijing Normal University · Shadow AI
World-action models (WAMs) predict scene changes to guide robot actions, yet their RGB futures entangle state changes with appearance and lack 3D geometry. Recent WAMs add geometry as a training signal or a control interface, or build native 3D models, but decide what geometry to model rather than how the action should read it. Intuitively, control-relevant cues lie at embodiment--object interactions and their transitions, yet probing existing WAMs reveals that their action attention underweights these interactions in favor of the rest of the embodiment. To focus the action on these interactions, we present Gnomon4D, a WAM that jointly generates future video, robot actions, and GeoDelta, a video-aligned 4D geometric motion field recording the image-plane displacement and depth change of embodiment and object points. With the embodiment and objects in one field, their relative motion, which marks an interaction, can be read off. Guided by this motion and the task, Interaction-Salient Attention (InSAttn) selects where the action reads, and Transition-Anchored Aggregation (TransAgg) selects when, around grasp and release. As GeoDelta encodes surface motion rather than embodiment-specific controls, it serves as a human--robot quasi-action that links action-free pretraining on EgoManip-5000H, which we curate, to robot mid-training on Robot-30H. Gnomon4D leads on RoboTwin~2.0, VLABench, and RoboDojo, and outperforms real-robot baselines, especially under visual shifts. Project page: https://droliven.github.io/Gnomon4D.
Figures & tables
Figure 1: Gnomon4D teaches the action where and when to focus: it jointly generates future video, actions, and GeoDelta, the 4D motion (Δu,Δv,Δd) of embodiment and object points, and reads GeoDelta at interaction regions and transition frames (top left). As a human–robot quasi-action, GeoDelta links pretraining on 5,000+ hours of egocentric video with mid-training on 30+ hours of robot data (top right). Gnomon4D leads in simulation (bottom left) and on a real robot, especially under visual shifts (bottom right).
Figure 2: Three routes to geometry in WAMs. Each route leaves a limitation (red). Gnomon4D (bottom right) generates GeoDelta jointly with video and actions and reads it where and when interactions occur, addressing the two limitations in the top row. Because GeoDelta is embodiment-agnostic, it also enables human-video pretraining and avoids the scarce 3D data limiting native 3D WAMs.
Figure 3: Overview of Gnomon4D. (a) The Video, GeoDelta, and Action experts generate their future modalities through joint attention, and an interaction readout enters the Action expert residually. (b) GeoDelta and Action share a denoising clock that finishes ahead of Video, so actions are ready before the video is refined. (c) InSAttn turns task-conditioned embodiment–object relative motion into saliency that selects interaction regions, and TransAgg aggregates the frames where this relation changes. Image tiles show RoboTwin examples.
Figure 4: GeoDelta supervision in the four domains. Each row shows four head-camera frames of one training demonstration with its instruction, and the GeoDelta from each frame t to t+4 encoded by Eq. ( 2 ). Neutral gray marks static embodiment or object pixels, and black marks background.
World action models (WAMs) have recently shown promise in jointly modeling future observations and executable robot actions. However, most existing WAMs still operate in 2D video or latent spaces, where visually plausible rollouts miss the 3D spatial constraints and occluded contact geometry required for precise manipulation. While geometric foundation models offer strong priors for recovering dense 3D structure and motion from visual observations, forcing WAMs to predict the dense 4D representation introduces costly geometric decoding and slows down causal action generation. To address the trade-off, we present WAM4D, a fast 4D world action model that uses lightweight spatial register tokens as training-time future-depth readouts to transfer pretrained geometric priors into a causal video-action transformer, then removes the register branch for lightweight action inference. To prevent non-causal shortcuts, we further design causal mixture attention for the Mixture-of-Transformers (MoT) WAM backbone, defining modality-specific visibility among video, action, and geometry tokens. Comprehensive experiments on RoboTwin 2.0 and challenging real-world manipulation tasks show that WAM4D improves spatial consistency and achieves competitive action prediction while maintaining efficient inference.
Ying Li, Xiaobao Wei, Jiajun Cao +10
1Peking University · 2The Hong Kong University of Science and Technology · 3Beijing Innovation Center of Humanoid Robotics
World Action Models (WAMs) have shown strong potential for robotic manipulation by jointly modeling visual future dynamics and executable action sequences. However, existing video-action co-training methods primarily optimize appearance-oriented video latents, which may insufficiently capture the temporally evolving geometry required for precise manipulation. We propose MECo-WAM, a Multi-Expert Co-Training World Action Model that injects action-relevant 4D geometric priors into video-action representations while preserving the original lightweight inference graph. During training, MECo-WAM combines video and action experts with a lightweight 4D expert supervised by relational targets from a frozen VGGT encoder. Asymmetric expert visibility prevents non-causal shortcuts from auxiliary geometry to action generation. To transfer geometric knowledge into the deployed video-action pathway, we introduce decayed 4D read-mask attention, which provides restricted current-frame geometric guidance early in training and progressively removes this dependency. We further propose action-aware temporal geometric distillation, which aligns within-frame geometric relations and their temporal evolution while emphasizing visual regions most relevant to robot actions. At deployment, all auxiliary 4D components are removed. Experiments on LIBERO (98.2%), RoboTwin 2.0 (92.6%), and challenging real-world manipulation tasks show that MECo-WAM improves manipulation performance without increasing inference cost.
Building on recent advances in world models, World Action Models (WAMs) jointly model video prediction and action generation. However, they typically represent videos in 2D pixel space, creating a representation gap with 3D space in which robotic actions are executed. Recent 3D approaches introduce 3D information, but fail to fully exploit the dynamics of 3D structures. In this work, we propose 4D-WAM, a model-agnostic training strategy that injects spatiotemporal knowledge from 3D trajectory fields into WAMs through representation alignment. To this end, we introduce two complementary objectives: 1) motion alignment, which aligns temporal feature variations across adjacent frames and encourages the model to build local 4D awareness during training, and 2) destination alignment, which guides the model to infer the final destination from the source frame by minimizing the gap between their attention-like similarity distributions. Together, these objectives provide both local motion supervision and long-horizon goal guidance, enabling WAMs to learn trajectory-level spatiotemporal representations. Extensive in-distribution and out-of-distribution experiments across different base models demonstrate the model's improvements in spatial understanding, execution precision, robustness, generalization, and versatility.
Lishan Yang, Wenxuan Song, Xi Wang +14
Adelaide University · HKUST(GZ) · Tsinghua University +3