Organizations: South China University of Technology · Tsinghua University · KlingAI Research · Nanjing University · Beijing Normal University · Shadow AI
World-action models (WAMs) predict scene changes to guide robot actions, yet their RGB futures entangle state changes with appearance and lack 3D geometry. Recent WAMs add geometry as a training signal or a control interface, or build native 3D models, but decide what geometry to model rather than how the action should read it. Intuitively, control-relevant cues lie at embodiment--object interactions and their transitions, yet probing existing WAMs reveals that their action attention underweights these interactions in favor of the rest of the embodiment. To focus the action on these interactions, we present Gnomon4D, a WAM that jointly generates future video, robot actions, and GeoDelta, a video-aligned 4D geometric motion field recording the image-plane displacement and depth change of embodiment and object points. With the embodiment and objects in one field, their relative motion, which marks an interaction, can be read off. Guided by this motion and the task, Interaction-Salient Attention (InSAttn) selects where the action reads, and Transition-Anchored Aggregation (TransAgg) selects when, around grasp and release. As GeoDelta encodes surface motion rather than embodiment-specific controls, it serves as a human--robot quasi-action that links action-free pretraining on EgoManip-5000H, which we curate, to robot mid-training on Robot-30H. Gnomon4D leads on RoboTwin~2.0, VLABench, and RoboDojo, and outperforms real-robot baselines, especially under visual shifts. Project page: https://droliven.github.io/Gnomon4D.
Figures & tables
Figure 1: Gnomon4D teaches the action where and when to focus: it jointly generates future video, actions, and GeoDelta, the 4D motion (Δu,Δv,Δd) of embodiment and object points, and reads GeoDelta at interaction regions and transition frames (top left). As a human–robot quasi-action, GeoDelta links pretraining on 5,000+ hours of egocentric video with mid-training on 30+ hours of robot data (top right). Gnomon4D leads in simulation (bottom left) and on a real robot, especially under visual shifts (bottom right).
Figure 2: Three routes to geometry in WAMs. Each route leaves a limitation (red). Gnomon4D (bottom right) generates GeoDelta jointly with video and actions and reads it where and when interactions occur, addressing the two limitations in the top row. Because GeoDelta is embodiment-agnostic, it also enables human-video pretraining and avoids the scarce 3D data limiting native 3D WAMs.
Figure 3: Overview of Gnomon4D. (a) The Video, GeoDelta, and Action experts generate their future modalities through joint attention, and an interaction readout enters the Action expert residually. (b) GeoDelta and Action share a denoising clock that finishes ahead of Video, so actions are ready before the video is refined. (c) InSAttn turns task-conditioned embodiment–object relative motion into saliency that selects interaction regions, and TransAgg aggregates the frames where this relation changes. Image tiles show RoboTwin examples.
Figure 4: GeoDelta supervision in the four domains. Each row shows four head-camera frames of one training demonstration with its instruction, and the GeoDelta from each frame t to t+4 encoded by Eq. ( 2 ). Neutral gray marks static embodiment or object pixels, and black marks background.