Adapting Video Models for Robot Control via Motion-guided Asymmetric Prediction

Taeyoung Kim; Junyoung Sung; Heeseung Kwon; Minseop Kim; Jinwoo Shin

cs.CVOct 27, 2026

Adapting Video Models for Robot Control via Motion-guided Asymmetric Prediction

Authors: Taeyoung Kim, Junyoung Sung, Heeseung Kwon, Minseop Kim, Jinwoo Shin

Organizations: KAIST · RLWRLD

Abstract

World Action Models (WAMs) leverage pretrained video generative models for robotic control by jointly predicting future visual observations and actions. The visual prediction objective plays a key role in adapting pretrained video representations to the target robot domain. Yet standard visual prediction inherited from video pretraining treats all visual regions uniformly, despite their differing relevance to robotic control. We therefore investigate the spatial allocation of visual supervision in WAMs and show that focusing supervision on dynamic regions can substantially improve policy performance. Building on this finding, we introduce Motion-Guided Asymmetric Prediction (MoGAP), which augments WAM training with motion-focused visual prediction. MoGAP allocates noise asymmetrically across regions, training the model to reconstruct more heavily corrupted dynamic regions using less corrupted static regions as context. MoGAP is trained alongside standard video-action prediction through a training-only auxiliary prediction path within the shared video diffusion backbone. It uses motion cues derived directly from training videos, requiring neither an external teacher nor additional modalities. Across robot manipulation benchmarks and WAM architectures, MoGAP consistently improves policy performance, increasing success rates on the GR-1 tabletop benchmark from 38.8% to 47.5% for Fast-WAM (Wan2.2-5B) and from 52.3% to 62.3% for Cosmos3-Nano.

Figures & tables

Figure 1: Overview. (a) We spatially reallocate visual supervision in World Action Models (WAMs) toward dynamic regions. This reallocation emphasizes robot motion and object interactions while downweighting static, task-irrelevant visual details, such as complex tabletop textures. (b) Our method, Motion-Guided Asymmetric Prediction (MoGAP), substantially improves policy performance across different WAM setups on GR-1 Tabletop, a challenging humanoid manipulation benchmark.

Figure 2: Motion-Guided Asymmetric Prediction (MoGAP). MoGAP is a visual prediction task that promotes control-relevant representation learning in World Action Models. MoGAP introduces regional asymmetry in noise levels, guided by temporal differences between video latents: dynamic regions receive higher noise levels, while less corrupted static regions serve as context for prediction.

Figure 3: Motivating study. Redistributing supervision toward dynamic regions.

Figure 4: Attention mask of MoGAP . The MoGAP tokens ${\mathbf{z}}_{t}^{\bm{\tau}_{M}}$ are omitted at inference time.

Method	PnP	Articulated	24 Tasks Average
VLA-based methods
GR00T-N1.6 ( Bjorck et al., 2025 )	55.4	24.2	47.6
StarVLA- $\alpha$ ( Ye et al., 2026a )	58.4	54.0	57.3
RLDX-1 ( Kim et al., 2026a )	59.0	57.7	58.7
WAM-based methods
Fast-WAM ( Yuan et al., 2026 )	33.8	53.7	38.8

Table 1: Comparison with state-of-the-art VLAs & WAMs. Results include the success rate (%) of representative VLAs and WAMs on GR-1 tabletop. Results of Fast-WAM and Cosmos3 are reproduced while others are from the original work. Best results are highlighted in bold .

Method	Teacher	24 Tasks Average
Cosmos3	–	52.3
w/ Geometry Forcing ( Wu et al., 2026 )	VGGT	53.9
w/ AGRA ( Qiu et al., 2026 )	DINOv2	55.4
w/ TSA ( Kim et al., 2026b )	VideoMAEv2	56.8
w/ VideoREPA ( Zhang et al., 2025 )	VideoMAEv2	57.8
w/ VideoREPA ( Zhang et al., 2025 )	V-JEPA2.1-L	59.2

Table 2: Comparison to teacher-based supervision. Results report the average success rate (%) on GR-1 tabletop, with each method applied to the same baseline WAM, Cosmos3-Nano ( Agarwal et al., 2026 ) . Best results are highlighted in bold .

Table 3: Ablation studies on MoGAP. We study (a) the effectiveness of its key components through ablations and (b) the generalization of MoGAP across WAM architectures. Results report average success rates (%) on GR-1 Tabletop. Best results within each comparison are highlighted in bold .

Figure 5: RoboCasa-Kitchen success rate (%) across execution timesteps.

Figure 6: Real-world experiment success rate (%). Results on the OpenArm humanoid real-world benchmark. We report success rates with standard errors for each method across task suites.

Figure 7: Object relocation task example. The target object’s location is moved during execution. In set II, the object is relocated farther than in set I. MoGAP successfully adapts its trajectory.

Figure 8: Qualitative analysis. The columns show, from left to right, current and future ground-truth observations, latent motion scores, action-to-vision attention maps for the baseline and MoGAP, and their generated future observations.

Table 4: Quantitative analysis. We conduct additional analyses of the latent motion signal, and the trained policy models on GR-1 Tabletop.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Method	24 Tasks Average
Cosmos3 (22K)	55.8
MoGAP (16K)	62.3

Appendix

Table 5: Compute-matched comparison

Figure 9: Task examples of (a) RoboCasa GR-1 Tabletop, (b) RoboCasa-Kitchen, and (c) the multi-view observation input configuration.

Figure 11: Unseen objects task examples.

Figure 12: Unseen environments task examples.

Figure 13: Object relocation task examples (left: Set I; right: Set II)

Figure 14: Visualization of the OpenArm dataset distribution.

World Action Models

World Action Models connect predictions of how scenes change under action to executable policies for robots, vehicles, and other embodied agents.

See all 221 papers

Trace-WAM: Learning to Act via 3D Motion Prediction for World Action Models
Oct 27, 2026Xu Hu, Changjie Wu, Linzhe Shi +8World Model Learning Robotic Manipulation
The Hong Kong Polytechnic University · Amap, Alibaba Group · Tsinghua University +2
DC-WAM: Dynamic-Centric Visual Supervision and Reasoning for World-Action Models
Jul 28, 2026Haoyuan Ji, Lingxiang Fan, Shang Su +4World Models for Robotics Visuomotor Policy Learning
Tsinghua University · Dense-AI · University of Michigan
Video Prediction Policy 2: Predict Better, Act Better
Oct 7, 2026Yanjiang Guo, Haodong Yan, Zhide Zhong +15Visuomotor Policy Learning World Action Models
Robotera · Tsinghua University · HKUST (GZ) +2

Oct 27, 2026J/K move · Enter open · S save

Trace-WAM: Learning to Act via 3D Motion Prediction for World Action Models

Xu Hu, Changjie Wu, Linzhe Shi, Hang Zhang+7

The Hong Kong Polytechnic University · Amap, Alibaba Group · Tsinghua University+2

World Model Learning Robotic Manipulation Robot Motion Prediction

Open paper AskPDF arXiv

World Action Models (WAMs) learn representations for robot control by coupling future visual prediction with action generation. Successful manipulation depends primarily on how the robot and task-relevant objects evolve, whereas future-video prediction must also reconstruct the appearance and background of the entire scene. We therefore propose Trace-WAM, which augments video-action learning with future 3D motion prediction as an auxiliary objective. This objective supervises temporal point correspondence together with image-plane displacement and depth change, directly describing the interaction dynamics and spatial changes involved in manipulation. To this end, we adopt a three-branch Mixture-of-Transformer (Tri-MoT) architecture to jointly learn video prediction, motion prediction, and action generation. The motion expert predicts future trajectories from current visual features, allowing motion supervision to shape the representations used for action generation. Because the action expert does not consume predicted trajectories, inference retains the current-observation policy path: a single visual forward pass followed by action denoising. Experiments on RoboCasa-GR1, LIBERO, and LIBERO-Plus demonstrate strong in-distribution performance and out-of-distribution generalization, with particularly clear gains over Fast-WAM under limited demonstrations and distribution shifts.

Abstract

World Action Models

Explore similar work

Trace-WAM: Learning to Act via 3D Motion Prediction for World Action Models

DC-WAM: Dynamic-Centric Visual Supervision and Reasoning for World-Action Models

Video Prediction Policy 2: Predict Better, Act Better