Action-Conditioned Video Generation

Follow this topic Ask about this topic

Action-conditioned video generation studies how to make generated scenes respond predictably to inputs such as robot trajectories, camera motion, keyboard controls, or natural-language instructions. Common approaches adapt diffusion and autoregressive video models, injecting action signals through tokens, spatial maps, optical flow, 3D geometry, or learned latent actions; some jointly predict video and executable actions. The goal is not only visual realism but also accurate action following, physical plausibility, and consistency over long or interactive rollouts, with applications in robotics, driving, surgery, and games. A central challenge is balancing these properties with real-time speed and reliable generalization to unfamiliar actions, embodiments, and environments; visual quality metrics alone often do not measure whether a model is controllable or physically sound.

Latest papers

114 papers24 papers in the last four weeks, up 118% on the four weeks before. 0.1% of all new papers.

ACWM-Phys: Investigating Generalized Physical Interaction in Action-Conditioned Video World Models
May 9, 2026Haotian Xue, Yipu Chen, Liqian Ma +4Action-Conditioned Video Generation World Models
Georgia Institute of Technology
NoiseGate: Learning Per-Latent Timestep Schedules as Information Gating in World Action Models
May 8, 2026Wen Huang, Haoran Sun, Yongjian Guo +8Action-Conditioned Video Generation Action-Conditioned World Models
Tsinghua University · Peking University · JDT AI Infra +1
EA-WM: Event-Aware Generative World Model with Structured Kinematic-to-Visual Action Fields
May 7, 2026Zhaoyang Yang, Yurun Jin, Lizhe Qi +2Action-Conditioned Video Generation Video Diffusion Models
Fudan University · Zhongguancun Institute of Artificial Intelligence · University of Science and Technology of China +2
Unified 4D World Action Modeling from Video Priors with Asynchronous Denoising
Apr 29, 2026Jun Guo, Qiwei Li, Peiyan Li +7Action-Conditioned Video Generation Video Diffusion Models
Tsinghua University · Xiaomi Robotics · Peking University +1
Ego-InBetween: Generating Object State Transitions in Ego-Centric Videos
Apr 20, 2026Mengmeng Ge, Takashi Isobe, Xu Jia +7HOI Generation Action-Conditioned Video Generation
Advanced Micro Devices, Inc. · Dalian University of Technology · The Hong Kong University of Science and Technology +2
VAG: Dual-Stream Video-Action Generation for Embodied Data Synthesis
Apr 10, 2026Xiaolei Lang, Yang Wang, Yukun Zhou +10Synthetic Data Generation for Robotics World Action Models
GigaAI · Zhejiang University · Peking University +2
Action Images: End-to-End Policy Learning via Multiview Video Generation
Apr 7, 2026Haoyu Zhen, Zixian Gao, Qiao Sun +7Action Representation Learning World Action Models
UMass Amherst · NVIDIA · Harvard University +1
DriveVA: Video Action Models are Zero-Shot Drivers
Apr 5, 2026Mengmeng Liu, Diankun Zhang, Jiuming Liu +7Action-Conditioned Video Generation World Models
University of Twente, The Netherlands · Xiaomi EV, China · University of Cambridge, United Kingdom +1
LOME: Learning Human-Object Manipulation with Action-Conditioned Egocentric World Model
Mar 28, 2026Quankai Gao, Jiawei Yang, Qiangeng Xu +2HOI Generation Action-Conditioned Video Generation
University of Southern California, Los Angeles, USA · Waymo, Mountain View, USA · Max Planck Institute for Intelligent Syetems, Tübingen, Germany
ABot-PhysWorld: Interactive World Foundation Model for Robotic Manipulation with Physics Alignment
Mar 24, 2026Yuzhi Chen, Ronghan Chen, Dongjie Huo +11Embodied World Models Action-Conditioned Video Generation
AMAP CV Lab Alibaba Group
X-World: Controllable Ego-Centric Multi-Camera World Models for Scalable End-to-End Driving
Mar 20, 2026Chaoda Zheng, Sean Li, Jinhao Deng +9World Models for Autonomous Driving Action-Conditioned Video Generation
GWM Team, XPeng Inc.
GigaWorld-Policy: An Efficient Action-Centered World--Action Model
Mar 18, 2026Angen Ye, Boyuan Wang, Chaojun Ni +21World Action Models Visuomotor Policy Learning
Kinema4D: Kinematic 4D World Modeling for Spatiotemporal Embodied Simulation
Mar 17, 2026Mutian Xu, Tianbao Zhang, Tianqi Liu +3Robotics Simulation Embodied World Models
S-Lab, Nanyang Technological University · SSE, CUHKSZ
SAW: Toward a Surgical Action World Model via Controllable and Scalable Video Generation
Mar 13, 2026Sampath Rapuri, Lalithkumar Seenivasan, Dominik Schneider +9Trajectory-Guided Video Generation Action-Conditioned Video Generation
Johns Hopkins University, Baltimore MD, USA · NVIDIA
Composing Driving Worlds through Disentangled Control for Adversarial Scenario Generation
Mar 13, 2026Yifan Zhan, Zhengqing Chen, Qingjie Wang +7Compositional Video Generation Autonomous Driving Simulation
The University of Tokyo · Horizon Robotics · University of Glasgow
SPIRAL: Self-Evolving Action-Conditioned Video Generation via Reflective Planning Agents
Mar 9, 2026Yu Yang, Yue Liao, Jianbiao Mei +11Action-Conditioned Video Generation Long-Horizon Video Generation
Zhejiang University · KnowledgeXLab at Shanghai AI Lab · National University of Singapore +4
RealWonder: Real-Time Physical Action-Conditioned Video Generation
Mar 5, 2026Wei Liu, Ziyu Chen, Zizhang Li +3Physics-Guided Video Generation Action-Conditioned Video Generation
Stanford University · University of Southern California
Solaris: Building a Multiplayer Video World Model in Minecraft
Feb 25, 2026Georgy Savva, Oscar Michel, Daohan Lu +6Video World Models Action-Conditioned Video Generation
New York University
StoryTailor:A Zero-Shot Pipeline for Action-Rich Multi-Subject Visual Narratives
Feb 24, 2026Jinghao Hu, Yuhe Zhang, GuoHua Geng +2Visual Storytelling Multi-Shot Video Generation
College of Computer Science, Northwest University
Learning Physics from Pretrained Video Models: A Multimodal Continuous and Sequential World Interaction Models for Robotic Manipulation
Feb 18, 2026Zijian Song, Qichang Li, Sihan Qin +4Robotic Manipulation World Models for Robotics
Sun Yat-sen University · X-Era AI Lab · Guangdong University of Technology +1
Factored Latent Action World Models
Feb 18, 2026Zizhao Wang, Chang Shi, Jiaheng Hu +4Latent World Models Latent Action Learning
University of Texas at Austin · Sony AI
CustomX: Unified Character, Action, and Scene Customization in Video World Models
Dec 18, 2025Yitong Wang, Fangyun Wei, Hongyang Zhang +2HOI Generation Action-Conditioned Video Generation
Fudan University · Microsoft Research · University of Waterloo +1
OmniNWM: Omniscient Driving Navigation World Models
Oct 21, 2025Bohan Li, Zhuang Ma, Dalong Du +10Action-Conditioned Video Generation World Model Learning
Shanghai Jiao Tong University · Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo, China · PhiGent +3
Matrix-game 2.0: An open-source, real-time, and streaming interactive world model
Aug 18, 2025Xianglong He, Chunli Peng, Zexiang Liu +16Action-Conditioned Video Generation Real-Time Interactive Video Generation
Skywork AI

May 9, 2026J/K move · Enter open · S save

ACWM-Phys: Investigating Generalized Physical Interaction in Action-Conditioned Video World Models

Haotian Xue, Yipu Chen, Liqian Ma, Zelin Zhao+3

Georgia Institute of Technology

Action-Conditioned Video Generation World Models Action-Conditioned World Models OOD Generalization World Model Evaluation

Open paper AskPDF arXiv

Action-conditioned world models (ACWMs) have shown strong promise for video prediction and decision-making. However, existing benchmarks are largely restricted to egocentric navigation or narrow, task-specific robotics datasets, offering only limited coverage of the rich physical interactions required for generalized world understanding. We introduce ACWM-Phys, a new benchmark for evaluating action-conditioned prediction under diverse physical dynamics in a clean, controllable simulation environment with a carefully designed action space. ACWM-Phys contains training and evaluation data spanning rigid-body dynamics, kinematics, deformable-object interactions, and particle dynamics. To evaluate both interpolation and generalization, we design in-distribution and out-of-distribution protocols with controlled shifts in interaction patterns or scene configurations. By building the benchmark in a fully controllable simulator, ACWM-Phys enables precise data collection, reproducible evaluation, and systematic analysis of model capabilities for physically grounded world modeling. Through systematic experiments on ACWM-DiT, we find that OoD generalization depends not only on the physical regime but also on effective task complexity: models generalize well on visually simple, low-dimensional interactions with clear geometric structure, but suffer larger drops on deformable contacts, high-dimensional control, and complex articulated motion. This suggests that the model still relies heavily on visual appearance patterns instead of fully learning the underlying physics. Ablations show that cross-attention improves high-dimensional action conditioning, causal VAEs outperform frame-wise encoders, and larger action spaces are harder to model but can improve generalization by providing richer control signals. These findings guide the design of physically grounded world models.