Action-Conditioned Video Generation

Overview

Action-conditioned video generation studies how to make generated scenes respond predictably to inputs such as robot trajectories, camera motion, keyboard controls, or natural-language instructions. Common approaches adapt diffusion and autoregressive video models, injecting action signals through tokens, spatial maps, optical flow, 3D geometry, or learned latent actions; some jointly predict video and executable actions. The goal is not only visual realism but also accurate action following, physical plausibility, and consistency over long or interactive rollouts, with applications in robotics, driving, surgery, and games. A central challenge is balancing these properties with real-time speed and reliable generalization to unfamiliar actions, embodiments, and environments; visual quality metrics alone often do not measure whether a model is controllable or physically sound.

Latest papers

CardsList

123 papers15 papers in the last four weeks, down 25% on the four weeks before. 0.1% of all new papers.

  1. Render the Robot, Generate the World: One Video World Model Across Embodiments

    Nov 18, 2026Yash Jangir, Karan Mirakhor, Renos Zabounidis +4World Models for RoboticsWorld Models

  2. Inertial Video World Models

    Nov 17, 2026Yuan Gao, Fan Xu, Yu Guo +4Video World ModelsMemory-Augmented World Models

  3. GameWM: Toward Next-Generation Video Games with Interaction-Centric World Models

    Nov 14, 2026Bing He, Runze Wang, Fei Zha +5Game World ModelsVideo Games

  4. QCC-WM: Linking Macro-State Dynamics and Visual Generation via Quotient Causal Construction

    Nov 9, 2026Zhizhong Fu, Wei Zhou, Zhaoyang Jiang +7Action-Conditioned World ModelsVideo World Models

  5. EgoGeneralist: Unifying Egocentric Action-Video Tasks through Video-Native Modeling

    Nov 5, 2026Yanlin Jin, Bowen Yang, Cheng Zhang +4Action-Conditioned Video GenerationEgocentric Vision

  6. ReactWorld: Structuring Interaction Responses for Embodied Video Generation

    Nov 4, 2026Hanwei Liu, Huijia Zhang, Hanzhang Wang +6Embodied World ModelsAction-Conditioned Video Generation

  7. Cycle WAM: Coupling Action and Video through Cycle Consistency

    Nov 3, 2026Qinglin Yang, Jiepeng Wang, Yuxing Huang +5World Action ModelsAction-Conditioned Video Generation

  8. FutureWorld: A Plug-and-Play Causal Next-State Predictor for Interactive Video World Exploration

    Oct 29, 2026Guangyuan Li, Kelu Yao, Shuolin Xu +6Interactive World ModelsVideo World Models

  9. RynnWorld-Latent: A Cross-Embodiment World Model Conditioned on Universal Latent Actions

    Oct 27, 2026Haoyu Zhao, Yinzhou Tang, Zixuan Wang +11Embodied World ModelsWorld Models

  10. FlockWorld: Toward Population-Scale Multi-Agent Video World Models through Artificial Life Simulation

    Oct 25, 2026Meryem Koksal, Yunzhe Wang, Volkan UstunVideo World ModelsAction-Conditioned Video Generation

  11. ACWM-RM: A Reference-Free Reward Model from Complementary Mismatch Preferences

    Oct 23, 2026Shibo Hong, Siyu Ren, FengJiao Chen +3Multimodal Reward ModelingRL for Video Generation

  12. Decoupling Action from Egocentric Observation for World Simulation

    Oct 20, 2026Yue Ma, Pengjie Song, Xinyu Wang +10Action-Conditioned Video GenerationHand-Object Interaction

  13. Training Controllable World Models via Action Influence Maximization

    Oct 19, 2026Samuel M. Bateman, Tenny Yin, Chongyi Zheng +5Action-Conditioned World ModelsAction-Conditioned Video Generation

  14. State World Models: Execute the Rules, Generate the Pixels

    Oct 19, 2026Chan Yan Ki, Yanran Zhang, Wenzhao Zheng +5World ModelsAction-Conditioned World Models

  15. ActionRoPE: Action as Coordinates for Spatially Consistent Control in Planar-Camera Game World Models

    Oct 19, 2026Zhiyang Deng, Yuzhe Wu, Zijun Lin +1Game World ModelsAction-Conditioned Video Generation

  16. Beckmann World Models: Direct Terminal-Map Learning for One-Step Action-Conditioned Prediction

    Oct 16, 2026Adam Lee, Shobhit Agarwal, June Pyo SuhWorld Model LearningAction-Conditioned World Models

  17. RIGen: Beyond Action Following with Reciprocal Interaction in Generative Gameplay

    Oct 14, 2026Mengfei Li, Yi Zhang, Xingtong Ge +8Real-Time Interactive Video GenerationInteractive World Models

  18. OneLA: Learning Many Worlds from Latent Actions

    Oct 14, 2026Ziyu Zhu, Zhaoxu Sun, Fan Jiang +3World Model LearningAction-Conditioned World Models

  19. StateCraft: Augmenting Video World Models with Abstract-State Reasoning

    Oct 14, 2026Lorenzo Venturoli, Nedko Savov, Xi Wang +2Video World ModelsLong-Horizon Video Generation

  20. SurgCast: Action-Conditioned Future Skeletons for Controllable Surgical World Models

    Oct 11, 2026Wanhao Liu, Rulin Zhou, Liangjing Shao +9World Models for RoboticsAction-Conditioned Video Generation

  21. UltraWorld: Learning Interactive Ultrasound World Models from Untracked Clinical Videos with Acoustic Sampling Map

    Oct 7, 2026Keke Yang, Erqi Wang, Sainan Guan +1Interactive World ModelsWorld Models for Robotics

  22. KineWorld: Action-Induced Transport Fields for Embodied World Modeling

    Oct 5, 2026Ziying Song, Yuchen Liu, Zhuoran Xu +6Action-Conditioned Video GenerationVideo Diffusion Models