Action-Conditioned Video Generation

Overview

Action-conditioned video generation studies how to make generated scenes respond predictably to inputs such as robot trajectories, camera motion, keyboard controls, or natural-language instructions. Common approaches adapt diffusion and autoregressive video models, injecting action signals through tokens, spatial maps, optical flow, 3D geometry, or learned latent actions; some jointly predict video and executable actions. The goal is not only visual realism but also accurate action following, physical plausibility, and consistency over long or interactive rollouts, with applications in robotics, driving, surgery, and games. A central challenge is balancing these properties with real-time speed and reliable generalization to unfamiliar actions, embodiments, and environments; visual quality metrics alone often do not measure whether a model is controllable or physically sound.

Latest papers

CardsList

126 papers15 papers in the last four weeks, down 25% on the four weeks before. 0.1% of all new papers.

  1. StoryTailor:A Zero-Shot Pipeline for Action-Rich Multi-Subject Visual Narratives

    Feb 24, 2026Jinghao Hu, Yuhe Zhang, GuoHua Geng +2Visual StorytellingMulti-Shot Video Generation

  2. Learning Physics from Pretrained Video Models: A Multimodal Continuous and Sequential World Interaction Models for Robotic Manipulation

    Feb 18, 2026Zijian Song, Qichang Li, Sihan Qin +4Robotic ManipulationWorld Models for Robotics

  3. Factored Latent Action World Models

    Feb 18, 2026Zizhao Wang, Chang Shi, Jiaheng Hu +4Latent World ModelsLatent Action Learning

  4. CustomX: Unified Character, Action, and Scene Customization in Video World Models

    Dec 18, 2025Yitong Wang, Fangyun Wei, Hongyang Zhang +2HOI GenerationAction-Conditioned Video Generation

  5. OmniNWM: Omniscient Driving Navigation World Models

    Oct 21, 2025Bohan Li, Zhuang Ma, Dalong Du +10Action-Conditioned Video GenerationWorld Model Learning