Action-Conditioned Video Generation

Action-conditioned video generation studies how to make generated scenes respond predictably to inputs such as robot trajectories, camera motion, keyboard controls, or natural-language instructions. Common approaches adapt diffusion and autoregressive video models, injecting action signals through tokens, spatial maps, optical flow, 3D geometry, or learned latent actions; some jointly predict video and executable actions. The goal is not only visual realism but also accurate action following, physical plausibility, and consistency over long or interactive rollouts, with applications in robotics, driving, surgery, and games. A central challenge is balancing these properties with real-time speed and reliable generalization to unfamiliar actions, embodiments, and environments; visual quality metrics alone often do not measure whether a model is controllable or physically sound.