Autoregressive Video Generation

Follow this topic Ask about this topic

Autoregressive video generation studies how to produce a sequence of frames or video chunks conditioned on earlier content, enabling streaming output and interactive control. Common approaches use causal diffusion transformers, often distilled from bidirectional models to reduce denoising steps, and add action, camera, audio, or text conditioning for world models and editing. To extend generation beyond short clips, methods train on model-generated histories and use retrieval, compressed KV caches, or structured memory to preserve identities, scenes, and motion. The central challenge is balancing speed and bounded memory with visual quality, diverse motion, and consistency over long rollouts, which benchmarks such as VBench-Long and task-specific interaction tests attempt to measure.

Latest papers

211 papers46 papers in the last four weeks, up 207% on the four weeks before. 0.3% of all new papers.

Lifelong Learning of Video Diffusion Models From a Single Video Stream
Jun 7, 2024Jason Yoo, Yingchen He, Saeid Naderiparizi +4Video Diffusion Models Video Generation
University of British Columbia · KU Leuven · Vector Institute +1

Jun 7, 2024J/K move · Enter open · S save

Lifelong Learning of Video Diffusion Models From a Single Video Stream

Jason Yoo, Yingchen He, Saeid Naderiparizi, Dylan Green+3

University of British Columbia · KU Leuven · Vector Institute+1

Video Diffusion Models Video Generation Autoregressive Video Generation

Open paper AskPDF arXiv

Video diffusion models can enable embodied agents to anticipate plausible futures from the recent past, but they are typically trained offline on curated datasets--a mismatch with the agents' learning setup at deployment: online, from a single video stream that sequentially outputs one frame at a time. We bridge this training gap and demonstrate that training autoregressive video diffusion models from such a stream, resembling the experience of embodied agents, is not only possible but can also perform comparably to standard offline training given the same number of gradient steps. We find that this robustness to video stream autocorrelation and nonstationarity can be achieved using experience replay methods that retain a subset of the video stream. To support training and evaluation in this setting, we introduce five new datasets for streaming lifelong generative video modeling: Lifelong Bouncing Balls (O), Lifelong Bouncing Balls (C), Lifelong 3D Maze, Lifelong Drive, and Lifelong PLAICraft, each consisting of one million consecutive frames from environments of increasing complexity. Together, our datasets and experiments lay the groundwork for video generative models and world models that continuously learn from single-sensor video streams rather than fixed datasets.