Real-Time Interactive Video Generation

Overview

The field studies how to generate video that responds as people interact, including controllable worlds, avatars, and live video editing. Common approaches adapt video diffusion or autoregressive models for causal streaming, then use few-step distillation, caching, and bounded memory to reduce latency; 3D geometry and physics can provide more explicit camera and action control. Systems are tested for visual quality, control accuracy, throughput, and latency, with benchmarks increasingly measuring long streams and interactive responsiveness. A central challenge is balancing fast reactions with realistic motion and consistent scenes or identities over long interactions, where drift and limited memory remain concerns.

Latest papers

92 papers16 papers in the last four weeks, up 23% on the four weeks before. 0.1% of all new papers.

Aug 18, 2025

Matrix-game 2.0: An open-source, real-time, and streaming interactive world model

Recent advances in interactive video generations have demonstrated diffusion model's potential as world models by capturing complex physical dynamics and interactive behaviors. However, existing interactive world models depend on bidirectional attention and lengthy inference steps, severely limiting real-time performance. Consequently, they are hard to simulate real-world dynamics, where outcomes must update instantaneously based on historical context and current actions. To address this, we present Matrix-Game 2.0, an interactive world model generates long videos on-the-fly via few-step auto-regressive diffusion. Our framework consists of three key components: (1) A scalable data production pipeline for Unreal Engine and GTA5 environments to effectively produce massive amounts (about 1200 hours) of video data with diverse interaction annotations; (2) An action injection module that enables frame-level mouse and keyboard inputs as interactive conditions; (3) A few-step distillation based on the casual architecture for real-time and streaming video generation. Matrix Game 2.0 can generate high-quality minute-level videos across diverse scenes at an ultra-fast speed of 25 FPS. We open-source our model weights and codebase to advance research in interactive world modeling.
Mar 21, 2025

Position: Next-Generation Game Engines Should Be Built on Interactive Generative Video

Modern game development faces significant challenges in creativity and cost due to predetermined content in traditional game engines. Recent breakthroughs in video generation models, capable of synthesizing realistic and interactive virtual environments, present an opportunity to revolutionize game creation. This position paper argues that next-generation game engines should be built on Interactive Generative Video (IGV). We term such engines Generative Game Engines (GGE), with IGV as the perceptual and world-generation core under designer control. We argue why IGV is uniquely suited for this role, identify what is missing, and examine progress across six functional modules to show how each supports our position, culminating in a five-level maturity roadmap (L0-L4). Drawing on discussions with researchers, game developers, and community participants, we present nine alternative views and ethical considerations to stimulate debate.