Real-Time Interactive Video Generation

Overview

The field studies how to generate video that responds as people interact, including controllable worlds, avatars, and live video editing. Common approaches adapt video diffusion or autoregressive models for causal streaming, then use few-step distillation, caching, and bounded memory to reduce latency; 3D geometry and physics can provide more explicit camera and action control. Systems are tested for visual quality, control accuracy, throughput, and latency, with benchmarks increasingly measuring long streams and interactive responsiveness. A central challenge is balancing fast reactions with realistic motion and consistent scenes or identities over long interactions, where drift and limited memory remain concerns.

Latest papers

CardsList

92 papers16 papers in the last four weeks, up 23% on the four weeks before. 0.1% of all new papers.

  1. FastFrames: Video Generation at Interactive Latency

    Nov 17, 2026Gene Chou, Guandao Yang, Fujun Luan +4Video Diffusion ModelsReal-Time Interactive Video Generation

  2. Code World Model 2.0: Towards Infinite Controllable Video Generation

    Nov 17, 2026Ruoyu Wang, Hao Zhong, Yanming Yang +8Controllable Video GenerationReal-Time Interactive Video Generation

  3. GameWM: Toward Next-Generation Video Games with Interaction-Centric World Models

    Nov 14, 2026Bing He, Runze Wang, Fei Zha +5Game World ModelsVideo Games

  4. Back to Pixels: Lifting Latent Video Diffusion into Pixel Space

    Nov 12, 2026Yanke Song, Shuo Chen, Chris Rockwell +9Video Diffusion ModelsHigh-Resolution Video Generation

  5. Consistent Video-to-Video Translation via Explicit Correspondences

    Nov 12, 2026Gaurav Parmar, Zhengqi Li, Richard Zhang +4Video-to-Video TranslationTemporal Consistency in Video Generation

  6. TouchéWorld tech report : Towards Online Unified Camera--Hand Exploration in Generative Worlds

    Nov 4, 2026Jiaao Yu, Yanjian Zhang, Haoyang Jiang +3Controllable Video GenerationReal-Time Interactive Video Generation

  7. Wave Forcing: Towards Speed-of-Light Streaming Video Generation

    Oct 31, 2026Yijie Jin, Yuxuan Liu, Jiajun Xu +7Streaming Video GenerationLong Video Generation

  8. L3G-World: Real-Time Infinite World Generation with Latent 3D Gaussian Memory

    Oct 29, 2026Jinhyeok Choi, JoungBin Lee, Jongmin Lee +5Video World ModelsReal-Time Interactive Video Generation

  9. ReferStream: Interactive Subject Injection for Streaming Video Generation

    Oct 27, 2026Shuolin Xu, Guanyuan Li, Boyu Li +5Subject-Driven Video GenerationStreaming Video Generation

  10. BoxWorld: Let World Engines Control and Video Generators Render

    Oct 26, 2026Zike Wu, Ahmed Hatem, Chintan Trivedi +4Real-Time Interactive Video GenerationInteractive World Models

  11. UniWorld: A Unified World Model for Real-Time Visual World Simulation

    Oct 23, 2026Xin Zhang, Yabo Chen, Shuaiwei Wang +4Video World ModelsVideo Generation

  12. LiveAct: Towards Real-Time Multi-Agent World Models with Heterogeneous Action Interfaces

    Oct 21, 2026Xinyu Wang, Huafeng Shi, Yue Ma +9Video World ModelsReal-Time Interactive Video Generation

  13. Matrix-Game 4.0: Harnessing Video Models for Real-Time Playable Game Worlds

    Oct 20, 2026Jihai Zhang, Runjia Qian, Kaiqin Kong +15Real-Time Interactive Video GenerationAutoregressive Video Generation

  14. TriTD: Tri-Partite Trajectory-Distribution Distillation for Real-Time Autoregressive Video Generation

    Oct 19, 2026Jia Li, Xurui Peng, Haowei Zhu +7Autoregressive Video GenerationReal-Time Interactive Video Generation

  15. RIGen: Beyond Action Following with Reciprocal Interaction in Generative Gameplay

    Oct 14, 2026Mengfei Li, Yi Zhang, Xingtong Ge +8Real-Time Interactive Video GenerationInteractive World Models

  16. Speculative Sampling in Interactive Video Generation

    Oct 14, 2026Yubo Huang, Yuxuan Zhang, Jiaming Liu +2Real-Time Interactive Video GenerationStreaming Video Generation

  17. World-T3: Test-Time Training as Persistent Memory for Real-Time Video World Models

    Oct 12, 2026Junyi Wu, Zekun Li, Chenhongyi Yang +9Video World ModelsMemory-Augmented Video Generation

  18. WorldCast: Distributed Multiplayer World Models

    Oct 8, 2026Ziyang Ye, Junchao Huang, Evelyn Zhang +9World ModelsInteractive World Models

  19. HelixWorld: A Real-time Interactive Audio-Visual World Model

    Sep 29, 2026Lei Ke, Jiahao Pan, Zeyue Tian +13World Model LearningReal-Time Interactive Video Generation

  20. Waypoint-1.5: A Real-Time Video World Model for Consumer Hardware

    Sep 29, 2026Rajit Rajpal, Shahbuland Matiana, Liew Wei Pyn +12Game World ModelsVideo Diffusion Models