Real-Time Interactive Video Generation

Follow this topic Ask about this topic

Momentum

8 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 43

All topics

Real-Time Joint Audio-Video Generation by Parallel Adapter Composition
Oct 7, 2026Jingyu Li, Xiaoxiao Xiang, Yiwen GuoAudio-Video Generation Real-Time Interactive Video Generation
LIGHTSPEED · Independent Researcher
CtrlCache: Accelerating Interactive Video World Models with Control-Aware Caching
Oct 6, 2026Shangye Song, Dong Gong, Hong Jia +2Action-Conditioned Video Generation Real-Time Interactive Video Generation
University of Auckland · UNSW Sydney
HelixWorld: A Real-time Interactive Audio-Visual World Model
Sep 29, 2026Lei Ke, Jiahao Pan, Zeyue Tian +13World Model Learning Real-Time Interactive Video Generation
The Hong Kong University of Science and Technology · Noiz AI
Waypoint-1.5: A Real-Time Video World Model for Consumer Hardware
Sep 29, 2026Rajit Rajpal, Shahbuland Matiana, Liew Wei Pyn +12Video Diffusion Models Real-Time Interactive Video Generation
HuggingFace
FlowAct-R2: Beyond Talking Avatar via Streaming Multimodal References and Proactive Agent Planning
Sep 28, 2026Ziyao Huang, Zhengkun Rong, Shiyang Qin +5Action-Conditioned Video Generation Real-Time Interactive Video Generation
Bytedance Intelligent Creation
WorldAttention: An Efficient Attention Architecture for Interactive Video World Models
Sep 28, 2026Zeyu Zhang, Jinyuan Mao, Dakai An +6Real-Time Interactive Video Generation Long-Horizon Video Generation
DAMO Academy, Alibaba Group · Zhejiang University · Hong Kong University of Science and Technology +2
CAST: Reconstruction-Coupled Acceleration of Interactive World Models
Sep 28, 2026Leyang Chen, Junyi Wu, Fanqing Kong +2Real-Time Interactive Video Generation Inference Acceleration
Shanghai Jiao Tong University
Astronex-World 1.0: Real-Time Interactive World Model Foundation
Sep 17, 2026Xin Zhou, Cong MiaoCamera-Conditioned Video Generation Video Diffusion Models
Astronex Robotics · Nanjing University of Information Science and Technology
Zing-0.5: Toward Playable Worlds with Real-Time Joint Action and Text Control
Sep 15, 2026Mingyang Chen, Shengdong Chen, Xiaoxiao Fu +13Action-Conditioned Video Generation Real-Time Interactive Video Generation
Zing Team, SeedLeap.ai
Vidu S2: Real-Time Interactive, Editable, and Spatial Video Generation
Sep 10, 2026Jintao Zhang, Kai Jiang, Jintao Chen +32Real-Time Interactive Video Generation Video Editing
Tsinghua University, Shengshu Technology
Matrix-Game 3.5: Enhancing Real-Time Streaming Interactive World Models with Patch Memory
Aug 30, 2026Runjia Qian, Zile Wang, Jihai Zhang +14Camera-Controlled Video Generation Real-Time Interactive Video Generation
Riemann Dynamics
4DStreamCtrl: Interactive Video Generation with Online 4D Control
Aug 26, 2026Shiqian Li, Chenguo Lin, Zhiguang Liu +4Camera-Controlled Video Generation Video Diffusion Models
Peking University · Tencent Hunyuan
Avatar-Forever: Decoupled Parallel Training for High-Quality Real-Time Infinite Avatars
Aug 12, 2026Ruibin Li, Tao Yang, Zhiyuan Ma +3Audio-Video Generation Real-Time Interactive Video Generation
The Hong Kong Polytechnic University · ByteDance · AMD
LiveAnimate: Stable Long-Form Streaming Human Animation in Real-Time
Aug 12, 2026Yuxuan Zhang, Haozhong Xiong, Yubo Huang +6Diffusion Transformer Real-Time Interactive Video Generation
The Chinese University of Hong Kong · Qwen Applications Business Group of Alibaba · Liblib AI
Wan-Animate-2: Pushing the Application Boundaries of Character Animation
Aug 6, 2026Guangyuan Wang, Li Hu, Dechao Meng +11Video Diffusion Models Real-Time Interactive Video Generation
Tongyi Lab, Alibaba Group
ReGenVC: End-to-End Real-Time Generative Video Coding at Ultra-Low Bitrate
Jul 30, 2026Zheyuan Zhang, Johnson WuVideo Diffusion Models Real-Time Interactive Video Generation
Intel Corporation
Visko Orbis 1.0: A Live Model for Real-Time Interactive Long Video Generation
Jul 29, 2026Xiangbo Gao, Siyuan Yang, Ping He +12Physical Consistency in Video Generation Real-Time Interactive Video Generation
Wonder: Video World Model Done Better
Jul 28, 2026Jiacong Xu, Hanwen Jiang, Zhixin Shu +3Camera-Controlled Video Generation Real-Time Interactive Video Generation
Adobe Research · Johns Hopkins University
OmniMate: Open-Ended Real-Time Streaming Audio-Visual Generation for Interactive Avatars
Jul 25, 2026Quanyue Song, Yishan He, Yanbo Ding +4Audio-Video Generation Real-Time Interactive Video Generation
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, China · China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd., China · Shenzhen Key Laboratory of Computer Vision and Pattern Recognition, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
Generative World Renderer at the Speed of Play
Jul 21, 2026Guixu Lin, Zheng-Hui Huang, Siqi Yang +3Real-Time Interactive Video Generation Streaming Video Generation
1Alaya Lab · University of California, Merced
AlayaWorld: Interactive Long-Horizon World Modeling -- Full Technical Report
Jul 20, 2026AlayaWorld Team, Kaipeng Zhang, Chuanhao Li +15Video Diffusion Models Real-Time Interactive Video Generation
Alaya Lab
Infinite Worlds with Versatile Interactions
Jul 8, 2026Zelin Gao, Qiuyu Wang, Jiapeng Zhu +17Real-Time Interactive Video Generation World Models
AlayaWorld: Long-Horizon and Playable Video World Generation
Jul 7, 2026AlayaWorld Team, Kaipeng Zhang, Chuanhao Li +14Real-Time Interactive Video Generation Long-Horizon Video Generation
Alaya Lab
Wan-Streamer v0.2: Higher Resolution, Same Latency
Jul 5, 2026Lianghua Huang, Zhi-Fan Wu, Yupeng Shi +23Audio-Video Generation Real-Time Interactive Video Generation
Wan Team, Alibaba Group
Vidu S1: A Real-Time Interactive Video Generation Model
Jul 3, 2026Jintao Zhang, Kai Jiang, Jintao Chen +24Video Diffusion Models Real-Time Interactive Video Generation
Tsinghua University, Shengshu Technology · Shengshu Technology
DreamForge-World 0.1 Preview: A Low-Compute Real-Time Controllable World Model
Jun 29, 2026Daniyel Ayupov, Artur Markov-TsoyReal-Time Interactive Video Generation World Models
DreamForge AI Lab, Kazakhstan
InteractiveAvatar: Real-Time Streaming Video Generation for Consistent and Intent-Aware Avatars
Jun 22, 2026Quanyue Song, Yishan He, Yanfei Zhang +6Real-Time Interactive Video Generation Memory-Augmented Video Generation
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, China · China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd., China · State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, China +1
MaineCoon: Pursuing A Real-Time Audio-Visual Social World Model
Jun 16, 2026Lichen Bai, Tianhao Zhang, Shitong Shao +14Audio-Video Generation Real-Time Interactive Video Generation
Catnip AI Team
DreamX-World 1.0: A General-Purpose Interactive World Model
Jun 15, 2026DreamX Team, Yancheng Bai, Rui Chen +20Camera-Controlled Video Generation Real-Time Interactive Video Generation
DreamX Team
MoVerse: Real-Time Video World Modeling with Panoramic Gaussian Scaffold
Jun 11, 2026Yang Zhou, Ziheng Wang, Yuqin Lu +43D Gaussian Splatting Camera-Conditioned Video Generation
South China University of Technology · Orange Team, Moku Lab, HUJING Digital Media & Entertainment Group · Columbia University +1

Oct 7, 2026, cs.CVJ/K move · Enter open · S save

Real-Time Joint Audio-Video Generation by Parallel Adapter Composition

Jingyu Li, Xiaoxiao Xiang, Yiwen Guo

LIGHTSPEED · Independent Researcher

Audio-Video Generation Real-Time Interactive Video Generation Streaming Video Generation Autoregressive Video Generation

Open paper AskPDF arXiv

Deploying a joint audio-video diffusion transformer for real-time, interactive generation normally requires two essential modifications: block-autoregressive attention, so frames can be emitted before the whole clip is finished, and few-step sampling, so each block is cheap. Conventionally, the streaming video literature obtains both capabilities from a chained pipeline. It first distills a bidirectional teacher into a causal student, then into a few-step one, or proceeds in reverse order. Each stage of such a chain fine-tunes the weights the previous one produced, so a later objective can undo an earlier capability. Following the idea of model merging, we show that on a packed audio-video backbone the two capabilities can be acquired in parallel. A causal adapter is trained against the frozen backbone, and an off-the-shelf few-step adapter provides the few-step capability. As the two edit different functional axes, we predict, and then verify, that their weight-update directions are near-orthogonal, without any explicit orthogonality constraint during training. Orthogonal updates should combine without interfering, so parallel composition is a direct sum. The two adapters are simply added at inference, with no joint training, yielding few-step, streaming audio-video whose image quality tracks the bidirectional teacher. Compared to the chained baselines, the composed model matches or beats them on most metrics, making parallel composition a practical approach. The resulting streaming system generates joint audio-video in real time,

\approx

26 fps at

480\times832

without quantization, and sustains 30 s of continuous generation with stable image quality. Demo Page: https://pac-demo-2027.github.io/demo/

Momentum

Related topics

Latest papers 43