Matrix-game 2.0: An open-source, real-time, and streaming interactive world model

Xianglong He; Chunli Peng; Zexiang Liu; Boyang Wang; Yifan Zhang; Qi Cui; Fei Kang; Biao Jiang

cs.CVAug 18, 2025

Matrix-game 2.0: An open-source, real-time, and streaming interactive world model

Authors: Xianglong He, Chunli Peng, Zexiang Liu, Boyang Wang, Yifan Zhang, Qi Cui, Fei Kang, Biao Jiang, +11 more

Organizations: Skywork AI

Abstract

Recent advances in interactive video generations have demonstrated diffusion model's potential as world models by capturing complex physical dynamics and interactive behaviors. However, existing interactive world models depend on bidirectional attention and lengthy inference steps, severely limiting real-time performance. Consequently, they are hard to simulate real-world dynamics, where outcomes must update instantaneously based on historical context and current actions. To address this, we present Matrix-Game 2.0, an interactive world model generates long videos on-the-fly via few-step auto-regressive diffusion. Our framework consists of three key components: (1) A scalable data production pipeline for Unreal Engine and GTA5 environments to effectively produce massive amounts (about 1200 hours) of video data with diverse interaction annotations; (2) An action injection module that enables frame-level mouse and keyboard inputs as interactive conditions; (3) A few-step distillation based on the casual architecture for real-time and streaming video generation. Matrix Game 2.0 can generate high-quality minute-level videos across diverse scenes at an ultra-fast speed of 25 FPS. We open-source our model weights and codebase to advance research in interactive world modeling.

Figures & tables

Figure 1 : Real-time Interactive Generation Results. We introduce Matrix-Game 2.0, a real-time interactive video generation model. By integrating action modules and few-step distillation, it can auto-regressively produce high-quality interactive videos given an input image in 25 FPS. The demonstrated results cover various scenes and diverse styles, demonstrating its powerful generation capabilities.

Figure 2 : Pipelines of Matrix-Game 2.0.

Figure 3 : Overview of Our Data Production Pipeline based on Unreal Engine.

Figure 4 : An example for Our Navigation System.

Figure 5 : Trajectory Examples of Collected Unreal Engine Data.

Figure 6 : Overview of Our Gaming Interactive Data Recording System.

Figure 7 : Overview of Matrix-Game 2.0 Architecture. The foundation model is derived from the Wan Wang et al. (2025) I2V design. By removing the text branch and adding action modules as in Matrix-Game Zhang et al. (2025) , the model predicts next frames only from visual contents and corresponding actions.

Figure 8 : Causal Student Model Initialization via ODE Trajectories. The proposed initialization method stabilizes subsequent distillation training by deriving a few-step causal student model from the bidirectional teacher model through optimal ODE trajectory sampling.

Figure 9 : Overview of Causal Diffusion Model Training via Self-Forcing. The distillation process aligns the student model’s distributions with the teacher model’s through self-conditioned generation. This approach effectively mitigates error accumulation while maintaining the generation quality.

Figure 10 : Qualitative Comparisons on Minecraft Scene Generations. Compared to Oasis Decart (2024) , our model shows superior visual performance in long interactive video generations.

Model	Visual Quality		Temporal Quality		Action Controllability		Physical Understanding
Model	Image Quality $\uparrow$	Aesthetic $\uparrow$	Temporal Cons. $\uparrow$	Motion smooth. $\uparrow$	Keyboard Acc. $\uparrow$	Mouse Acc. $\uparrow$	Obj. Cons. $\uparrow$	Scenario Cons. $\uparrow$
Oasis Decart (2024)	0.27	0.27	0.82	0.99	0.73	0.56	0.18	0.84
Ours	0.61	0.50	0.94	0.98	0.91	0.95	0.64	0.80

Table 1 : Quantitative Comparisons on Minecraft Scene Generations.

Figure 11 : Qualitative Comparisons on Wild Scene Generations. For wild image inputs, Matrix-Game 2.0 exhibits strong generalization capabilities, fast generation speed, and accurate interaction responses.

Model	Visual Quality		Temporal Quality		Physical Understanding
Model	Image Quality $\uparrow$	Aesthetic $\uparrow$	Temporal Cons. $\uparrow$	Motion smooth. $\uparrow$	Obj. Cons. $\uparrow$	Scenario Cons. $\uparrow$
YUME Mao et al. (2025)	0.65	0.48	0.85	0.99	0.77	0.80
Ours	0.67	0.51	0.86	0.98	0.71	0.76

Table 2 : Quantitative Comparisons on Wild Scene Generations.

Figure 12 : Long Video Generations of Matrix-Game 2.0. The real-time generation results demonstrate excellent visual quality and precise action controllability when generating long videos.

Figure 13 : Generation Results under the Parkour Game TempleRun scene.

Figure 14 : Qualitative Comparison on Different Local Size for KV-cache . Larger local size cause artifacts in long sequences while smaller local size can keep a balance between visual quality and content fidelity.

Acceleration Techniques	Visual Quality		Temporal Quality		Action Controllability		Physical Understanding		Speed
Acceleration Techniques	Image $\uparrow$	Aesthetic $\uparrow$	Temporal $\uparrow$	Motion $\uparrow$	Keyboard $\uparrow$	Mouse $\uparrow$	Object $\uparrow$	Scenario $\uparrow$	FPS $\uparrow$
(1) +VAE Cache	0.61	0.51	0.93	0.97	0.91	0.95	0.68	0.81	15.49
(2) (1)+Halving action modules	0.61	0.51	0.94	0.97	0.92	0.95	0.63	0.81	21.03
(3) (2)+Reducing denoising steps (4 $\rightarrow$ 3)	0.61	0.50	0.94	0.98	0.91	0.95	0.64	0.80	25.15

Table 3 : Quantitative Comparisons of Different Acceleration Techniques. While maintaining comparable generation quality metrics, our combined acceleration techniques achieve 25 FPS throughput, enabling on-the-fly video generation.

Figure 15 : Bad cases. Matrix-Game-V2 sometimes fails when handling out-of-domain scenes, like producing over-saturated (left) or degraded (right) results.

Explore similar work

Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory
Apr 10, 2026Zile Wang, Zexiang Liu, Jiaxing Li +20
Skywork AI
Matrix-Game 3.5: Enhancing Real-Time Streaming Interactive World Models with Patch Memory
Aug 30, 2026Runjia Qian, Zile Wang, Jihai Zhang +14World Models
Riemann Dynamics
minWM: A Full-Stack Open-Source Framework for Real-Time Interactive Video World Models
May 28, 2026Min Zhao, Hongzhou Zhu, Bokai Yan +9Video World Models Real-Time Systems
1ShengShu 2THU · 1ShengShu 3RUC · 1ShengShu +4

Apr 10, 2026, cs.CVJ/K move · Enter open · S save

Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory

Zile Wang, Zexiang Liu, Jiaxing Li, Kaichen Huang+19

Skywork AI

Open paper AskPDF arXiv

With the advancement of interactive video generation, diffusion models have increasingly demonstrated their potential as world models. However, existing approaches still struggle to simultaneously achieve memory-enabled long-term temporal consistency and high-resolution real-time generation, limiting their applicability in real-world scenarios. To address this, we present Matrix-Game 3.0, a memory-augmented interactive world model designed for 720p real-time longform video generation. Building upon Matrix-Game 2.0, we introduce systematic improvements across data, model, and inference. First, we develop an upgraded industrial-scale infinite data engine that integrates Unreal Engine-based synthetic data, large-scale automated collection from AAA games, and real-world video augmentation to produce high-quality Video-Pose-Action-Prompt quadruplet data at scale. Second, we propose a training framework for long-horizon consistency: by modeling prediction residuals and re-injecting imperfect generated frames during training, the base model learns self-correction; meanwhile, camera-aware memory retrieval and injection enable the base model to achieve long horizon spatiotemporal consistency. Third, we design a multi-segment autoregressive distillation strategy based on Distribution Matching Distillation (DMD), combined with model quantization and VAE decoder pruning, to achieve efficient real-time inference. Experimental results show that Matrix-Game 3.0 achieves up to 40 FPS real-time generation at 720p resolution with a 5B model, while maintaining stable memory consistency over minute-long sequences. Scaling up to a 2x14B model further improves generation quality, dynamics, and generalization. Our approach provides a practical pathway toward industrial-scale deployable world models.

Abstract

Explore similar work

Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory

Matrix-Game 3.5: Enhancing Real-Time Streaming Interactive World Models with Patch Memory

minWM: A Full-Stack Open-Source Framework for Real-Time Interactive Video World Models