Existing action-conditioned video generation models (video world models) are limited to single-agent perspectives, failing to capture the multi-agent interactions of real-world environments. We introduce Solaris, a multiplayer video world model that simulates consistent multi-view observations. To enable this, we develop a multiplayer data system designed for robust, continuous, and automated data collection on video games such as Minecraft. Unlike prior platforms built for single-player settings, our system supports coordinated multi-agent interaction and synchronized videos + actions capture. Using this system, we collect 12.64 million multiplayer frames and propose an evaluation framework for multiplayer movement, memory, grounding, building, and view consistency. We train Solaris using a staged pipeline that progressively transitions from single-player to multiplayer modeling, combining bidirectional, causal, and Self Forcing training. In the final stage, we introduce Checkpointed Self Forcing, a memory-efficient Self Forcing variant that enables a longer-horizon teacher. Results show our architecture and training design outperform existing baselines. Through open-sourcing our system and models, we hope to lay the groundwork for a new generation of multi-agent world models.
Figures & tables
Figure 1: Selected samples from our model. Our model takes in starting frames from each player as input and generates action-conditioned videos. The action descriptions shown here are summaries of the fine-grained action sequences given to the model that span many frames. The third-person ground truth visualizations are not given to the model.
Figure 2: SolarisEngine Overview. (Left) Docker-based orchestration of containerized game server, camera, and controller bots. Cameras mirror Controllers’ state and actions via a custom server-side plugin; Controllers are Mineflayer bots that run episode code and log low-level actions. (Right) Episodes compose reusable skill primitives from a shared library. Simplified “collector” episode code is shown.
Method
Controllability
Multiplayer
Graphics
Malmo [ 33 ]
×
✓
✓
MineRL [ 20 ]
×
×
✓
MineDojo [ 17 ]
×
×
✓
Voyager [ 55 ]
✓
×
×
Mineflayer [ 48 ]
✓
✓
×
SolarisEngine
✓
✓
✓
Table 1: Comparison of Minecraft AI frameworks. Unlike prior systems, SolarisEngine enables controlled multiplayer gameplay collection with real Minecraft graphics. RL-based frameworks such as Malmo, MineRL, and MineDojo produce visual observations but offer limited controllability, as their low-level action spaces require training RL agents to collect meaningful data, which is prohibitively expensive and counterproductive for world-modeling data collection. Voyager and Mineflayer provide high-level behavioral control but operate in text-only mode without visual output. SolarisEngine supports multiplayer gameplay, fine-grained programmatic control and rich visual observations.
Figure 3: Dataset Statistics of our training dataset. (Left) The dataset consists of four different episode categories focusing on building, combat, movement, and mining scenarios, respectively. (Middle) It has a total of 9,240 episodes and 6.32 M frames per player, for a combined 12.64 M frames. Episode types are chosen randomly with weights that decrease with respect to the typical episode length. (Right) Most episode lengths range from 128 to 512 frames or 6.4 to 25.6 seconds (we record at 20 fps).
Figure 4: Episode Demonstrations from our training dataset. We show the recorded frames from 3 different training episodes at various points in time. Note that the third-person “start state” and “end state” screenshots are for visualization only and are not part of the dataset.
Figure 5: Our modified DiT block achieves multiplayer modeling through visual interleaving along the sequence dimension. We denote the number of players with N and the number of tokens per video with M . Multiplayer information is exchanged through a shared self-attention block. The other modules are unchanged from Matrix Game 2.0 and applied independently per player.
Figure 6: An overview of the full training pipeline. Starting with a pretrained bidirectional video diffusion model, we first finetune it with single-player and then multiplayer data. We then finetune it with a causal mask before using Self Forcing to achieve stable long-horizon autoregressive generations.
Figure 7: An overview of Checkpointed Self Forcing. (A) A video is generated with a sliding-window KV cache. The computation graph in Jax contains redundant tensors, making naive backpropagation memory prohibitive. (B) Our recomputation step simulates the last step of denoising for every frame in parallel. Clean context frames and final noisy frames are cached from the previous step. (C) An illustration of the attention mask used in part B (black squares denote attention). This attention mask allows us to simulate the final denoising step of the rollout in parallel. (D) Leveraging these memory savings, we enable backpropagation through KV layers (avoiding the standard stop-gradient of Self Forcing), which we show improves visual generation in Section 7 . Specifically, we allow gradients to flow in line 29 of Algorithm 1 . (E) Peak memory comparison between naive and Checkpointed Self Forcing across varying depths. Scaled-down networks are used to prevent OOM errors in the naive baseline.
Figure 8: Diff-style pseudocode comparing a single generator train step of naively applying sliding-window self forcing to our efficient implementation, Checkpointed Self Forcing. Green “+" lines denote lines from our algorithm that have been added while red “-" denote lines that have been removed. Please also see Section E.2 for mask pseudocode and Fig. 7 for a visualization.
Figure 9: Qualitative comparison. Our model Solaris is able to produce stable and coherent frame generations for long-horizon, as shown here with 224 frames. Unlike the baselines, it maintains realistic fighting gameplay and displays complex terrain that maintains realistic texture. In contrast, the frame concatenation baseline shows severe degradation for the second player and flattened texture for the first player. “Solaris w/o pretrain" exhibits unnatural behavior such as duplicating player bodies, showing incorrect pop-up notifications, and degenerating to an unrealistic underwater setting.
Figure 10: Qualitative examples of learned capabilities. We present generated videos demonstrating the model’s ability to simulate complex game dynamics. The rows illustrate: (1) fine-grained state tracking, specifically inventory counter synchronization following block placement; (2) global environmental consistency, shown by the simultaneous onset of rain; (3) inventory active item synchronization, torch placement, and generating accurate mining animations; and (4) coherent PVP on complex terrain.
Movement
Grounding
Memory
Building
Consistency
Method
VLM ↑
FID ↓
VLM ↑
FID ↓
VLM ↑
FID ↓
VLM ↑
FID ↓
VLM ↑
FID ↓
Frame concat
77.1 ± 2.9
68.9
53.1 ± 0.0
66.6
37.5 ± 2.6
74.4
0.0 ± 0.0
103.2
49.5 ± 5.3
129.4
Solaris w/o pretrain
69.3 ± 0.7
42.5
29.2 ± 1.5
49.9
18.8 ± 0.0
67.8
0.0 ± 0.0
86.6
49.5 ± 3.2
121.4
Solaris
68.2 ± 2.7
38.5
62.5 ± 0.0
38.0
37.5 ± 2.6
55.1
20.8 ± 1.5
83.6
71.4 ± 7.1
99.4
Table 2: Quantitative comparison across tasks. We compare our method against concatenating player observations along the channel dimension following Enigma Multiverse [ 16 ] and training from scratch without single-player pretraining. “VLM” shows the VLM accuracy ± 1 (estimated) standard deviation. The standard deviation is estimated by repeating the VLM eval 3 times.
Components
Movement
Grounding
Memory
Building
Consistency
Init.
Pre-DMD
KV-BP
VLM ↑
FID ↓
VLM ↑
FID ↓
VLM ↑
FID ↓
VLM ↑
FID ↓
VLM ↑
FID ↓
ODE Reg
×
✓
23.4 ± 1.5
65.3
3.1 ± 0.0
56.6
0.0 ± 0.0
99.5
3.1 ± 0.0
95.7
49.0 ± 4.2
142.3
Causal FT
✓
✓
21.4 ± 2.7
49.1
3.1 ± 0.0
40.4
0.0 ± 0.0
55.7
8.3 ± 1.5
90.5
55.2 ± 3.7
160.1
Causal FT
×
×
78.6 ± 0.7
60.3
72.9 ± 1.5
55.2
49.0 ± 2.9
63.8
15.6 ± 0.0
87.4
70.8 ± 6.6
105.1
Causal FT
×
✓
68.2 ± 2.7
38.5
62.5 ± 0.0
38.0
37.5 ± 2.6
55.1
20.8 ± 1.5
83.6
71.4 ± 7.1
99.4
Table 3: Ablations on Self Forcing training variants. We study the initialization of the causal model, finding simple causal finetuning with Diffusion Forcing [ 10 ] to suffice. We also find that doing few-step distillation before Self-Forcing is ineffective. Finally, we find that enabling KV cache backpropagation improves visual quality. “VLM” shows the VLM accuracy ± 1 (estimated) standard deviation. The standard deviation is estimated by repeating the VLM eval 3 times.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Episode Name
Category
Description
buildStructure
Building
Each bot builds a wall or tower in front of the other, or bots build a platform together at midpoint
buildTower
Building
Both bots build a tall 1-block tower by jumping and placing blocks underneath themselves.
buildHouse
Building
Bots build a house together.
towerBridge
Building
Bots build a 1-block tower by jumping, then build a bridge connecting the two towers, and meet.
pve
Combat
Bots defend two random positions on the ground, facing each other against spawning mobs.
pvp
Combat
Bots equip swords and fight each other.
Appendix
Table 4: Episode types in our training dataset.
Figure 11: An episode where a bot has been teleported underwater.
Figure 12: The “bubbles” HUD template PNG used to filter underwater episodes.
Action key
Type
Description
forward
bool/sustained
Player moving forward (W).
back
bool/sustained
Player moving backward (S).
left
bool/sustained
Player strafing left (A).
right
bool/sustained
Player strafing right (D).
jump
bool/sustained
Player jumping.
sprint
bool/sustained
Player sprinting.
Appendix
Table 5: Action space of our multiplayer dataset. “Sustained” denotes action is recorded for as long as it lasts. “Once” denotes action is recorded for one tick only, usually applicable to instantaneous events.
Figure 13: A histogram showing the mouse action magnitude distribution in our training dataset.
Setting
Learning Rate
Batch Size
Num Steps
Adam β1
Adam β2
Weight Decay
Bidirectional Single-player
1e−4
64
120K
0.9
0.95
0
Bidirectional Multiplayer
1e−4
32
120K
0.9
0.95
0
Causal Multiplayer
1e−4
32
60K
0.9
0.95
0
Self Forcing (generator)
3e−6
32
240
0.9
0.95
0
Self Forcing (critic)
3e−7
32
1200
0.9
0.95
0
Appendix
Table 6: Training hyperparameters across all training stages.
Figure 14: An illustration of the frames and prompts provided to the VLM for evaluation. Each scenario is a ground truth video for illustration. Above the third-person visualization, we show the frames that are sent to the VLM along with the corresponding question and correct answer. Prompts are abbreviated; please see Table 7 for the full text.
Video world models have achieved remarkable success in simulating environmental dynamics in response to actions by users or agents. They are modeled as action-conditioned video generation models that take historical frames and current actions as input to predict future frames. Yet, most existing approaches are limited to single-agent scenarios and fail to capture the complex interactions inherent in real-world multi-agent systems. We present MultiWorld, a unified framework for multi-agent multi-view world modeling that enables accurate control of multiple agents while maintaining multi-view consistency. We introduce the Multi-Agent Condition Module to achieve precise multi-agent controllability, and the Global State Encoder to ensure coherent observations across different views. MultiWorld supports flexible scaling of agent and view counts, and synthesizes different views in parallel for high efficiency. Experiments on multi-player game environments and multi-robot manipulation tasks demonstrate that MultiWorld outperforms baselines in video fidelity, action-following ability, and multi-view consistency. Project page: https://multi-world.github.io/
Video world models are a foundational generative technology for embodied AI and the Metaverse, yet existing approaches are inherently limited to a single agent observing from a single perspective. Extending these models to multi-agent settings introduces two critical challenges: data scarcity (coordinated multi-view recordings are prohibitively expensive to collect for general open-domain scenarios) and world state alignment (independently generated video streams cannot ensure that shared physical environments and events evolve consistently across views). To address these challenges, we propose MetaWorld, a novel framework that scales multi-agent video world models to open-domain environments directly from single-view videos. First, we introduce Monocular World-State Unrolling (MWSU) to explicitly decompose monocular footage into the camera operator's ego-motion and the visible subject's spatial trajectory. This camera-trajectory decomposition naturally extracts synchronized multi-agent motion data within a shared 3D space, completely bypassing the need for multi-camera setups. Second, for precise visual control, we develop the Subject-Aware World Generator to enable appearance-driven simulation conditioned on per-agent identity images. Finally, to ensure both views are grounded in the identical physical reality, we propose World-State Alignment, a per-frame inter-branch cross-attention mechanism inserted at every transformer layer of the video DiT. By jointly synchronizing the denoising process, WSA enforces both static geometric consistency and dynamic motion consistency, encouraging that the shared 3D environment and physical events remain well-aligned across both egocentric views. Extensive experiments demonstrate that MetaWorld achieves superior cross-view consistency and identity fidelity, establishing a highly scalable, physics-driven paradigm for multi-agent video world modeling.
Teng Hu, Mingchun Lu, Yating Wang +6
Shanghai Jiao Tong University · Zhejiang University · Nanyang Technological University
Multi-player interactive world models aim to simulate shared environments from multiple players' perspectives, but achieving robust generation while scaling to more players remains challenging. Existing approaches that rely on fully-connected communication (e.g., Solaris) incur quadratic attention costs as the number of players grows, while centralized approaches (e.g., γ-World) suffer from interference between unrelated events in a shared global representation. To address these challenges, we present Multics, a multi-player video generation framework with hierarchical communication that structures information exchange across player, group, and global levels. This hierarchy introduces group tokens into self-attention as intermediate group-level representations that aggregate locally relevant player information. These tokens connect to global tokens for cross-group communication, while the hierarchical topology limits direct mixing between unrelated local events. Group-token parameters are shared across group slots, allowing the same communication module to accommodate varying numbers of players without introducing new group-specific parameters. We further propose budget-constrained dynamic grouping based on recent camera trajectories, which adaptively groups relevant players while limiting unnecessary communication. Experiments demonstrate that Multics achieves state-of-the-art performance in 4- and 6-player settings, with improved cross-player consistency, player-specific action and camera controllability, and interference robustness. Moreover, Multics generalizes from 4-player training to 6-, 8-, and 10-player inference without retraining, while reducing self-attention computation compared with fully-connected communication.
Kaiyi Huang, Wangbo Yu, Yukun Huang +6
The University of Hong Kong · ARC Lab, Tencent IEG