cs.CVFeb 25, 2026
SaveSolaris: Building a Multiplayer Video World Model in Minecraft
Organizations: New York University
Abstract
Existing action-conditioned video generation models (video world models) are limited to single-agent perspectives, failing to capture the multi-agent interactions of real-world environments. We introduce Solaris, a multiplayer video world model that simulates consistent multi-view observations. To enable this, we develop a multiplayer data system designed for robust, continuous, and automated data collection on video games such as Minecraft. Unlike prior platforms built for single-player settings, our system supports coordinated multi-agent interaction and synchronized videos + actions capture. Using this system, we collect 12.64 million multiplayer frames and propose an evaluation framework for multiplayer movement, memory, grounding, building, and view consistency. We train Solaris using a staged pipeline that progressively transitions from single-player to multiplayer modeling, combining bidirectional, causal, and Self Forcing training. In the final stage, we introduce Checkpointed Self Forcing, a memory-efficient Self Forcing variant that enables a longer-horizon teacher. Results show our architecture and training design outperform existing baselines. Through open-sourcing our system and models, we hope to lay the groundwork for a new generation of multi-agent world models.
Figures & tables
Figure 1: Selected samples from our model. Our model takes in starting frames from each player as input and generates action-conditioned videos. The action descriptions shown here are summaries of the fine-grained action sequences given to the model that span many frames. The third-person ground truth visualizations are not given to the model.
Figure 2: SolarisEngine Overview. (Left) Docker-based orchestration of containerized game server, camera, and controller bots. Cameras mirror Controllers’ state and actions via a custom server-side plugin; Controllers are Mineflayer bots that run episode code and log low-level actions. (Right) Episodes compose reusable skill primitives from a shared library. Simplified “collector” episode code is shown.
| Method | Controllability | Multiplayer | Graphics |
| Malmo [ 33 ] | × | ✓ | ✓ |
| MineRL [ 20 ] | × | × | ✓ |
| MineDojo [ 17 ] | × | × | ✓ |
| Voyager [ 55 ] | ✓ | × | × |
| Mineflayer [ 48 ] | ✓ | ✓ | × |
| SolarisEngine | ✓ | ✓ | ✓ |
Table 1: Comparison of Minecraft AI frameworks. Unlike prior systems, SolarisEngine enables controlled multiplayer gameplay collection with real Minecraft graphics. RL-based frameworks such as Malmo, MineRL, and MineDojo produce visual observations but offer limited controllability, as their low-level action spaces require training RL agents to collect meaningful data, which is prohibitively expensive and counterproductive for world-modeling data collection. Voyager and Mineflayer provide high-level behavioral control but operate in text-only mode without visual output. SolarisEngine supports multiplayer gameplay, fine-grained programmatic control and rich visual observations.
Figure 3: Dataset Statistics of our training dataset. (Left) The dataset consists of four different episode categories focusing on building, combat, movement, and mining scenarios, respectively. (Middle) It has a total of 9,240 episodes and 6.32 M frames per player, for a combined 12.64 M frames. Episode types are chosen randomly with weights that decrease with respect to the typical episode length. (Right) Most episode lengths range from 128 to 512 frames or 6.4 to 25.6 seconds (we record at 20 fps).
Figure 4: Episode Demonstrations from our training dataset. We show the recorded frames from 3 different training episodes at various points in time. Note that the third-person “start state” and “end state” screenshots are for visualization only and are not part of the dataset.
Figure 5: Our modified DiT block achieves multiplayer modeling through visual interleaving along the sequence dimension. We denote the number of players with and the number of tokens per video with . Multiplayer information is exchanged through a shared self-attention block. The other modules are unchanged from Matrix Game 2.0 and applied independently per player.
Figure 6: An overview of the full training pipeline. Starting with a pretrained bidirectional video diffusion model, we first finetune it with single-player and then multiplayer data. We then finetune it with a causal mask before using Self Forcing to achieve stable long-horizon autoregressive generations.
Figure 7: An overview of Checkpointed Self Forcing. (A) A video is generated with a sliding-window KV cache. The computation graph in Jax contains redundant tensors, making naive backpropagation memory prohibitive. (B) Our recomputation step simulates the last step of denoising for every frame in parallel. Clean context frames and final noisy frames are cached from the previous step. (C) An illustration of the attention mask used in part B (black squares denote attention). This attention mask allows us to simulate the final denoising step of the rollout in parallel. (D) Leveraging these memory savings, we enable backpropagation through KV layers (avoiding the standard stop-gradient of Self Forcing), which we show improves visual generation in Section 7 . Specifically, we allow gradients to flow in line 29 of Algorithm 1 . (E) Peak memory comparison between naive and Checkpointed Self Forcing across varying depths. Scaled-down networks are used to prevent OOM errors in the naive baseline.
Figure 8: Diff-style pseudocode comparing a single generator train step of naively applying sliding-window self forcing to our efficient implementation, Checkpointed Self Forcing. Green “+" lines denote lines from our algorithm that have been added while red “-" denote lines that have been removed. Please also see Section E.2 for mask pseudocode and Fig. 7 for a visualization.
Figure 9: Qualitative comparison. Our model Solaris is able to produce stable and coherent frame generations for long-horizon, as shown here with 224 frames. Unlike the baselines, it maintains realistic fighting gameplay and displays complex terrain that maintains realistic texture. In contrast, the frame concatenation baseline shows severe degradation for the second player and flattened texture for the first player. “Solaris w/o pretrain" exhibits unnatural behavior such as duplicating player bodies, showing incorrect pop-up notifications, and degenerating to an unrealistic underwater setting.
Figure 10: Qualitative examples of learned capabilities. We present generated videos demonstrating the model’s ability to simulate complex game dynamics. The rows illustrate: (1) fine-grained state tracking, specifically inventory counter synchronization following block placement; (2) global environmental consistency, shown by the simultaneous onset of rain; (3) inventory active item synchronization, torch placement, and generating accurate mining animations; and (4) coherent PVP on complex terrain.
| Movement | Grounding | Memory | Building | Consistency | ||||||
| Method | VLM | FID | VLM | FID | VLM | FID | VLM | FID | VLM | FID |
| Frame concat | 77.1 2.9 | 68.9 | 53.1 0.0 | 66.6 | 37.5 2.6 | 74.4 | 0.0 0.0 | 103.2 | 49.5 5.3 | 129.4 |
| Solaris w/o pretrain | 69.3 0.7 | 42.5 | 29.2 1.5 | 49.9 | 18.8 0.0 | 67.8 | 0.0 0.0 | 86.6 | 49.5 3.2 | 121.4 |
| Solaris | 68.2 2.7 | 38.5 | 62.5 0.0 | 38.0 | 37.5 2.6 | 55.1 | 20.8 1.5 | 83.6 | 71.4 7.1 | 99.4 |
Table 2: Quantitative comparison across tasks. We compare our method against concatenating player observations along the channel dimension following Enigma Multiverse [ 16 ] and training from scratch without single-player pretraining. “VLM” shows the VLM accuracy 1 (estimated) standard deviation. The standard deviation is estimated by repeating the VLM eval 3 times.
| Components | Movement | Grounding | Memory | Building | Consistency | |||||||
| Init. | Pre-DMD | KV-BP | VLM | FID | VLM | FID | VLM | FID | VLM | FID | VLM | FID |
| ODE Reg | × | ✓ | 23.4 1.5 | 65.3 | 3.1 0.0 | 56.6 | 0.0 0.0 | 99.5 | 3.1 0.0 | 95.7 | 49.0 4.2 | 142.3 |
| Causal FT | ✓ | ✓ | 21.4 2.7 | 49.1 | 3.1 0.0 | 40.4 | 0.0 0.0 | 55.7 | 8.3 1.5 | 90.5 | 55.2 3.7 | 160.1 |
| Causal FT | × | × | 78.6 0.7 | 60.3 | 72.9 1.5 | 55.2 | 49.0 2.9 | 63.8 | 15.6 0.0 | 87.4 | 70.8 6.6 | 105.1 |
| Causal FT | × | ✓ | 68.2 2.7 | 38.5 | 62.5 0.0 | 38.0 | 37.5 2.6 | 55.1 | 20.8 1.5 | 83.6 | 71.4 7.1 | 99.4 |
Table 3: Ablations on Self Forcing training variants. We study the initialization of the causal model, finding simple causal finetuning with Diffusion Forcing [ 10 ] to suffice. We also find that doing few-step distillation before Self-Forcing is ineffective. Finally, we find that enabling KV cache backpropagation improves visual quality. “VLM” shows the VLM accuracy 1 (estimated) standard deviation. The standard deviation is estimated by repeating the VLM eval 3 times.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Episode Name | Category | Description |
| buildStructure | Building | Each bot builds a wall or tower in front of the other, or bots build a platform together at midpoint |
| buildTower | Building | Both bots build a tall 1-block tower by jumping and placing blocks underneath themselves. |
| buildHouse | Building | Bots build a house together. |
| towerBridge | Building | Bots build a 1-block tower by jumping, then build a bridge connecting the two towers, and meet. |
| pve | Combat | Bots defend two random positions on the ground, facing each other against spawning mobs. |
| pvp | Combat | Bots equip swords and fight each other. |
Table 4: Episode types in our training dataset.
Figure 11: An episode where a bot has been teleported underwater.
Figure 12: The “bubbles” HUD template PNG used to filter underwater episodes.
| Action key | Type | Description |
| forward | bool/sustained | Player moving forward (W). |
| back | bool/sustained | Player moving backward (S). |
| left | bool/sustained | Player strafing left (A). |
| right | bool/sustained | Player strafing right (D). |
| jump | bool/sustained | Player jumping. |
| sprint | bool/sustained | Player sprinting. |
Table 5: Action space of our multiplayer dataset. “Sustained” denotes action is recorded for as long as it lasts. “Once” denotes action is recorded for one tick only, usually applicable to instantaneous events.
Figure 13: A histogram showing the mouse action magnitude distribution in our training dataset.
| Setting | Learning Rate | Batch Size | Num Steps | Adam | Adam | Weight Decay |
| Bidirectional Single-player | 64 | 120K | 0.9 | 0.95 | 0 | |
| Bidirectional Multiplayer | 32 | 120K | 0.9 | 0.95 | 0 | |
| Causal Multiplayer | 32 | 60K | 0.9 | 0.95 | 0 | |
| Self Forcing (generator) | 32 | 240 | 0.9 | 0.95 | 0 | |
| Self Forcing (critic) | 32 | 1200 | 0.9 | 0.95 | 0 |
Table 6: Training hyperparameters across all training stages.
Figure 14: An illustration of the frames and prompts provided to the VLM for evaluation. Each scenario is a ground truth video for illustration. Above the third-person visualization, we show the frames that are sent to the VLM along with the corresponding question and correct answer. Prompts are abbreviated; please see Table 7 for the full text.
More in this topic
Video World Models
Video world models generate and predict action-conditioned visual futures to simulate changing scenes for embodied agents.
See all 257 papers