A multiplayer world model must predict how the independent actions of several players change a shared environment, including changes outside a player's current field of view. A point-of-view (POV) video captures such changes only after they enter its field of view, whereas panoramic video records the surroundings in every viewing direction and has served as an intermediate representation of the world state in recent world models. Learning this task therefore requires synchronized POV and panoramic videos of interacting players together with their actions, yet no existing dataset provides this combination in Minecraft, an established testbed in which players act independently in a shared, modifiable 3D world. We present PAWA, a dataset and framework for action-conditioned panoramic multiplayer world modeling. The dataset pairs each player's POV video with world-stabilized panoramic video and aligned actions. Its distributed pipeline decouples CPU gameplay capture from GPU rendering, scales beyond 64 simultaneous players, and re-renders each recorded session under new observation configurations without replaying the gameplay. On this dataset, we develop a dual-tower model that jointly predicts POV and panoramic video and couples the two streams through cross-attention that aligns tokens observing the same direction in a shared world frame. We further introduce a quadratic-phase player encoding that provides many more player codes than existing rotary codes at the same embedding dimension. Experiments show that PAWA outperforms state-of-the-art multiplayer world models in POV prediction quality while also producing high-fidelity panoramic predictions. The dataset and collection pipeline are open source to support further research on multiplayer world models.
Figures & tables
Figure 1: Panoramic multiplayer world modeling with PAWA. From each player’s initial POV and panoramic observations and the actions of both players, PAWA jointly forecasts every player’s POV video and world-stabilized panoramic video. Apart from the green-bordered inputs, all frames are forecasts of the distilled model. The dashed outline projects the dashed POV frame at 2 s onto the panorama at the same time. Circles mark the co-player, which appears in the matching direction in the POV and the panorama. Panoramas are cropped to ±55∘ latitude for display.
Figure 2: Distributed data generation. Coding agents write action scripts, CPU capture records sessions with more than 64 players without graphics clients, and GPU workers render the replays offline. Behavioral QC and VLM review return failed scripts for revision.
Figure 3: PAWA architecture. (a) POV and panorama towers share weights across players and are coupled by ray-aligned cross-attention at seven sites. Actions ap and appearance attributes ep condition self-attention, and the player code θp rotates Q and K. (b) Tokens that share a viewing ray, shown by the POV field of view projected onto the panorama. Each direction rotates the normalized Q and K by their viewing rays and adds its output through a zero-initialized gate.
Model
Stream
FVD ↓
FID ↓
LPIPS ↓
PSNR ↑
SSIM ↑
Gamma-World (Bidirectional)
POV
267.1
56.63
0.567
13.08
0.478
Gamma-World (Causal)
POV
306.7
52.00
0.606
12.46
0.451
Gamma-World (Distilled)
POV
279.7
58.36
0.584
11.83
0.431
Solaris
POV
614.9
77.31
0.634
10.42
0.447
MultiWorld
POV
771.2
186.06
0.667
11.76
0.395
PAWA (Causal)
POV
120.1
30.80
0.421
16.86
0.562
Table 1: Two-player interaction tier with 30 episodes and 239 forecast frames per player. Baselines support only POV input, and PAWA additionally receives the initial panoramas. Bold and underline mark the best and second best within each stream and evaluation group.
Figure 4: Synchronized two-player forecasts. Reference denotes the recorded continuation, and POV and panorama rows share time columns. The co-player in each POV can be compared with the one in the Reference row. MultiWorld uses 50 denoising steps per block and the other models use four.
Figure 5: A co-player leaving and re-entering the view. In (a) the co-player walks around behind the player, and in (b) it steps out to the side and returns. Green boxes mark a correct co-player, red boxes a co-player that should be hidden or a wrong character, and dashed red boxes a co-player the model has lost. The baselines keep the co-player in view, replace it or lose it, whereas the panorama of PAWA (Distilled) tracks the hidden co-player and its POV shows it again after the return.
4 players
8 players
16 players
Model
FVD ↓
LPIPS ↓
FVD ↓
LPIPS ↓
FVD ↓
LPIPS ↓
Gamma-World (Bidirectional)
752.3
0.633
1707.7
0.681
–
–
Gamma-World (Causal)
373.0
0.587
297.3
0.606
–
–
Gamma-World (Distilled)
323.6
0.577
267.1
0.586
–
–
PAWA (Causal)
138.0
0.427
111.5
0.458
92.2
0.467
PAWA (Distilled)
167.3
0.481
141.3
0.517
116.7
0.520
Table 2: POV prediction with larger populations. Bold and underline mark the best and second best.
Episodes
Player views
Episodes by player count
N=1
N=2
N=4
N=8
N=16
Collection snapshot
8162
45716
350
2555
2770
1327
1160
Table 3: Collection snapshot. Player views count complete per-player renders, each carrying a POV stream, a panorama stream and a per-frame action record.
Model
Stream
FVD ↓
FID ↓
LPIPS ↓
PSNR ↑
SSIM ↑
MAE ↓
Gamma-World (Bidirectional)
POV
333.4
67.25
0.557
12.87
0.453
37.70
Gamma-World (Causal)
POV
323.7
70.16
0.572
13.16
0.435
37.67
Gamma-World (Distilled)
POV
340.1
69.14
0.551
12.98
0.416
38.12
Solaris
POV
482.7
79.50
0.593
11.49
0.429
46.33
MultiWorld
POV
715.8
185.33
0.642
12.68
0.381
43.49
PAWA (Causal)
POV
157.0
41.61
0.380
16.95
0.546
20.97
Table 4: Two-player low-motion tier with 12 episodes and 239 forecast frames per player. Bold and underline mark the best and second best within each stream and evaluation group.
Model
Stream
FVD ↓
FID ↓
LPIPS ↓
PSNR ↑
SSIM ↑
MAE ↓
Gamma-World (Bidirectional)
POV
752.3
142.95
0.633
13.34
0.487
38.14
Gamma-World (Causal)
POV
373.0
57.02
0.587
13.06
0.465
40.38
Gamma-World (Distilled)
POV
323.6
56.65
0.577
12.39
0.440
42.73
PAWA (Causal)
POV
138.0
33.49
0.427
16.41
0.550
24.68
PAWA (Causal)
Panorama
161.9
32.79
0.254
19.08
0.630
17.42
PAWA (Distilled)
POV
167.3
33.86
0.481
15.08
0.506
30.54
Table 5: Four-player tier with 12 episodes and 239 forecast frames per player. Bold and underline mark the best and second best within each stream and evaluation group.
Model
Stream
FVD ↓
FID ↓
LPIPS ↓
PSNR ↑
SSIM ↑
MAE ↓
8 players
Gamma-World (Bidirectional)
POV
1707.7
213.93
0.681
12.83
0.497
43.22
Gamma-World (Causal)
POV
297.3
48.84
0.606
12.21
0.453
45.16
Gamma-World (Distilled)
POV
267.1
48.27
0.586
11.56
0.432
47.28
PAWA (Causal)
POV
111.5
32.32
0.458
15.06
0.536
28.92
PAWA (Causal)
Panorama
127.9
27.42
0.294
17.31
0.591
20.52
Table 6: Eight- and sixteen-player tiers with 12 episodes each and 239 forecast frames per player. At eight players, Gamma-World runs with its simplex player encoding extended from four to eight codes. Bold and underline mark the best and second best within each stream and evaluation group.
Players
Events
PAWA
Best baseline
Static
PAWA − best
2
48
0.582
0.654
0.635
−0.071
[−0.116,−0.023]
4
111
0.603
0.649
0.647
−0.046
[−0.076,−0.018]
8
505
0.649
0.710
0.688
−0.061
[−0.081,−0.040]
Table 7: Co-player LPIPS in the POV over the 10 frames after a co-player enters a player’s view after at least 20 frames outside it. Static repeats the conditioning frame, and the last column is the paired change of PAWA (Distilled) against the best baseline.
Players
Yaw error ↓
2, low-motion
2, interaction
4
8
16
Gamma-World (Bidirectional)
4.65
8.55
11.16
12.23
N/A
Gamma-World (Causal)
4.77
8.80
8.55
8.45
N/A
Gamma-World (Distilled)
3.68
8.35
7.77
8.00
N/A
Solaris
7.72
13.10
N/A
N/A
N/A
MultiWorld
6.26
13.90
N/A
N/A
N/A
Table 8: Action following. Top, mean absolute error in degrees per 0.5 s window between the yaw rotation estimated from the generated POV and the same estimate on the recorded video. Bottom, the fraction of players whose own actions best explain the rotation of their generated stream among all players whose camera turns noticeably, with chance in parentheses and the recorded videos as a reference.
Figure 6: Two-player comparison with all baselines. The episode of Figure 4 with frames 0, 80, 160 and 239 at 20 Hz. Separate Solaris and MultiWorld references preserve their native preprocessing. Gamma-World (Bidirectional) and Gamma-World (Causal) use 35 steps, MultiWorld uses 50 denoising steps per block and the other models use four.
Figure 7: Synchronized four-player forecasts. Both distilled models use four steps, and the first column contains the recorded inputs of the four players.
Stream
Sites
FVD ↓
FID ↓
LPIPS ↓
PSNR ↑
SSIM ↑
POV
Enabled
138.1
28.40
0.4648
15.382
0.5157
POV
Disabled
179.0
37.71
0.5096
14.749
0.5094
Panorama
Enabled
98.1
28.26
0.2425
19.765
0.6172
Panorama
Disabled
104.0
28.42
0.2444
19.645
0.6142
Table 9: Inference-time removal of the seven cross-attention sites from PAWA (Distilled). Both configurations use the same final distilled checkpoint, 30 two-player interaction episodes, recorded noise, seed 1 and four-step sampler. No model is retrained. Bold and underline mark the best and second best within each stream and evaluation group.
POV
Panorama
Removed
Δ FVD
Δ FID
Δ LPIPS
95% interval
Δ PSNR
Δ FVD
Δ LPIPS
All sites
+36.0
+10.04
+0.052
[0.039,0.067]
−1.09
+8.0
+0.002
Pano → POV only
+36.0
+10.04
+0.052
[0.039,0.067]
−1.09
+1.2
−0.001
POV → pano only
+3.8
+0.86
+0.009
[0.004,0.014]
−0.34
+8.0
+0.002
Early sites
+10.3
+3.87
+0.024
[0.013,0.035]
−0.58
−2.9
−0.001
Late sites
+6.8
+2.61
+0.025
[0.016,0.035]
−0.56
+6.0
+0.001
Table 10: Inference-time removal of cross-attention directions and sites from PAWA (Distilled) on the 30 two-player interaction episodes. Values are changes against the outputs of the full model regenerated in the same run with identical checkpoint, recorded noise and sampler, and positive FVD and LPIPS changes mean degradation. Pano → POV denotes POV queries reading panoramic keys, early sites follow blocks 3, 7 and 11, and late sites follow blocks 19, 23 and 27.
Updates
Cross-attention
FVD ↓
FID ↓
LPIPS ↓
PSNR ↑
0
with
103.5
27.68
0.452
16.08
0
without
131.1
30.67
0.464
15.90
1000
with
139.8
29.98
0.455
15.90
1000
without
152.1
34.23
0.462
15.66
2000
with
143.7
32.35
0.448
16.14
2000
without
164.7
38.22
0.478
15.56
Table 11: Training with and without the cross-attention. Both arms continue the causal model from the same checkpoint with the same data distribution, learning rate and batch size, and differ in the cross-attention sites. Their batches are identical for the first 2000 updates. At 0 updates the arm without cross-attention is the starting checkpoint with its sites disabled. POV scores on the 30 two-player interaction episodes. Bold marks the better arm at each training length.
Model
Site
Angle ↓
Frustum mass ↑
Mass within 15∘↑
Argmax error ↓
PAWA (Distilled)
19
51∘
0.43
0.25
9∘
23
50∘
0.46
0.29
7∘
27
65∘
0.28
0.16
24∘
PAWA (Causal)
19
47∘
0.47
0.29
8∘
23
38∘
0.57
0.43
5∘
27
55∘
0.36
0.25
12∘
Table 12: Geometry of pano → POV attention at the late cross-attention sites, averaged over heads. For each POV query we report the expected angle between its ray and the attended panoramic key rays, the attention mass inside the POV frustum, the mass within 15∘ of the query ray, and the median angle to the most attended key. Chance is uniform attention over the panorama, and the control re-initializes the cross-attention sites of the distilled model while keeping its ray encoding.
Whole frame
Co-player regions
Model
1–44
45–92
93–188
189–239
1–44
45–92
93–188
189–239
Gamma-World (Bidirectional)
0.447
0.565
0.595
0.621
0.505
0.617
0.656
0.646
Gamma-World (Causal)
0.452
0.591
0.645
0.678
0.514
0.609
0.679
0.710
Gamma-World (Distilled)
0.446
0.571
0.617
0.651
0.507
0.607
0.661
0.718
Solaris
0.471
0.619
0.679
0.702
0.512
0.680
0.763
0.783
MultiWorld
0.588
0.677
0.690
0.683
0.618
0.704
0.728
0.694
Table 13: POV LPIPS by forecast-frame band on the 30 two-player interaction episodes, over the whole frame and within projected co-player regions. Repeat frame 0 copies the conditioning frame for the whole rollout and serves as a static reference.
Existing action-conditioned video generation models (video world models) are limited to single-agent perspectives, failing to capture the multi-agent interactions of real-world environments. We introduce Solaris, a multiplayer video world model that simulates consistent multi-view observations. To enable this, we develop a multiplayer data system designed for robust, continuous, and automated data collection on video games such as Minecraft. Unlike prior platforms built for single-player settings, our system supports coordinated multi-agent interaction and synchronized videos + actions capture. Using this system, we collect 12.64 million multiplayer frames and propose an evaluation framework for multiplayer movement, memory, grounding, building, and view consistency. We train Solaris using a staged pipeline that progressively transitions from single-player to multiplayer modeling, combining bidirectional, causal, and Self Forcing training. In the final stage, we introduce Checkpointed Self Forcing, a memory-efficient Self Forcing variant that enables a longer-horizon teacher. Results show our architecture and training design outperform existing baselines. Through open-sourcing our system and models, we hope to lay the groundwork for a new generation of multi-agent world models.
Multiplayer world models must generate independently controlled views with consistent representations of both players and their shared environment. Most existing approaches coordinate multiple players through joint multi-view generation, whose cost grows with each additional player. We present WorldCast, a distributed multiplayer world model in which each player runs a local client comprising a video generator and a state model. Using recorded player positions and map geometry during training, the state model estimates the player's position from generated video and control inputs. Clients exchange player states and project them into camera-aligned player state fields that guide where and how other players are rendered. Shared scene state enables clients to reuse one another's generated observations to maintain consistent scene appearance across views. Experiments on Counter-Strike 2 demonstrate WorldCast's consistency, real-time performance, and distributed scalability. The camera-aligned player state field improves player rendering rates by over an order of magnitude over joint-generation methods, while shared scene state improves visual consistency over whole rounds. Each client runs in real time and exchanges only player and scene states, enabling scalable multiplayer generation without a centralized computational bottleneck. Image quality remains stable over hour-long rollouts.
Current video world models struggle in multiplayer environments because they entangle world state with view-dependent visual latents, leading to redundant compute, view inconsistencies, and poor scalability. We propose MASS (Multiplayer world models with Authoritative Shared State) to resolve this limitation. Inspired by multiplayer game architectures, MASS disentangles world dynamics and view rendering. A learned Logic Engine advances a global, authoritative typed state from joint actions without any hand-written transition function, acting as the sole recurrent memory and synchronization reference. From this shared state, a learned Rendering Engine generates independent and consistent views for any requested camera on demand. This explicit disentangling allows MASS to achieve superior state accuracy and lower cross-view inconsistency compared to state-of-the-art multi-view baselines on a matched multiplayer Snake benchmark. It advances predicted worlds with 1,024 concurrent players for 10,000 recurrent steps. Our results show that explicit, authoritative state modeling provides a practical foundation for scalable and consistent multi-agent world simulation.
Ziqi Cai, Siqi Yang, Yimu Wang +6
1Alaya Lab 2Peking University 3Institute of Science Tokyo