PAWA: Panoramic Action-conditioned World Modeling for Multiple Agents

Siqi Yang; Yimu Wang; Ziqi Cai; Chengxuan Zhu; Shuchen Weng; Erwin Wu; Zhaofei Yu; Kaipeng Zhang

cs.CVOct 14, 2026

PAWA: Panoramic Action-conditioned World Modeling for Multiple Agents

Authors: Siqi Yang, Yimu Wang, Ziqi Cai, Chengxuan Zhu, Shuchen Weng, Erwin Wu, Zhaofei Yu, Kaipeng Zhang,

Organizations: Alaya Lab · Peking University · Institute of Science Tokyo

Abstract

A multiplayer world model must predict how the independent actions of several players change a shared environment, including changes outside a player's current field of view. A point-of-view (POV) video captures such changes only after they enter its field of view, whereas panoramic video records the surroundings in every viewing direction and has served as an intermediate representation of the world state in recent world models. Learning this task therefore requires synchronized POV and panoramic videos of interacting players together with their actions, yet no existing dataset provides this combination in Minecraft, an established testbed in which players act independently in a shared, modifiable 3D world. We present PAWA, a dataset and framework for action-conditioned panoramic multiplayer world modeling. The dataset pairs each player's POV video with world-stabilized panoramic video and aligned actions. Its distributed pipeline decouples CPU gameplay capture from GPU rendering, scales beyond 64 simultaneous players, and re-renders each recorded session under new observation configurations without replaying the gameplay. On this dataset, we develop a dual-tower model that jointly predicts POV and panoramic video and couples the two streams through cross-attention that aligns tokens observing the same direction in a shared world frame. We further introduce a quadratic-phase player encoding that provides many more player codes than existing rotary codes at the same embedding dimension. Experiments show that PAWA outperforms state-of-the-art multiplayer world models in POV prediction quality while also producing high-fidelity panoramic predictions. The dataset and collection pipeline are open source to support further research on multiplayer world models.

Figures & tables

Figure 1: Panoramic multiplayer world modeling with PAWA. From each player’s initial POV and panoramic observations and the actions of both players, PAWA jointly forecasts every player’s POV video and world-stabilized panoramic video. Apart from the green-bordered inputs, all frames are forecasts of the distilled model. The dashed outline projects the dashed POV frame at 2 s onto the panorama at the same time. Circles mark the co-player, which appears in the matching direction in the POV and the panorama. Panoramas are cropped to $\pm 55^{\circ}$ latitude for display.

Figure 2: Distributed data generation. Coding agents write action scripts, CPU capture records sessions with more than 64 players without graphics clients, and GPU workers render the replays offline. Behavioral QC and VLM review return failed scripts for revision.

Figure 3: PAWA architecture. (a) POV and panorama towers share weights across players and are coupled by ray-aligned cross-attention at seven sites. Actions $a_{p}$ and appearance attributes $e_{p}$ condition self-attention, and the player code $\theta_{p}$ rotates Q and K. (b) Tokens that share a viewing ray, shown by the POV field of view projected onto the panorama. Each direction rotates the normalized Q and K by their viewing rays and adds its output through a zero-initialized gate.

Model	Stream	FVD $\downarrow$	FID $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$
Gamma-World (Bidirectional)	POV	$267.1$	$56.63$	$0.567$	$13.08$	$0.478$
Gamma-World (Causal)	POV	$306.7$	$52.00$	$0.606$	$12.46$	$0.451$
Gamma-World (Distilled)	POV	$279.7$	$58.36$	$0.584$	$11.83$	$0.431$
Solaris	POV	$614.9$	$77.31$	$0.634$	$10.42$	$0.447$
MultiWorld	POV	$771.2$	$186.06$	$0.667$	$11.76$	$0.395$
PAWA (Causal)	POV	120.1	30.80	0.421	16.86	0.562

Table 1: Two-player interaction tier with 30 episodes and 239 forecast frames per player. Baselines support only POV input, and PAWA additionally receives the initial panoramas. Bold and underline mark the best and second best within each stream and evaluation group.

Figure 4: Synchronized two-player forecasts. Reference denotes the recorded continuation, and POV and panorama rows share time columns. The co-player in each POV can be compared with the one in the Reference row. MultiWorld uses 50 denoising steps per block and the other models use four.

Figure 5: A co-player leaving and re-entering the view. In (a) the co-player walks around behind the player, and in (b) it steps out to the side and returns. Green boxes mark a correct co-player, red boxes a co-player that should be hidden or a wrong character, and dashed red boxes a co-player the model has lost. The baselines keep the co-player in view, replace it or lose it, whereas the panorama of PAWA (Distilled) tracks the hidden co-player and its POV shows it again after the return.

	4 players		8 players		16 players
Model	FVD $\downarrow$	LPIPS $\downarrow$	FVD $\downarrow$	LPIPS $\downarrow$	FVD $\downarrow$	LPIPS $\downarrow$
Gamma-World (Bidirectional)	$752.3$	$0.633$	$1707.7$	$0.681$	–	–
Gamma-World (Causal)	$373.0$	$0.587$	$297.3$	$0.606$	–	–
Gamma-World (Distilled)	$323.6$	$0.577$	$267.1$	$0.586$	–	–
PAWA (Causal)	138.0	0.427	111.5	0.458	92.2	0.467
PAWA (Distilled)	167.3	0.481	141.3	0.517	116.7	0.520

Table 2: POV prediction with larger populations. Bold and underline mark the best and second best.

	Episodes	Player views	Episodes by player count
			$N{=}1$	$N{=}2$	$N{=}4$	$N{=}8$	$N{=}16$
Collection snapshot	$8162$	$45716$	$350$	$2555$	$2770$	$1327$	$1160$

Table 3: Collection snapshot. Player views count complete per-player renders, each carrying a POV stream, a panorama stream and a per-frame action record.

Model	Stream	FVD $\downarrow$	FID $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	MAE $\downarrow$
Gamma-World (Bidirectional)	POV	$333.4$	$67.25$	$0.557$	$12.87$	$0.453$	$37.70$
Gamma-World (Causal)	POV	$323.7$	$70.16$	$0.572$	$13.16$	$0.435$	$37.67$
Gamma-World (Distilled)	POV	$340.1$	$69.14$	$0.551$	$12.98$	$0.416$	$38.12$
Solaris	POV	$482.7$	$79.50$	$0.593$	$11.49$	$0.429$	$46.33$
MultiWorld	POV	$715.8$	$185.33$	$0.642$	$12.68$	$0.381$	$43.49$
PAWA (Causal)	POV	157.0	41.61	0.380	16.95	0.546	20.97

Table 4: Two-player low-motion tier with 12 episodes and 239 forecast frames per player. Bold and underline mark the best and second best within each stream and evaluation group.

Model	Stream	FVD $\downarrow$	FID $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	MAE $\downarrow$
Gamma-World (Bidirectional)	POV	$752.3$	$142.95$	$0.633$	$13.34$	$0.487$	$38.14$
Gamma-World (Causal)	POV	$373.0$	$57.02$	$0.587$	$13.06$	$0.465$	$40.38$
Gamma-World (Distilled)	POV	$323.6$	$56.65$	$0.577$	$12.39$	$0.440$	$42.73$
PAWA (Causal)	POV	138.0	33.49	0.427	16.41	0.550	24.68
PAWA (Causal)	Panorama	161.9	32.79	0.254	19.08	0.630	17.42
PAWA (Distilled)	POV	167.3	33.86	0.481	15.08	0.506	30.54

Table 5: Four-player tier with 12 episodes and 239 forecast frames per player. Bold and underline mark the best and second best within each stream and evaluation group.

Model	Stream	FVD $\downarrow$	FID $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	MAE $\downarrow$
8 players
Gamma-World (Bidirectional)	POV	$1707.7$	$213.93$	$0.681$	$12.83$	0.497	$43.22$
Gamma-World (Causal)	POV	$297.3$	$48.84$	$0.606$	$12.21$	$0.453$	$45.16$
Gamma-World (Distilled)	POV	$267.1$	$48.27$	$0.586$	$11.56$	$0.432$	$47.28$
PAWA (Causal)	POV	111.5	32.32	0.458	15.06	0.536	28.92
PAWA (Causal)	Panorama	127.9	27.42	0.294	17.31	0.591	20.52

Table 6: Eight- and sixteen-player tiers with 12 episodes each and 239 forecast frames per player. At eight players, Gamma-World runs with its simplex player encoding extended from four to eight codes. Bold and underline mark the best and second best within each stream and evaluation group.

Players	Events	PAWA	Best baseline	Static	PAWA $-$ best
$2$	$48$	0.582	$0.654$	$0.635$	$-0.071$	$[-0.116,-0.023]$
$4$	$111$	0.603	$0.649$	$0.647$	$-0.046$	$[-0.076,-0.018]$
$8$	$505$	0.649	$0.710$	$0.688$	$-0.061$	$[-0.081,-0.040]$

Table 7: Co-player LPIPS in the POV over the 10 frames after a co-player enters a player’s view after at least 20 frames outside it. Static repeats the conditioning frame, and the last column is the paired change of PAWA (Distilled) against the best baseline.

	Players
Yaw error $\downarrow$	2, low-motion	2, interaction	$4$	$8$	$16$
Gamma-World (Bidirectional)	4.65	8.55	$11.16$	$12.23$	N/A
Gamma-World (Causal)	4.77	8.80	$8.55$	$8.45$	N/A
Gamma-World (Distilled)	3.68	8.35	$7.77$	$8.00$	N/A
Solaris	7.72	13.10	N/A	N/A	N/A
MultiWorld	6.26	13.90	N/A	N/A	N/A

Table 8: Action following. Top, mean absolute error in degrees per 0.5 s window between the yaw rotation estimated from the generated POV and the same estimate on the recorded video. Bottom, the fraction of players whose own actions best explain the rotation of their generated stream among all players whose camera turns noticeably, with chance in parentheses and the recorded videos as a reference.

Figure 6: Two-player comparison with all baselines. The episode of Figure 4 with frames 0, 80, 160 and 239 at 20 Hz. Separate Solaris and MultiWorld references preserve their native preprocessing. Gamma-World (Bidirectional) and Gamma-World (Causal) use 35 steps, MultiWorld uses 50 denoising steps per block and the other models use four.

Figure 7: Synchronized four-player forecasts. Both distilled models use four steps, and the first column contains the recorded inputs of the four players.

Stream	Sites	FVD $\downarrow$	FID $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$
POV	Enabled	138.1	28.40	0.4648	15.382	0.5157
POV	Disabled	179.0	37.71	0.5096	14.749	0.5094
Panorama	Enabled	98.1	28.26	0.2425	19.765	0.6172
Panorama	Disabled	104.0	28.42	0.2444	19.645	0.6142

Table 9: Inference-time removal of the seven cross-attention sites from PAWA (Distilled). Both configurations use the same final distilled checkpoint, 30 two-player interaction episodes, recorded noise, seed 1 and four-step sampler. No model is retrained. Bold and underline mark the best and second best within each stream and evaluation group.

Removed	$\Delta$ FVD	$\Delta$ FID	$\Delta$ LPIPS	95% interval	$\Delta$ PSNR	$\Delta$ FVD	$\Delta$ LPIPS
	POV					Panorama
All sites	$+36.0$	$+10.04$	$+0.052$	$[0.039,0.067]$	$-1.09$	$+8.0$	$+0.002$
Pano $\to$ POV only	$+36.0$	$+10.04$	$+0.052$	$[0.039,0.067]$	$-1.09$	$+1.2$	$-0.001$
POV $\to$ pano only	$+3.8$	$+0.86$	$+0.009$	$[0.004,0.014]$	$-0.34$	$+8.0$	$+0.002$
Early sites	$+10.3$	$+3.87$	$+0.024$	$[0.013,0.035]$	$-0.58$	$-2.9$	$-0.001$
Late sites	$+6.8$	$+2.61$	$+0.025$	$[0.016,0.035]$	$-0.56$	$+6.0$	$+0.001$

Table 10: Inference-time removal of cross-attention directions and sites from PAWA (Distilled) on the 30 two-player interaction episodes. Values are changes against the outputs of the full model regenerated in the same run with identical checkpoint, recorded noise and sampler, and positive FVD and LPIPS changes mean degradation. Pano

\to

POV denotes POV queries reading panoramic keys, early sites follow blocks 3, 7 and 11, and late sites follow blocks 19, 23 and 27.

Updates	Cross-attention	FVD $\downarrow$	FID $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$
$0$	with	103.5	27.68	0.452	16.08
$0$	without	$131.1$	$30.67$	$0.464$	$15.90$
$1000$	with	139.8	29.98	0.455	15.90
$1000$	without	$152.1$	$34.23$	$0.462$	$15.66$
$2000$	with	143.7	32.35	0.448	16.14
$2000$	without	$164.7$	$38.22$	$0.478$	$15.56$

Table 11: Training with and without the cross-attention. Both arms continue the causal model from the same checkpoint with the same data distribution, learning rate and batch size, and differ in the cross-attention sites. Their batches are identical for the first 2000 updates. At 0 updates the arm without cross-attention is the starting checkpoint with its sites disabled. POV scores on the 30 two-player interaction episodes. Bold marks the better arm at each training length.

Model	Site	Angle $\downarrow$	Frustum mass $\uparrow$	Mass within $15^{\circ}$ $\uparrow$	Argmax error $\downarrow$
PAWA (Distilled)	19	$51^{\circ}$	$0.43$	$0.25$	$9^{\circ}$
	23	$50^{\circ}$	$0.46$	$0.29$	$7^{\circ}$
	27	$65^{\circ}$	$0.28$	$0.16$	$24^{\circ}$
PAWA (Causal)	19	$47^{\circ}$	$0.47$	$0.29$	$8^{\circ}$
	23	$38^{\circ}$	$0.57$	$0.43$	$5^{\circ}$
	27	$55^{\circ}$	$0.36$	$0.25$	$12^{\circ}$

Table 12: Geometry of pano

\to

POV attention at the late cross-attention sites, averaged over heads. For each POV query we report the expected angle between its ray and the attended panoramic key rays, the attention mass inside the POV frustum, the mass within

15^{\circ}

of the query ray, and the median angle to the most attended key. Chance is uniform attention over the panorama, and the control re-initializes the cross-attention sites of the distilled model while keeping its ray encoding.

	Whole frame				Co-player regions
Model	1–44	45–92	93–188	189–239	1–44	45–92	93–188	189–239
Gamma-World (Bidirectional)	$0.447$	$0.565$	$0.595$	$0.621$	$0.505$	$0.617$	$0.656$	$0.646$
Gamma-World (Causal)	$0.452$	$0.591$	$0.645$	$0.678$	$0.514$	$0.609$	$0.679$	$0.710$
Gamma-World (Distilled)	$0.446$	$0.571$	$0.617$	$0.651$	$0.507$	$0.607$	$0.661$	$0.718$
Solaris	$0.471$	$0.619$	$0.679$	$0.702$	$0.512$	$0.680$	$0.763$	$0.783$
MultiWorld	$0.588$	$0.677$	$0.690$	$0.683$	$0.618$	$0.704$	$0.728$	$0.694$

Table 13: POV LPIPS by forecast-frame band on the 30 two-player interaction episodes, over the whole frame and within projected co-player regions. Repeat frame 0 copies the conditioning frame for the whole rollout and serves as a static reference.

World Model Learning

World model learning builds predictive models of how environments change under actions to support planning, simulation, and decision-making.

See all 414 papers

Solaris: Building a Multiplayer Video World Model in Minecraft
Feb 25, 2026Georgy Savva, Oscar Michel, Daohan Lu +6Video World Models Action-Conditioned Video Generation
New York University
WorldCast: Distributed Multiplayer World Models
Oct 8, 2026Ziyang Ye, Junchao Huang, Evelyn Zhang +9World Models Interactive World Models
CUHK-Shenzhen · SLAI · Tsinghua SIGS +2
MASS: Multiplayer World Models with Authoritative Shared State
Aug 6, 2026Ziqi Cai, Siqi Yang, Yimu Wang +6Multi-View Consistency World Model Learning
1Alaya Lab 2Peking University 3Institute of Science Tokyo

Feb 25, 2026J/K move · Enter open · S save

Solaris: Building a Multiplayer Video World Model in Minecraft

Georgy Savva, Oscar Michel, Daohan Lu, Suppakit Waiwitlikhit+5

New York University

Video World Models Action-Conditioned Video Generation Multi-View Video Generation Long-Horizon Video Generation

Open paper AskPDF arXiv

Existing action-conditioned video generation models (video world models) are limited to single-agent perspectives, failing to capture the multi-agent interactions of real-world environments. We introduce Solaris, a multiplayer video world model that simulates consistent multi-view observations. To enable this, we develop a multiplayer data system designed for robust, continuous, and automated data collection on video games such as Minecraft. Unlike prior platforms built for single-player settings, our system supports coordinated multi-agent interaction and synchronized videos + actions capture. Using this system, we collect 12.64 million multiplayer frames and propose an evaluation framework for multiplayer movement, memory, grounding, building, and view consistency. We train Solaris using a staged pipeline that progressively transitions from single-player to multiplayer modeling, combining bidirectional, causal, and Self Forcing training. In the final stage, we introduce Checkpointed Self Forcing, a memory-efficient Self Forcing variant that enables a longer-horizon teacher. Results show our architecture and training design outperform existing baselines. Through open-sourcing our system and models, we hope to lay the groundwork for a new generation of multi-agent world models.

Abstract

World Model Learning

Explore similar work

Solaris: Building a Multiplayer Video World Model in Minecraft

WorldCast: Distributed Multiplayer World Models

MASS: Multiplayer World Models with Authoritative Shared State