Despite the success of video world models, extending them to multiple agents is difficult because it requires generating several agent-specific video streams that are consistent in time and between agents. To address this gap, we introduce OverSeer, a multi-agent world model that augments egocentric agent views with auxiliary camera views. We suggest that, by fixing the auxiliary views in space, they act as a proxy for a world-centric representation: Unlike the agents' views, which can change entirely due to a simple change in agent position, changes in the auxiliary views directly map to localized changes in the world state. We show that these fixed views substantially help the model to maintain consistency. We also show that, due to their redundant and semi-static nature, the auxiliary views can be efficiently compressed with a delta-VAE that encodes only the changes relative to a static reference image, reducing the number of tokens required to process them. Experiments in Minecraft and Doom demonstrate improved world and agent consistency, with qualitative results on real-world multi-camera captures indicating that the approach is not tied to game engines. Our compression analysis further shows that delta-VAE substantially reduces the computational overhead of auxiliary-view generation. We conclude that these fixed views provide a simple and efficient way to improve multi-agent world models.
Figures & tables
Figure 1: OverSeer sees beyond what agents see. Two agents interact in a shared Minecraft environment (Left). In addition to generating each agent’s egocentric view (Bottom), OverSeer models six fixed-pose auxiliary cameras (Right). These auxiliary streams establish a persistent shared reference, enforcing world consistency over time and agent consistency across views.
Figure 2: OverSeer design overview. A δ -VAE (Left) compresses each fixed auxiliary stream into compact residual codes relative to its first-frame reference, and OverSeer (Right) jointly generates agent latents and these residual codes, conditioned on T5-encoded actions via cross-attention and Plücker-ray camera poses via AdaLN.
Figure 3: OverSeer keeps the agents consistent across views. Generated videos in Minecraft , GZDoom and a multi-camera Real Capture , each at two moments (before, after). Each agent’s trail is one 3D track projected into every auxiliary camera (Agent 1 blue, Agent 2 orange); it lands on the generated agent in each of them, so the agents are consistent across views.
Figure 4: δ -VAE residual codes carry the movers, not the scene. Four cross-scene transfers: a host anchor is decoded with a donor clip’s residual code (transfer, purple band) instead of its own (control), at two moments (before, after). The donor’s mover (white box) appears in the host scene (purple box) while the host scene itself is preserved.
Solaris Benchmark
MAC-Bench
Movement
Grounding
Memory
Building
Consistency
Reunite at Start
Reunite Far
Dynamic Event
Method
VLM ↑
FID ↓
VLM ↑
FID ↓
VLM ↑
FID ↓
VLM ↑
FID ↓
VLM ↑
FID ↓
VLM ↑
FID ↓
VLM ↑
FID ↓
VLM ↑
FID ↓
Solaris
68.8
22.38
66.7
14.92
20.8
47.11
22.9
76.01
56.3
68.42
0.0
17.06
6.2
20.23
12.5
26.52
γ -World
64.6
17.13
62.5
29.39
16.7
29.57
43.8
24.62
47.9
56.39
0.0
23.83
0.0
25.71
8.3
31.72
OverSeer
59.1
18.27
91.7
11.33
87.5
12.95
41.7
32.50
60.4
26.47
60.2
13.99
37.9
12.79
40.5
17.61
Table 1: Results on Minecraft. Comparison on the Solaris benchmark and our MAC-Bench. VLM denotes VLM-judged accuracy (%) and FID denotes frame-level FID; MAC-Bench results are averaged over 5 runs. Bold and underline denote the best and second-best results in each column, respectively.
Figure 5: OverSeer keeps the world consistent over time. On the three revisiting tasks (reunite at start, reunite far, dynamic event), OverSeer shows the partner and the crater where the ground truth does, while Solaris and γ -World mostly do not.
Minecraft
Doom
Reunite at Start
Reunite Far
Dynamic Event
Reunite at Start
Reunite Far
Variant
VLM ↑
FID ↓
VLM ↑
FID ↓
VLM ↑
FID ↓
VLM ↑
FID ↓
VLM ↑
FID ↓
OverSeer
60.2
14.0
37.9
12.8
40.5
17.6
62.0
23.2
67.0
20.9
w/o Aux. Views
25.0
24.6
16.7
27.2
20.8
33.1
12.0
50.6
0.0
87.4
w/o δ -VAE
59.4
8.2
40.6
13.4
41.7
10.3
78.0
15.8
84.0
15.4
Table 2: Ablation of auxiliary views and δ -VAE. Comparison on MAC-Bench in Minecraft and Doom. w/o δ -VAE models auxiliary views with uncompressed video-VAE latents, using 8× more auxiliary tokens. Bold denotes the best result in each column.
Figure 6: Effect of auxiliary camera count. Adding auxiliary views systematically improves FID and VLM accuracies on consistency and memory metrics across both benchmarks, with diminishing returns once spatial coverage saturates.
Figure 7: Effect of δ -VAE compression. δ -VAE keeps compute nearly flat as auxiliary views are added. Stronger compression trades reconstruction fidelity for disentanglement: donor leakage drops steadily, while mover transfer peaks at the default 8× before degrading.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Start
Mid-episode
End
Query
Reunite at Start
13 blocks apart, facing
each wanders ≤9 blocks
exact start poses
Is the partner visible?
Reunite Far
17 blocks apart
∼59 blocks each, converging; partner 86∘ off-axis
2.9 blocks apart, facing
Is the partner visible?
Dynamic Event
13 blocks apart, event at midpoint
separate to 40 blocks by frame 76
4.3 blocks apart
What happened to the event?
Appendix
Table 3: MAC-Bench task specifications in Minecraft. Mid-episode angles are medians over episodes.
δ -VAE
Minecraft Multi-agent Consistency Benchmark
PSNR ↑
Reunite at Start
Reunite Far
Dynamic Event
Variant
dB
VLM ↑
FID ↓
VLM ↑
FID ↓
VLM ↑
FID ↓
w/o Lalign
25.1
56.9
13.47
33.4
12.26
38.1
17.08
Full δ -VAE
24.7
60.2
13.99
37.9
12.79
40.5
17.61
Appendix
Table 4: Ablation of the feature alignment loss. We evaluate its effect on δ -VAE reconstruction quality and downstream world consistency in Minecraft.
Entry
Sentence
forward
The player is walking forward
left
The player is strafing to the left
back
The player is walking backward
right
The player is strafing to the right
jump
The player is jumping into the air
sneak
The player is sneaking and crouching down
Appendix
Table 5: The action vocabulary, one sentence per entry. The nine hotbar entries differ only in the number they name. drop , inv , esc , tab and swap_hands are never pressed in our episodes and are kept only so that the vocabulary matches the game’s input.
Figure 8: Additional MAC-Bench results in Minecraft. Top: the three benchmark tasks. Bottom: three further dynamic events. OverSeer recovers the partner or the evolved event state where the ground truth does, while the egocentric-only variant does not.
Figure 9: MAC-Bench results in Doom. Reunite tasks on three Freedoom maps. OverSeer restores the partner after the agents separate; the egocentric-only variant does not.
Figure 10: Long rollouts in Minecraft. Each block shows both agent views and all six auxiliary views over 30 s.
Figure 11: Long rollouts in Doom. Each block shows both agent views and all eight auxiliary views over 21 s.
Figure 12: Long rollouts on real-world multi-camera captures. Left: both agent views and all six auxiliary views over 30 s. Right: top-down layout of the auxiliary cameras and agent trajectories.
Video world models have achieved remarkable success in simulating environmental dynamics in response to actions by users or agents. They are modeled as action-conditioned video generation models that take historical frames and current actions as input to predict future frames. Yet, most existing approaches are limited to single-agent scenarios and fail to capture the complex interactions inherent in real-world multi-agent systems. We present MultiWorld, a unified framework for multi-agent multi-view world modeling that enables accurate control of multiple agents while maintaining multi-view consistency. We introduce the Multi-Agent Condition Module to achieve precise multi-agent controllability, and the Global State Encoder to ensure coherent observations across different views. MultiWorld supports flexible scaling of agent and view counts, and synthesizes different views in parallel for high efficiency. Experiments on multi-player game environments and multi-robot manipulation tasks demonstrate that MultiWorld outperforms baselines in video fidelity, action-following ability, and multi-view consistency. Project page: https://multi-world.github.io/
Video world models have made rapid progress in generating controllable visual experiences, but most of them still simulate the world from a single observer. Extending such models to multiple agents raises a central challenge: if each agent's future state is generated independently, overlapping views may instantiate different versions of the same scene, leading to inconsistent objects, layouts, and appearances across agents. Conventional camera conditioning controls individual trajectories, but it does not explicitly couple the generation of views that should agree under shared scene geometry. We introduce Prisma-World, a camera-controllable multi-agent world model that formulates multi-agent generation as a joint geometry-aware denoising process for cross-view consistency. Prisma-World processes all agent videos within one full-attention sequence, uses a multi-agent RoPE design to distinguish agent identities while preserving synchronized temporal coordinates, and injects relative camera geometry into attention to bias overlapping viewpoints toward shared scene evidence. To further strengthen multi-view consistency and enhance global spatial perception, we augment our framework with an overlap-decaying curriculum training paradigm alongside minimap-conditioned structural guidance. To facilitate the training and evaluation of multi-agent models, we introduce PrismaDataset, a large-scale UE5 dataset with panoramic acquisition across diverse scenes, composable multi-agent view groups with flexible agent counts and complex camera trajectories, and precise camera/action annotations for consistency training and evaluation. Experiments show that a single Prisma-World model can generate high-fidelity multi-agent videos with flexible agent numbers, camera controllability, improved cross-view consistency, and spatial grounding under minimap guidance.
Huiqiang Sun, Zhan Peng, Size Wu +9
School of AIA, HUST · S-Lab, NTU · SenseTime Research +4
Autoregressive video world models enable temporally coherent generation for a single observer. Extending them to multiple agents requires consistency across independently controlled views and temporal gaps under causal streaming. We present ConsistWorld, a multi-agent world model that generates camera-controlled video streams of a static scene from one shared image. We formulate consistency as routing evidence from committed multi-agent history and concurrently generated peer views to the tokens being generated. Pose Conditioned Memory Retrieval selects relevant historical observations from all agents, recovering evidence beyond the recent context window. Visibility-Gated Peer Sharing regulates current peer information according to estimated historical coverage and current-view overlap. Together, they determine which historical observations enter the context and where concurrent peer information contributes, supporting long-term recall and coordinated exploration. Both mechanisms use camera geometry and maintain a bounded active context for a fixed agent count and retrieval budget. Experiments on evidence sharing cases and video length and agent number generalizations show that ConsistWorld achieves a strong cross-time and cross-agent consistency while preserving competitive generation quality.
Qianxun Xu, Xianfang Zeng, Xinyao Liao +3
Westlake University · StepFun · University of California, Los Angeles