Organizations: Shenzhen International Graduate School, Tsinghua University · Kling Team · The Hong Kong University of Science and Technology · Peking University · University of Science and Technology of China
Interactive video world models predict how an environment evolves under user actions, but most existing systems assume one controlled agent and one task-specific action interface. We study multi-agent video world modeling, where one model generates a joint visual future for multiple controlled agents while interpreting heterogeneous controls ranging from binary game buttons to continuous robot commands. This task presents two complementary challenges. First, every agent-indexed action stream must control its intended visual entity independently. Second, the model must adapt the meaning of each control variable across different tasks, where the same tensor dimension may denote an attack, steering command, or robot command. We introduce LiveAct, centered on a Heterogeneous Action Interface (HAI) whose two components mirror this decomposition. To support independent control, Agent-Aware Spatial Modulation uses language-driven spatial gates and agent-specific attention to bind each action stream to its intended visual agent. To accommodate heterogeneous controls, Schema-Guided Action Adaptation describes each control variable in language and modulates its semantic token with the synchronized numerical value. We further transfer the complete interface to a blockwise causal generator through teacher forcing and rollout-aware distribution matching, enabling efficient streaming generation while retaining agent-specific control and action semantics. We instantiate the formulation on six multi-agent sources spanning games and multi-robot manipulation, and evaluate visual quality, action following, agent attribution, and latency. Across games and robot manipulation, LiveAct reaches 0.898 and 0.942 agent-attribution accuracy without agent locations or identity crops, and the causal generator attains 17.70 FPS at 512×448.
Figures & tables
Figure 2: Two challenges for multi-agent action interfaces. (a) Streams that differ only by input order entangle fighters under index-only binding; LiveAct’s agent gate keeps each on its fighter. (b) Dimension j=7 changes meaning across tasks. A raw vector keeps the value but not the meaning, free text the reverse; LiveAct modulates each schema token by its value.
Figure 3: Overview of LiveAct. (a) Foundational Ability Acquisition trains a bidirectional teacher with a Heterogeneous Action Interface, which integrates Agent-Aware Spatial Modulation and Schema-Guided Action Adaptation. (b) Teacher Forcing for AR Tuning transfers the same HAI to causal attention. (c) DMD for Streaming Video distills few-step streaming rollouts.
Figure 4: Data curation pipeline. Game and robotic episodes are curated into aligned samples containing a scene prompt, per-agent action streams, agent descriptions, and task-specific schemas.
Figure 5: Qualitative comparison on shared initial frames and action streams. Columns show ground truth, the five baselines, and LiveAct; Rows are frames sampled at different durations.
Reconstruction
Control
Diagnostics
Method
FVD ↓
LPIPS ↓
PSNR ↑
SSIM ↑
Act-F1 ↑ / NRMSE ↓
Attr-Acc ↑
MME ↓
Aesthetic ↑
Multi-Agent Video Games
LingBot-World 2.0 [ 6 ]
1256.6
0.322
13.25
0.490
0.146
0.553
0.57
5.093
MultiWorld (1-view) a [ 50 ]
2661.4
0.719
10.40
0.125
0.163
0.548
2.67
4.059
Incantation [ 63 ]
1989.5
0.593
10.45
0.226
0.162
0.534
8.77
4.539
ActionParty [ 30 ]
1672.5
0.272
14.56
0.467
0.086
0.547
0.47
4.974
Table 2: Main comparison across multi-agent video games and multi-robot manipulation. Best results are bolded. a Native MultiWorld uses synchronized multi-view observations; we have set it to a fixed same perspective. b The code is not released; this is our reproduction.
Variant
Game Act-F1 ↑
Game Attr-Acc ↑
Robot NRMSE ↓
Robot Attr-Acc ↑
Full LiveAct
0.594
0.898
0.558
0.942
w/o AASM
0.172
0.592
0.931
0.810
w/o SGAA
0.225
0.603
1.120
0.484
Table 3: Quantitative ablation study.
Figure 6: Qualitative ablation study.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1 : Showcase of our dataset. Frames from the six sources, covering versus fighting, shared-map competition, shared-track racing, and multi-arm manipulation.
Street Fighter II
Fatal Fury 2
Mega Bomberman
Micro Machines
RoboTwin
RoboFactory
Total
Trajectories
5460
3200
2050
2150
2532
2559
17951
Agents
2
2
2–4
2–4
2
2–3
2–4
Action type
binary
binary
binary
binary
continuous
continuous
mixed
Frame rate (fps)
60
60
60
60
30 / 50
30
–
Raw frames (M)
9.83
5.76
3.69
3.87
0.65
0.91
24.71
Raw duration (h)
45.50
26.67
17.08
17.92
3.93
8.44
119.54
Appendix
Table S1 : Dataset statistics for four multi-agent games and two multi-robot manipulation sources. Frame counts and durations refer to the raw recordings at their native frame rates (RoboTwin: 30 fps for self-collected and 50 fps for augmented episodes). Training and evaluation use 81-frame clips drawn from these trajectories.
81 frames at 512×448 , 16 fps; 1 conditioning and 20 future latents
Frame sampling
Games: consecutive native frames; robots: uniform over the episode
Actions
12 description-modulated controls per agent; one-latent window
Conditioner
Text projections 4096→1024→3072 ; value/gate/delta bottleneck 768
Appendix
Table S2 : Training and inference configuration of LiveAct. Stages 2 and 3 are trained with FSDP, and Stage 3 samples at timesteps [1000,750,500,250] with the EMA weights; “–” marks settings that do not apply.
Stage 1 Foundational Ability Acquisition
Stage 2 Teacher Forcing for AR Tuning
Stage 3 DMD for Streaming Video
Role
Full-context teacher
Causal initialization
Rollout distillation
Attention
Bidirectional
Blockwise causal
Blockwise causal
Training history
Full video
Clean chunks
Model rollouts
Streaming
×
✓
✓
Chunk size
Full sequence
4 latent frames
4 latent frames
Denoising steps
50
50
4
Appendix
Table S3 : Three-stage causal transfer. Generation settings and results of each stage; throughput is measured at 512×448 on a single GPU, and Stage 3 samples without classifier-free guidance (CFG). Best results are bolded.
Variant
Control identity
Tokens per frame
Value fusion
Training
Control token
Full LiveAct
schema uj
C
affine modulation
trained
hτ,j=uj⊙(1+γτ,j)+βτ,j
w/o SGAA
–
1
flat MLP
trained
hτ=MLPflat(Aτ)
w/o value modulation
schema uj
C
addition
trained
hτ,j=uj+MLPval(aτ,j)
w/o schema semantics
learned ID ejID
C
affine modulation
trained
hτ,j=ejID⊙(1+γτ,j)+βτ,j
shuffled schema
schema uπ(j)
C
affine modulation
inference only
hτ,j=uπ(j)⊙(1+γτ,j)+βτ,j
Appendix
Table S4 : Design of the Schema-Guided Action Adaptation variants. Each variant changes only how the control tokens of agent i are constructed; the superscript (i) is omitted. All variants keep Agent-Aware Spatial Modulation, per-agent cross-attention, the one-latent action window, and the time and agent embeddings of Eq. 5 . uj is the projected schema embedding of Eq. 3 , (γτ,j,βτ,j)=MLPmod(aτ,j) as in Eq. 4 , ejID is a learned per-dimension embedding, and π is a fixed permutation of the control dimensions.
Variant
Game Act-F1 ↑
Game Attr-Acc ↑
Robot NRMSE ↓
Robot Attr-Acc ↑
Full LiveAct ( L=1 )
0.594
0.898
0.558
0.942
Schema-Guided Action Adaptation
w/o SGAA (flat MLP)
0.225
0.603
1.120
0.484
w/o value modulation (additive value)
0.209
0.513
1.071
0.480
w/o schema semantics (ID tokens)
0.230
0.588
0.799
0.778
shuffled schema (inference only)
0.207
0.504
1.063
0.496
Appendix
Table S5 : Detailed ablations of Schema-Guided Action Adaptation and temporal alignment. The SGAA variants are defined in Tab. S4 . All trained variants use the same data and optimization budget as the full model. L is the action window length in latent frames (App. B.2 ). Best results are bolded.
Figure S2 : Spatial localization of agent-specific control residuals. Panels (a)–(d) show representative spatial residual-magnitude heatmaps from the agent-specific branches. Red and blue denote agents 1 and 2, respectively, and higher intensity indicates a larger response. Across the examples, the high-response regions concentrate around the corresponding visual agents.
Figure S3 : Additional qualitative comparison (set 1). Same layout and methods as Fig. 5 .
Figure S4 : Additional qualitative comparison (set 2). Same layout and methods as Fig. 5 .
Figure S5 : Additional LiveAct showcase.
Figure S6 : Showcase beyond two agents. Scenes with more than two concurrently controlled agents. For each scene, the top row shows the ground truth and the bottom row the LiveAct generation from the same initial frame and per-agent action streams.
Figure S7 : Single-case long-horizon rollout as a dense frame matrix. We show 24 generated frames from the same 241-frame rollout, sampled every ten frames and ordered from left to right and top to bottom.
Video world models have achieved remarkable success in simulating environmental dynamics in response to actions by users or agents. They are modeled as action-conditioned video generation models that take historical frames and current actions as input to predict future frames. Yet, most existing approaches are limited to single-agent scenarios and fail to capture the complex interactions inherent in real-world multi-agent systems. We present MultiWorld, a unified framework for multi-agent multi-view world modeling that enables accurate control of multiple agents while maintaining multi-view consistency. We introduce the Multi-Agent Condition Module to achieve precise multi-agent controllability, and the Global State Encoder to ensure coherent observations across different views. MultiWorld supports flexible scaling of agent and view counts, and synthesizes different views in parallel for high efficiency. Experiments on multi-player game environments and multi-robot manipulation tasks demonstrate that MultiWorld outperforms baselines in video fidelity, action-following ability, and multi-view consistency. Project page: https://multi-world.github.io/
World models for interactive video generation have largely focused on single-agent settings, where future observations are generated from a single control signal. However, many generated environments require multi-agent interaction: multiple players, robots, or embodied agents act simultaneously within a shared space. Scaling world models to such settings requires a principled multi-agent design: agents should remain independently controllable, permutation-symmetric, and support efficient inference while maintaining consistency across time and perspectives. In this paper, we present our generative multi-agent world model for interactive simulation. It introduces Simplex Rotary Agent Encoding, a parameter-free extension of 3D RoPE that represents agents as vertices of a regular simplex in rotary angle space. This gives each agent a distinct phase while making all agents permutation-equivalent, enabling scalable agent identity without learned per-slot identities or a fixed agent ordering. To avoid dense all-to-all attention across agents, we further propose Sparse Hub Attention, where learnable hub tokens mediate token interaction across agents, reducing cross-agent attention cost from quadratic to linear in the number of agents. For real-time rollout, we distill a full-context diffusion teacher into a causal student that generates temporal blocks sequentially with KV caching, enabling action-responsive generation at 24 FPS. Experiments in multiplayer virtual environments show that our model improves video fidelity, action controllability, and inter-agent consistency over slot-based and dense-attention baselines, while generalizing from two to four players without additional training.
Fangfu Liu, Kai He, Tianchang Shen +7
NVIDIA · Tsinghua University · University of Toronto +1
Video world models have made rapid progress in generating controllable visual experiences, but most of them still simulate the world from a single observer. Extending such models to multiple agents raises a central challenge: if each agent's future state is generated independently, overlapping views may instantiate different versions of the same scene, leading to inconsistent objects, layouts, and appearances across agents. Conventional camera conditioning controls individual trajectories, but it does not explicitly couple the generation of views that should agree under shared scene geometry. We introduce Prisma-World, a camera-controllable multi-agent world model that formulates multi-agent generation as a joint geometry-aware denoising process for cross-view consistency. Prisma-World processes all agent videos within one full-attention sequence, uses a multi-agent RoPE design to distinguish agent identities while preserving synchronized temporal coordinates, and injects relative camera geometry into attention to bias overlapping viewpoints toward shared scene evidence. To further strengthen multi-view consistency and enhance global spatial perception, we augment our framework with an overlap-decaying curriculum training paradigm alongside minimap-conditioned structural guidance. To facilitate the training and evaluation of multi-agent models, we introduce PrismaDataset, a large-scale UE5 dataset with panoramic acquisition across diverse scenes, composable multi-agent view groups with flexible agent counts and complex camera trajectories, and precise camera/action annotations for consistency training and evaluation. Experiments show that a single Prisma-World model can generate high-fidelity multi-agent videos with flexible agent numbers, camera controllability, improved cross-view consistency, and spatial grounding under minimap guidance.
Huiqiang Sun, Zhan Peng, Size Wu +9
School of AIA, HUST · S-Lab, NTU · SenseTime Research +4