Recent advances in video diffusion have enabled the development of "world models" capable of simulating interactive environments. However, these models are largely restricted to single-agent settings, failing to control multiple agents simultaneously in a scene. In this work, we tackle a fundamental issue of action binding in existing video diffusion models, which struggle to associate specific actions with their corresponding subjects. For this purpose, we propose ActionParty, an action controllable multi-subject world model for generative video games. It introduces subject state tokens, i.e. latent variables that persistently capture the state of each subject in the scene. By jointly modeling state tokens and video latents with a spatial biasing mechanism, we disentangle global video frame rendering from individual action-controlled subject updates. We evaluate ActionParty on the Melting Pot benchmark, demonstrating the first video world model capable of controlling up to seven players simultaneously across 46 diverse environments. Our results show significant improvements in action-following accuracy and identity consistency, while enabling robust autoregressive tracking of subjects through complex interactions.
Figures & tables
Figure 1: Left: Action binding failure case of text-to-video models with prompt: ‘The red triangle moves right and the blue square moves up. Then the red triangle moves down and the blue square moves left. Then the red triangle moves up and the blue square moves right. Then the red triangle moves left and the blue square moves down.’ Right: ActionParty enables action control of multiple subjects in a scene.
Figure 2: ActionParty pipeline. Given initial video frames x0:t and subject states z0:t as context, we aim to generate the next video frame xt+1 conditioned on action inputs a0:t and a text description of the game c . We concatenate the video and subject state tokens along the sequence dimension and feed them into a diffusion transformer (DiT) for joint denoising. Each DiT block first runs self-attention with an attention mask MSA and 3D RoPE biasing to render subject states to pixels in the video. It then runs cross-attention with another attention mask MCA with explicit subject-action binding to update subject states using action inputs.
Figure 3: Attention mechanisms in ActionParty DiT. (a) In self-attention, we use RoPE to link a subject in a video frame to its state token zit . We encode the state token with the subject’s coordinates in the previous timestep, biasing it to attend to video tokens close to the subject. (b) In cross-attention, subject i ’s state tokens zi is only allowed to attend to its own actions ai , ensuring correct subject-action binding. We also allow the text embedding of the environment description c to attend to video tokens x .
Method
Action Binding
Visual Quality
MA ↑
SP ↑
DR ↑
LPIPS ↓
PSNR ↑
FVD ↓
Pretrained AR
0.065
0.571
0.330
0.0494
26.98
59.30
Zero-shot I2V
0.027
0.422
0.440
0.0906
23.17
281.55
Text-Action
0.158
0.668
0.433
0.0353
29.14
56.74
Ours (ActionParty)
0.779
0.903
0.886
0.0102
36.35
17.16
Table 1: Baseline comparison on action binding metrics and visual fidelity. ActionParty is the only method able to associate actions to the correct characters (MA), preserve the subject appearance (SP), and execute rollouts with correct subject trajectories (DR). We also compute fidelity metrics, LPIPS, PSNR, and FVD, which show that videos generated by ActionParty are consistent with ground-truth.
Method
Effect Accuracy ↑
Idle
Move
Turn
Interact
Overall
Pretrained AR
0.399
0.414
0.462
0.346
0.412
Zero-shot I2V
0.000
0.101
0.150
0.000
0.126
Text-Action
0.357
0.420
0.553
0.326
0.433
Ours (ActionParty)
0.899
0.867
0.914
0.774
0.861
Table 2: We evaluate the Effect Accuracy of our generated videos across different action types. ActionParty achieves the highest accuracy on “Interact” actions compared to other methods, indicating that it most often creates the desired visual effect on neighboring tiles across different games.
Figure 4: Qualitative comparison with baselines. We display ground truth subject positions and orientations at each step with an arrow for each subject, which reflects the ground truth subjects (notice that the arrow is consistent in all methods). Our method is the only one able to follow the ground truth actions with appropriate action binding.
Figure 5: Movement accuracy (MA) over autoregressive steps. ActionParty maintains stable action binding across multiple rollout steps, whereas baselines degrade over time and get close to 0.
Figure 6: Results of ActionParty with xt and zt (circle) predictions for 2-, 4-, and 7-player games with a diversity of subjects. The full frames are generated and we show a centred crop around 2 subjects for visualization purposes with action annotations on the resulting frame. Note that actions are relative to orientation , so the subject moving forward is in the direction of where it is facing.
Method
MA ↑
zt Error ↓
SP ↑
DR ↑
w/o MSA
0.580
0.108
1.00
0.716
w/o MCA
0.052
0.215
0.88
0.307
Frame-wise MCA
0.052
0.240
0.80
0.293
No RoPE in SA
0.032
0.291
1.00
0.278
Ours (ActionParty)
0.872
0.072
1.00
0.913
Table 3: Ablations on the 4-action Coins game. The full ActionParty model achieves the most accurate localization of players indicated by the highest movement accuracy (MA) and the lowest zt error. Our MCA is important for consistent subject preservation according to the subject appearance (SP) and detection rate (DR).
Figure 7: Qualitative comparison of ablations run on the same initial frame and action trajectory. The blue subject is still, while the purple one moves and returns to its previous position. The zt trajectory is plotted for each subject. If we remove our introduced components, we lose the possibility to control characters.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Method
# Players
Action Binding
Visual Quality
MA ↑
SP ↑
DR ↑
LPIPS ↓
PSNR ↑
FVD ↓
Pretrained AR
Sparse
0.079
0.649
0.366
0.0281
29.27
53.31
Dense
0.064
0.534
0.317
0.0597
25.71
84.53
Crowded
0.051
0.580
0.331
0.0673
25.33
293.25
Zero-shot I2V
Sparse
0.031
0.485
0.435
0.0605
24.57
229.20
Dense
0.018
0.385
0.418
0.1055
22.32
427.58
Appendix
Table 5: Metrics split by number of subjects per game. All methods degrade with more subjects, but ActionParty remains the most robust across all metrics.
Method
# Players
Effect Accuracy ↑
Idle
Move
Turn
Interact
Overall
Pretrained AR
Sparse
0.512
0.438
0.488
0.411
0.451
Dense
0.379
0.461
0.481
0.368
0.443
Crowded
0.300
0.243
0.374
0.221
0.276
Zero-shot I2V
Sparse
0.000
0.122
0.211
0.000
0.167
Dense
0.000
0.119
0.155
0.000
0.137
Appendix
Table 6: Effect Accuracy split by scene density. ActionParty maintains high accuracy across all action types even in Crowded scenes, while baselines degrade significantly. Zero-shot I2V produces only a single next-frame prediction, so Idle and Interact actions (which occur at later steps in validation set) are not evaluated and reported as 0.
Game
Prompt
Coins
Two player sprites move around a black-floored room surrounded by gray brick walls. Small pink and yellow coin sprites periodically appear at random positions on the floor. Players walk over coins to pick them up and coins disappear on contact.
Clean up
Player sprites move across a map split into a sandy upper area and a green lower area with a blue-green polluted river section between them. Gray brick walls surround the map. Pink apple sprites appear on the green grass. Players fire cleaning beams at the river pollution.
Chemistry
Player sprites move across a white open field scattered with colorful arrow-shaped molecule sprites in blue, green, red, and cyan. Players walk into molecules to pick them up and carry them. Molecules near each other react and transform into different colored molecules.
Collaborative cooking
Two player sprites move in a very small kitchen made of brown counter tiles with minimal dark floor space. Red tomato sprites, gray cooking pots, and white plate sprites sit on counters. Players squeeze past each other carrying items between stations.
Paintball
Two teams of player sprites move through a gray-walled arena bordered by purple indicator tiles. Players fire beams that paint the floor red or blue in large patches. Small flag sprites sit at each team’s base. Players pick up the opposing flag and carry it back.
Appendix
Table 7: Example game description prompts used for text conditioning. Each prompt describes the visual content of the environment. Full prompts for all 46 games are provided in the supplementary materials.
Video world models have achieved remarkable success in simulating environmental dynamics in response to actions by users or agents. They are modeled as action-conditioned video generation models that take historical frames and current actions as input to predict future frames. Yet, most existing approaches are limited to single-agent scenarios and fail to capture the complex interactions inherent in real-world multi-agent systems. We present MultiWorld, a unified framework for multi-agent multi-view world modeling that enables accurate control of multiple agents while maintaining multi-view consistency. We introduce the Multi-Agent Condition Module to achieve precise multi-agent controllability, and the Global State Encoder to ensure coherent observations across different views. MultiWorld supports flexible scaling of agent and view counts, and synthesizes different views in parallel for high efficiency. Experiments on multi-player game environments and multi-robot manipulation tasks demonstrate that MultiWorld outperforms baselines in video fidelity, action-following ability, and multi-view consistency. Project page: https://multi-world.github.io/
Video world models are a foundational generative technology for embodied AI and the Metaverse, yet existing approaches are inherently limited to a single agent observing from a single perspective. Extending these models to multi-agent settings introduces two critical challenges: data scarcity (coordinated multi-view recordings are prohibitively expensive to collect for general open-domain scenarios) and world state alignment (independently generated video streams cannot ensure that shared physical environments and events evolve consistently across views). To address these challenges, we propose MetaWorld, a novel framework that scales multi-agent video world models to open-domain environments directly from single-view videos. First, we introduce Monocular World-State Unrolling (MWSU) to explicitly decompose monocular footage into the camera operator's ego-motion and the visible subject's spatial trajectory. This camera-trajectory decomposition naturally extracts synchronized multi-agent motion data within a shared 3D space, completely bypassing the need for multi-camera setups. Second, for precise visual control, we develop the Subject-Aware World Generator to enable appearance-driven simulation conditioned on per-agent identity images. Finally, to ensure both views are grounded in the identical physical reality, we propose World-State Alignment, a per-frame inter-branch cross-attention mechanism inserted at every transformer layer of the video DiT. By jointly synchronizing the denoising process, WSA enforces both static geometric consistency and dynamic motion consistency, encouraging that the shared 3D environment and physical events remain well-aligned across both egocentric views. Extensive experiments demonstrate that MetaWorld achieves superior cross-view consistency and identity fidelity, establishing a highly scalable, physics-driven paradigm for multi-agent video world modeling.
Teng Hu, Mingchun Lu, Yating Wang +6
Shanghai Jiao Tong University · Zhejiang University · Nanyang Technological University
This paper presents ShareVerse, a video generation framework enabling multi-agent shared world modeling, addressing the gap in existing works that lack support for unified shared world construction with multi-agent interaction. ShareVerse leverages the generation capability of large video models and integrates three key innovations: 1) A dataset for large-scale multi-agent interactive world modeling is built on the CARLA simulation platform, featuring diverse scenes, weather conditions, and interactive trajectories with paired multi-view videos (front/ rear/ left/ right views per agent) and camera data. 2) We propose a spatial concatenation strategy for four-view videos of independent agents to model a broader environment and to ensure internal multi-view geometric consistency. 3) We integrate cross-agent attention blocks into the pretrained video model, which enable interactive transmission of spatial-temporal information across agents, guaranteeing shared world consistency in overlapping regions and reasonable generation in non-overlapping regions. ShareVerse, which supports 49-frame large-scale video generation, accurately perceives the position of dynamic agents and achieves consistent shared world modeling.
Jiayi Zhu, Jianing Zhang, Yiying Yang +2
Shanghai Jiao Tong University, China · Fudan University, China · StepFun, China