ActionParty: Multi-Subject Action Binding in Generative Video Games

Alexander Pondaven; Ziyi Wu; Igor Gilitschenski; Philip Torr; Sergey Tulyakov; Fabio Pizzati; Aliaksandr Siarohin

cs.CVApr 2, 2026

ActionParty: Multi-Subject Action Binding in Generative Video Games

Authors: Alexander Pondaven, Ziyi Wu, Igor Gilitschenski, Philip Torr, Sergey Tulyakov, Fabio Pizzati, Aliaksandr Siarohin

Organizations: Snap Research · University of Oxford · University of Toronto · MBZUAI

Abstract

Recent advances in video diffusion have enabled the development of "world models" capable of simulating interactive environments. However, these models are largely restricted to single-agent settings, failing to control multiple agents simultaneously in a scene. In this work, we tackle a fundamental issue of action binding in existing video diffusion models, which struggle to associate specific actions with their corresponding subjects. For this purpose, we propose ActionParty, an action controllable multi-subject world model for generative video games. It introduces subject state tokens, i.e. latent variables that persistently capture the state of each subject in the scene. By jointly modeling state tokens and video latents with a spatial biasing mechanism, we disentangle global video frame rendering from individual action-controlled subject updates. We evaluate ActionParty on the Melting Pot benchmark, demonstrating the first video world model capable of controlling up to seven players simultaneously across 46 diverse environments. Our results show significant improvements in action-following accuracy and identity consistency, while enabling robust autoregressive tracking of subjects through complex interactions.

Figures & tables

Figure 1: Left: Action binding failure case of text-to-video models with prompt: ‘The red triangle moves right and the blue square moves up. Then the red triangle moves down and the blue square moves left. Then the red triangle moves up and the blue square moves right. Then the red triangle moves left and the blue square moves down.’ Right: ActionParty enables action control of multiple subjects in a scene.

Figure 2: ActionParty pipeline. Given initial video frames $x_{0:t}$ and subject states $z_{0:t}$ as context, we aim to generate the next video frame $x_{t+1}$ conditioned on action inputs $a_{0:t}$ and a text description of the game $c$ . We concatenate the video and subject state tokens along the sequence dimension and feed them into a diffusion transformer (DiT) for joint denoising. Each DiT block first runs self-attention with an attention mask $\mathcal{M}_{SA}$ and 3D RoPE biasing to render subject states to pixels in the video. It then runs cross-attention with another attention mask $\mathcal{M}_{CA}$ with explicit subject-action binding to update subject states using action inputs.

Panel of Figure 3 — Figure 3: Attention mechanisms in ActionParty DiT. (a) In self-attention, we use RoPE to link a subject in a video frame to its state token $z_{i}^{t}$ . We encode the state token with the subject’s coordinates in the previous timestep, biasing it to attend to video tokens close to the subject. (b) In cross-attention, subject $i$ ’s state tokens $z^{i}$ is only allowed to attend to its own actions $a^{i}$ , ensuring correct subject-action binding. We also allow the text embedding of the environment description $c$ to attend to video tokens $x$ .

Method	Action Binding			Visual Quality
Method	MA $\uparrow$	SP $\uparrow$	DR $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	FVD $\downarrow$
Pretrained AR	0.065	0.571	0.330	0.0494	26.98	59.30
Zero-shot I2V	0.027	0.422	0.440	0.0906	23.17	281.55
Text-Action	0.158	0.668	0.433	0.0353	29.14	56.74
Ours (ActionParty)	0.779	0.903	0.886	0.0102	36.35	17.16

Table 1: Baseline comparison on action binding metrics and visual fidelity. ActionParty is the only method able to associate actions to the correct characters (MA), preserve the subject appearance (SP), and execute rollouts with correct subject trajectories (DR). We also compute fidelity metrics, LPIPS, PSNR, and FVD, which show that videos generated by ActionParty are consistent with ground-truth.

Method	Effect Accuracy $\uparrow$
Method	Idle	Move	Turn	Interact	Overall
Pretrained AR	0.399	0.414	0.462	0.346	0.412
Zero-shot I2V	0.000	0.101	0.150	0.000	0.126
Text-Action	0.357	0.420	0.553	0.326	0.433
Ours (ActionParty)	0.899	0.867	0.914	0.774	0.861

Table 2: We evaluate the Effect Accuracy of our generated videos across different action types. ActionParty achieves the highest accuracy on “Interact” actions compared to other methods, indicating that it most often creates the desired visual effect on neighboring tiles across different games.

Figure 4: Qualitative comparison with baselines. We display ground truth subject positions and orientations at each step with an arrow for each subject, which reflects the ground truth subjects (notice that the arrow is consistent in all methods). Our method is the only one able to follow the ground truth actions with appropriate action binding.

Figure 5: Movement accuracy (MA) over autoregressive steps. ActionParty maintains stable action binding across multiple rollout steps, whereas baselines degrade over time and get close to 0.

Figure 6: Results of ActionParty with $x_{t}$ and $z_{t}$ (circle) predictions for 2-, 4-, and 7-player games with a diversity of subjects. The full frames are generated and we show a centred crop around 2 subjects for visualization purposes with action annotations on the resulting frame. Note that actions are relative to orientation , so the subject moving forward is in the direction of where it is facing.

Method	MA $\uparrow$	$z_{t}$ Error $\downarrow$	SP $\uparrow$	DR $\uparrow$
w/o $\mathcal{M}_{SA}$	0.580	0.108	1.00	0.716
w/o $\mathcal{M}_{CA}$	0.052	0.215	0.88	0.307
Frame-wise $\mathcal{M}_{CA}$	0.052	0.240	0.80	0.293
No RoPE in SA	0.032	0.291	1.00	0.278
Ours (ActionParty)	0.872	0.072	1.00	0.913

Table 3: Ablations on the 4-action Coins game. The full ActionParty model achieves the most accurate localization of players indicated by the highest movement accuracy (MA) and the lowest

z_{t}

error. Our

\mathcal{M}_{\text{CA}}

is important for consistent subject preservation according to the subject appearance (SP) and detection rate (DR).

Figure 7: Qualitative comparison of ablations run on the same initial frame and action trajectory. The blue subject is still, while the purple one moves and returns to its previous position. The $z_{t}$ trajectory is plotted for each subject. If we remove our introduced components, we lose the possibility to control characters.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Method	# Players	Action Binding			Visual Quality
Method	# Players	MA $\uparrow$	SP $\uparrow$	DR $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	FVD $\downarrow$
Pretrained AR	Sparse	0.079	0.649	0.366	0.0281	29.27	53.31
	Dense	0.064	0.534	0.317	0.0597	25.71	84.53
	Crowded	0.051	0.580	0.331	0.0673	25.33	293.25
Zero-shot I2V	Sparse	0.031	0.485	0.435	0.0605	24.57	229.20
	Dense	0.018	0.385	0.418	0.1055	22.32	427.58

Appendix

Table 5: Metrics split by number of subjects per game. All methods degrade with more subjects, but ActionParty remains the most robust across all metrics.

Method	# Players	Effect Accuracy $\uparrow$
Method	# Players	Idle	Move	Turn	Interact	Overall
Pretrained AR	Sparse	0.512	0.438	0.488	0.411	0.451
	Dense	0.379	0.461	0.481	0.368	0.443
	Crowded	0.300	0.243	0.374	0.221	0.276
Zero-shot I2V	Sparse	0.000	0.122	0.211	0.000	0.167
	Dense	0.000	0.119	0.155	0.000	0.137

Appendix

Table 6: Effect Accuracy split by scene density. ActionParty maintains high accuracy across all action types even in Crowded scenes, while baselines degrade significantly. Zero-shot I2V produces only a single next-frame prediction, so Idle and Interact actions (which occur at later steps in validation set) are not evaluated and reported as 0.

Game	Prompt
Coins	Two player sprites move around a black-floored room surrounded by gray brick walls. Small pink and yellow coin sprites periodically appear at random positions on the floor. Players walk over coins to pick them up and coins disappear on contact.
Clean up	Player sprites move across a map split into a sandy upper area and a green lower area with a blue-green polluted river section between them. Gray brick walls surround the map. Pink apple sprites appear on the green grass. Players fire cleaning beams at the river pollution.
Chemistry	Player sprites move across a white open field scattered with colorful arrow-shaped molecule sprites in blue, green, red, and cyan. Players walk into molecules to pick them up and carry them. Molecules near each other react and transform into different colored molecules.
Collaborative cooking	Two player sprites move in a very small kitchen made of brown counter tiles with minimal dark floor space. Red tomato sprites, gray cooking pots, and white plate sprites sit on counters. Players squeeze past each other carrying items between stations.
Paintball	Two teams of player sprites move through a gray-walled arena bordered by purple indicator tiles. Players fire beams that paint the floor red or blue in large patches. Small flag sprites sit at each team’s base. Players pick up the opposing flag and carry it back.

Appendix

Table 7: Example game description prompts used for text conditioning. Each prompt describes the visual content of the environment. Full prompts for all 46 games are provided in the supplementary materials.

Explore similar work

MultiWorld: Scalable Multi-Agent Multi-View Video World Models
Apr 20, 2026Haoyu Wu, Jiwen Yu, Yingtian Zou +1Multi-View Consistency Action-Conditioned World Models
The University of Hong Kong · Sreal AI
MetaWorld: Scaling Multi-Agent Video World Model from Single-view Video Data
Jun 1, 2026Teng Hu, Mingchun Lu, Yating Wang +6Multi-View Consistency Video Diffusion Models
Shanghai Jiao Tong University · Zhejiang University · Nanyang Technological University
ShareVerse: Multi-Agent Consistent Video Generation for Shared World Modeling
Mar 3, 2026Jiayi Zhu, Jianing Zhang, Yiying Yang +2Video World Models Multi-View Video Generation
Shanghai Jiao Tong University, China · Fudan University, China · StepFun, China

Apr 20, 2026J/K move · Enter open · S save

MultiWorld: Scalable Multi-Agent Multi-View Video World Models

Haoyu Wu, Jiwen Yu, Yingtian Zou, Xihui Liu

The University of Hong Kong · Sreal AI

Multi-View Consistency Action-Conditioned World Models Multi-Agent Control Controllable Video Generation Video World Models

Open paper AskPDF arXiv

Video world models have achieved remarkable success in simulating environmental dynamics in response to actions by users or agents. They are modeled as action-conditioned video generation models that take historical frames and current actions as input to predict future frames. Yet, most existing approaches are limited to single-agent scenarios and fail to capture the complex interactions inherent in real-world multi-agent systems. We present MultiWorld, a unified framework for multi-agent multi-view world modeling that enables accurate control of multiple agents while maintaining multi-view consistency. We introduce the Multi-Agent Condition Module to achieve precise multi-agent controllability, and the Global State Encoder to ensure coherent observations across different views. MultiWorld supports flexible scaling of agent and view counts, and synthesizes different views in parallel for high efficiency. Experiments on multi-player game environments and multi-robot manipulation tasks demonstrate that MultiWorld outperforms baselines in video fidelity, action-following ability, and multi-view consistency. Project page: https://multi-world.github.io/

Abstract

Explore similar work

MultiWorld: Scalable Multi-Agent Multi-View Video World Models

MetaWorld: Scaling Multi-Agent Video World Model from Single-view Video Data

ShareVerse: Multi-Agent Consistent Video Generation for Shared World Modeling