LiveAct: Towards Real-Time Multi-Agent World Models with Heterogeneous Action Interfaces

Xinyu Wang; Huafeng Shi; Yue Ma; Zijie Meng; Teng Xi; Yuzhuo Chen; Yucheng Wang; Ruicheng Zhang

cs.CVOct 21, 2026

LiveAct: Towards Real-Time Multi-Agent World Models with Heterogeneous Action Interfaces

Authors: Xinyu Wang, Huafeng Shi, Yue Ma, Zijie Meng, Teng Xi, Yuzhuo Chen, Yucheng Wang, Ruicheng Zhang,

Organizations: Shenzhen International Graduate School, Tsinghua University · Kling Team · The Hong Kong University of Science and Technology · Peking University ·

Abstract

Interactive video world models predict how an environment evolves under user actions, but most existing systems assume one controlled agent and one task-specific action interface. We study multi-agent video world modeling, where one model generates a joint visual future for multiple controlled agents while interpreting heterogeneous controls ranging from binary game buttons to continuous robot commands. This task presents two complementary challenges. First, every agent-indexed action stream must control its intended visual entity independently. Second, the model must adapt the meaning of each control variable across different tasks, where the same tensor dimension may denote an attack, steering command, or robot command. We introduce LiveAct, centered on a Heterogeneous Action Interface (HAI) whose two components mirror this decomposition. To support independent control, Agent-Aware Spatial Modulation uses language-driven spatial gates and agent-specific attention to bind each action stream to its intended visual agent. To accommodate heterogeneous controls, Schema-Guided Action Adaptation describes each control variable in language and modulates its semantic token with the synchronized numerical value. We further transfer the complete interface to a blockwise causal generator through teacher forcing and rollout-aware distribution matching, enabling efficient streaming generation while retaining agent-specific control and action semantics. We instantiate the formulation on six multi-agent sources spanning games and multi-robot manipulation, and evaluate visual quality, action following, agent attribution, and latency. Across games and robot manipulation, LiveAct reaches 0.898 and 0.942 agent-attribution accuracy without agent locations or identity crops, and the causal generator attains 17.70 FPS at $512\times 448$ .

Figures & tables

Figure 2: Two challenges for multi-agent action interfaces. (a) Streams that differ only by input order entangle fighters under index-only binding; LiveAct’s agent gate keeps each on its fighter. (b) Dimension $j{=}7$ changes meaning across tasks. A raw vector keeps the value but not the meaning, free text the reverse; LiveAct modulates each schema token by its value.

Method	Domain	Multi-Agent	Real-Time	Semantic	Numerical
LingBot-World 2.0 [ 6 ]	General visual worlds	$\times$	$\checkmark$	$\checkmark$	$\checkmark$
Solaris [ 36 ]	Single ( Minecraft )	$\checkmark$	$\times$	$\times$	$\checkmark$
MultiWorld [ 50 ]	Single ( It Takes Two )	$\checkmark$	$\times$	$\times$	$\checkmark$
ActionParty [ 30 ]	Single ( Melting Pot )	$\checkmark$	$\times$	$\times$	$\checkmark$
Incantation [ 63 ]	Single ( Elden Ring )	$\checkmark$	$\checkmark$	$\checkmark$	$\times$
WanToFight [ 13 ]	Single (KOF ’97)	$\checkmark$	$\checkmark$	$\times$	$\checkmark$

Table 1: Native capability comparison.

\checkmark

: supported;

\times

: unsupported.

Figure 3: Overview of LiveAct. (a) Foundational Ability Acquisition trains a bidirectional teacher with a Heterogeneous Action Interface, which integrates Agent-Aware Spatial Modulation and Schema-Guided Action Adaptation. (b) Teacher Forcing for AR Tuning transfers the same HAI to causal attention. (c) DMD for Streaming Video distills few-step streaming rollouts.

Figure 4: Data curation pipeline. Game and robotic episodes are curated into aligned samples containing a scene prompt, per-agent action streams, agent descriptions, and task-specific schemas.

Figure 5: Qualitative comparison on shared initial frames and action streams. Columns show ground truth, the five baselines, and LiveAct; Rows are frames sampled at different durations.

	Reconstruction				Control		Diagnostics
Method	FVD $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	Act-F1 $\uparrow$ / NRMSE $\downarrow$	Attr-Acc $\uparrow$	MME $\downarrow$	Aesthetic $\uparrow$
Multi-Agent Video Games
LingBot-World 2.0 [ 6 ]	1256.6	0.322	13.25	0.490	0.146	0.553	0.57	5.093
MultiWorld (1-view) a [ 50 ]	2661.4	0.719	10.40	0.125	0.163	0.548	2.67	4.059
Incantation [ 63 ]	1989.5	0.593	10.45	0.226	0.162	0.534	8.77	4.539
ActionParty [ 30 ]	1672.5	0.272	14.56	0.467	0.086	0.547	0.47	4.974

Table 2: Main comparison across multi-agent video games and multi-robot manipulation. Best results are bolded. a Native MultiWorld uses synchronized multi-view observations; we have set it to a fixed same perspective. b The code is not released; this is our reproduction.

Variant	Game Act-F1 $\uparrow$	Game Attr-Acc $\uparrow$	Robot NRMSE $\downarrow$	Robot Attr-Acc $\uparrow$
Full LiveAct	0.594	0.898	0.558	0.942
w/o AASM	0.172	0.592	0.931	0.810
w/o SGAA	0.225	0.603	1.120	0.484

Table 3: Quantitative ablation study.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1 : Showcase of our dataset. Frames from the six sources, covering versus fighting, shared-map competition, shared-track racing, and multi-arm manipulation.

	Street Fighter II	Fatal Fury 2	Mega Bomberman	Micro Machines	RoboTwin	RoboFactory	Total
Trajectories	5460	3200	2050	2150	2532	2559	17951
Agents	2	2	2–4	2–4	2	2–3	2–4
Action type	binary	binary	binary	binary	continuous	continuous	mixed
Frame rate (fps)	60	60	60	60	30 / 50	30	–
Raw frames (M)	9.83	5.76	3.69	3.87	0.65	0.91	24.71
Raw duration (h)	45.50	26.67	17.08	17.92	3.93	8.44	119.54

Appendix

Table S1 : Dataset statistics for four multi-agent games and two multi-robot manipulation sources. Frame counts and durations refer to the raw recordings at their native frame rates (RoboTwin: 30 fps for self-collected and 50 fps for augmented episodes). Training and evaluation use 81-frame clips drawn from these trajectories.

Model and data
Backbone	Wan2.2-TI2V-5B; 30 blocks, width 3072, 24 heads; VAE $4\times 16\times 16$
Trainable	Full DiT and HAI; umT5 encoder and VAE frozen
Clip	81 frames at $512\times 448$ , 16 fps; 1 conditioning and 20 future latents
Frame sampling	Games: consecutive native frames; robots: uniform over the episode
Actions	12 description-modulated controls per agent; one-latent window
Conditioner	Text projections $4096\rightarrow 1024\rightarrow 3072$ ; value/gate/delta bottleneck 768

Appendix

Table S2 : Training and inference configuration of LiveAct. Stages 2 and 3 are trained with FSDP, and Stage 3 samples at timesteps

[1000,750,500,250]

with the EMA weights; “–” marks settings that do not apply.

	Stage 1 Foundational Ability Acquisition	Stage 2 Teacher Forcing for AR Tuning	Stage 3 DMD for Streaming Video
Role	Full-context teacher	Causal initialization	Rollout distillation
Attention	Bidirectional	Blockwise causal	Blockwise causal
Training history	Full video	Clean chunks	Model rollouts
Streaming	$\times$	$\checkmark$	$\checkmark$
Chunk size	Full sequence	4 latent frames	4 latent frames
Denoising steps	50	50	4

Appendix

Table S3 : Three-stage causal transfer. Generation settings and results of each stage; throughput is measured at

512\times 448

on a single GPU, and Stage 3 samples without classifier-free guidance (CFG). Best results are bolded.

Variant	Control identity	Tokens per frame	Value fusion	Training	Control token
Full LiveAct	schema $u_{j}$	$C$	affine modulation	trained	$h_{\tau,j}=u_{j}\odot(1+\gamma_{\tau,j})+\beta_{\tau,j}$
w/o SGAA	–	$1$	flat MLP	trained	$h_{\tau}=\operatorname{MLP}_{\mathrm{flat}}(A_{\tau})$
w/o value modulation	schema $u_{j}$	$C$	addition	trained	$h_{\tau,j}=u_{j}+\operatorname{MLP}_{\mathrm{val}}(a_{\tau,j})$
w/o schema semantics	learned ID $e_{j}^{\mathrm{ID}}$	$C$	affine modulation	trained	$h_{\tau,j}=e_{j}^{\mathrm{ID}}\odot(1+\gamma_{\tau,j})+\beta_{\tau,j}$
shuffled schema	schema $u_{\pi(j)}$	$C$	affine modulation	inference only	$h_{\tau,j}=u_{\pi(j)}\odot(1+\gamma_{\tau,j})+\beta_{\tau,j}$

Appendix

Table S4 : Design of the Schema-Guided Action Adaptation variants. Each variant changes only how the control tokens of agent

i

are constructed; the superscript

(i)

is omitted. All variants keep Agent-Aware Spatial Modulation, per-agent cross-attention, the one-latent action window, and the time and agent embeddings of Eq. 5 .

u_{j}

is the projected schema embedding of Eq. 3 ,

(\gamma_{\tau,j},\beta_{\tau,j})=\operatorname{MLP}_{\mathrm{mod}}(a_{\tau,j})

as in Eq. 4 ,

e_{j}^{\mathrm{ID}}

is a learned per-dimension embedding, and

\pi

is a fixed permutation of the control dimensions.

Variant	Game Act-F1 $\uparrow$	Game Attr-Acc $\uparrow$	Robot NRMSE $\downarrow$	Robot Attr-Acc $\uparrow$
Full LiveAct ( $L{=}1$ )	0.594	0.898	0.558	0.942
Schema-Guided Action Adaptation
w/o SGAA (flat MLP)	0.225	0.603	1.120	0.484
w/o value modulation (additive value)	0.209	0.513	1.071	0.480
w/o schema semantics (ID tokens)	0.230	0.588	0.799	0.778
shuffled schema (inference only)	0.207	0.504	1.063	0.496

Appendix

Table S5 : Detailed ablations of Schema-Guided Action Adaptation and temporal alignment. The SGAA variants are defined in Tab. S4 . All trained variants use the same data and optimization budget as the full model.

L

is the action window length in latent frames (App. B.2 ). Best results are bolded.

Figure S2 : Spatial localization of agent-specific control residuals. Panels (a)–(d) show representative spatial residual-magnitude heatmaps from the agent-specific branches. Red and blue denote agents 1 and 2, respectively, and higher intensity indicates a larger response. Across the examples, the high-response regions concentrate around the corresponding visual agents.

Figure S3 : Additional qualitative comparison (set 1). Same layout and methods as Fig. 5 .

Figure S4 : Additional qualitative comparison (set 2). Same layout and methods as Fig. 5 .

Figure S5 : Additional LiveAct showcase.

Figure S6 : Showcase beyond two agents. Scenes with more than two concurrently controlled agents. For each scene, the top row shows the ground truth and the bottom row the LiveAct generation from the same initial frame and per-agent action streams.

Figure S7 : Single-case long-horizon rollout as a dense frame matrix. We show 24 generated frames from the same 241-frame rollout, sampled every ten frames and ordered from left to right and top to bottom.

Video World Models

Video world models generate and predict action-conditioned visual futures to simulate changing scenes for embodied agents.

See all 257 papers

MultiWorld: Scalable Multi-Agent Multi-View Video World Models
Apr 20, 2026Haoyu Wu, Jiwen Yu, Yingtian Zou +1Multi-View Consistency Action-Conditioned World Models
The University of Hong Kong · Sreal AI
Gamma-World: Generative Multi-Agent World Modeling Beyond Two Players
May 27, 2026Fangfu Liu, Kai He, Tianchang Shen +7Interactive World Models Video World Models
NVIDIA · Tsinghua University · University of Toronto +1
Prisma-World: Camera-Controllable Multi-Agent Video World Model
Jun 8, 2026Huiqiang Sun, Zhan Peng, Size Wu +9Multi-View Consistency Camera-Controlled Video Generation
School of AIA, HUST · S-Lab, NTU · SenseTime Research +4

Apr 20, 2026J/K move · Enter open · S save

MultiWorld: Scalable Multi-Agent Multi-View Video World Models

Haoyu Wu, Jiwen Yu, Yingtian Zou, Xihui Liu

The University of Hong Kong · Sreal AI

Multi-View Consistency Action-Conditioned World Models Multi-Agent Control Controllable Video Generation Video World Models

Open paper AskPDF arXiv

Video world models have achieved remarkable success in simulating environmental dynamics in response to actions by users or agents. They are modeled as action-conditioned video generation models that take historical frames and current actions as input to predict future frames. Yet, most existing approaches are limited to single-agent scenarios and fail to capture the complex interactions inherent in real-world multi-agent systems. We present MultiWorld, a unified framework for multi-agent multi-view world modeling that enables accurate control of multiple agents while maintaining multi-view consistency. We introduce the Multi-Agent Condition Module to achieve precise multi-agent controllability, and the Global State Encoder to ensure coherent observations across different views. MultiWorld supports flexible scaling of agent and view counts, and synthesizes different views in parallel for high efficiency. Experiments on multi-player game environments and multi-robot manipulation tasks demonstrate that MultiWorld outperforms baselines in video fidelity, action-following ability, and multi-view consistency. Project page: https://multi-world.github.io/

Abstract

Video World Models

Explore similar work

MultiWorld: Scalable Multi-Agent Multi-View Video World Models

Gamma-World: Generative Multi-Agent World Modeling Beyond Two Players

Prisma-World: Camera-Controllable Multi-Agent Video World Model