MultiGen: Level-Design for Editable Multiplayer Worlds in Diffusion Game Engines

Ryan Po; David Junhao Zhang; Amir Hertz; Gordon Wetzstein; Neal Wadhwa; Nataniel Ruiz

cs.AIMar 3, 2026

MultiGen: Level-Design for Editable Multiplayer Worlds in Diffusion Game Engines

Authors: Ryan Po, David Junhao Zhang, Amir Hertz, Gordon Wetzstein, Neal Wadhwa, Nataniel Ruiz

Organizations: Stanford University · Google

Abstract

Video world models have shown immense promise for interactive simulation and entertainment, but current systems still struggle with two important aspects of interactivity: user control over the environment for reproducible, editable experiences, and shared inference where players hold influence over a common world. To address these limitations, we introduce an explicit external memory into the system, a persistent state operating independent of the model's context window, that is continually updated by user actions and queried throughout the generation roll-out. Unlike conventional diffusion game engines that operate as next-frame predictors, our approach decomposes generation into Memory, Observation, and Dynamics modules. This design gives users direct, editable control over environment structure via an editable memory representation, and it naturally extends to real-time multiplayer rollouts with coherent viewpoints and consistent cross-player interactions.

Figures & tables

Figure 1: Level Design via Editable Memory. Users define a level through coarse 2D geometry (left). During inference, the diffusion model generates first-person observations consistent with the top-down level layout (right).

Figure 2: Method overview. We introduce an explicit external memory and factor the diffusion game engine into three modules: Memory (map geometry and pose; Sec. 3.1 ), Observation (next-frame generation conditioned on history and memory readouts; Sec. 3.2 ), and Dynamics (pose update for state progression; Sec. 3.3 ).

Figure 3: Example rollouts under an authored map and action sequence. Top: minimap $M$ with pose $p_{t}$ (red arrow). Middle: generated first-person observations $\hat{o}_{t}$ . Bottom: actions $a_{t}$ . The viewpoint evolves coherently with the action inputs while adhering with the designed layout.

	SSIM $\uparrow$			PSNR $\uparrow$			LPIPS $\downarrow$
Method	All	1–128	128–256	All	1–128	128–256	All	1–128	128–256
IP-Adapter [ 81 ]	0.415	0.433	0.396	18.74	20.30	17.19	0.488	0.397	0.578
ControlNet [ 87 ]	0.411	0.422	0.401	18.51	19.58	17.45	0.524	0.453	0.596
GameNGen [ 68 ]	0.405	0.427	0.384	18.77	20.23	17.33	0.471	0.379	0.562
Ours (MultiGen)	0.418	0.429	0.406	19.32	20.06	18.59	0.453	0.400	0.505

Table 1: SSIM/PSNR/LPIPS between generated frames and ground truth. Our method consistently outperforms baselines, with the largest gains appearing in later rollout stages where consistent memory is the most important.

Figure 4: Example Two-Player Gameplay Roll-out. Our method generates consistent first-person views for both players by maintaining a shared world memory. The roll-out shows a short two-player interaction: the players meet, and Player 1 kills Player 2, after which Player 2 is removed from the shared state. Player 1 then explores the map while Player 2 respawns and is re-added to the shared state. The players meet again, and Player 1 kills Player 2 once more. Note that both views are are consistent with each other, as actions from one player directly effects the next-frame observation generated by the other model. All game play frames are generated using the observation module. Frames shown during player death are not part of the model output, and are only shown for illustrative purposes.

Figure 5: Real-Time Interactive Multiplayer Generative Experiences. A shared consistent world state (left) enables consistent multiplayer generative experiences (right). Our method leverages a diffusion model conditioned on past frame observations, the next player action, and the external world state to generate gameplay roll-outs in real-time. The shared world state enables meaningful interactions between players, such as one player killing another (right).

Figure 6: Consistent Views in Three-Player. Our distributed model system supports an arbitrary number of players in the shared state despite only being trained on data from a single viewpoint. Player positions are accurately represented in all three viewpoints in the example above.

Method	Accuracy $\uparrow$	Precision $\uparrow$	Recall $\uparrow$
IP-Adapter [ 81 ]	62.12%	84.62%	40.73%
ControlNet [ 88 ]	60.71%	82.16%	39.20%
Split-screen [ 65 ]	65.31%	86.86%	44.59%
Ours (MultiGen)	75.38%	88.12%	65.07%

Table 2: Multiplayer consistency evaluation using opponent-presence detection. We report accuracy, precision, and recall of a VLM-based judge on generated frames against ground-truth visibility labels. MultiGen consistently outperforms all baselines.

$L$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$
2	0.709	27.6	0.121
4	0.775	29.5	0.097
8	0.783	29.8	0.094
16	0.782	29.8	0.093
32	0.789	30.0	0.089

Table 3: Context frame ablation.

Interactive World Models

Interactive world models predict and generate how environments change in response to actions, supporting simulation, planning, and control.

See all 117 papers

Code Plans, Diffusion Renders: Open-Ended Generative World Modeling
Sep 22, 2026Zixun Fang, Yawen Shao, Kai Zhu +7World Models Code World Models
USTC · TongYi Lab
Matrix-game 2.0: An open-source, real-time, and streaming interactive world model
Aug 18, 2025Xianglong He, Chunli Peng, Zexiang Liu +16Action-Conditioned Video Generation Real-Time Interactive Video Generation
Skywork AI
Precise Editing and Flexible Referencing for Interactable Worlds
Sep 28, 2026Xinyao Liao, Xianfang Zeng, Zhu Liang +5Instruction-Based Video Editing Autoregressive Video Generation
Nanyang Technological University · StepFun

Sep 22, 2026J/K move · Enter open · S save

Code Plans, Diffusion Renders: Open-Ended Generative World Modeling

Zixun Fang, Yawen Shao, Kai Zhu, Jie Xiao+6

USTC · TongYi Lab

World Models Code World Models Interactive World Models Video World Models

Open paper AskPDF arXiv

We introduce CoDeR, a new paradigm for world modeling. Unlike existing video world models that implicitly represent world dynamics through visual observations, our system explicitly constructs an executable world with code and employs video generation models for visual realization. Specifically, we coordinate five complementary roles to translate high-level concepts into structured world rules, executable dynamics, and perceptual observations. This design enables long-term memory, open-ended interactions, autonomous world evolution, and multi-agent scenarios, where multiple entities can act, interact, and evolve persistently beyond the current observation. Extensive experiments demonstrate that our framework substantially extends the capabilities of existing world models, enabling long-term memory, open-ended interactions, autonomous evolution, and persistent multi-agent dynamics, while achieving state-of-the-art performance across multiple evaluation settings. Code and model weights will be made publicly available. Project Page: CoDeR.

Abstract

Interactive World Models

Explore similar work

Code Plans, Diffusion Renders: Open-Ended Generative World Modeling

Matrix-game 2.0: An open-source, real-time, and streaming interactive world model

Precise Editing and Flexible Referencing for Interactable Worlds