PlayGen2: Learning Real-Time Playable Worlds with Explicit 3D State

Jihai Zhang; Yuhan Zhou; Junyou Li; Deheng Ye; Qi Dou; Yu Cheng

cs.CVNov 5, 2026

PlayGen2: Learning Real-Time Playable Worlds with Explicit 3D State

Authors: Jihai Zhang, Yuhan Zhou, Junyou Li, Deheng Ye, Qi Dou, Yu Cheng

Organizations: The Chinese University of Hong Kong · Wellington College · Tencent · Nanyang Technological University

Abstract

Building a playable world requires more than generating plausible video: it demands responsive control, persistent spatial structure, and coherent interactions with dynamic entities. Using generated frames as the sole recurrent world representation ties these requirements to view-dependent appearance synthesis, making it difficult to maintain long-term spatial and temporal consistency in generation and interaction. We present PlayGen2, a real-time interactive world model that separates learned 3D state evolution from neural rendering. PlayGen2 maintains a persistent world memory comprising static geometry represented by a volumetric occupancy grid and identity-indexed 3D states for the player and other entities. Within a fixed-size local region, its world-state model jointly learns action-conditioned entity dynamics, geometry completion, and entity discovery, committing updates to a persistent world memory. A separate neural renderer synthesizes first-person observations from depth and semantic projections of the world state, without feeding generated images back into state prediction. Using Doom as a testbed, we demonstrate that the model successfully learns world dynamics including geometry-aware player motion and player--NPC interactions. The framework is computationally efficient, enabling the complete simulation and rendering pipeline to achieve approximately 160ms backend latency on a MacBook Pro with an M4 Pro chip. Beyond single-player interaction, we demonstrate zero-shot deployment on unseen Doom maps, multiplayer interaction through the composition of single-player models, and programmable gameplay through rule-based state interventions. These capabilities demonstrate the transferability and extensibility of the learned game-world model. Code and pretrained model weights are publicly available.

Figures & tables

Figure 1: PlayGen2 overview. An action-conditioned world-state model updates an ego-centric region of persistent world memory. A separate diffusion renderer synthesizes observations from projected depth and semantics, actions, and visual history.

Figure 2: World-state model architecture. The Shared Context Transformer encodes geometry, entity histories, and actions. The Discovery Transformer processes Geometry Queries and Spatial Entity Queries to complete geometry and generate new entities at time $t$ . The Transition Transformer predicts player and entity states at $t+1$ , additionally attending to Current Entity Tokens that include newly accepted entities. The state-commit and re-embedding path is omitted for clarity.

	CER (%) $\downarrow$			ActAcc (%) $\uparrow$
Setting	1 min	5 min	10 min	1 min	5 min	10 min
Seen/global	2.0	4.8	5.3	93.7	85.4	83.9
Seen/local	3.1	4.3	6.7	91.2	87.8	82.1
Unseen/global	3.7	4.4	5.2	94.6	84.1	85.0
Unseen/local	3.6	5.6	6.8	93.3	88.2	85.6

Table 1: Interaction over time. CER is cumulative through each time horizon; ActAcc measures action recognition from generated RGB. Global/local denotes map initialization.

Figure 3: Learned monster motion toward a stationary player. Blue marks the player; green and red mark the monster’s start and end. Arrows indicate temporal direction. These selected successful proximity trajectories use a 300-step budget, distinct from the 200-step success statistic in the text: D reaches the threshold at step 211, and E is a separately initialized wall-side example. The model updates monster states without scripted pursuit or runtime collision correction.

Figure 4: Long-run rendering on unseen maps. Two trajectories sampled at 1, 5, and 10 simulated minutes. Each cell shows the first and last frames of a 32-frame clip, with generated images above pose-aligned engine references.

			Global					Local
Setting	Method	Min	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$	FID $\downarrow$	FVD $\downarrow$
Seen	PlayGen2	1	18.3	0.40	0.41	223.1	1302.7	233.4	1720.6
		5	19.4	0.39	0.42	237.4	1298.3	253.2	1638.5
		10	17.3	0.39	0.44	250.3	1261.9	243.6	1673.1
	PlayGen	1	19.3	0.39	0.38	210.9	1097.5	N/A
	PlayGen	5	16.1	0.29	0.69	373.4	2386.9	N/A

Table 2: Visual quality over time. Global and Local denote PlayGen2’s world-memory initialization. PlayGen receives trajectory-derived actions and is compared only on seen maps. Local initialized generation is evaluated against unpaired references. Times denote simulated minutes; N/A indicates an inapplicable setting.

Figure 5: World maps generated from local initialization. Five rollout snapshots at 1 minute (A–C) and 10 minutes (D–E). A starts from a local neighborhood of the training map; B–E start from unseen maps. Cutaway views distinguish initialized geometry (gray), generated extensions (cyan), and class-colored entity boxes. The player is omitted.

Figure 6: Paired multiplayer rollouts in Freedoom2. Synchronized views and trajectories (P1: blue; P2: red); A–D mark displayed times. Top: P1 receives idle actions as P2 approaches and leaves its view. Bottom: P1 follows P2. Both players use learned transitions and separate rendering histories, with no runtime collision correction. Setup details are in Appendix C.1 .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Level	Resolution	Channels
Encoder 1	$40\times 160\times 160$	8
Encoder 2	$40\times 80\times 80$	16
Encoder 3	$20\times 40\times 40$	32
Encoder 4	$10\times 20\times 20$	64
Encoder 5	$5\times 10\times 10$	128

Appendix

Table 4: World-state architecture. Volume dimensions follow

Z,Y,X

order. Main Transformer blocks use width 512 and feed-forward width 2048; entity-local attention uses width 128.

Family	Sampling pattern	Maximum count
Boundary	Motion-leading faces and six-face safety	64
Active entity	Base, center, top, and nearby offsets	32
Viewing ray	4 depths from 40 to 400; height offsets $-20,0,20$	12
Interior	$3\times 3\times 2$ grid within the crop	18

Appendix

Table 5: Candidate construction. Budgets precede clipping and deduplication. Matching uses a 100-unit Euclidean radius. The default objectness and occupancy probability thresholds are both 0.5.

Phase	Previous pair valid	Previous motion	Target motion
Continue	Yes	Moving	Moving
Start	Yes	Stationary	Moving
Stop	Yes	Moving	Stationary
Stay	Yes	Stationary	Stationary
Unknown-moving	No	Unavailable	Moving
Unknown-stationary	No	Unavailable	Stationary

Appendix

Table 6: Motion phases. Moving means a 3D displacement norm greater than one world unit. Previous motion uses clean history with the post-truncation validity mask; target motion uses clean

t\to t+1

states.

Stage	RGB history	Objective
Base training	Ground-truth	$\mathcal{L}_{\mathrm{diff}}$
Refinement	Ground-truth	$\mathcal{L}_{\mathrm{diff}}$ + value L1 (0.25) + gradient L1 (0.1)
Mixed-history	50% ground-truth, 40% generated, 10% none	$\mathcal{L}_{\mathrm{diff}}$

Appendix

Table 7: Renderer training stages. Intermediate refinement adds clean-latent value and spatial-gradient L1 terms for

n\leq 200

, averaged over eligible samples. Gradient error averages horizontal and vertical finite differences. Both auxiliary terms are disabled for the final stage.

Schedule	Steps	FPS	Latency	World	Render
Serial	4	6.19	161.2	82.8	77.9
Pipeline	4	8.97	212.5	104.7	102.3
Serial	8	4.98	200.6	83.3	116.7
Pipeline	8	7.16	275.7	91.7	137.8

Appendix

Table 8: M4 Pro backend efficiency. Steps denotes renderer denoising steps. Times are mean milliseconds; latency spans world-update start to completed RGB encoding. World includes projection and state handling; Render includes VAE decoding and output encoding.

Model	Penetration $\downarrow$	CER $\downarrow$	Proximity $\uparrow$
Without local map	24/28	24/27 (88.9%)	4/28
With local map	12/28	12/22 (54.5%)	15/28

Appendix

Table 9: Local-map conditioning on challenging NPC placements. Penetration and proximity count episodes; CER counts failed/resolved horizontal encounters.

Figure 8: Local-map conditioning and NPC trajectories. Paired predictions from identical initial states, with matched spatial scales. Both models fail in the first example; local conditioning improves approach in the other two.

Metric	2D	3D
Occupancy IoU (%) $\uparrow$	61.66	64.30
Precision (%) $\uparrow$	92.32	89.57
Recall (%) $\uparrow$	64.99	69.50
Fragment density $\downarrow$	48.61	9.94

Appendix

Table 10: Geometry quality after 60-second rollouts. Scores pool generated reference voxels over eight paths. Fragment density counts isolated six-connected components of at most 27 voxels per million generated voxels, excluding components touching initialization or unknown/crop boundaries.

Figure 9: Geometry after continuous generation. Two matched trajectories, with identical framing within each row. Gray denotes initialized geometry; teal denotes reference or generated extensions.

Training stage	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Before	17.1	0.31	0.62
After (Table 2 )	18.3	0.40	0.41

Appendix

Table 11: Reported renderer quality with condition-only startup. Seen map at one simulated minute. After reproduces the main-evaluation result; Before is from a separate checkpoint evaluation.

Game World Models

See all 8 papers

BoxWorld: Let World Engines Control and Video Generators Render
Oct 26, 2026Zike Wu, Ahmed Hatem, Chintan Trivedi +4Real-Time Interactive Video Generation Interactive World Models
University of British Columbia · Vector Institute for AI · Huawei Canada +1
Programmable World Model
Sep 9, 2026Zheng-Hui Huang, Guixu Lin, Jiacheng Lin +8World Models Controllable Video Generation
Alaya Lab
StatePlay: State-Aware Game World Models for Mechanics-Consistent Generation
Jul 29, 2026Zijun Lin, Zeqing Wang, Cheston Tan +2Game World Models World Models
Tencent · Nanyang Technological University · Centre for Frontier AI Research, A*STAR +1

Oct 26, 2026J/K move · Enter open · S save

BoxWorld: Let World Engines Control and Video Generators Render

Zike Wu, Ahmed Hatem, Chintan Trivedi, Wallace Michel Pinto Lira+3

University of British Columbia · Vector Institute for AI · Huawei Canada+1

Real-Time Interactive Video Generation Interactive World Models Controllable Video Generation World Models Autoregressive Video Generation

Open paper AskPDF arXiv

We tackle the challenge of building real-time interactive video worlds with user-defined behaviors. Such worlds must track object states, apply interaction rules, and render the resulting consequences as users act. Yet existing end-to-end video world models typically learn interaction dynamics within the generator, making it difficult to specify object-level rules and reason about their consequences. In this work, we introduce BoxWorld, which separates interaction reasoning from generative rendering. Given user-specified rules, a coding agent creates a Box World Engine that determines interaction outcomes and exposes the resulting object states through a lightweight interface of layout boxes, persistent object attributes, and dynamic status variables, leaving appearance and physical effects to generative rendering. A Box-State Generative Renderer then renders this evolving state stream into video autoregressively, enabling online control over object motion, state switches, and conditional events. Rendering this state stream is non-trivial: the renderer must respond to object-level state updates in real time while remaining coherent with the generated history. To this end, we introduce a three-stage training strategy that learns object grounding from open-domain videos and state transitions from gameplay videos, both annotated automatically, and finally performs rollout distillation on self-generated histories. We further propose PairedStateBench, a paired benchmark for fine-grained interaction control. BoxWorld substantially outperforms end-to-end world models and remains competitive across multiple public benchmarks. Project page: https://boxworld-web.github.io.

Abstract

Game World Models

Explore similar work

BoxWorld: Let World Engines Control and Video Generators Render

Programmable World Model

StatePlay: State-Aware Game World Models for Mechanics-Consistent Generation