Building a playable world requires more than generating plausible video: it demands responsive control, persistent spatial structure, and coherent interactions with dynamic entities. Using generated frames as the sole recurrent world representation ties these requirements to view-dependent appearance synthesis, making it difficult to maintain long-term spatial and temporal consistency in generation and interaction. We present PlayGen2, a real-time interactive world model that separates learned 3D state evolution from neural rendering. PlayGen2 maintains a persistent world memory comprising static geometry represented by a volumetric occupancy grid and identity-indexed 3D states for the player and other entities. Within a fixed-size local region, its world-state model jointly learns action-conditioned entity dynamics, geometry completion, and entity discovery, committing updates to a persistent world memory. A separate neural renderer synthesizes first-person observations from depth and semantic projections of the world state, without feeding generated images back into state prediction. Using Doom as a testbed, we demonstrate that the model successfully learns world dynamics including geometry-aware player motion and player--NPC interactions. The framework is computationally efficient, enabling the complete simulation and rendering pipeline to achieve approximately 160ms backend latency on a MacBook Pro with an M4 Pro chip. Beyond single-player interaction, we demonstrate zero-shot deployment on unseen Doom maps, multiplayer interaction through the composition of single-player models, and programmable gameplay through rule-based state interventions. These capabilities demonstrate the transferability and extensibility of the learned game-world model. Code and pretrained model weights are publicly available.
Figures & tables
Figure 1: PlayGen2 overview. An action-conditioned world-state model updates an ego-centric region of persistent world memory. A separate diffusion renderer synthesizes observations from projected depth and semantics, actions, and visual history.
Figure 2: World-state model architecture. The Shared Context Transformer encodes geometry, entity histories, and actions. The Discovery Transformer processes Geometry Queries and Spatial Entity Queries to complete geometry and generate new entities at time t . The Transition Transformer predicts player and entity states at t+1 , additionally attending to Current Entity Tokens that include newly accepted entities. The state-commit and re-embedding path is omitted for clarity.
CER (%) ↓
ActAcc (%) ↑
Setting
1 min
5 min
10 min
1 min
5 min
10 min
Seen/global
2.0
4.8
5.3
93.7
85.4
83.9
Seen/local
3.1
4.3
6.7
91.2
87.8
82.1
Unseen/global
3.7
4.4
5.2
94.6
84.1
85.0
Unseen/local
3.6
5.6
6.8
93.3
88.2
85.6
Table 1: Interaction over time. CER is cumulative through each time horizon; ActAcc measures action recognition from generated RGB. Global/local denotes map initialization.
Figure 3: Learned monster motion toward a stationary player. Blue marks the player; green and red mark the monster’s start and end. Arrows indicate temporal direction. These selected successful proximity trajectories use a 300-step budget, distinct from the 200-step success statistic in the text: D reaches the threshold at step 211, and E is a separately initialized wall-side example. The model updates monster states without scripted pursuit or runtime collision correction.
Figure 4: Long-run rendering on unseen maps. Two trajectories sampled at 1, 5, and 10 simulated minutes. Each cell shows the first and last frames of a 32-frame clip, with generated images above pose-aligned engine references.
Global
Local
Setting
Method
Min
PSNR ↑
SSIM ↑
LPIPS ↓
FID ↓
FVD ↓
FID ↓
FVD ↓
Seen
PlayGen2
1
18.3
0.40
0.41
223.1
1302.7
233.4
1720.6
5
19.4
0.39
0.42
237.4
1298.3
253.2
1638.5
10
17.3
0.39
0.44
250.3
1261.9
243.6
1673.1
PlayGen
1
19.3
0.39
0.38
210.9
1097.5
N/A
5
16.1
0.29
0.69
373.4
2386.9
Table 2: Visual quality over time. Global and Local denote PlayGen2’s world-memory initialization. PlayGen receives trajectory-derived actions and is compared only on seen maps. Local initialized generation is evaluated against unpaired references. Times denote simulated minutes; N/A indicates an inapplicable setting.
Figure 5: World maps generated from local initialization. Five rollout snapshots at 1 minute (A–C) and 10 minutes (D–E). A starts from a local neighborhood of the training map; B–E start from unseen maps. Cutaway views distinguish initialized geometry (gray), generated extensions (cyan), and class-colored entity boxes. The player is omitted.
Figure 6: Paired multiplayer rollouts in Freedoom2. Synchronized views and trajectories (P1: blue; P2: red); A–D mark displayed times. Top: P1 receives idle actions as P2 approaches and leaves its view. Bottom: P1 follows P2. Both players use learned transitions and separate rendering histories, with no runtime collision correction. Setup details are in Appendix C.1 .
Table 9
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Level
Resolution
Channels
Encoder 1
40×160×160
8
Encoder 2
40×80×80
16
Encoder 3
20×40×40
32
Encoder 4
10×20×20
64
Encoder 5
5×10×10
128
Appendix
Table 4: World-state architecture. Volume dimensions follow Z,Y,X order. Main Transformer blocks use width 512 and feed-forward width 2048; entity-local attention uses width 128.
Family
Sampling pattern
Maximum count
Boundary
Motion-leading faces and six-face safety
64
Active entity
Base, center, top, and nearby offsets
32
Viewing ray
4 depths from 40 to 400; height offsets −20,0,20
12
Interior
3×3×2 grid within the crop
18
Appendix
Table 5: Candidate construction. Budgets precede clipping and deduplication. Matching uses a 100-unit Euclidean radius. The default objectness and occupancy probability thresholds are both 0.5.
Phase
Previous pair valid
Previous motion
Target motion
Continue
Yes
Moving
Moving
Start
Yes
Stationary
Moving
Stop
Yes
Moving
Stationary
Stay
Yes
Stationary
Stationary
Unknown-moving
No
Unavailable
Moving
Unknown-stationary
No
Unavailable
Stationary
Appendix
Table 6: Motion phases. Moving means a 3D displacement norm greater than one world unit. Previous motion uses clean history with the post-truncation validity mask; target motion uses clean t→t+1 states.
Stage
RGB history
Objective
Base training
Ground-truth
Ldiff
Refinement
Ground-truth
Ldiff + value L1 (0.25) + gradient L1 (0.1)
Mixed-history
50% ground-truth, 40% generated, 10% none
Ldiff
Appendix
Table 7: Renderer training stages. Intermediate refinement adds clean-latent value and spatial-gradient L1 terms for n≤200 , averaged over eligible samples. Gradient error averages horizontal and vertical finite differences. Both auxiliary terms are disabled for the final stage.
Schedule
Steps
FPS
Latency
World
Render
Serial
4
6.19
161.2
82.8
77.9
Pipeline
4
8.97
212.5
104.7
102.3
Serial
8
4.98
200.6
83.3
116.7
Pipeline
8
7.16
275.7
91.7
137.8
Appendix
Table 8: M4 Pro backend efficiency. Steps denotes renderer denoising steps. Times are mean milliseconds; latency spans world-update start to completed RGB encoding. World includes projection and state handling; Render includes VAE decoding and output encoding.
Model
Penetration ↓
CER ↓
Proximity ↑
Without local map
24/28
24/27 (88.9%)
4/28
With local map
12/28
12/22 (54.5%)
15/28
Appendix
Table 9: Local-map conditioning on challenging NPC placements. Penetration and proximity count episodes; CER counts failed/resolved horizontal encounters.
Figure 8: Local-map conditioning and NPC trajectories. Paired predictions from identical initial states, with matched spatial scales. Both models fail in the first example; local conditioning improves approach in the other two.
Metric
2D
3D
Occupancy IoU (%) ↑
61.66
64.30
Precision (%) ↑
92.32
89.57
Recall (%) ↑
64.99
69.50
Fragment density ↓
48.61
9.94
Appendix
Table 10: Geometry quality after 60-second rollouts. Scores pool generated reference voxels over eight paths. Fragment density counts isolated six-connected components of at most 27 voxels per million generated voxels, excluding components touching initialization or unknown/crop boundaries.
Figure 9: Geometry after continuous generation. Two matched trajectories, with identical framing within each row. Gray denotes initialized geometry; teal denotes reference or generated extensions.
Training stage
PSNR ↑
SSIM ↑
LPIPS ↓
Before
17.1
0.31
0.62
After (Table 2 )
18.3
0.40
0.41
Appendix
Table 11: Reported renderer quality with condition-only startup. Seen map at one simulated minute. After reproduces the main-evaluation result; Before is from a separate checkpoint evaluation.
We tackle the challenge of building real-time interactive video worlds with user-defined behaviors. Such worlds must track object states, apply interaction rules, and render the resulting consequences as users act. Yet existing end-to-end video world models typically learn interaction dynamics within the generator, making it difficult to specify object-level rules and reason about their consequences. In this work, we introduce BoxWorld, which separates interaction reasoning from generative rendering. Given user-specified rules, a coding agent creates a Box World Engine that determines interaction outcomes and exposes the resulting object states through a lightweight interface of layout boxes, persistent object attributes, and dynamic status variables, leaving appearance and physical effects to generative rendering. A Box-State Generative Renderer then renders this evolving state stream into video autoregressively, enabling online control over object motion, state switches, and conditional events. Rendering this state stream is non-trivial: the renderer must respond to object-level state updates in real time while remaining coherent with the generated history. To this end, we introduce a three-stage training strategy that learns object grounding from open-domain videos and state transitions from gameplay videos, both annotated automatically, and finally performs rollout distillation on self-generated histories. We further propose PairedStateBench, a paired benchmark for fine-grained interaction control. BoxWorld substantially outperforms end-to-end world models and remains competitive across multiple public benchmarks. Project page: https://boxworld-web.github.io.
Zike Wu, Ahmed Hatem, Chintan Trivedi +4
University of British Columbia · Vector Institute for AI · Huawei Canada +1
Recent video world models generate increasingly realistic and interactive visual experiences, yet lack reliable mechanisms for maintaining persistent world state and enforcing programmable rules over extended interactions. We introduce Programmable World Model, a framework that decouples world-state evolution from visual observation generation. An agent translates natural-language instructions into executable programs that specify entity states and state-transition rules, enabling direct control over individual entities and their interactions. A lightweight engine executes these programs to update and maintain an explicit, persistent global world state, including off-screen entities and non-visual attributes. To connect world state with visual generation, we introduce state-augmented 3D oriented bounding boxes (OBBs) as an intermediate representation. This representation, together with the target camera trajectory, is deterministically compiled into pixel-aligned spatiotemporal conditioning signals for a pretrained video model serving as the generative renderer. This design allows users to create playable games with predefined mechanics, direct control over individual entities, and persistent world state throughout gameplay. We further introduce CombatStateBench, a benchmark for evaluating programmable world models. On CombatStateBench, our method achieves 94% Count Accuracy and 98% State Accuracy, substantially outperforming existing interactive video world models while supporting coherent long-horizon generation. These results demonstrate the effectiveness of separating explicit state evolution from generative rendering for building persistent, programmable worlds.
Recent game world models can generate visually realistic and interactive environments conditioned on player actions. However, games are not defined by pixels alone; they are governed by explicit mechanics, namely state-dependent rules that control health reduction, skill activation, and game termination. These mechanics depend on precise internal states, such as health points, skill meters, and timers, which are tightly coupled with visual observations and determine how gameplay evolves. Without modeling these state dynamics, existing game world models may generate visually plausible rollouts but violate the underlying game rules. In this paper, we propose StatePlay, a novel state-aware game world model that jointly predicts visual content and game states to promote mechanics-consistent generation. StatePlay adopts a mixture-of-transformers (MoT)-style architecture that preserves specialized visual and state representations while enabling cross-modal interaction, allowing predicted states to guide frame generation. Each branch is further optimized with a distinct objective suited to its modality. Experiments show that StatePlay achieves an average normalized L1 distance below 0.06 for state prediction. Furthermore, compared with models without explicit state modeling, our method improves mechanics fidelity in generated game rollouts by 18.6%. Overall, our work highlights the importance of state-aware game world modeling and advances beyond pixel-level realism toward complete and mechanically faithful game generation.
Zijun Lin, Zeqing Wang, Cheston Tan +2
Tencent · Nanyang Technological University · Centre for Frontier AI Research, A*STAR +1