For more than half a century, video games have pursued better graphics, more natural interaction, and more engaging gameplay. Generative video models (VMs) now bear on all three. Neural networks already draw most of the pixels in shipped games, and recent world models let a lightweight game engine keep the world state while a VM turns the engine's coarse proxy video into final frames. These systems, however, treat the VM as a renderer that only restyles motion the engine has already authored, which leaves most of its generative capacity unused and limits the diversity a game can offer. We present GameWM, a design for next-generation games built on a clear division of labor. The GameWM-engine keeps what must be exact: rules, state, camera, and the layout of the world, which also spares the video model from having to remember it. The GameWM-VM generates what is expensive to author by hand: the look of the world and the detail of the player's interactions. One game mechanic can then be realized in many visually and physically distinct ways, while its outcome stays exactly as the engine computed it. To realize this division of labor, we build GameWM as a complete system: a data engine centered on interactions rather than exploration, an interface through which authored character sheets, scene art, and skill descriptions enter the model, and a model family from 5B to 33B parameters with a real-time causal variant for play. Together they give every player a fresh version of the same game: each player's own character, world, and skills turn the same engine mechanics into a new experience, while every event keeps the timing and outcome the GameWM-engine computed.
Figures & tables
Figure 1: Seven decades of games along graphics, interaction, and gameplay. Eras are delimited by transitions in graphics, and each dated landmark is placed in the era of its release. Interaction is split as in Section 1 : coarse-grained interaction moves the camera and the protagonist; fine-grained interaction is what the protagonist does to objects and characters.
System
Year
Control / proxy
Backbone
Paired data
Real time
VM decides
Fine-grained interaction
First generation: action-conditioned, state in the generated frames
GameNGen
2024
DOOM key actions
Stable Diffusion v1.4, augmented
RL-agent play of DOOM
20 FPS, 1 TPU
Everything
The game’s own actions
Oasis
2024
Keyboard
DiT + ViT autoencoder, 500M open
Minecraft
20 FPS
Everything
Mine, place, pick up, open chests
Hunyuan-GameCraft
2025
Keys + mouse as camera motion
HunyuanVideo
> 1M recordings, 100+ AAA games; synthetic
6.6 FPS, distilled
Everything
Navigation only
Matrix-Game 2.0
2025
Frame-level keys + mouse
SkyReels-V2-1.3B (Wan2.1)
UE and game capture ( ∼ 1,200 h); Sekai
25 FPS
Everything
Navigation only
Genie 3
2025
Navigation; text world events
n/r
n/r
720p, 24 FPS
Everything
Text events (weather, objects, characters), not by the agent
Table 1: Interactive world models compared by design choice. Entries come from each paper or project page; inputs, resolutions, and hardware differ, so the table compares design choices rather than measured performance. “Fine-grained interaction”: what the user can do beyond moving and looking, and who produces it. n/r: not reported in the sources we read.
Figure 2: GameWM maximizes generative capability subject to game stability. Designs run from traditional CG (left) to pure world models (right). Each is a stack of the four modules of a frame, with world state at the bottom: purple blocks are produced by a neural network, grey blocks by the engine, and the label above each gap names the module the network takes over between two neighboring designs. The curves are schematic, not measured.
Figure 3: From authored assets to play. Left: authored once, a first frame and a character sheet, and a skill tree that fuses an attribute of the sheet (fire) with each mechanic. Middle: at run time the GameWM-engine renders low-poly proxy frames and resolves the event, here a ranged attack that becomes Fireball. Right: the four inputs fill the slots of the GameWM-VM’s reference interface (H3’s Ref2VA, Section 6 ); the output column shows frames of a BD-Sheet sample generated from them.
Figure 4: The interaction-centric data engine. Left: both source paths yield the same three proxies and an event caption, from which H3 Ref2VA renders m low-poly and n high-fidelity versions of the clip. After VLM screening, each of the three proxies and the m low-poly videos is paired with every high-fidelity target and with the source recording. Right: one “enter vehicle” clip with m=n=4 , giving 7×4=28 stylized pairs, plus seven with the recording (not drawn); the outlined cell is the pair shown above the grid.
Figure 5: Composition of the data engine by source clips. Left: the share of each source. Right: the event of every clip, with granularity on the inner ring and the event on the outer ring; hue gives the source, and height the granularity (fine-grained, coarse, untagged). ABot-World is the released 500-hour, action-annotated part of the training data of ABot-World-0 [ ABot-World Team, 2026 ] , with depth and semantics estimated by MoGe-3 [ Kong et al., 2026 ] and SAM 2 [ Ravi et al., 2025 ] ; CARLA [ Dosovitskiy et al., 2017 ] and Magpie [ Zhan and others, 2026 ] come from SolarRenderer-Data [ Fang et al., 2026 ] . Shares are of the snapshot in Table 6 .
Figure 6: One training pair per engine event. Each block shows a clip’s capsule re-render, which is the proxy, above the source recording. The first two rows are fine-grained GTA V events; the rest are coarse or untagged clips from GTA V, Magpie, ABot-World, and CARLA. Red ticks mark engine state changes. The capsules carry the camera, the layout, and the timing of each event, but no limbs, faces, or effects.
Model
Backbone
Proxy entry
Adaptation
Clip
GameWM-H3-33B
MiniMax-H3 (33B)
Native Ref2VA rows
LoRA, rank 128
1280×704 , 124 frames, 24 fps
GameWM-Wan2.2-14B
Wan2.2-I2V-A14B, high-noise expert (14B)
Prefix, half resolution
LoRA, rank 128
832×480 , 81 frames, 16 fps
GameWM-Wan2.2-5B
Wan2.2-TI2V-5B
Prefix, half resolution
Full fine-tuning
1280×704 , 121 frames, 24 fps
Table 2: The GameWM models. All three share the interface of Section 6.2 , are trained on the same corpus, and are distilled to four-step causal students by the stages of Section 6.3 .
Model
Initialized from
Training
Steps
Sampling
BD
H3
Eq. ( 1 ), lr 2×10−4 , batch 8
900
30, full clip
AR
BD
Teacher-forced AnyFlow, lr 10−4 , batch 8
600
few-step, causal
DMD
AR; teacher and critic from BD
Eq. ( 3 ), 5 critic updates per student update
1,000
4, causal
BD-Sheet
BD
Eq. ( 1 ) with character sheet
+600
30, full clip
AR-Sheet
BD-Sheet
Teacher-forced AnyFlow
900
few-step, causal
DMD-Sheet
AR-Sheet; teacher and critic from BD-Sheet
Eq. ( 3 )
1,000
4, causal
Table 3: Checkpoints of GameWM-H3-33B in the experiments. “Sheet” variants add a character sheet as a second anchor image. Sampling steps are per clip for BD and per chunk for the causal models.
Figure 7: Every proxy on one held-out vehicle entry. Top: the recorded game. Below, each group shows a proxy above the bidirectional output it drives: three engine buffers (DUV, depth, semantic), three coarse re-renders (low-poly, voxel, capsule), and a static control in which the first DUV frame is held for the whole clip. Red ticks mark the engine’s entry start and seated frames.
Figure 8: One mechanic, many realizations: ranged and melee. The left four columns are the shooting clip and the right four a melee clip; each clip keeps one capsule proxy and one caption template (the box above its columns). Each row is one character: its sheet and name (far left), a first frame of that character in each scene, and its skill for each mechanic (the tag on the first frame of each clip). BD-Sheet, 30 steps; red ticks mark the shot, the strike, and the fall as read off the proxies. The DMD-Sheet version of the ranged clip is in Figure 13 .
Figure 9: One character across scenes. Four characters (left, with their sheets), each in four scenes: three shooting clips and a held-out vehicle entry, all driven by capsule proxies. Each first frame is authored for its scene from the sheet. BD-Sheet model; columns from 0 to 5 s. The DMD-Sheet version is in Figure 14 .
With sheet
Sheet removed
Model
Own ↑
Others
Margin
Own ↑
Others
Margin
BD-Sheet
0.588 ± 0.096
0.481
0.107
0.564 ± 0.102
0.463
0.101
DMD-Sheet
0.584 ± 0.072
0.474
0.110
0.570 ± 0.079
0.461
0.110
Table 4: Identity across scenes. DINOv2 cosine similarity between the tracked protagonist and a character sheet, averaged over the sixteen scene–character pairs of Figure 9 (mean ± standard deviation over pairs). “Own”: the protagonist’s own sheet. “Others”: mean over the other three characters’ sheets. “Margin”: own minus others. “Sheet removed”: the same model, first frame, and caption, run without the sheet; similarity is still measured against the sheets.
Figure 10: Bidirectional teacher and causal student on four events. Each group shows the proxy above the bidirectional output (30 steps, whole clip) and the DMD output (4 steps per chunk). Shoot: the shooting clip with its capsule proxy; Enter, Exit, Melee: held-out clips under semantic, DUV, and capsule proxies. Columns are chosen per event; red ticks mark the engine’s event frames (for the two attacks, read off the proxy).
Figure 11: Four-step causal students across backbones. The same capsule proxy (top), fire-mage sheet, first frame, and caption for every model; for each model, the three of eight seeds that follow the proxy most closely. GameWM-H3-33B is DMD-Sheet; the Wan2.2 students are distilled from their sheet-conditioned bidirectional models. Columns are times on the 24 fps clip (the 14B student renders 16 fps); red ticks mark the shot and the fall in the proxy.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Source
Proxy obtained from
Engine events
Clips
Low-poly pass
High-fid. pass
GTA V
Engine buffers (exact)
Fine-grained and coarse
1,460
94%
70%
ABot-World
Monocular estimates
Coarse (exploration)
2,765
55%
49%
CARLA
Simulator (exact)
None (street traffic)
1,000
75%
–
Magpie
Simulator, gray-box geometry
Coarse (locomotion)
1,030
30%
49%
Appendix
Table 6: Corpus snapshot (2026-11-01). Source clips in the stylization pool and the share of renders that pass the VLM screen, before human review. The corpus keeps growing as sources are converted; Section 7.1 states which data each checkpoint was trained on.
Figure 12: Characters and their skills, causal model. One capsule proxy and one caption template (top); each row adds a character sheet (left), a first frame of that character, and the character’s skill (row label). DMD-Sheet, 4 steps per chunk; one frame from the middle of each chunk, dashed lines at chunk boundaries.
Figure 14: One character across scenes, causal model. The scenes, sheets, first frames, and captions of Figure 9 , rendered by DMD-Sheet at 4 steps per chunk; columns from 0 to 5 s.
Building interactive worlds that respond coherently to player actions has long been a shared goal of computer graphics, games, and artificial intelligence. Recent video generative models provide a data-driven route toward this goal by predicting future observations conditioned on user actions, and are increasingly regarded as potential next-generation game engines. Realizing a genuinely interactive game world, however, requires interaction outcomes that follow rules over evolving game conditions, consequences that persist over long horizons, and a generation loop that operates in real time. Conventional game engines realize these properties through a recurrent action-state-observation loop, in which player actions update an explicit game state according to predefined rules and observations are rendered from the resulting state. Taking this loop as an organizing lens, this paper examines interactive game world modeling along four dimensions: player action control, game state dynamics, state-observation persistence, and real-time interactive generation. For each dimension, we start from the capabilities required by an interactive game world, group existing approaches into representative families, and discuss the strengths and trade-offs of each family. Complementing this analysis, we present a scalable data engine for Black Myth: Wukong that collects over 90 hours of gameplay with frame-aligned player actions, ground-truth game states, and visual observations, together with structured and semantic annotations, as a resource for state-aware game world modeling. We hope this paper offers a clear picture of where the field stands and fosters progress toward interactive game worlds.
Modern video games combine first-person perception, rapid visual changes, persistent world state, and heterogeneous native controls. Existing game agents map visual and task context directly to actions but lack explicit world dynamics modeling, whereas interactive game world models predict visual futures from supplied actions but do not serve as task policies. World-Action Models (WAMs) unify these objectives, but remain largely unexplored under the dynamics and open-ended interaction of video games. We introduce GameWAM, to our knowledge the first WAM for native closed-loop gameplay and GUI control. GameWAM jointly generates future visual observations and executable keyboard-mouse trajectories through parallel visual and action generative processes with block-causal conditioning and flow matching. To support joint world-action learning, we construct synchronized gameplay and GUI trajectories. To handle heterogeneous native control, GameWAM predicts a gameplay/GUI mode per action step and generates actions with mode-specific prediction distributions and continuous-action normalization. For long-horizon interaction, block-cycle control coordinates prediction, execution, and temporal context: it predicts beyond the committed horizon, executes short action blocks, replans from new observations, and hierarchically structures context from fine-grained within-cycle history to persistent cross-cycle history. Experiments demonstrate competitive task success with fewer executed native actions than the compared agents. We further uncover Low-Frequency Action Source Imprinting (LASI), in which low-frequency components of the sampled action source systematically steer coarse generated camera motion under fixed conditioning, revealing a source-sensitivity failure mode in generative control. Project page is available at https://yunncheng.github.io/GameWAM/.
Yuncheng Guo, Zhanqiu Zhang, Yiwen Guo +1
Fudan University · LIGHTSPEED · Independent Researcher +1
Modern game development faces significant challenges in creativity and cost due to predetermined content in traditional game engines. Recent breakthroughs in video generation models, capable of synthesizing realistic and interactive virtual environments, present an opportunity to revolutionize game creation. This position paper argues that next-generation game engines should be built on Interactive Generative Video (IGV). We term such engines Generative Game Engines (GGE), with IGV as the perceptual and world-generation core under designer control. We argue why IGV is uniquely suited for this role, identify what is missing, and examine progress across six functional modules to show how each supports our position, culminating in a five-level maturity roadmap (L0-L4). Drawing on discussions with researchers, game developers, and community participants, we present nine alternative views and ethical considerations to stimulate debate.