GameWM: Toward Next-Generation Video Games with Interaction-Centric World Models
Organizations: Lightspeed Studios, Tencent
Abstract
For more than half a century, video games have pursued better graphics, more natural interaction, and more engaging gameplay. Generative video models (VMs) now bear on all three. Neural networks already draw most of the pixels in shipped games, and recent world models let a lightweight game engine keep the world state while a VM turns the engine's coarse proxy video into final frames. These systems, however, treat the VM as a renderer that only restyles motion the engine has already authored, which leaves most of its generative capacity unused and limits the diversity a game can offer. We present GameWM, a design for next-generation games built on a clear division of labor. The GameWM-engine keeps what must be exact: rules, state, camera, and the layout of the world, which also spares the video model from having to remember it. The GameWM-VM generates what is expensive to author by hand: the look of the world and the detail of the player's interactions. One game mechanic can then be realized in many visually and physically distinct ways, while its outcome stays exactly as the engine computed it. To realize this division of labor, we build GameWM as a complete system: a data engine centered on interactions rather than exploration, an interface through which authored character sheets, scene art, and skill descriptions enter the model, and a model family from 5B to 33B parameters with a real-time causal variant for play. Together they give every player a fresh version of the same game: each player's own character, world, and skills turn the same engine mechanics into a new experience, while every event keeps the timing and outcome the GameWM-engine computed.
Figures & tables
| System | Year | Control / proxy | Backbone | Paired data | Real time | VM decides | Fine-grained interaction |
| First generation: action-conditioned, state in the generated frames | |||||||
| GameNGen | 2024 | DOOM key actions | Stable Diffusion v1.4, augmented | RL-agent play of DOOM | 20 FPS, 1 TPU | Everything | The game’s own actions |
| Oasis | 2024 | Keyboard | DiT + ViT autoencoder, 500M open | Minecraft | 20 FPS | Everything | Mine, place, pick up, open chests |
| Hunyuan-GameCraft | 2025 | Keys + mouse as camera motion | HunyuanVideo | 1M recordings, 100+ AAA games; synthetic | 6.6 FPS, distilled | Everything | Navigation only |
| Matrix-Game 2.0 | 2025 | Frame-level keys + mouse | SkyReels-V2-1.3B (Wan2.1) | UE and game capture ( 1,200 h); Sekai | 25 FPS | Everything | Navigation only |
| Genie 3 | 2025 | Navigation; text world events | n/r | n/r | 720p, 24 FPS | Everything | Text events (weather, objects, characters), not by the agent |
| Model | Backbone | Proxy entry | Adaptation | Clip |
| GameWM-H3-33B | MiniMax-H3 (33B) | Native Ref2VA rows | LoRA, rank 128 | , 124 frames, 24 fps |
| GameWM-Wan2.2-14B | Wan2.2-I2V-A14B, high-noise expert (14B) | Prefix, half resolution | LoRA, rank 128 | , 81 frames, 16 fps |
| GameWM-Wan2.2-5B | Wan2.2-TI2V-5B | Prefix, half resolution | Full fine-tuning | , 121 frames, 24 fps |
| Model | Initialized from | Training | Steps | Sampling |
| BD | H3 | Eq. ( 1 ), lr , batch 8 | 900 | 30, full clip |
| AR | BD | Teacher-forced AnyFlow, lr , batch 8 | 600 | few-step, causal |
| DMD | AR; teacher and critic from BD | Eq. ( 3 ), 5 critic updates per student update | 1,000 | 4, causal |
| BD-Sheet | BD | Eq. ( 1 ) with character sheet | +600 | 30, full clip |
| AR-Sheet | BD-Sheet | Teacher-forced AnyFlow | 900 | few-step, causal |
| DMD-Sheet | AR-Sheet; teacher and critic from BD-Sheet | Eq. ( 3 ) | 1,000 | 4, causal |
| With sheet | Sheet removed | |||||
| Model | Own | Others | Margin | Own | Others | Margin |
| BD-Sheet | 0.588 0.096 | 0.481 | 0.107 | 0.564 0.102 | 0.463 | 0.101 |
| DMD-Sheet | 0.584 0.072 | 0.474 | 0.110 | 0.570 0.079 | 0.461 | 0.110 |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Source | Proxy obtained from | Engine events | Clips | Low-poly pass | High-fid. pass |
| GTA V | Engine buffers (exact) | Fine-grained and coarse | 1,460 | 94% | 70% |
| ABot-World | Monocular estimates | Coarse (exploration) | 2,765 | 55% | 49% |
| CARLA | Simulator (exact) | None (street traffic) | 1,000 | 75% | – |
| Magpie | Simulator, gray-box geometry | Coarse (locomotion) | 1,030 | 30% | 49% |