Rules govern how player actions change interactive worlds. Different rules create distinct gameplay experiences within similar scenes and action spaces. For customizable gameplay, generative world models must follow user-defined rules beyond real-world behaviors. However, realizing user-defined rules presents two challenges: Rule-State Persistence and Rule-to-Visual Realization. The former requires preserving rule definitions, dependencies, and world states across saving and loading, so that restored worlds follow the established rules. The latter requires accurately realizing rule-induced changes on designated objects and attributes, particularly when affected objects are unseen. In this paper, we propose WorldRuler, which organizes rule definitions, dependencies, and object states in an executable 3D rule graph and connects rule execution, video generation, and perceptual feedback in a closed loop. The graph supports joint saving and restoration of rules and world states, while dynamic object binding grounds rule-specified entities in visual instances and uses stored states to constrain generation from their first appearance. We further introduce object attribute state replacement and counterfactual reconstruction training, enabling accurate attribute control and compatible appearance and motion changes while limiting interference with unrelated content. We develop an automated data collection and annotation pipeline using GTA-V and Habitat to construct RuleVerse, a dataset of rule-driven interactions with detailed state and event annotations, including cases where rules take effect before affected objects become visible. Extensive experiments demonstrate the effectiveness of WorldRuler in preserving rule-state consistency across saving and loading and accurately realizing rule outcomes on designated objects and attributes.
Figures & tables
Figure 1: WorldRuler enables persistent, rule-governed generative worlds. The saved rules and world state allow the player to collect the key and open the door after loading, demonstrating Rule-State Persistence and Rule-to-Visual Realization .
Figure 2: Overview of WorldRuler. Rule execution updates persistent object states, dynamic binding associates rule entities with 3D instances and their current image regions, and local attribute replacement corrects denoising estimates for the next sampling update. Rules, states, execution progress, and bindings are saved and restored jointly. Independent perceptual feedback informs subsequent windows. Scene and point cloud images come from the demo visualization, while the remaining flows and gate-state comparison illustrate the mechanism.
Figure 3: Local attribute replacement and paired training. (a) A frozen reader estimates current attributes, and rule targets determine their discrepancies. Attribute corrections and compatible adjustments modify local features before insertion into the sampling update. (b) Counterfactual reconstruction supervises two target states and their difference from shared generated features. Bidirectional swapping starts from real paired features and supervises replacement with the other state. All branches share the replacement operator Tϕ . Dashed links indicate supervision.
Figure 4: RuleVerse data collection. Scene capabilities constrain rule and script authoring, and compiled actions are executed with the designated rules enabled and disabled. Simulator states and events provide annotations of rule outcomes. Videos and annotations are checked before splitting by scene or location group.
Figure 5: Statistics of RuleVerse. (a) Distributions of sample pairs over primary trigger and effect categories, normalized within each displayed heatmap. (b) Rule proportions for single-object binding, cross-object binding, and environment effects. (c) First activation in the first or second half of the video (top), and dependence on action history through ordered actions or cumulative event counts (bottom).
Method
Rule Success Rate ↑
Full Completion Rate ↑
Identity
Trigger
Effect
Timing
Hold
avg.
Stages
Order
Goal
avg.
Habitat
MiniMax-H3 MiniMax (2026)
78.85
53.37
17.79
9.13
9.62
33.75
11.21
13.64
5.26
10.03
CWM Chen et al. (2026b)
93.85
49.23
3.85
0.77
0.77
29.69
13.52
12.48
16.81
14.27
SWM Ki et al. (2026)
98.16
68.66
20.74
15.67
17.05
44.06
15.55
17.58
20.82
17.98
WorldRuler
98.61
79.72
58.47
37.36
50.83
64.99
52.61
44.38
47.12
48.04
Table 1: Comparison on the Habitat and GTA-V subsets of RuleVerse. We report component-wise and average Rule Success Rate (RSR) and Full Completion Rate (FCR). All values are percentages.
Method
EndFollow ↑
PairFollow ↑
TwinContrast ↑
Strict ↑
CntAbsent ↑
PairStrict ↑
Wan2.2 A14B Wan et al. (2025)
65
32
38
54
63
26
HunyuanVideo 1.5 Wu et al. (2025)
57
20
26
29
30
4
MiniMax-H3 MiniMax (2026)
66
32
32
29
30
4
LingBot-World 2.0 Gao et al. (2026)
48
16
38
44
76
14
AlayaWorld Team et al. (2026)
37
12
28
33
85
6
SWM Ki et al. (2026)
96
92
96
95
98
90
Table 2: Evaluation of WorldRuler and baseline methods on WorldState-Bench using the benchmark’s original judging protocol.
Variant
RSR ↑
FCR ↑
First-appearance ↑
Preservation ↑
Fixed object binding
48.09
44.81
20.58
19.49
Without attribute state replacement
46.43
38.50
12.90
17.16
Without Lswap
51.24
45.75
6.45
20.63
Full Method
64.99
48.04
29.69
28.71
Table 3: Component ablations on the Habitat subset of RuleVerse. All configurations use the complete test set, with rule realization evaluated on rule-enabled branches and rule-disabled branches retained for counterfactual comparison.
Restoration
Consistency ↑
Post-load RSR ↑
Post-load FCR ↑
Last image and rule text
94.25
49.23
40.11
Rules and attributes, no progress
95.13
51.26
45.53
Joint rules, execution state, and bindings
98.17
54.36
47.89
Table 4: Save and load evaluation. Consistency uses all predetermined checkpoints, while post-load metrics use predetermined continuation events and tasks.
Figure 6: Qualitative results on the GTA-V subset of RuleVerse from the WorldRuler runtime with MiniMax-H3 Ref2VA. The two cases show fire and smoke after shooting a fuel tank, and smoke after entering a vehicle. Enabled and disabled branches are displayed at matching video times, with red boxes marking visible effect regions.
Recent video world models generate increasingly realistic and interactive visual experiences, yet lack reliable mechanisms for maintaining persistent world state and enforcing programmable rules over extended interactions. We introduce Programmable World Model, a framework that decouples world-state evolution from visual observation generation. An agent translates natural-language instructions into executable programs that specify entity states and state-transition rules, enabling direct control over individual entities and their interactions. A lightweight engine executes these programs to update and maintain an explicit, persistent global world state, including off-screen entities and non-visual attributes. To connect world state with visual generation, we introduce state-augmented 3D oriented bounding boxes (OBBs) as an intermediate representation. This representation, together with the target camera trajectory, is deterministically compiled into pixel-aligned spatiotemporal conditioning signals for a pretrained video model serving as the generative renderer. This design allows users to create playable games with predefined mechanics, direct control over individual entities, and persistent world state throughout gameplay. We further introduce CombatStateBench, a benchmark for evaluating programmable world models. On CombatStateBench, our method achieves 94% Count Accuracy and 98% State Accuracy, substantially outperforming existing interactive video world models while supporting coherent long-horizon generation. These results demonstrate the effectiveness of separating explicit state evolution from generative rendering for building persistent, programmable worlds.
Existing world models can generate action-conditioned video from a single image. Many focus on camera movement, allowing users to explore the generated scene. However, navigation alone does not ensure that an action follows persistent rules when its outcome depends on state that is invisible in the current frame. We present Stateful Harness, a framework that separates world evolution from video rendering to support stateful and rule-governed interaction. It maintains an evolving world state under specified rules, renders each resulting event with a frozen video model, and uses the generated clip to update the world state for the next turn. At each turn, the harness resolves the action selected by the user and compiles the resulting event into an instruction, while a preview clip from an editable 3D scene provides spatial and camera guidance. To evaluate rule compliance, we build WorldState-Bench, a benchmark of 50 counterfactual twin pairs that share an initial frame and action but differ in hidden state and required outcome. Under rule-only prompting, the best of the 11 evaluated video backends produces the required endings for both cases in 32% of pairs. For MiniMax-H3, specifying the required endings raises this rate from 32% to 64%, while the full harness reaches 92% with the same frozen model. These results support separating rule execution from video rendering for world interactions with hidden state.
Chan Yan Ki, Yanran Zhang, Wenzhao Zheng +5
Department of Automation, Tsinghua University, China · GigaAI
We tackle the challenge of building real-time interactive video worlds with user-defined behaviors. Such worlds must track object states, apply interaction rules, and render the resulting consequences as users act. Yet existing end-to-end video world models typically learn interaction dynamics within the generator, making it difficult to specify object-level rules and reason about their consequences. In this work, we introduce BoxWorld, which separates interaction reasoning from generative rendering. Given user-specified rules, a coding agent creates a Box World Engine that determines interaction outcomes and exposes the resulting object states through a lightweight interface of layout boxes, persistent object attributes, and dynamic status variables, leaving appearance and physical effects to generative rendering. A Box-State Generative Renderer then renders this evolving state stream into video autoregressively, enabling online control over object motion, state switches, and conditional events. Rendering this state stream is non-trivial: the renderer must respond to object-level state updates in real time while remaining coherent with the generated history. To this end, we introduce a three-stage training strategy that learns object grounding from open-domain videos and state transitions from gameplay videos, both annotated automatically, and finally performs rollout distillation on self-generated histories. We further propose PairedStateBench, a paired benchmark for fine-grained interaction control. BoxWorld substantially outperforms end-to-end world models and remains competitive across multiple public benchmarks. Project page: https://boxworld-web.github.io.
Zike Wu, Ahmed Hatem, Chintan Trivedi +4
University of British Columbia · Vector Institute for AI · Huawei Canada +1