State World Models: Execute the Rules, Generate the Pixels

Chan Yan Ki; Yanran Zhang; Wenzhao Zheng; Xiaofeng Wang; Zheng Zhu; Lei Chen; Jie Zhou; Jiwen Lu

cs.CVOct 19, 2026

State World Models: Execute the Rules, Generate the Pixels

Authors: Chan Yan Ki, Yanran Zhang, Wenzhao Zheng, Xiaofeng Wang, Zheng Zhu, Lei Chen, Jie Zhou, Jiwen Lu

Organizations: Department of Automation, Tsinghua University, China · GigaAI

Abstract

Existing world models can generate action-conditioned video from a single image. Many focus on camera movement, allowing users to explore the generated scene. However, navigation alone does not ensure that an action follows persistent rules when its outcome depends on state that is invisible in the current frame. We present Stateful Harness, a framework that separates world evolution from video rendering to support stateful and rule-governed interaction. It maintains an evolving world state under specified rules, renders each resulting event with a frozen video model, and uses the generated clip to update the world state for the next turn. At each turn, the harness resolves the action selected by the user and compiles the resulting event into an instruction, while a preview clip from an editable 3D scene provides spatial and camera guidance. To evaluate rule compliance, we build WorldState-Bench, a benchmark of 50 counterfactual twin pairs that share an initial frame and action but differ in hidden state and required outcome. Under rule-only prompting, the best of the 11 evaluated video backends produces the required endings for both cases in 32% of pairs. For MiniMax-H3, specifying the required endings raises this rate from 32% to 64%, while the full harness reaches 92% with the same frozen model. These results support separating rule execution from video rendering for world interactions with hidden state.

Figures & tables

Figure 1: State World Models with Stateful Harness . The same initial frame and action can lead to different outcomes under different world states. The harness executes predefined rules to determine the event before a frozen video model renders it, illustrated by the red and blue elemental states. The generated clip provides the visual anchor for the next turn.

Figure 2: Counterfactual Pairs Reveal the Role of Hidden State. The two cases share the first frame $I_{0}$ and action $a_{t}$ but differ in the hidden flag has_key . In the illustrated failure, an unwrapped video model opens the chest in both cases, violating the rule when no key is available. The harness maintains semantic and spatial state, executes the shared rule to determine whether the chest remains locked or opens, and renders each outcome with the same frozen video model.

Figure 3: The Stateful Harness . In Stage 0 , a coding agent constructs an executable game definition specifying the state, actions, rules, events, goals, and visual semantics. In Stage 1 , the runtime evaluates the rules, updates the state, and determines the event and goal status for each action. The resolved event is compiled into an instruction for the frozen video model $V_{\theta}$ . Visual observations of the generated clip inform the state used in the next turn. Adaptation to a new game changes the game definition while retaining the shared runtime and video model weights.

	A. Rule-only probe			B. Rule+end probe
Backend	EndFollow $\uparrow$	PairFollow $\uparrow$	TwinContrast $\uparrow$	Strict $\uparrow$	CntAbsent $\uparrow$	PairStrict $\uparrow$
Image-to-video generators
Wan2.2 A14B ( Wan et al., 2025 )	65 %	32 %	38 %	54 %	63 %	26 %
LongCat-Video 13.6B ( LongCat Team et al., 2025 )	47 %	8 %	24 %	46 %	62 %	8 %
HunyuanVideo 1.5 ( Wu et al., 2025 )	57 %	20 %	26 %	29 %	30 %	4 %
MiniMax-H3 ( Chen et al., 2026a )	66 %	32 %	32 %	29 %	30 %	4 %

Table 1: Main results on WorldState-Bench under Rule-only instructions (Track A) and Rule+end instructions (Track B). Bold indicates the best result and underline the best among the unwrapped baselines. The highlighted row reports Stateful Harness with text and first-frame reference inputs. Rows within a track share the probe, first frame, and judge. The matched comparison for the highlighted row is the same backbone under the Rule+end probe (Appendix Table 4 ).

Figure 4: Rendered twin cases under the two probes. Each cell contains four evenly spaced frames from a 5 s clip. A ✓indicates that only the required ending appears, while a $\times$ indicates forbidden content or a missing required ending. The illustrated baselines reproduce the same ending across a pair or include forbidden content, whereas the harness generates the distinct required outcomes.

Panel of Figure 5 — Figure 5: Human battles won by Stateful Harness


Metric group	Text-driven											Camera-controlled						Action-conditioned							Ours
	Seedance 1.5	Wan 2.7	Kling 3.0	YUME 1.5	HY-Video 1.5	LTX 2.3	LongCat-Video	Kairos 3.0	Cosmos 2.5	H3 (official)	Average	LingBot-World	HY-World 1.5	Fantasy-World	InSpatio-World	Astra	Average	Happy Oyster	Matrix-Game 3.0	Genie 3	Matrix-Game 2.0	HY-GameCraft	Infinite-World	Average	Ours

Video quality (6)	82.0	81.0	80.0	77.6	76.7	77.0	77.2	73.1	71.7	77.9	77.4	78.9	78.1	72.4	71.5	67.1	73.6	77.3	75.5	75.2	73.8	73.0	77.0	75.3	78.0
Setting (2)	82.9	91.4	91.0	72.4	85.6	85.2	72.3	70.3	83.3	71.1	81.6	72.6	72.2	71.3	71.4	59.6	69.4	74.2	63.6	72.5	67.1	66.6	69.3	68.9	87.2
Interaction (4)	67.9	71.7	72.8	48.2	54.6	49.0	44.8	41.4	43.2	68.9	54.8	–	–	–	–	–	–	–	–	–	–	–	–	–	77.7

Table 2: WBench averages for the twenty models of Ying et al. (2026) plus our two entries (excluded from class averages). Text entries use the full split, camera/action the navigation split. Gray indicates class averages, and “–” denotes undefined values.

Panel of Figure 6 — Figure 6: Turn sequences on the same frozen weights. All three rows are real harness episodes, with two adjacent frames per chunk at native aspect, each chunk headed by the video’s own banner and HUD state (stamina and quest progress at the top, affection, relationship, and mood in the middle, and arbiter HP, eve HP, and mana at the bottom). Elided turns are named on the arrows.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Prompt example for the orb pair, twin A. The panels show the Rule-only prompt, the Rule+end prompt with its appended outcome instruction highlighted, and the compiled VisualScript instruction. The annotations below the panels describe the rule program and settled state.

Figure 8: Prompt example for the sign pair, twin A, using the layout of Figure 7 . Highlighted clauses identify the appended Rule+end sentence and the ending specified by the compiled instruction. The annotations below the panels show the corresponding rule program and settled state.

Opponent (Rule-only)	Ours win	Split
MiniMax-H3 baseline	96 %	4 %
AlayaWorld	100 %	0 %
Echo-WM Flash	100 %	0 %
Wan2.2 A14B	96 %	4 %
All opponents	98 %	2 %

Appendix

Table 3: Human preference for the full harness in pairwise comparisons with each baseline. Each comparison receives two ratings using randomized positions and forced choice without a tie option. The remaining comparisons contain split auditor votes, and none favors a baseline under the aggregate decision rule.

Figure 9: Clip-verification interface (auditor view). The left panel displays the required ending, forbidden outcome, and questions Q1 and Q2 with an “unclear” option. The right panel displays the clip at 2 $\times$ speed. System identity and probe are hidden.

Figure 10: Interface for human pairwise preference evaluation. The required and forbidden outcomes for both hidden states appear above a grid of four videos. Auditors select the system that more accurately follows the specified rules. System positions are randomized.

	EndFollow $\uparrow$	PairFollow $\uparrow$	TwinContrast $\uparrow$	Strict $\uparrow$	CntAbsent $\uparrow$	PairStrict $\uparrow$	Unclear $\downarrow$
Rule-only probe
Wan2.2 A14B	65	32	38	45	54	14	4
LongCat-Video 13.6B	47	8	24	39	57	0	14
HunyuanVideo 1.5	57	20	26	32	36	6	1
MiniMax-H3	66	32	32	26	28	0	0
LingBot-World v2	48	16	38	39	69	8	21

Appendix

Table 4: Results under both instruction conditions on WorldState-Bench. All metrics are percentages, and pair metrics are computed over 50 pairs. The harness uses one compiled instruction, derived from the case program and independent of probe wording, that always specifies the required ending.

	$\Delta$ Strict	Leakage-only	Direction err.	CopyPaste	Branch gap	PairStrict
		R-only $\to$ R+E	R-only $\to$ R+E	R+E	R+E	R+E
External backends
Wan2.2 A14B	$+9$	20 $\to$ 18	36 $\to$ 28	16	0.24	13/50
LongCat-Video 13.6B	$+7$	8 $\to$ 8	53 $\to$ 47	16	0.48	4/50
HunyuanVideo 1.5	$-3$	25 $\to$ 34	42 $\to$ 38	36	0.22	2/50
MiniMax-H3	$+3$	40 $\to$ 53	34 $\to$ 18	14	0.14	2/50

Appendix

Table 5: Failure diagnostics corresponding to Table 4 .

\Delta

Strict is the difference between Rule+end and Rule-only Strict . Leakage and direction errors are shown for both conditions in that order. Copy-paste, branch gap, and PairStrict are reported under Rule+end for 50 pairs.

Panel of Figure 11 — Figure 11: Change in Strict from Rule-only to Rule+end on WorldState-Bench (Table 5 ). Among the ten backends other than MiniMax-H3, six improve (green) and four decline (red). MiniMax-H3 (blue) provides the frozen weights used by the harness, and the dashed line indicates the mean change of $+2.3$ across the other ten backends.

Condition	Spatial ref.	CopyPaste	TwinContrast	Direction err.	Leakage-only	Branch gap
Rule-only	none	34.0	32.0	34.0	40.0	0.20
Rule+end	none	14.0	64.0	18.0	53.0	0.14
Rule-only	first frame	30.0	46.0	29.0	44.0	0.14
Rule+end	first frame	18.0	66.0	18.0	53.0	0.10
Harness	none	0.0	96.0	6.0	5.0	0.02
Harness	first frame	2.0	96.0	4.0	1.0	0.02

Appendix

Table 6: Counterfactual pair diagnostics for MiniMax-H3 under different spatial reference configurations. All values are percentages computed over 50 pairs. The deployed configuration uses a first-frame reference.

Panel of Figure 12 — Figure 12: PairStrict and TwinContrast under Rule+end instructions (Table 4 ). Distinct endings do not necessarily imply strict compliance. The harness, marked by a star, achieves high scores on both metrics.


Group	Metric	Text-driven											Camera-controlled						Action-conditioned
		Seedance 1.5	Wan 2.7	Kling 3.0	YUME 1.5	HY-Video 1.5	LTX 2.3	LongCat-Video	Kairos 3.0	Cosmos 2.5	H3 (official)	Average	LingBot-World	HY-World 1.5	Fantasy-World	InSpatio-World	Astra	Average	Happy Oyster	Matrix-Game 3.0	Genie 3	Matrix-Game 2.0	HY-GameCraft	Infinite-World	Average	Ours

Video Quality	Aesthetic	59.7	59.6	61.3	59.3	61.9	56.9	64.7	58.4	60.1	63.1	60.2	66.9	60.1	63.0	64.4	48.6	60.6	56.6	46.4	51.6	54.0	52.6	58.7	53.3	59.1
	Imaging	69.8	68.1	67.7	65.7	67.4	62.3	69.8	63.6	67.2	66.2	66.8	67.9	65.4	62.8	67.6	52.5	63.2	63.9	70.0	59.3	60.3	58.7	66.1	63.1	64.6
	Flickering	93.4	93.0	94.5	94.8	95.5	94.1	94.9	96.3	96.0	96.2	94.7	94.1	93.5	95.8	96.0	96.0	95.1	94.0	86.3	95.0	94.6	93.7	94.1	93.0	95.7

Appendix

Table 7: WBench per-metric grid, scores as reported by Ying et al. 2026 (gray

=

class average). Our two configurations are excluded from the class averages. Their dimension averages correspond to the leaderboard scores in Table 2 . For our entry, the Interaction average is the official leaderboard value and is not the mean of the sub-scores shown.

	Navigation	Full split
Configuration	Overall	Quality $\uparrow$	Setting $\uparrow$	Interaction $\uparrow$	Consistency $\uparrow$	Physical $\uparrow$	Overall $\uparrow$
H3 (raw text protocol)	74.7	77.9	71.1	68.9	83.8	69.3	74.2
H3 (raw text protocol) + control video	65.7	79.3	75.3	55.7	67.1	58.2	67.1
H3 + harness, first-frame reference	76.8	77.6	89.6	69.0	75.3	69.0	76.1
Stateful Harness	79.0	78.0	87.2	77.7	77.8	68.8	77.9

Appendix

Table 8: Reference channel ablation on WBench with frozen MiniMax-H3 weights. The configurations compare first-frame anchoring and spatial preview conditioning, including the same preview supplied without the harness. Bold indicates the best overall score on each split.

Configuration	Quality $\uparrow$	Setting $\uparrow$	Nav. traj. $\uparrow$	Consistency $\uparrow$	Physical $\uparrow$	Overall $\uparrow$
H3 (raw text protocol)	77.5	72.7	68.9	85.0	69.4	74.7
H3 (raw text protocol) + control video	73.3	53.0	79.3	64.8	58.0	65.7
H3 + harness, first-frame reference	79.5	91.5	69.0	75.0	69.0	76.8
Stateful Harness	80.1	89.1	77.7	79.2	68.8	79.0

Appendix

Table 9: Navigation split dimensions for the reference channel ablation using frozen MiniMax-H3 weights and WBench leaderboard scores. The interaction column is the navigation trajectory metric.

Figure 13: Four additional twin pairs under the Rule-only probe, in the format of Figure 4 (badge conventions as there). The harness depicts only the required ending in all eight illustrated clips.

Figure 14: Four twin pairs under the Rule+end probe, same format as Figure 13 . The harness depicts only the required ending in all eight illustrated clips.

Panel of Figure 15 — Figure 15: The full fifteen-act town-exploration episode of Figure 6 (Case A), released video of our harness on frozen MiniMax-H3. Two frames per act (native 16:9), with headers giving the act and the Traveler HUD (stamina, status, quest progress) at the act’s first frame. Stamina drains 28 $\to$ 0 on the walk, the status line switches to “Exhausted — slow walk only”, and rest restores it to 60. A second drain-and-recovery cycle follows before the backstage door, while the quest counter advances only forward, 1/3 to 2/3 to 3/3.

Panel of Figure 16 — Figure 16: The full twenty-nine-turn date episode of Figure 6 (Case B), released video of our harness on frozen MiniMax-H3. Two frames per turn (native 16:9), with headers giving the turn, scene, and HUD state (affection, relationship, mood) at the turn’s first frame. Affection rises monotonically 47 $\to$ 75, one point per turn. The relationship state flips friends $\to$ flirting at turn 14 and flirting $\to$ lovers at turn 29, and the closing confession is accepted only under this accumulated state.

Panel of Figure 17 — Figure 17: The full eleven-turn boss-duel episode of Figure 6 (Case C), released video of our harness on frozen MiniMax-H3. Two frames per turn (native 16:9), with headers giving the skill and the HUD state (arbiter HP, eve HP, mana) at the turn’s first frame. The arbiter’s HP drops monotonically 128 $\to$ 0 across the eleven turns. Turn 8 plays the defensive Gate of the End, and the closing Falling Red Sun ends the duel in the victory screen.

Step	Operation	Input $\to$ output
0	setup (once)	domain declaration $\to$ runtime registry, service preflight, initial anchor $f_{0}$
1	gate check	terminal status, turn budget, legal actions $\to$ continue or stop
2	action selection	$s_{t}$ , legal actions $\to a_{t}$ from a scripted, random, human, or LLM player
3	settle	$s_{t},a_{t}\to(\bar{z}_{t+1},\epsilon_{t},e_{t})$ through Eq. 2
4	direct	$\bar{s}_{t+1},e_{t}\to$ VisualScript script
5	compile	script and scene $\to(p_{t},r_{t})$ , with invalid records rejected before generation

Appendix

Table 10: The full turn lifecycle with per-step inputs and outputs. Step 0 initializes the world, steps 1 to 9 repeat for each turn, and steps 10 and 11 produce the final presentation. The table extends Algorithm 1 with the harness operations surrounding the render call.

Slot	Content
style	one sentence fixing art style, palette, and quality register
first-frame anchor	description that must match the anchor image
camera	camera motion, drawn from a fixed phrase vocabulary
actor (+ appearance)	acting entity and appearance attributes that preserve identity across scenes
action	what the actor does, with prepositions
target (+ reaction)	the acted-upon party and its response

Appendix

Table 11: The VisualScript record: typed slots filled by the game layer, each with a machine-checked validation rule.

	State representation	Control interface	Spatial state	Evaluation
PWM ( Huang et al., 2026 )	engine and 3D OBB IR	custom pixel-level conditioning	per-segment signal	combat count/death
CWM ( Chen et al., 2026b )	coding-agent code and proxy video	fine-tuning	geometry constraint (proxy)	qualitative adherence
WorldMind ( Deng et al., 2026 )	recovered from frames	Wan backend	recovered from pixels	human preference
Unbounded ( Li et al., 2025a )	character-life variables	language router	none (2D narrative)	user study (20 raters)
Ours	ledger + VisualScript IR + clay model	native ref. + compiled text	persistent scene asset	Change $\to$ Comply twins

Appendix

Table 12: Positioning against concurrent explicit-state systems. “Native ref.” denotes the standard reference inputs of a frozen model.

World Models

World models represent and predict how environments change, helping agents plan, act, and reason about possible futures.

See all 358 papers

StateCraft: Augmenting Video World Models with Abstract-State Reasoning
Oct 14, 2026Lorenzo Venturoli, Nedko Savov, Xi Wang +2Video World Models Long-Horizon Video Generation
INSAIT, Sofia University "St. Kliment Ohridski" · ETH Zurich · University of Ulm +3
Programmable World Model
Sep 9, 2026Zheng-Hui Huang, Guixu Lin, Jiacheng Lin +8World Models Controllable Video Generation
Alaya Lab
Oneira: From Open-Ended Generation to Open-World Interaction in Video World Models
Oct 1, 2026Xindi Yang, Baolu Li, Liam Lee +8Long-Horizon Video Generation Action-Conditioned World Models
Monash University · Dalian University of Technology · The Chinese University of Hong Kong, Shenzhen +2

Oct 14, 2026J/K move · Enter open · S save

StateCraft: Augmenting Video World Models with Abstract-State Reasoning

Lorenzo Venturoli, Nedko Savov, Xi Wang, Marc Pollefeys+1

INSAIT, Sofia University "St. Kliment Ohridski" · ETH Zurich · University of Ulm+3

Video World Models Long-Horizon Video Generation Memory-Augmented Video Generation Action-Conditioned Video Generation Video Diffusion Models

Open paper AskPDF arXiv

Video world models aim to simulate complex environments with realistic appearance and dynamics. Maintaining consistency over long interactions requires tracking past actions and understanding how those actions affect the environment. Video world models can only generate partial observations of the entire world at a time. Remembering past observations alone may not be sufficient in every scenario, as they may no longer reflect the world after a sequence of actions. We introduce a paradigm that decouples a persistent, updatable abstract state from the visual state. The abstract state keeps track of the entities' changes as the world evolves, and is used to condition the visual state, which represents the current view. We propose StateCraft, a novel architecture that pairs a language model with a video diffusion model to evolve the dynamic world. We evaluate our approach on generative interfaces, which pose a substantial challenge because a single action may change parts of the state that are not currently visible. To support training and evaluation, we develop a data collection pipeline that produces informative, densely annotated trajectories at scale without human or agent supervision, together with a benchmark measuring generation quality, memory, and controllability. On this benchmark, our approach improves surpasses the state-of-the-art baselines. We demonstrate that StateCraft maintains high self-consistency and strong generation quality over long trajectories. Because the abstract state is explicit, it can be edited directly, and generation follows edits to states never observed in the data. Finally, a proof-of-concept 3D experiment, in which actions change the illumination of an unobserved room, shows that the paradigm can also be applied to spatial environments. Project page: https://insait-institute.github.io/StateCraft/.

Abstract

World Models

Explore similar work

StateCraft: Augmenting Video World Models with Abstract-State Reasoning

Programmable World Model

Oneira: From Open-Ended Generation to Open-World Interaction in Video World Models