StateCraft: Augmenting Video World Models with Abstract-State Reasoning
Authors: Lorenzo Venturoli, Nedko Savov, Xi Wang, Marc Pollefeys, Luc Van Gool
Organizations: INSAIT, Sofia University "St. Kliment Ohridski" · ETH Zurich · University of Ulm · TUM · MCML - Munich Center for Machine Learning · Microsoft
Video world models aim to simulate complex environments with realistic appearance and dynamics. Maintaining consistency over long interactions requires tracking past actions and understanding how those actions affect the environment. Video world models can only generate partial observations of the entire world at a time. Remembering past observations alone may not be sufficient in every scenario, as they may no longer reflect the world after a sequence of actions. We introduce a paradigm that decouples a persistent, updatable abstract state from the visual state. The abstract state keeps track of the entities' changes as the world evolves, and is used to condition the visual state, which represents the current view. We propose StateCraft, a novel architecture that pairs a language model with a video diffusion model to evolve the dynamic world. We evaluate our approach on generative interfaces, which pose a substantial challenge because a single action may change parts of the state that are not currently visible. To support training and evaluation, we develop a data collection pipeline that produces informative, densely annotated trajectories at scale without human or agent supervision, together with a benchmark measuring generation quality, memory, and controllability. On this benchmark, our approach improves surpasses the state-of-the-art baselines. We demonstrate that StateCraft maintains high self-consistency and strong generation quality over long trajectories. Because the abstract state is explicit, it can be edited directly, and generation follows edits to states never observed in the data. Finally, a proof-of-concept 3D experiment, in which actions change the illumination of an unobserved room, shows that the paradigm can also be applied to spatial environments. Project page: https://insait-institute.github.io/StateCraft/.
Figures & tables
Figure 1: Abstract and visual state decoupling for consistent video world models. StateCraft (top) couples a VLM that reasons about action effects and updates an abstract state with a video world model that generates the corresponding observations. This state preserves information beyond the current view, enabling consistent folder revisits, while prior methods relying on latent memory (bottom) may generate plausible but inconsistent content.
Figure 2: Overview of StateCraft for generative UI. The abstract-thinking module (left) maintains the abstract state using a graph. Given action ats , observed node ot , and retrieved context nodes ctx1..K , a VLM predicts a state diff and the next node ot+1 . A frozen text encoder maps the predicted node’s serialized components ct+11..M to embeddings ft+11..M . The observation-design module (right) generates video frames conditioned on past frames, action embeddings at−H..te , and component embeddings ft+11..M . Bottom-right panels detail the schema elements.
Figure 3: Autoregressive rollout on a held-out trajectory. The model conditioned on the state signal tracks folders, filenames, and icons across the whole rollout, while NeuralOS keeps a convincing shell but drifts in content and loses high-frequency details.
Model
SSIM ↑
PSNR ↑
ifreq-SSIM ↑
ifreq-PSNR ↑
NeuralOS
0.85
20.92
0.82
19.04
DFoT
0.86
23.27
0.81
19.45
CF++
0.78
18.49
0.79
18.99
StateCraft (Ours; DFoT)
0.92
27.63
0.92
28.06
StateCraft (Ours; CF++)
0.95
28.78
0.95
28.71
Upper bound: ground-truth conditioning
Table 1: Generation quality with first-visit initialization. Our method improves every metric over the unconditioned backbones and NeuralOS, and stays close to the reconstruction quality obtained with ground-truth state updates.
Breadcrumb
Canvas
Side panel
Model
SSIM ↑
PSNR ↑
SSIM ↑
PSNR ↑
SSIM ↑
PSNR ↑
Full region
NeuralOS
0.85
30.12
0.61
19.74
0.96
34.28
CF++
0.77
21.86
0.59
17.51
0.88
23.99
StateCraft (Ours; CF++)
0.91
32.01
0.89
30.80
0.95
33.06
StateCraft (Ours; CF++), GT diffs
0.93
32.20
0.94
32.35
0.95
33.53
Table 2: Render quality by screen region, over the full region (top) and inside OCR-detected text boxes only (bottom). Gains over the baselines are largest on the canvas, the region with the highest variability. Side panel is mostly constant and the models do not need external conditioning to perform well. Last row of each block: upper bound with ground-truth diffs.
Figure 4: Memory across revisit horizons. (a) StateCraft maintains high self-consistency across horizons. Most baselines score around 0.5–0.6, trained MoC declines at the longest horizons. (b) StateCraft preserves target content, whereas NeuralOS generates a plausible interface with hallucinated content.
Edit
Context at the edited frame
Follow
E-PSNR
C-PSNR
Label change (same file type: new name with icon fixed)
Rename
ground truth
0.64
25.1
21.3
Rename
model-generated context
0.92
30.9
22.1
Type change (new file type and name)
Retype
ground truth
0.84
28.4
22.6
Retype
model-generated context
0.86
26.9
24.6
Table 3: Controllability by edit type and context. Edit-following rate and PSNR within edited (E) and unedited control (C) crops. The baseline uses the same model without the state intervention. All predictions are evaluated against the same edited targets using the metrics of Section 3.4 .
Figure 5: Hidden-state conditioned generation. Left: two rollouts along the same path with different state signals. Our model generates the corresponding target-room illumination (last column) while preserving the distractor-room state (second column). The floor plan marks the camera path and four rendered poses. Right: another example from the second held-out apartment.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
DFoT
CF++
Parameters (base / added)
288M / 76M
1.42B / 470M
Training clip length (frames)
32
21
Context at inference (frames)
8
21
Denoising steps
50 (DDIM)
2
Optimizer / learning rate
AdamW / 1×10−4
AdamW / 2×10−6
Batch size
4 × 2 accum × 4 GPU = 32
8 (1 per GPU × 8)
Appendix
Table 4: Training hyperparameters.
Grounded
NeuralOS-style
Blind (our env.)
Recorded actions/transitions
586,210
89,800
7,200
Semantic state-change rate
47.7%
—
—
Pixel-change rate
63.9%
4.0%
21.3%
Interactable-element hit probability
100%
—
47.8%
State-advancing-element hit probability
—
—
28.7%
Unique states in corpus
5,615
—
—
Appendix
Table 5: Collection statistics. Pixel changes use an MSE threshold of 10−2 ; NeuralOS-style rates exclude cursor motion. Dashes indicate unavailable or unreported measurements. Blind-click hit probabilities are geometric estimates over recorded states, not empirical success rates of the blind trajectories. The blind state-count reference ( † ) is a conservative comparison value, not a measured mean.
Figure 6: Visual-change rates across thresholds. Fraction of transitions meeting each pixel-MSE threshold for grounded collection and NeuralOS-style collection, with and without cursor masking. The dotted line indicates the grounded semantic state-change rate (47.7%).
Figure 7: Coverage and click geometry by depth. Left: folder coverage relative to each session’s generated filesystem. Right: probability that a uniform click hits an element observed to advance state.
Figure 8: Observed state accumulation. Unique states against cumulative actions over five shuffled trajectory orders. The dashed line indicates the Chao1 support estimate, S≈6362 .
Figure 9: Exploration under simulated policies. Unique states against action budget, with means and 10th–90th percentile bands over 2,000 walks. The measured grounded count and conservative blind reference are shown for comparison. Simulations use corpus-estimated navigation transitions and omit dialog traps.
Figure 10: Revisit reconstruction correctness. ROI-level SSIM against ground truth for the designated target. StateCraft maintains high reconstruction accuracy across horizons, complementing the self-consistency results in Figure 4 .
Breadcrumb
Canvas
Side panel
Model
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
Full region
NeuralOS
0.85
30.12
0.61
19.74
0.96
34.28
DFoT
0.83
31.27
0.62
21.48
0.95
34.71
StateCraft (DFoT)
0.91
32.88
0.89
31.83
0.92
33.81
StateCraft (DFoT), GT diffs
0.93
33.41
0.92
32.87
0.92
33.92
Appendix
Table 6: Render quality by screen region for the DFoT backbone, over the full region (top) and inside OCR-detected text boxes only (bottom). Gains over the baselines are largest on the canvas, where filesystem state is displayed, and largest of all on canvas text, which the baselines fail to render legibly. On the static side panel, all models perform comparably. Last row of each block: upper bound with ground-truth diffs.
In-domain
OOD
Model
Node
Trans.
Schema
Ops
Games
TPS
Qwen3.5-0.8B [ Qwen Team, 2026 ]
FT
0.86
0.96
0.91
0.00
0.18
250
FS
0.48
0.52
0.85
1.00
0.83
250
DiffusionGemma-26B [ DiffusionGemma Team et al., 2026 ]
FT
0.77
0.85
0.79
1.00
0.86
1500
FS
0.34
0.40
0.70
1.00
0.85
1500
Appendix
Table 7: State predictor evaluation: finetuned (FT) against base with few-shot prompts (FS), both backbones on the same corpus and schedule. In-domain: node accuracy, transition accuracy and schema validity on held-out navigation. OOD: node accuracy on unseen operations and cross-domain games. Finetuning Qwen3.5-0.8B trades generality for in-domain accuracy; DiffusionGemma-26B scores are lower over the task-specific metrics, but it maintains generalization capabilities and decodes faster.
Region / lag
None
Path
Flat-node
Graph (full)
Full-region reconstruction: SSIM / PSNR
Breadcrumb
0.83 / 31.27
0.92 / 31.18
0.91 / 32.55
0.93 / 33.41
Canvas
0.62 / 21.48
0.62 / 21.80
0.76 / 25.92
0.92 / 32.87
Side panel
0.95 / 34.71
0.94 / 34.29
0.93 / 33.70
0.92 / 33.92
Text reconstruction: SSIM / PSNR
Breadcrumb
0.39 / 29.51
0.80 / 31.65
0.41 / 30.95
0.79 / 31.60
Appendix
Table 8: Conditioning representation ablation on DFoT. Reconstruction cells report SSIM / PSNR (dB); text scores use OCR-detected boxes. Memory scores measure revisit similarity by action lag, with approximately 0.5 indicating plausible but unrelated content. None denotes no state conditioning. All models receive actions, and all conditioning variants use ground-truth states, isolating the representation supplied to the video model. Best values are bold and second-best underlined, including ties.
CF++
DFoT
SSIM ↑
full
− act.
− graph
full
− act.
− graph
Whole frame
0.934
0.895
0.824
0.929
0.889
0.857
Canvas
0.897
0.836
0.564
0.907
0.856
0.616
Breadcrumb
0.890
0.855
0.759
0.941
0.922
0.833
Semantic (all)
0.891
0.859
0.788
0.946
0.916
0.876
Revisit k=64
0.878
0.874
0.656
0.898
0.806
0.688
Appendix
Table 9: Effect of the different conditioning signals on the main metrics. Each channel is removed at inference by zeroing it, on the same 240 test trajectories with an 8-frame ground-truth seed and closed-loop generation. Graph tokens carry what is on screen: removing them costs the canvas 0.33 (CF++) and 0.29 (DFoT) SSIM and roughly a fifth of revisit memory, and drops breadcrumb agreement with the ground-truth path by 14–19 points. Removing the action channel is far cheaper on both backbones. Compare columns within a backbone: since the ablation is meant to compare the effectiveness of the components within an architecture, both CF++ and DFoT are scored here at their native resolution ( 640×480 and 256×192 ).
Features
Dim.
AUROC
AUROC 2
Acc5
F1
Acc5lin
Skip
Purity
Majority class
–
0.500
0.500
0.496
0.133
0.496
–
–
Action
8
0.944
0.919
0.790
0.736
0.790
0.329
0.981
Symbolic features
26
0.965
0.941
0.835
0.805
0.801
0.395
0.991
Graph conditioning
2056
0.978
0.960
0.870
0.820
0.819
0.397
0.990
Raw pixels
200
0.985
0.974
0.911
0.882
0.869
0.427
0.991
Denoiser latents
4616
0.987
0.975
0.916
0.898
0.901
0.410
0.990
Appendix
Table 10: Transition probing across feature families. Results on 35,880 test decisions from trajectories disjoint from training and calibration. All representations include eight action features. We report MLP results and logistic-regression accuracy ( Acc5lin ). AUROC is five-class one-vs-rest macro AUROC; AUROC 2 measures change versus no change. Acc5 is five-class accuracy and F1 is macro-F1. Skip is the fraction of omitted calls; purity is the fraction of skips requiring no state change. Thresholds calibrated for 99% purity are applied unchanged to test. These are held-out probe results, closed-loop results are reported in the text.
Figure 11: Controllability interventions by type (multiple edits, new element on background, rename with type change). Per row: original state (left), commanded graph diff, removals in red and additions in green (centre), and the causally generated frame for the edited state, which never occurs in the data (right; red boxes mark edited regions). The render follows the state edit rather than the pixels of its own context.
Video world models enable interactive exploration of dynamic environments, yet struggle to respect prior observations over long horizons and across viewpoints. We present WorldCrafter, a video world model that learns a camera-queryable implicit 3D-aware memory for this purpose. The key insight is to let the requested viewpoint shape how multi-view evidence is compressed into the video generator's limited token budget. Trained jointly with the video generator, a memory encoder and pose-conditioned readout module integrate historical observations into a fixed set of target view-specific tokens before denoising, without explicit depth-based correspondences. By combining this memory with recent temporal context and few-step distillation, WorldCrafter enables streaming scene exploration from a single input image or text prompt. Experiments across static and dynamic scenes show substantial gains in long-horizon consistency and camera-control accuracy while preserving visual quality during minute-scale exploration.
Recent video world models generate increasingly realistic and interactive visual experiences, yet lack reliable mechanisms for maintaining persistent world state and enforcing programmable rules over extended interactions. We introduce Programmable World Model, a framework that decouples world-state evolution from visual observation generation. An agent translates natural-language instructions into executable programs that specify entity states and state-transition rules, enabling direct control over individual entities and their interactions. A lightweight engine executes these programs to update and maintain an explicit, persistent global world state, including off-screen entities and non-visual attributes. To connect world state with visual generation, we introduce state-augmented 3D oriented bounding boxes (OBBs) as an intermediate representation. This representation, together with the target camera trajectory, is deterministically compiled into pixel-aligned spatiotemporal conditioning signals for a pretrained video model serving as the generative renderer. This design allows users to create playable games with predefined mechanics, direct control over individual entities, and persistent world state throughout gameplay. We further introduce CombatStateBench, a benchmark for evaluating programmable world models. On CombatStateBench, our method achieves 94% Count Accuracy and 98% State Accuracy, substantially outperforming existing interactive video world models while supporting coherent long-horizon generation. These results demonstrate the effectiveness of separating explicit state evolution from generative rendering for building persistent, programmable worlds.
Existing world models can generate action-conditioned video from a single image. Many focus on camera movement, allowing users to explore the generated scene. However, navigation alone does not ensure that an action follows persistent rules when its outcome depends on state that is invisible in the current frame. We present Stateful Harness, a framework that separates world evolution from video rendering to support stateful and rule-governed interaction. It maintains an evolving world state under specified rules, renders each resulting event with a frozen video model, and uses the generated clip to update the world state for the next turn. At each turn, the harness resolves the action selected by the user and compiles the resulting event into an instruction, while a preview clip from an editable 3D scene provides spatial and camera guidance. To evaluate rule compliance, we build WorldState-Bench, a benchmark of 50 counterfactual twin pairs that share an initial frame and action but differ in hidden state and required outcome. Under rule-only prompting, the best of the 11 evaluated video backends produces the required endings for both cases in 32% of pairs. For MiniMax-H3, specifying the required endings raises this rate from 32% to 64%, while the full harness reaches 92% with the same frozen model. These results support separating rule execution from video rendering for world interactions with hidden state.
Chan Yan Ki, Yanran Zhang, Wenzhao Zheng +5
Department of Automation, Tsinghua University, China · GigaAI