StateCraft: Augmenting Video World Models with Abstract-State Reasoning

Lorenzo Venturoli; Nedko Savov; Xi Wang; Marc Pollefeys; Luc Van Gool

cs.CVOct 14, 2026

StateCraft: Augmenting Video World Models with Abstract-State Reasoning

Authors: Lorenzo Venturoli, Nedko Savov, Xi Wang, Marc Pollefeys, Luc Van Gool

Organizations: INSAIT, Sofia University "St. Kliment Ohridski" · ETH Zurich · University of Ulm · TUM ·

Abstract

Video world models aim to simulate complex environments with realistic appearance and dynamics. Maintaining consistency over long interactions requires tracking past actions and understanding how those actions affect the environment. Video world models can only generate partial observations of the entire world at a time. Remembering past observations alone may not be sufficient in every scenario, as they may no longer reflect the world after a sequence of actions. We introduce a paradigm that decouples a persistent, updatable abstract state from the visual state. The abstract state keeps track of the entities' changes as the world evolves, and is used to condition the visual state, which represents the current view. We propose StateCraft, a novel architecture that pairs a language model with a video diffusion model to evolve the dynamic world. We evaluate our approach on generative interfaces, which pose a substantial challenge because a single action may change parts of the state that are not currently visible. To support training and evaluation, we develop a data collection pipeline that produces informative, densely annotated trajectories at scale without human or agent supervision, together with a benchmark measuring generation quality, memory, and controllability. On this benchmark, our approach improves surpasses the state-of-the-art baselines. We demonstrate that StateCraft maintains high self-consistency and strong generation quality over long trajectories. Because the abstract state is explicit, it can be edited directly, and generation follows edits to states never observed in the data. Finally, a proof-of-concept 3D experiment, in which actions change the illumination of an unobserved room, shows that the paradigm can also be applied to spatial environments. Project page: https://insait-institute.github.io/StateCraft/.

Figures & tables

Figure 1: Abstract and visual state decoupling for consistent video world models. StateCraft (top) couples a VLM that reasons about action effects and updates an abstract state with a video world model that generates the corresponding observations. This state preserves information beyond the current view, enabling consistent folder revisits, while prior methods relying on latent memory (bottom) may generate plausible but inconsistent content.

Figure 2: Overview of StateCraft for generative UI. The abstract-thinking module (left) maintains the abstract state using a graph. Given action $a^{s}_{t}$ , observed node $o_{t}$ , and retrieved context nodes $\mathrm{ctx}_{1..K}$ , a VLM predicts a state diff and the next node $o_{t+1}$ . A frozen text encoder maps the predicted node’s serialized components $c^{1..M}_{t+1}$ to embeddings $f^{1..M}_{t+1}$ . The observation-design module (right) generates video frames conditioned on past frames, action embeddings $a^{e}_{t-H..t}$ , and component embeddings $f^{1..M}_{t+1}$ . Bottom-right panels detail the schema elements.

Figure 3: Autoregressive rollout on a held-out trajectory. The model conditioned on the state signal tracks folders, filenames, and icons across the whole rollout, while NeuralOS keeps a convincing shell but drifts in content and loses high-frequency details.

Upper bound: ground-truth conditioning
Model	SSIM $\uparrow$	PSNR $\uparrow$	ifreq-SSIM $\uparrow$	ifreq-PSNR $\uparrow$
NeuralOS	0.85	20.92	0.82	19.04
DFoT	0.86	23.27	0.81	19.45
CF++	0.78	18.49	0.79	18.99
StateCraft (Ours; DFoT)	0.92	27.63	0.92	28.06
StateCraft (Ours; CF++)	0.95	28.78	0.95	28.71

Table 1: Generation quality with first-visit initialization. Our method improves every metric over the unconditioned backbones and NeuralOS, and stays close to the reconstruction quality obtained with ground-truth state updates.

Full region
	Breadcrumb		Canvas		Side panel
Model	SSIM $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$
NeuralOS	0.85	30.12	0.61	19.74	0.96	34.28
CF++	0.77	21.86	0.59	17.51	0.88	23.99
StateCraft (Ours; CF++)	0.91	32.01	0.89	30.80	0.95	33.06
StateCraft (Ours; CF++), GT diffs	0.93	32.20	0.94	32.35	0.95	33.53

Table 2: Render quality by screen region, over the full region (top) and inside OCR-detected text boxes only (bottom). Gains over the baselines are largest on the canvas, the region with the highest variability. Side panel is mostly constant and the models do not need external conditioning to perform well. Last row of each block: upper bound with ground-truth diffs.

Panel of Figure 4 — Figure 4: Memory across revisit horizons. (a) StateCraft maintains high self-consistency across horizons. Most baselines score around 0.5–0.6, trained MoC declines at the longest horizons. (b) StateCraft preserves target content, whereas NeuralOS generates a plausible interface with hallucinated content.

Label change (same file type: new name with icon fixed)
Edit	Context at the edited frame	Follow	E-PSNR	C-PSNR
Rename	ground truth	0.64	25.1	21.3
Rename	model-generated context	0.92	30.9	22.1
Type change (new file type and name)
Retype	ground truth	0.84	28.4	22.6
Retype	model-generated context	0.86	26.9	24.6

Table 3: Controllability by edit type and context. Edit-following rate and PSNR within edited (E) and unedited control (C) crops. The baseline uses the same model without the state intervention. All predictions are evaluated against the same edited targets using the metrics of Section 3.4 .

Figure 5: Hidden-state conditioned generation. Left: two rollouts along the same path with different state signals. Our model generates the corresponding target-room illumination (last column) while preserving the distractor-room state (second column). The floor plan marks the camera path and four rendered poses. Right: another example from the second held-out apartment.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

	DFoT	CF++
Parameters (base / added)	288M / 76M	1.42B / 470M
Training clip length (frames)	32	21
Context at inference (frames)	8	21
Denoising steps	50 (DDIM)	2
Optimizer / learning rate	AdamW / $1\times 10^{-4}$	AdamW / $2\times 10^{-6}$
Batch size	4 $\times$ 2 accum $\times$ 4 GPU = 32	8 (1 per GPU $\times$ 8)

Appendix

Table 4: Training hyperparameters.

	Grounded	NeuralOS-style	Blind (our env.)
Recorded actions/transitions	586,210	89,800	7,200
Semantic state-change rate	47.7%	—	—
Pixel-change rate	63.9%	4.0%	21.3%
Interactable-element hit probability	100%	—	47.8%
State-advancing-element hit probability	—	—	28.7%
Unique states in corpus	5,615	—	—

Appendix

Table 5: Collection statistics. Pixel changes use an MSE threshold of

10^{-2}

; NeuralOS-style rates exclude cursor motion. Dashes indicate unavailable or unreported measurements. Blind-click hit probabilities are geometric estimates over recorded states, not empirical success rates of the blind trajectories. The blind state-count reference (

\dagger

) is a conservative comparison value, not a measured mean.

Figure 6: Visual-change rates across thresholds. Fraction of transitions meeting each pixel-MSE threshold for grounded collection and NeuralOS-style collection, with and without cursor masking. The dotted line indicates the grounded semantic state-change rate (47.7%).

Figure 7: Coverage and click geometry by depth. Left: folder coverage relative to each session’s generated filesystem. Right: probability that a uniform click hits an element observed to advance state.

Figure 8: Observed state accumulation. Unique states against cumulative actions over five shuffled trajectory orders. The dashed line indicates the Chao1 support estimate, $\widehat{S}\approx 6362$ .

Figure 9: Exploration under simulated policies. Unique states against action budget, with means and 10th–90th percentile bands over 2,000 walks. The measured grounded count and conservative blind reference are shown for comparison. Simulations use corpus-estimated navigation transitions and omit dialog traps.

Figure 10: Revisit reconstruction correctness. ROI-level SSIM against ground truth for the designated target. StateCraft maintains high reconstruction accuracy across horizons, complementing the self-consistency results in Figure 4 .

Full region
	Breadcrumb		Canvas		Side panel
Model	SSIM	PSNR	SSIM	PSNR	SSIM	PSNR
NeuralOS	0.85	30.12	0.61	19.74	0.96	34.28
DFoT	0.83	31.27	0.62	21.48	0.95	34.71
StateCraft (DFoT)	0.91	32.88	0.89	31.83	0.92	33.81
StateCraft (DFoT), GT diffs	0.93	33.41	0.92	32.87	0.92	33.92

Appendix

Table 6: Render quality by screen region for the DFoT backbone, over the full region (top) and inside OCR-detected text boxes only (bottom). Gains over the baselines are largest on the canvas, where filesystem state is displayed, and largest of all on canvas text, which the baselines fail to render legibly. On the static side panel, all models perform comparably. Last row of each block: upper bound with ground-truth diffs.

Model		Node	Trans.	Schema	Ops	Games	TPS
		In-domain			OOD
Qwen3.5-0.8B [ Qwen Team, 2026 ]	FT	0.86	0.96	0.91	0.00	0.18	250
	FS	0.48	0.52	0.85	1.00	0.83	250
DiffusionGemma-26B [ DiffusionGemma Team et al., 2026 ]	FT	0.77	0.85	0.79	1.00	0.86	1500
	FS	0.34	0.40	0.70	1.00	0.85	1500

Appendix

Table 7: State predictor evaluation: finetuned (FT) against base with few-shot prompts (FS), both backbones on the same corpus and schedule. In-domain: node accuracy, transition accuracy and schema validity on held-out navigation. OOD: node accuracy on unseen operations and cross-domain games. Finetuning Qwen3.5-0.8B trades generality for in-domain accuracy; DiffusionGemma-26B scores are lower over the task-specific metrics, but it maintains generalization capabilities and decodes faster.

Full-region reconstruction: SSIM / PSNR
Region / lag	None	Path	Flat-node	Graph (full)
Breadcrumb	0.83 / 31.27	0.92 / 31.18	0.91 / 32.55	0.93 / 33.41
Canvas	0.62 / 21.48	0.62 / 21.80	0.76 / 25.92	0.92 / 32.87
Side panel	0.95 / 34.71	0.94 / 34.29	0.93 / 33.70	0.92 / 33.92
Text reconstruction: SSIM / PSNR
Breadcrumb	0.39 / 29.51	0.80 / 31.65	0.41 / 30.95	0.79 / 31.60

Appendix

Table 8: Conditioning representation ablation on DFoT. Reconstruction cells report SSIM / PSNR (dB); text scores use OCR-detected boxes. Memory scores measure revisit similarity by action lag, with approximately 0.5 indicating plausible but unrelated content. None denotes no state conditioning. All models receive actions, and all conditioning variants use ground-truth states, isolating the representation supplied to the video model. Best values are bold and second-best underlined, including ties.

	CF++			DFoT
SSIM $\uparrow$	full	$-$ act.	$-$ graph	full	$-$ act.	$-$ graph
Whole frame	0.934	0.895	0.824	0.929	0.889	0.857
Canvas	0.897	0.836	0.564	0.907	0.856	0.616
Breadcrumb	0.890	0.855	0.759	0.941	0.922	0.833
Semantic (all)	0.891	0.859	0.788	0.946	0.916	0.876
Revisit $k{=}64$	0.878	0.874	0.656	0.898	0.806	0.688

Appendix

Table 9: Effect of the different conditioning signals on the main metrics. Each channel is removed at inference by zeroing it, on the same 240 test trajectories with an 8-frame ground-truth seed and closed-loop generation. Graph tokens carry what is on screen: removing them costs the canvas 0.33 (CF++) and 0.29 (DFoT) SSIM and roughly a fifth of revisit memory, and drops breadcrumb agreement with the ground-truth path by 14–19 points. Removing the action channel is far cheaper on both backbones. Compare columns within a backbone: since the ablation is meant to compare the effectiveness of the components within an architecture, both CF++ and DFoT are scored here at their native resolution (

640{\times}480

and

256{\times}192

Features	Dim.	AUROC	AUROC 2	$\mathrm{Acc}_{5}$	F1	$\mathrm{Acc}_{5}^{\mathrm{lin}}$	Skip	Purity
Majority class	–	0.500	0.500	0.496	0.133	0.496	–	–
Action	8	0.944	0.919	0.790	0.736	0.790	0.329	0.981
Symbolic features	26	0.965	0.941	0.835	0.805	0.801	0.395	0.991
Graph conditioning	2056	0.978	0.960	0.870	0.820	0.819	0.397	0.990
Raw pixels	200	0.985	0.974	0.911	0.882	0.869	0.427	0.991
Denoiser latents	4616	0.987	0.975	0.916	0.898	0.901	0.410	0.990

Appendix

Table 10: Transition probing across feature families. Results on 35,880 test decisions from trajectories disjoint from training and calibration. All representations include eight action features. We report MLP results and logistic-regression accuracy (

\mathrm{Acc}_{5}^{\mathrm{lin}}

). AUROC is five-class one-vs-rest macro AUROC; AUROC 2 measures change versus no change.

\mathrm{Acc}_{5}

is five-class accuracy and F1 is macro-F1. Skip is the fraction of omitted calls; purity is the fraction of skips requiring no state change. Thresholds calibrated for 99% purity are applied unchanged to test. These are held-out probe results, closed-loop results are reported in the text.

Figure 11: Controllability interventions by type (multiple edits, new element on background, rename with type change). Per row: original state (left), commanded graph diff, removals in red and additions in green (centre), and the causally generated frame for the edited state, which never occurs in the data (right; red boxes mark edited regions). The render follows the state edit rather than the pixels of its own context.

Video World Models

Video world models generate and predict action-conditioned visual futures to simulate changing scenes for embodied agents.

See all 245 papers

WorldCrafter: Consistent Video World Model with Implicit 3D-aware Memory
Sep 21, 2026Wangbo Yu, Kunhao Liu, Wenbo Hu +83D-Aware Video Generation Camera-Conditioned Video Generation
ARC Lab, Tencent IEG · Peking University
Programmable World Model
Sep 9, 2026Zheng-Hui Huang, Guixu Lin, Jiacheng Lin +8World Models Controllable Video Generation
Alaya Lab
State World Models: Execute the Rules, Generate the Pixels
Oct 19, 2026Chan Yan Ki, Yanran Zhang, Wenzhao Zheng +5World Models Action-Conditioned World Models
Department of Automation, Tsinghua University, China · GigaAI

Sep 21, 2026J/K move · Enter open · S save

WorldCrafter: Consistent Video World Model with Implicit 3D-aware Memory

Wangbo Yu, Kunhao Liu, Wenbo Hu, Shenghai Yuan+7

ARC Lab, Tencent IEG · Peking University

3D-Aware Video Generation Camera-Conditioned Video Generation Memory-Augmented Video Generation Temporal Consistency in Video Generation Video World Models

Open paper AskPDF arXiv

Video world models enable interactive exploration of dynamic environments, yet struggle to respect prior observations over long horizons and across viewpoints. We present WorldCrafter, a video world model that learns a camera-queryable implicit 3D-aware memory for this purpose. The key insight is to let the requested viewpoint shape how multi-view evidence is compressed into the video generator's limited token budget. Trained jointly with the video generator, a memory encoder and pose-conditioned readout module integrate historical observations into a fixed set of target view-specific tokens before denoising, without explicit depth-based correspondences. By combining this memory with recent temporal context and few-step distillation, WorldCrafter enables streaming scene exploration from a single input image or text prompt. Experiments across static and dynamic scenes show substantial gains in long-horizon consistency and camera-control accuracy while preserving visual quality during minute-scale exploration.

Abstract

Video World Models

Explore similar work

WorldCrafter: Consistent Video World Model with Implicit 3D-aware Memory

Programmable World Model

State World Models: Execute the Rules, Generate the Pixels