Reading, Writing, Sharing, and Interacting with the World through Video

Shengqiong Wu; Meng Luo; Bobo Li; Ziwei Liu; Shuicheng Yan; Ming-Hsuan Yang; Philip Torr; Hao Fei

cs.CVOct 22, 2026

Reading, Writing, Sharing, and Interacting with the World through Video

Authors: Shengqiong Wu, Meng Luo, Bobo Li, Ziwei Liu, Shuicheng Yan, Ming-Hsuan Yang, Philip Torr, Hao Fei

Organizations: University of Oxford · NUS · NTU · University of California, Merced

Abstract

Video foundation models have advanced rapidly in understanding and generating visual sequences, while video world models seek to capture the underlying dynamics of the world they depict. Despite their different objectives, both concern the relationship between video observations and an evolving world. This survey adopts world modeling as a unifying lens for examining these developments. Viewing an observed video as a partial, lossy, and temporally ordered projection of a latent temporal world, we organize the literature around four complementary operations: reading infers world properties from observed video; writing synthesizes video from specified or learned world conditions; sharing integrates understanding and generation through common modeling mechanisms; and interacting predicts and updates world states under actions and evolving observations. This perspective distinguishes temporal coherence in observed or generated sequences from consistency in the underlying world state. Across tasks, representations, models, datasets, and benchmarks, we trace a shift in emphasis from clip-level recognition and perceptual fidelity toward persistent state, controllable evolution, physical and causal validity, and evaluation grounded in interaction and downstream decisions, and we distill the open challenges that follow.

Figures & tables

Figure 1: Selected milestones in the parallel evolution of video understanding, generation, shared representations and workflows, and world prediction and control.

Panel of Figure 2 — Figure 2: Knowledge graph of the four research fields. Reading, writing, sharing, and interacting branch into technical streams and representative works.

Symbol	Meaning	Introduced in
World and video
$t\in\{1,\dots,T\}$	video time; $T$ is the number of frames	Section 3.1
$W=(w_{t})_{t\geq 1}$ , $\mathcal{W}$	latent temporal world and its state space	Definition
$\mathcal{T}(\,\cdot\mid w_{t},a_{t})$	world dynamics	Definition
$a_{t}$	action or external intervention; may be absent	Definition
$V=(x_{1},\dots,x_{T})$	observed video and its frames	Definition

Table 1: Notation used throughout the survey.

Operation	1. Plausible output	2. Faithful to evidence or conditions	3. Consistent world state	4. Decision utility
Reading	correct labels and answers on clips Kay et al.,2017 ; Fu et al.,2025	answers depend on the video, with shortcuts controlled Ok & Lee,2025 ; Song et al.,2026 ; Krojer et al.,2025	state kept across occlusion, revisits, and streams Ma et al.,2026 ; Li et al.,2026i ; Liu et al.,2026d	evidence supports agent or driving decisions An et al.,2026 ; Sima et al.,2024
Writing	perceptual quality and temporal coherence Huang et al.,2024a	adherence to text, motion, and reference conditions Sun et al.,2025 ; Chu et al.,2025	world consistency (Definition ): identity, layout, physics Bansal et al.,2026 ; Motamed et al.,2026	generated data or plans improve downstream models Jang et al.,2025 ; Shang et al.,2026
Sharing	both directions work in one system Li et al.,2024a ; Zheng et al.,2025	integrated, source-faithful transformations Wei et al.,2026b ; Liu et al.,2025d	measured transfer under matched data, capacity, and compute Fu et al.,2026a ; Poppi et al.,2026	shared prediction improves planning or action Lu et al.,2025 ; Wang et al.,2026b
Interacting	plausible predicted frames or features	predicted changes follow fixed or online controls Arai et al.,2024 ; Fang et al.,2026	persistent state and valid responses to interventions Ye et al.,2026 ; Chen et al.,2026a ; Li et al.,2025b ; Cai et al.,2026	better planning, policy training, or policy ranking Quevedo et al.,2026 ; Zhang et al.,2026d ; Shang et al.,2026

Table 2: Evidence ladder for claims about the world. Each operation can be supported at four increasing levels; stronger world-modeling claims require the higher levels. Benchmarks are representative, not exhaustive.

Dimension	State property / readout	Benchmarks
Semantic	Actions/events; clip–text matching	Kinetics Kay et al., 2017 , MSR-VTT Xu et al., 2016 , VATEX Wang et al., 2019 , ActivityNet-QA Yu et al.,2019
Semantic	VideoQA with temporal and knowledge demands	Video-MME series Fu et al., 2025 ; Fu et al., 2026 , MMBench-Video Fang et al., 2024 , MMVU Zhao et al., 2025a , Video-MMMU Hu et al.,2026c
Semantic	Evidence support; hallucination/omission	VideoHallucer Wang et al., 2024g , EventHallusion Zhang et al., 2024 , ARGUS Rawal et al.,2025
Spatial	Masks/tubes and identity continuity	DAVIS Pont-Tuset et al., 2017 , YouTube-VOS Xu et al., 2018 , Ref-YouTube-VOS Seo et al., 2020 , MeViS Ding et al., 2023 , MOTChallenge Milan et al., 2016 , LaSOT Fan et al., 2019 , VidSTG Zhang et al.,2020
Spatial	Geometry, viewpoint, and observer/object motion	VSI-Bench Yang et al., 2025b , MMSI-Video-Bench Lin et al., 2025a , STI-Bench Li et al., 2025j , DSI-Bench Zhang et al., 2025i ; NuScenes-SpatialQA Tian et al.,2025
Spatial	Out-of-view states and retained spatial context	SCP-Bench Zhao et al., 2026b , Beyond3D Ma et al., 2026 , OVO-S-Bench Li et al.,2026i

Table 4: Video-understanding benchmarks organized by the primary world-state property tested. A benchmark may probe more than one dimension.

Figure 3: Four families of video representation learning. They differ in how supervision targets are constructed and how predictions are compared with or generated from those targets.

Model / training stage	Vision encoder	Connector / compression	LLM
Video-ChatGPT Maaz et al.,2024 Instruction tuning	✗ CLIP ViT-L/14	✓ Spatiotemporal pooling; linear projector	✗ Vicuna-7B
Video-LLaVA Lin et al.,2024 Instruction tuning	✗ LanguageBind image/video encoders	✓ Shared two-layer MLP	✓ Vicuna-7B v1.5
LLaMA-VID Li et al.,2024c Instruction tuning	✗ EVA-G	✓ Q-Former context query; context/content projection	✓ Vicuna-7B/13B
LLaVA-OneVision Li et al.,2025 Stage 2: mixed modalities	✓ SigLIP	✓ Two-layer MLP; video pooling	✓ Qwen2
VideoLLaMA3 Zhang et al.,2025 Video-centric fine-tuning	✓ SigLIP, dynamic resolution	✓ Differential frame pruning; MLP	✓ Qwen2.5
Qwen2.5-VL Bai et al.,2025a Supervised fine-tuning	✗ Window-attention ViT	✓ $2\!\times\!2$ merge; MLP	✓ Qwen2.5

Table 5: Video-LLM architectures and component training status. ✓ denotes trained and ✗ frozen parameters in the stage named beneath each model. Connector marks refer to learned modules, not parameter-free pooling or pruning.

Web-scale video–text pairs (alignment pretraining)
Dataset	Scale	Description
HowTo100M Miech et al.,2019	136M clips / 1.2M videos	instructional videos paired with automatically transcribed narration
WebVid-10M Bain et al.,2021	10.7M clips	stock-footage clips paired with alt-text descriptions
YT-Temporal-180M Zellers et al.,2021	180M segments / 6M videos	transcribed speech aligned with frames for temporal script knowledge
HD-VILA-100M Xue et al.,2022	103M clips / 3.3M videos	371.5K hours of 720p video spanning 15 YouTube categories
InternVid Wang et al.,2024c	234M clips / 7.1M videos	scalable model-generated captions with ViCLIP filtering; 760K hours

Table 6: Datasets for Video-LLM pretraining and instruction tuning. Scale retains each source’s counting unit; mixed-modality and image-only auxiliary resources are identified explicitly.

Model	Backbone	Organization	Memory / context	Operations
VideoAgent (long-form) Wang et al.,2024a	GPT-4 controller; LaViLa or CogAgent captioner	Single	Selected-frame captions	Frame retrieval; captioning
VideoAgent (memory) Fan et al.,2024	GPT-4 controller	Single	Event captions; object tracks	Caption retrieval; video QA; temporal localization; object queries
VideoMind Liu et al.,2026i	Qwen2-VL + role LoRA	Multi-role	Grounded intervals; role context	Grounding; verification; answering
VideoMultiAgents Kugo et al.,2025	GPT-4o roles; Gemini 2.0 Flash video tool	Multi-role	Role histories; scene graph	Caption analysis; scene-graph analysis; video QA
Ego-R1 Tian et al.,2026a	Qwen2.5-3B controller	Single	Timestamped summaries	Hierarchical retrieval; video QA; image QA
M3-Agent Long et al.,2026a	Qwen3 controller; Qwen2.5-Omni memory model	Single	Episodic + semantic memory	Memory retrieval; face/speaker identification

Table 7: Agentic video systems. Organization : Single = one reasoning controller; Multi-role = coordinated specialist reasoning roles, which may share a backbone.

Benchmark	Task / data	Measures	Protocol
VideoSIAH-Eval Yang et al.,2026f	Sparse-evidence QA; 652 questions	Answer accuracy	Sparse/dense sampling; tool calling
LongVidSearch Yu et al.,2026	Multi-hop retrieval; 3,000 questions / 447 videos	Answer accuracy; tool-call cost	Fixed tools; 2–4 hops; gold-evidence oracle
OmniInteract Lu et al.,2026a	Audiovisual streams; 1,430 response slots / 250 videos	IA-QTF1 a	Causal replay; interruptions; nested context
IPIBench Li et al.,2026d	Proactive monitoring; interleaved queries	Trigger timing; response correctness	Add/edit/cancel tasks; past frames only
VideoGAIA Zhang et al.,2026	Video–web tasks; 271 tasks	pass@1, three-run mean	ReAct; 40-step budget
AgentVidBench An et al.,2026	Multi-hop QA; 100 questions / 71 videos	Answer accuracy; trajectory scores b	Evidence milestones; judged trajectories

Table 8: Video-agent benchmarks for evidence acquisition, interaction control, and persistent memory.

Resource	Condition	Scale	Annotations / construction
Panda-70M Chen et al.,2024d	Text	70.8M clips	Captions selected from multiple cross-modality teachers
Koala-36M Wang et al.,2025a	Text; motion, aesthetic, and clarity scores	36M clips	Structured captions; Video Training Suitability Score (VTSS) filtering
OpenVid-1M Nan et al.,2025	Text	1M pairs; 433K HD subset	Captions with aesthetic, temporal, motion, and clarity filtering
UltraVideo Xue et al.,2025	Text	42K short; 17K long clips	4K/8K footage; structured captions and summaries
RealEstate10K Zhou et al.,2018a	Reference image; camera parameters	$\sim$ 80K clips; 10M frames	Estimated camera intrinsics and poses for view synthesis
DL3DV-10K Ling et al.,2024	Reference image; camera parameters	10,510 videos; 51.2M frames	4K multi-view scenes; camera intrinsics, poses, and scene-property labels

Table 9: Training resources for conditioned video generation and editing. Condition lists available input modalities or control signals; their use depends on the training task. Scale retains the source’s counting unit.

Evaluation aspect	Metrics and scorers	What is measured
Perceptual and temporal quality
Distributional quality	IS Salimans et al., 2016 , FID Heusel et al., 2017 , FVD Unterthiner et al., 2018 , tFVD Guo et al., 2026 , JEDi Luo et al.,2025	classifier confidence/class coverage (IS); feature-distribution distances (FID/FVD/JEDi), not direct physical validity
Reference fidelity	PSNR, SSIM Wang et al., 2004 , LPIPS Zhang et al.,2018	pixel, structural, or perceptual agreement with an aligned target; unsuitable as sole criteria for open-ended futures
Conditional diversity	DiVid descriptors Hu et al.,2026a	variation across repeated samples in semantics, style, subject, scene, motion, and camera, alongside faithfulness
Motion and temporal quality	FVMD Liu et al., 2024 , STREAM Kim et al.,2024	motion-feature distributions and separated spatial/temporal quality; dependent on tracking or feature extraction
Condition fidelity

Table 10: Measurement functions and learned scorers for video generation. Benchmarks appear in Table 11 .

Benchmark	Condition	Evaluation target
VBench / EvalCrafter Huang et al.,2024a ; Liu et al.,2024c	Text	Frame appearance, temporal consistency, and text–video alignment
VBench++ Huang et al.,2026f	Text or reference image	Video appearance and motion; image–video subject/background consistency
FETV / T2V-CompBench Liu et al.,2023a ; Sun et al.,2025	Text	Content coverage; attribute, action, motion, and relation composition
VMBench Ling et al.,2025	Text	Object integrity, motion amplitude/smoothness, temporal coherence, and commonsense
MoveBench Chu et al.,2025	Text; reference image; point tracks or masks	Trajectory adherence and motion consistency
OmniVBench Li et al.,2026e	Text; reference image or video	Reference fidelity, attribute binding, and separation of content, motion, and style

Table 11: Benchmarks for conditioned video and audio-video generation and editing. Condition is the supplied modality or control signal; Evaluation target is the tested property. Alternative input settings are labeled where needed.

Figure 4: Latent representations for video generation. Continuous, discrete, hybrid, and semantically aligned designs differ in how tokenizers and VAEs shape reconstruction fidelity, compression, sequence modeling, and unification with understanding. Semantic alignment can be applied to either continuous or discrete representations.

Figure 5: Generative modeling paradigms. Diffusion and flow matching learn denoising or velocity targets over video latents. Autoregressive modeling factorizes an ordered sequence, while masked prediction fills selected positions from visible context through parallel conditional predictions.

System	Architecture	Learning	Visual space	MLLM online	Modes T I V AV L
Diffusion over pixels or continuous latents
Video Diffusion Models (text) Ho et al.,2022a	U-Net	Diffusion	Px	–	• – – – –
Video Diffusion Models (prediction) Ho et al.,2022a	U-Net	Diffusion	Px	–	– • • – –
Imagen Video Ho et al.,2022	Cascaded U-Nets	Diffusion	Px	–	• – – – –
Make-A-Video Singer et al.,2023	U-Net cascade	Diffusion	Px	–	• • • – –
Video LDM (text) Blattmann et al.,2023a	U-Net	Diffusion	VAE	–	• – – – –

Table 12: Video-generation systems. Visual space describes generated variables; MLLM online identifies a multimodally pretrained backbone used at inference, excluding offline annotators. Modes refer to the cited configurations; MAGVIT-v2 is a tokenizer without mode assignments. Closed systems appear only where an official technical source documents the listed capability.

Figure 6: System-level organization for video generation. An LLM or MLLM plans for a visual generator through an explicit plan, such as shot lists, storyboards, or layouts, or through learned plan embeddings, and agentic systems close the loop with scripting, generation, evaluation, and revision under a controller.

Figure 7: Control mechanisms by intervention site. Each route is identified by the component through which its principal control signal enters generation: the input, the attention maps, the latent state, block features, parameters and embeddings, or the sampling update.

Family	Where control enters	What is trained	Strengths	Limitations	Examples
Attention	Cross- and self-attention maps, replaced, reweighted, or reused from a source run	No separate control network	Local semantic edits that keep the source layout	Depends on learned correspondences; large motion and occlusion break them; strong map reuse blocks geometric change	Prompt-to-Prompt Hertz et al., 2023 , Video-P2P Liu et al., 2024a , FateZero Qi et al., 2023 , TokenFlow Geyer et al.,2024
Adapter	Block features of a frozen backbone, through a side branch	Adapter or side network	Keeps the pretrained generator; encodes dense spatial and temporal structure	Needs paired control data; one branch per signal; separately trained adapters may conflict	ControlNet Zhang et al., 2023a , T2I-Adapter Mou et al., 2024 , MotionCtrl Wang et al., 2024i , VideoControlNet Hu & Xu, 2023 , VACE Jiang et al.,2025
Latent space	Noisy latent or denoising trajectory: partial noising, inversion, or constructed and optimized latents	Nothing, or the base model for constructed conditions	Noise level sets fidelity against edit strength; accurate inversion preserves background, identity, and motion	Limited by what the latent retains; inversion errors accumulate into temporal drift	SDEdit Meng et al., 2022 , DDIM inversion Song et al., 2021 , Wan-Move Chu et al., 2025 , VLIPP Yang et al.,2025d
Concatenation / in-context	Model input, as extra channels or tokens, including in-context examples and retrieved evidence	The backbone and its condition encoders	Simple; many conditions combine in one sequence and one model	Training must teach the model to bind each token type; longer sequences raise attention cost; retrieved evidence can be wrong	SVD Blattmann et al., 2023 , VideoComposer Wang et al., 2023a , FullDiT Ju et al., 2025 , UNIC Ye et al., 2026a , PhysRAG Cheng et al.,2026a
Guidance	Sampling update at each denoising step	Nothing at test time; CFG needs condition dropout during training	Works at test time; can impose constraints absent from the training data	Extra model or gradient evaluations; high scales reduce diversity; only as good as the surrogate objective	CFG Ho & Salimans, 2022 , universal guidance Bansal et al., 2024 , FreeDoM Yu et al., 2023 , self-guidance Epstein et al.,2023
Tuning	Model parameters or text embeddings	Embeddings, full or low-rank weights, or a hypernetwork that predicts them	Preserves a specific subject or style more faithfully than prompts	Needs adaptation data and compute; adds parameters; can bind the subject to reference poses, backgrounds, or motion	Textual Inversion Gal et al., 2023 , DreamBooth Ruiz et al., 2023 , LoRA Hu et al., 2022 , Tune-A-Video Wu et al., 2023 , Video2LoRA Wu et al.,2026d

Table 13: Control strategies for video generation, grouped by where the control signal enters the generator (Figure 7 ). Practical systems often combine several families.

Figure 8: Four architecture families for unified video understanding and generation, ordered by the abstraction of what is shared: discrete-token unification shares a token sequence and autoregressive backbone; continuous latent-space unification shares a continuous representation with direction-specific heads; decoupled and modular unification links separate experts through a semantic bridge; and agentic and tool-augmented unification shares a task state maintained by a controller. The families can be combined: an agent may call tools built on any of the other three.

System	Core	Read interface	Write / act interface	Training / notes	U G E
Discrete-token unification
Emu3 Wang et al.,2024b	8B Transformer	Spatiotemporal VQ	VQ $\rightarrow$ decoder	Joint AR $\cdot$ C	✓ ✓ –
Emu3.5 Cui et al.,2025	34B Transformer	IBQ over frames	IBQ $\rightarrow$ frame decoder; keyframe diffusion	AR + SFT + RL $\cdot$ C	✓ ✓ –
Video-LaVIT Jin et al.,2024a	Llama-2-7B	Keyframe + motion VQ	Codes $\rightarrow$ diffusion decoders	Video–language AR $\cdot$ C	✓ ✓ –
Vega Wang et al.,2026e	Qwen2.5-3B	Tar semantic tokens	Tar $\rightarrow$ Wan2.1 renderer	Semantic AR + FM $\cdot$ C	✓ ✓ –
VILA-U Wu et al.,2025e	Llama-2-7B + RQ depth	SigLIP/RQ tower	RQ $\rightarrow$ RQ-VAE	Alignment + joint AR $\cdot$ C	✓ ✓ –

Table 14: Representative unified video systems, grouped by their integration mechanism. For agentic and tool-augmented systems, the core is a controller and the interfaces are perception, synthesis, or execution tools. Entries follow the cited configurations.

Figure 9: Learning paradigms in unified video models. Isolated learning keeps independently pretrained experts; dual learning relates inverse tasks or reconstruction cycles; co-learning optimizes both directions jointly; and synergistic learning targets cross-direction benefit. The paradigms can coexist.

Evidence level	Required test	Benchmarks and probes	Video-native status
Level 1 Separate competence	evaluate understanding and generation separately	Video-MME Fu et al., 2025 , MVBench Li et al., 2024a , LongVideoBench Wu et al., 2024 , LVBench Wang et al., 2025c , MLVU Zhou et al., 2025b ; VBench Huang et al., 2024a , VBench-2.0 Zheng et al.,2025	well established
Level 2 Integrated behavior	test integrated transformations and source/intent fidelity	UniVBench Wei et al., 2026b , UniM Li et al., 2026h ; agentic UniVA-Bench Liang et al., 2025a ; reasoning-conditioned RVE-Bench Liu et al., 2025d and VR-Bench Xing et al.,2026	emerging video coverage
Level 3 Measured synergy	ablate cross-direction information or training under matched settings	Video controls: Lance task mixtures Fu et al., 2026a , ReViSE reward Liu et al., 2025d , CounterVid preferences Poppi et al., 2026 , UnifiedReward critic transfer Wang et al., 2025f , EWAM future-feature interventions Wang et al., 2026b . Image–text: RealUnify Shi et al., 2026b , UniSandbox Niu et al.,2025a	directional video evidence; general bidirectional tests remain limited
Open gap Video round trip	specify invariants, block answer leakage, and compare cycle / non-cycle controls	image-based precedents: UniCorn Fang et al., 2026a , CyCLeGen Shan et al.,2026	no established video-native standard

Table 15: Evidence ladder for unified video models, from separate competence to measured cross-direction transfer.

Figure 10: The

2\times 3

taxonomy of video world models. Rows give the prediction space: observations (frames, video tokens, or video-decoding latents) or latent states (video-derived features predicted without required rendering). Columns give the transition: passive prediction without action input, action-conditioned prediction with controls fixed before prediction, and interactive prediction that receives new controls after feedback. Each cell shows its prediction form and representative models;

x

denotes observations,

z

features,

a

actions,

H

the horizon, and

C

and

Q

the context and target positions of masked feature prediction.

Model / protocol	Prediction space		Transition protocol
	Observe	Latent	Passive	Action	Interactive
SV2P, action-free Babaeizadeh et al.,2018	✓	–	✓	–	–
SV2P, fixed actions Babaeizadeh et al.,2018	✓	–	–	✓	–
GAIA-1, driving controls Hu et al.,2023	✓	–	–	✓	–
Vista Gao et al.,2024a	✓	–	–	✓	–
IRASim Zhu et al.,2025	✓	–	–	✓	–

Table 16: Video world-model implementations and evaluated protocols. Observe predicts video observations; Latent predicts feature states. Passive has no action input; Action uses a fixed control sequence; Interactive closes the loop. Marks refer to the listed protocol.

Method	Training mechanism	Intended effect
Teacher forcing	Clean ground-truth history; next-step prediction	Fit observed transitions; exposure mismatch remains
Diffusion Forcing Chen et al.,2024	Independent noise levels across sequence elements	Denoise from histories with varying uncertainty
CausVid Yin et al.,2025	DMD from a bidirectional teacher to a causal student	Few-step causal generation
Self-Forcing Huang et al.,2025a	Self-generated histories; video-level distribution matching	Align training context with autoregressive inference
Causal Forcing Zhu et al.,2026a	Causal-teacher ODE initialization followed by DMD on rollouts	Align initialization with causal prediction
Causal Forcing++ Zhao et al.,2026a	Causal consistency initialization followed by DMD	Reduce initialization cost for frame-wise few-step generation

Table 17: Training mechanisms for causal and streaming generation. DMD: distribution-matching distillation.

Memory type	Stored state	Horizon scaling	Examples	Main limitation
Visual retrieval	frames, tokens, or anchors selected by similarity or pose	growing or capped archive	WorldMem Xiao et al., 2025a , RELIC Hong et al., 2025 , Context-as-Memory Yu et al., 2025a , AnchorWeave Wang et al., 2026h , SMI Yang et al., 2026d , Keepsake Al et al., 2026 (capped)	archives grow; eviction loses detail; false matches corrupt recall
Spatial caches	static maps, surfels, projected features, and reference views	grows with retained scene coverage	VMem Li et al., 2025g , Long-term Spatial Memory Wu et al., 2025c , Matrix-Game 3.5 Qian et al.,2026	geometry errors persist; static caches alone do not model off-screen change
Planned dynamics with retrieval	object/camera trajectories, crops, and selected history	bounded selected context; growing video archive	WorldDirector Wang et al.,2026a	off-screen evolution is externally planned; chunks do not establish online interaction
Addressable KV summaries	recent window plus Field summaries or Landmark slots	fixed GPU cache; host scaling depends on writer	WorldTrace Wu et al., 2026b ; evaluated with LoopBench	host history grows by default; bounded aggregates lose information
Compressed context	selected chunks or summaries with unequal temporal budgets	bounded working context	StreamingT2V Henschel et al., 2025 , FramePack Zhang et al.,2025c	discarded detail requires another retained source or a new observation
Fixed-size memory	memory queries; spatial and spatiotemporal feature planes	fixed carrier; history budget accounted separately	Echo-Infinity Bian et al., 2026 , Owl-1 Huang et al., 2024 , GIM-World Wei et al., 2026c , Honeycomb Shi et al.,2026a	compression loses detail; state updates or history re-encoding add cost

Table 18: Memory mechanisms for long-horizon video generation and world modeling. Horizon scaling distinguishes active context from the full archive.

Evaluation target	Benchmarks / protocols	Interpretation limit
Video plausibility	VBench Huang et al., 2024a , VBench++ Huang et al., 2026f , VBench-2.0 Zheng et al.,2025	No closed-loop action or hidden-state test
Action response	ACT-Bench Arai et al., 2024 , iWorld-Bench Fang et al., 2026 , MiraBench Yang et al.,2026b	Action spaces and control interfaces differ
State persistence	MIND Ye et al., 2026 , MemoBench Chen et al., 2026a , LoopBench Wu et al.,2026b	Static revisits alone do not test off-screen dynamics
Interactive latency	Waypoint-1.5 Rajpal et al., 2026 and system protocols	Display, latent, and control rates differ
Physical validity	WorldModelBench Li et al., 2025b , Physics-IQ Motamed et al., 2026 , WorldBench Upadhyay et al., 2026 , CRONOS Begiristain et al., 2026 , What-If World Cai et al.,2026	Intervention modalities differ
Scene / semantic consistency	EWMBench Yue et al.,2025	Visual similarity alone does not establish state correctness

Table 19: World-model benchmarks and protocols by target, with limits on interpreting and comparing scores.

Figure 11: Hidden state evolves during occlusion; replaying an old observation can miss that change.

Figure 12: Changing gravity alters the trajectory while unrelated scene properties remain fixed.

Figure 13: Two systems with matched resources differ only in whether information passes between understanding and generation.

Figure 14: Video and feature predictions need different probes, but both face the same decision test.

Figure 15: Action effects mix with camera and background motion.

Figure 16: A held-out scene–body pair tests whether familiar factors compose while preserving action effects.

Figure 17: Valid futures differ in frequency, and invalid and unreadable samples must be counted separately.

Video Foundation Models

See all 17 papers

StateCraft: Augmenting Video World Models with Abstract-State Reasoning
Oct 14, 2026Lorenzo Venturoli, Nedko Savov, Xi Wang +2Video World Models Long-Horizon Video Generation
INSAIT, Sofia University "St. Kliment Ohridski" · ETH Zurich · University of Ulm +3
Out of Sight, Out of Mind? Evaluating State Evolution in Video World Models
Mar 13, 2026Ziqi Ma, Mengzhan Liufu, Georgia GkioxariVideo World Models World Model Evaluation
California Institute of Technology
Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends
May 31, 2026Jiuming Liu, Chaojun Ni, Mengmeng Liu +7Action-Conditioned Video Generation Long-Horizon Video Generation
Department of Engineering, University of Cambridge, U.K. · Peking University, China · University of Twente, Netherlands +4

Oct 14, 2026J/K move · Enter open · S save

StateCraft: Augmenting Video World Models with Abstract-State Reasoning

Lorenzo Venturoli, Nedko Savov, Xi Wang, Marc Pollefeys+1

INSAIT, Sofia University "St. Kliment Ohridski" · ETH Zurich · University of Ulm+3

Video World Models Long-Horizon Video Generation Memory-Augmented Video Generation Action-Conditioned Video Generation Video Diffusion Models

Open paper AskPDF arXiv

Video world models aim to simulate complex environments with realistic appearance and dynamics. Maintaining consistency over long interactions requires tracking past actions and understanding how those actions affect the environment. Video world models can only generate partial observations of the entire world at a time. Remembering past observations alone may not be sufficient in every scenario, as they may no longer reflect the world after a sequence of actions. We introduce a paradigm that decouples a persistent, updatable abstract state from the visual state. The abstract state keeps track of the entities' changes as the world evolves, and is used to condition the visual state, which represents the current view. We propose StateCraft, a novel architecture that pairs a language model with a video diffusion model to evolve the dynamic world. We evaluate our approach on generative interfaces, which pose a substantial challenge because a single action may change parts of the state that are not currently visible. To support training and evaluation, we develop a data collection pipeline that produces informative, densely annotated trajectories at scale without human or agent supervision, together with a benchmark measuring generation quality, memory, and controllability. On this benchmark, our approach improves surpasses the state-of-the-art baselines. We demonstrate that StateCraft maintains high self-consistency and strong generation quality over long trajectories. Because the abstract state is explicit, it can be edited directly, and generation follows edits to states never observed in the data. Finally, a proof-of-concept 3D experiment, in which actions change the illumination of an unobserved room, shows that the paradigm can also be applied to spatial environments. Project page: https://insait-institute.github.io/StateCraft/.

Abstract

Video Foundation Models

Explore similar work

StateCraft: Augmenting Video World Models with Abstract-State Reasoning

Out of Sight, Out of Mind? Evaluating State Evolution in Video World Models

Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends