cs.CVOct 22, 2026

Reading, Writing, Sharing, and Interacting with the World through Video

Authors: Shengqiong Wu, Meng Luo, Bobo Li, Ziwei Liu, Shuicheng Yan, Ming-Hsuan Yang, Philip Torr, Hao Fei

Organizations: University of Oxford · NUS · NTU · University of California, Merced

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Video foundation models have advanced rapidly in understanding and generating visual sequences, while video world models seek to capture the underlying dynamics of the world they depict. Despite their different objectives, both concern the relationship between video observations and an evolving world. This survey adopts world modeling as a unifying lens for examining these developments. Viewing an observed video as a partial, lossy, and temporally ordered projection of a latent temporal world, we organize the literature around four complementary operations: reading infers world properties from observed video; writing synthesizes video from specified or learned world conditions; sharing integrates understanding and generation through common modeling mechanisms; and interacting predicts and updates world states under actions and evolving observations. This perspective distinguishes temporal coherence in observed or generated sequences from consistency in the underlying world state. Across tasks, representations, models, datasets, and benchmarks, we trace a shift in emphasis from clip-level recognition and perceptual fidelity toward persistent state, controllable evolution, physical and causal validity, and evaluation grounded in interaction and downstream decisions, and we distill the open challenges that follow.

Figures & tables

Figure 1
Figure 1: Selected milestones in the parallel evolution of video understanding, generation, shared representations and workflows, and world prediction and control.
Figure 2: Knowledge graph of the four research fields. Reading, writing, sharing, and interacting branch into technical streams and representative works.
SymbolMeaningIntroduced in
World and video
t∈{1,…,T}t\in\{1,\dots,T\}video time; TT is the number of framesSection 3.1
W=(wt)t≥1W=(w_{t})_{t\geq 1} , W\mathcal{W}latent temporal world and its state spaceDefinition
T( ⋅∣wt,at)\mathcal{T}(\,\cdot\mid w_{t},a_{t})world dynamicsDefinition
ata_{t}action or external intervention; may be absentDefinition
V=(x1,…,xT)V=(x_{1},\dots,x_{T})observed video and its framesDefinition
Table 1: Notation used throughout the survey.
Operation1. Plausible output2. Faithful to evidence or conditions3. Consistent world state4. Decision utility
Readingcorrect labels and answers on clips Kay et al.,2017 ; Fu et al.,2025answers depend on the video, with shortcuts controlled Ok & Lee,2025 ; Song et al.,2026 ; Krojer et al.,2025state kept across occlusion, revisits, and streams Ma et al.,2026 ; Li et al.,2026i ; Liu et al.,2026devidence supports agent or driving decisions An et al.,2026 ; Sima et al.,2024
Writingperceptual quality and temporal coherence Huang et al.,2024aadherence to text, motion, and reference conditions Sun et al.,2025 ; Chu et al.,2025world consistency (Definition ): identity, layout, physics Bansal et al.,2026 ; Motamed et al.,2026generated data or plans improve downstream models Jang et al.,2025 ; Shang et al.,2026
Sharingboth directions work in one system Li et al.,2024a ; Zheng et al.,2025integrated, source-faithful transformations Wei et al.,2026b ; Liu et al.,2025dmeasured transfer under matched data, capacity, and compute Fu et al.,2026a ; Poppi et al.,2026shared prediction improves planning or action Lu et al.,2025 ; Wang et al.,2026b
Interactingplausible predicted frames or featurespredicted changes follow fixed or online controls Arai et al.,2024 ; Fang et al.,2026persistent state and valid responses to interventions Ye et al.,2026 ; Chen et al.,2026a ; Li et al.,2025b ; Cai et al.,2026better planning, policy training, or policy ranking Quevedo et al.,2026 ; Zhang et al.,2026d ; Shang et al.,2026
Table 2: Evidence ladder for claims about the world. Each operation can be supported at four increasing levels; stronger world-modeling claims require the higher levels. Benchmarks are representative, not exhaustive.
DimensionState property / readoutBenchmarks
SemanticActions/events; clip–text matchingKinetics Kay et al., 2017 , MSR-VTT Xu et al., 2016 , VATEX Wang et al., 2019 , ActivityNet-QA Yu et al.,2019
SemanticVideoQA with temporal and knowledge demandsVideo-MME series Fu et al., 2025 ; Fu et al., 2026 , MMBench-Video Fang et al., 2024 , MMVU Zhao et al., 2025a , Video-MMMU Hu et al.,2026c
SemanticEvidence support; hallucination/omissionVideoHallucer Wang et al., 2024g , EventHallusion Zhang et al., 2024 , ARGUS Rawal et al.,2025
SpatialMasks/tubes and identity continuityDAVIS Pont-Tuset et al., 2017 , YouTube-VOS Xu et al., 2018 , Ref-YouTube-VOS Seo et al., 2020 , MeViS Ding et al., 2023 , MOTChallenge Milan et al., 2016 , LaSOT Fan et al., 2019 , VidSTG Zhang et al.,2020
SpatialGeometry, viewpoint, and observer/object motionVSI-Bench Yang et al., 2025b , MMSI-Video-Bench Lin et al., 2025a , STI-Bench Li et al., 2025j , DSI-Bench Zhang et al., 2025i ; NuScenes-SpatialQA Tian et al.,2025
SpatialOut-of-view states and retained spatial contextSCP-Bench Zhao et al., 2026b , Beyond3D Ma et al., 2026 , OVO-S-Bench Li et al.,2026i
Table 4: Video-understanding benchmarks organized by the primary world-state property tested. A benchmark may probe more than one dimension.
Figure 3: Four families of video representation learning. They differ in how supervision targets are constructed and how predictions are compared with or generated from those targets.
Model / training stageVision encoderConnector / compressionLLM
Video-ChatGPT Maaz et al.,2024 Instruction tuning✗ CLIP ViT-L/14✓ Spatiotemporal pooling; linear projector✗ Vicuna-7B
Video-LLaVA Lin et al.,2024 Instruction tuning✗ LanguageBind image/video encoders✓ Shared two-layer MLP✓ Vicuna-7B v1.5
LLaMA-VID Li et al.,2024c Instruction tuning✗ EVA-G✓ Q-Former context query; context/content projection✓ Vicuna-7B/13B
LLaVA-OneVision Li et al.,2025 Stage 2: mixed modalities✓ SigLIP✓ Two-layer MLP; video pooling✓ Qwen2
VideoLLaMA3 Zhang et al.,2025 Video-centric fine-tuning✓ SigLIP, dynamic resolution✓ Differential frame pruning; MLP✓ Qwen2.5
Qwen2.5-VL Bai et al.,2025a Supervised fine-tuning✗ Window-attention ViT✓ 2 ⁣× ⁣22\!\times\!2 merge; MLP✓ Qwen2.5
Table 5: Video-LLM architectures and component training status. ✓ denotes trained and ✗ frozen parameters in the stage named beneath each model. Connector marks refer to learned modules, not parameter-free pooling or pruning.
DatasetScaleDescription
Web-scale video–text pairs (alignment pretraining)
HowTo100M Miech et al.,2019136M clips / 1.2M videosinstructional videos paired with automatically transcribed narration
WebVid-10M Bain et al.,202110.7M clipsstock-footage clips paired with alt-text descriptions
YT-Temporal-180M Zellers et al.,2021180M segments / 6M videostranscribed speech aligned with frames for temporal script knowledge
HD-VILA-100M Xue et al.,2022103M clips / 3.3M videos371.5K hours of 720p video spanning 15 YouTube categories
InternVid Wang et al.,2024c234M clips / 7.1M videosscalable model-generated captions with ViCLIP filtering; 760K hours
Table 6: Datasets for Video-LLM pretraining and instruction tuning. Scale retains each source’s counting unit; mixed-modality and image-only auxiliary resources are identified explicitly.
ModelBackboneOrganizationMemory / contextOperations
VideoAgent (long-form) Wang et al.,2024aGPT-4 controller; LaViLa or CogAgent captionerSingleSelected-frame captionsFrame retrieval; captioning
VideoAgent (memory) Fan et al.,2024GPT-4 controllerSingleEvent captions; object tracksCaption retrieval; video QA; temporal localization; object queries
VideoMind Liu et al.,2026iQwen2-VL + role LoRAMulti-roleGrounded intervals; role contextGrounding; verification; answering
VideoMultiAgents Kugo et al.,2025GPT-4o roles; Gemini 2.0 Flash video toolMulti-roleRole histories; scene graphCaption analysis; scene-graph analysis; video QA
Ego-R1 Tian et al.,2026aQwen2.5-3B controllerSingleTimestamped summariesHierarchical retrieval; video QA; image QA
M3-Agent Long et al.,2026aQwen3 controller; Qwen2.5-Omni memory modelSingleEpisodic + semantic memoryMemory retrieval; face/speaker identification
Table 7: Agentic video systems. Organization : Single = one reasoning controller; Multi-role = coordinated specialist reasoning roles, which may share a backbone.
BenchmarkTask / dataMeasuresProtocol
VideoSIAH-Eval Yang et al.,2026fSparse-evidence QA; 652 questionsAnswer accuracySparse/dense sampling; tool calling
LongVidSearch Yu et al.,2026Multi-hop retrieval; 3,000 questions / 447 videosAnswer accuracy; tool-call costFixed tools; 2–4 hops; gold-evidence oracle
OmniInteract Lu et al.,2026aAudiovisual streams; 1,430 response slots / 250 videosIA-QTF1 aCausal replay; interruptions; nested context
IPIBench Li et al.,2026dProactive monitoring; interleaved queriesTrigger timing; response correctnessAdd/edit/cancel tasks; past frames only
VideoGAIA Zhang et al.,2026Video–web tasks; 271 taskspass@1, three-run meanReAct; 40-step budget
AgentVidBench An et al.,2026Multi-hop QA; 100 questions / 71 videosAnswer accuracy; trajectory scores bEvidence milestones; judged trajectories
Table 8: Video-agent benchmarks for evidence acquisition, interaction control, and persistent memory.
ResourceConditionScaleAnnotations / construction
Panda-70M Chen et al.,2024dText70.8M clipsCaptions selected from multiple cross-modality teachers
Koala-36M Wang et al.,2025aText; motion, aesthetic, and clarity scores36M clipsStructured captions; Video Training Suitability Score (VTSS) filtering
OpenVid-1M Nan et al.,2025Text1M pairs; 433K HD subsetCaptions with aesthetic, temporal, motion, and clarity filtering
UltraVideo Xue et al.,2025Text42K short; 17K long clips4K/8K footage; structured captions and summaries
RealEstate10K Zhou et al.,2018aReference image; camera parameters∼\sim 80K clips; 10M framesEstimated camera intrinsics and poses for view synthesis
DL3DV-10K Ling et al.,2024Reference image; camera parameters10,510 videos; 51.2M frames4K multi-view scenes; camera intrinsics, poses, and scene-property labels
Table 9: Training resources for conditioned video generation and editing. Condition lists available input modalities or control signals; their use depends on the training task. Scale retains the source’s counting unit.
Evaluation aspectMetrics and scorersWhat is measured
Perceptual and temporal quality
Distributional qualityIS Salimans et al., 2016 , FID Heusel et al., 2017 , FVD Unterthiner et al., 2018 , tFVD Guo et al., 2026 , JEDi Luo et al.,2025classifier confidence/class coverage (IS); feature-distribution distances (FID/FVD/JEDi), not direct physical validity
Reference fidelityPSNR, SSIM Wang et al., 2004 , LPIPS Zhang et al.,2018pixel, structural, or perceptual agreement with an aligned target; unsuitable as sole criteria for open-ended futures
Conditional diversityDiVid descriptors Hu et al.,2026avariation across repeated samples in semantics, style, subject, scene, motion, and camera, alongside faithfulness
Motion and temporal qualityFVMD Liu et al., 2024 , STREAM Kim et al.,2024motion-feature distributions and separated spatial/temporal quality; dependent on tracking or feature extraction
Condition fidelity
Table 10: Measurement functions and learned scorers for video generation. Benchmarks appear in Table 11 .
BenchmarkConditionEvaluation target
VBench / EvalCrafter Huang et al.,2024a ; Liu et al.,2024cTextFrame appearance, temporal consistency, and text–video alignment
VBench++ Huang et al.,2026fText or reference imageVideo appearance and motion; image–video subject/background consistency
FETV / T2V-CompBench Liu et al.,2023a ; Sun et al.,2025TextContent coverage; attribute, action, motion, and relation composition
VMBench Ling et al.,2025TextObject integrity, motion amplitude/smoothness, temporal coherence, and commonsense
MoveBench Chu et al.,2025Text; reference image; point tracks or masksTrajectory adherence and motion consistency
OmniVBench Li et al.,2026eText; reference image or videoReference fidelity, attribute binding, and separation of content, motion, and style
Table 11: Benchmarks for conditioned video and audio-video generation and editing. Condition is the supplied modality or control signal; Evaluation target is the tested property. Alternative input settings are labeled where needed.
Figure 4: Latent representations for video generation. Continuous, discrete, hybrid, and semantically aligned designs differ in how tokenizers and VAEs shape reconstruction fidelity, compression, sequence modeling, and unification with understanding. Semantic alignment can be applied to either continuous or discrete representations.
Figure 5: Generative modeling paradigms. Diffusion and flow matching learn denoising or velocity targets over video latents. Autoregressive modeling factorizes an ordered sequence, while masked prediction fills selected positions from visible context through parallel conditional predictions.
SystemArchitectureLearningVisual spaceMLLM onlineModes T I V AV L
Diffusion over pixels or continuous latents
Video Diffusion Models (text) Ho et al.,2022aU-NetDiffusionPx–• – – – –
Video Diffusion Models (prediction) Ho et al.,2022aU-NetDiffusionPx–– • • – –
Imagen Video Ho et al.,2022Cascaded U-NetsDiffusionPx–• – – – –
Make-A-Video Singer et al.,2023U-Net cascadeDiffusionPx–• • • – –
Video LDM (text) Blattmann et al.,2023aU-NetDiffusionVAE–• – – – –
Table 12: Video-generation systems. Visual space describes generated variables; MLLM online identifies a multimodally pretrained backbone used at inference, excluding offline annotators. Modes refer to the cited configurations; MAGVIT-v2 is a tokenizer without mode assignments. Closed systems appear only where an official technical source documents the listed capability.
Figure 6: System-level organization for video generation. An LLM or MLLM plans for a visual generator through an explicit plan, such as shot lists, storyboards, or layouts, or through learned plan embeddings, and agentic systems close the loop with scripting, generation, evaluation, and revision under a controller.
Figure 7: Control mechanisms by intervention site. Each route is identified by the component through which its principal control signal enters generation: the input, the attention maps, the latent state, block features, parameters and embeddings, or the sampling update.
FamilyWhere control entersWhat is trainedStrengthsLimitationsExamples
AttentionCross- and self-attention maps, replaced, reweighted, or reused from a source runNo separate control networkLocal semantic edits that keep the source layoutDepends on learned correspondences; large motion and occlusion break them; strong map reuse blocks geometric changePrompt-to-Prompt Hertz et al., 2023 , Video-P2P Liu et al., 2024a , FateZero Qi et al., 2023 , TokenFlow Geyer et al.,2024
AdapterBlock features of a frozen backbone, through a side branchAdapter or side networkKeeps the pretrained generator; encodes dense spatial and temporal structureNeeds paired control data; one branch per signal; separately trained adapters may conflictControlNet Zhang et al., 2023a , T2I-Adapter Mou et al., 2024 , MotionCtrl Wang et al., 2024i , VideoControlNet Hu & Xu, 2023 , VACE Jiang et al.,2025
Latent spaceNoisy latent or denoising trajectory: partial noising, inversion, or constructed and optimized latentsNothing, or the base model for constructed conditionsNoise level sets fidelity against edit strength; accurate inversion preserves background, identity, and motionLimited by what the latent retains; inversion errors accumulate into temporal driftSDEdit Meng et al., 2022 , DDIM inversion Song et al., 2021 , Wan-Move Chu et al., 2025 , VLIPP Yang et al.,2025d
Concatenation / in-contextModel input, as extra channels or tokens, including in-context examples and retrieved evidenceThe backbone and its condition encodersSimple; many conditions combine in one sequence and one modelTraining must teach the model to bind each token type; longer sequences raise attention cost; retrieved evidence can be wrongSVD Blattmann et al., 2023 , VideoComposer Wang et al., 2023a , FullDiT Ju et al., 2025 , UNIC Ye et al., 2026a , PhysRAG Cheng et al.,2026a
GuidanceSampling update at each denoising stepNothing at test time; CFG needs condition dropout during trainingWorks at test time; can impose constraints absent from the training dataExtra model or gradient evaluations; high scales reduce diversity; only as good as the surrogate objectiveCFG Ho & Salimans, 2022 , universal guidance Bansal et al., 2024 , FreeDoM Yu et al., 2023 , self-guidance Epstein et al.,2023
TuningModel parameters or text embeddingsEmbeddings, full or low-rank weights, or a hypernetwork that predicts themPreserves a specific subject or style more faithfully than promptsNeeds adaptation data and compute; adds parameters; can bind the subject to reference poses, backgrounds, or motionTextual Inversion Gal et al., 2023 , DreamBooth Ruiz et al., 2023 , LoRA Hu et al., 2022 , Tune-A-Video Wu et al., 2023 , Video2LoRA Wu et al.,2026d
Table 13: Control strategies for video generation, grouped by where the control signal enters the generator (Figure 7 ). Practical systems often combine several families.
Figure 8: Four architecture families for unified video understanding and generation, ordered by the abstraction of what is shared: discrete-token unification shares a token sequence and autoregressive backbone; continuous latent-space unification shares a continuous representation with direction-specific heads; decoupled and modular unification links separate experts through a semantic bridge; and agentic and tool-augmented unification shares a task state maintained by a controller. The families can be combined: an agent may call tools built on any of the other three.
SystemCoreRead interfaceWrite / act interfaceTraining / notesU G E
Discrete-token unification
Emu3 Wang et al.,2024b8B TransformerSpatiotemporal VQVQ →\rightarrow decoderJoint AR ⋅\cdot C✓ ✓ –
Emu3.5 Cui et al.,202534B TransformerIBQ over framesIBQ →\rightarrow frame decoder; keyframe diffusionAR + SFT + RL ⋅\cdot C✓ ✓ –
Video-LaVIT Jin et al.,2024aLlama-2-7BKeyframe + motion VQCodes →\rightarrow diffusion decodersVideo–language AR ⋅\cdot C✓ ✓ –
Vega Wang et al.,2026eQwen2.5-3BTar semantic tokensTar →\rightarrow Wan2.1 rendererSemantic AR + FM ⋅\cdot C✓ ✓ –
VILA-U Wu et al.,2025eLlama-2-7B + RQ depthSigLIP/RQ towerRQ →\rightarrow RQ-VAEAlignment + joint AR ⋅\cdot C✓ ✓ –
Table 14: Representative unified video systems, grouped by their integration mechanism. For agentic and tool-augmented systems, the core is a controller and the interfaces are perception, synthesis, or execution tools. Entries follow the cited configurations.
Figure 9: Learning paradigms in unified video models. Isolated learning keeps independently pretrained experts; dual learning relates inverse tasks or reconstruction cycles; co-learning optimizes both directions jointly; and synergistic learning targets cross-direction benefit. The paradigms can coexist.
Evidence levelRequired testBenchmarks and probesVideo-native status
Level 1 Separate competenceevaluate understanding and generation separatelyVideo-MME Fu et al., 2025 , MVBench Li et al., 2024a , LongVideoBench Wu et al., 2024 , LVBench Wang et al., 2025c , MLVU Zhou et al., 2025b ; VBench Huang et al., 2024a , VBench-2.0 Zheng et al.,2025well established
Level 2 Integrated behaviortest integrated transformations and source/intent fidelityUniVBench Wei et al., 2026b , UniM Li et al., 2026h ; agentic UniVA-Bench Liang et al., 2025a ; reasoning-conditioned RVE-Bench Liu et al., 2025d and VR-Bench Xing et al.,2026emerging video coverage
Level 3 Measured synergyablate cross-direction information or training under matched settingsVideo controls: Lance task mixtures Fu et al., 2026a , ReViSE reward Liu et al., 2025d , CounterVid preferences Poppi et al., 2026 , UnifiedReward critic transfer Wang et al., 2025f , EWAM future-feature interventions Wang et al., 2026b . Image–text: RealUnify Shi et al., 2026b , UniSandbox Niu et al.,2025adirectional video evidence; general bidirectional tests remain limited
Open gap Video round tripspecify invariants, block answer leakage, and compare cycle / non-cycle controlsimage-based precedents: UniCorn Fang et al., 2026a , CyCLeGen Shan et al.,2026no established video-native standard
Table 15: Evidence ladder for unified video models, from separate competence to measured cross-direction transfer.
Figure 10: The 2×32\times 3 taxonomy of video world models. Rows give the prediction space: observations (frames, video tokens, or video-decoding latents) or latent states (video-derived features predicted without required rendering). Columns give the transition: passive prediction without action input, action-conditioned prediction with controls fixed before prediction, and interactive prediction that receives new controls after feedback. Each cell shows its prediction form and representative models; xx denotes observations, zz features, aa actions, HH the horizon, and CC and QQ the context and target positions of masked feature prediction.
Model / protocolPrediction spaceTransition protocol
 ObserveLatentPassiveActionInteractive
SV2P, action-free Babaeizadeh et al.,2018✓–✓––
SV2P, fixed actions Babaeizadeh et al.,2018✓––✓–
GAIA-1, driving controls Hu et al.,2023✓––✓–
Vista Gao et al.,2024a✓––✓–
IRASim Zhu et al.,2025✓––✓–
Table 16: Video world-model implementations and evaluated protocols. Observe predicts video observations; Latent predicts feature states. Passive has no action input; Action uses a fixed control sequence; Interactive closes the loop. Marks refer to the listed protocol.
MethodTraining mechanismIntended effect
Teacher forcingClean ground-truth history; next-step predictionFit observed transitions; exposure mismatch remains
Diffusion Forcing Chen et al.,2024Independent noise levels across sequence elementsDenoise from histories with varying uncertainty
CausVid Yin et al.,2025DMD from a bidirectional teacher to a causal studentFew-step causal generation
Self-Forcing Huang et al.,2025aSelf-generated histories; video-level distribution matchingAlign training context with autoregressive inference
Causal Forcing Zhu et al.,2026aCausal-teacher ODE initialization followed by DMD on rolloutsAlign initialization with causal prediction
Causal Forcing++ Zhao et al.,2026aCausal consistency initialization followed by DMDReduce initialization cost for frame-wise few-step generation
Table 17: Training mechanisms for causal and streaming generation. DMD: distribution-matching distillation.
Memory typeStored stateHorizon scalingExamplesMain limitation
Visual retrievalframes, tokens, or anchors selected by similarity or posegrowing or capped archiveWorldMem Xiao et al., 2025a , RELIC Hong et al., 2025 , Context-as-Memory Yu et al., 2025a , AnchorWeave Wang et al., 2026h , SMI Yang et al., 2026d , Keepsake Al et al., 2026 (capped)archives grow; eviction loses detail; false matches corrupt recall
Spatial cachesstatic maps, surfels, projected features, and reference viewsgrows with retained scene coverageVMem Li et al., 2025g , Long-term Spatial Memory Wu et al., 2025c , Matrix-Game 3.5 Qian et al.,2026geometry errors persist; static caches alone do not model off-screen change
Planned dynamics with retrievalobject/camera trajectories, crops, and selected historybounded selected context; growing video archiveWorldDirector Wang et al.,2026aoff-screen evolution is externally planned; chunks do not establish online interaction
Addressable KV summariesrecent window plus Field summaries or Landmark slotsfixed GPU cache; host scaling depends on writerWorldTrace Wu et al., 2026b ; evaluated with LoopBenchhost history grows by default; bounded aggregates lose information
Compressed contextselected chunks or summaries with unequal temporal budgetsbounded working contextStreamingT2V Henschel et al., 2025 , FramePack Zhang et al.,2025cdiscarded detail requires another retained source or a new observation
Fixed-size memorymemory queries; spatial and spatiotemporal feature planesfixed carrier; history budget accounted separatelyEcho-Infinity Bian et al., 2026 , Owl-1 Huang et al., 2024 , GIM-World Wei et al., 2026c , Honeycomb Shi et al.,2026acompression loses detail; state updates or history re-encoding add cost
Table 18: Memory mechanisms for long-horizon video generation and world modeling. Horizon scaling distinguishes active context from the full archive.
Evaluation targetBenchmarks / protocolsInterpretation limit
Video plausibilityVBench Huang et al., 2024a , VBench++ Huang et al., 2026f , VBench-2.0 Zheng et al.,2025No closed-loop action or hidden-state test
Action responseACT-Bench Arai et al., 2024 , iWorld-Bench Fang et al., 2026 , MiraBench Yang et al.,2026bAction spaces and control interfaces differ
State persistenceMIND Ye et al., 2026 , MemoBench Chen et al., 2026a , LoopBench Wu et al.,2026bStatic revisits alone do not test off-screen dynamics
Interactive latencyWaypoint-1.5 Rajpal et al., 2026 and system protocolsDisplay, latent, and control rates differ
Physical validityWorldModelBench Li et al., 2025b , Physics-IQ Motamed et al., 2026 , WorldBench Upadhyay et al., 2026 , CRONOS Begiristain et al., 2026 , What-If World Cai et al.,2026Intervention modalities differ
Scene / semantic consistencyEWMBench Yue et al.,2025Visual similarity alone does not establish state correctness
Table 19: World-model benchmarks and protocols by target, with limits on interpreting and comparing scores.
Figure 11: Hidden state evolves during occlusion; replaying an old observation can miss that change.
Figure 12: Changing gravity alters the trajectory while unrelated scene properties remain fixed.
Figure 13: Two systems with matched resources differ only in whether information passes between understanding and generation.
Figure 14: Video and feature predictions need different probes, but both face the same decision test.
Figure 15: Action effects mix with camera and background motion.
Figure 16: A held-out scene–body pair tests whether familiar factors compose while preserving action effects.
Figure 17: Valid futures differ in frequency, and invalid and unreadable samples must be counted separately.
More in this topic

Video Foundation Models

See all 17 papers

Explore similar work

CardsList
Oct 14, 2026

StateCraft: Augmenting Video World Models with Abstract-State Reasoning

Video world models aim to simulate complex environments with realistic appearance and dynamics. Maintaining consistency over long interactions requires tracking past actions and understanding how those actions affect the environment. Video world models can only generate partial observations of the entire world at a time. Remembering past observations alone may not be sufficient in every scenario, as they may no longer reflect the world after a sequence of actions. We introduce a paradigm that decouples a persistent, updatable abstract state from the visual state. The abstract state keeps track of the entities' changes as the world evolves, and is used to condition the visual state, which represents the current view. We propose StateCraft, a novel architecture that pairs a language model with a video diffusion model to evolve the dynamic world. We evaluate our approach on generative interfaces, which pose a substantial challenge because a single action may change parts of the state that are not currently visible. To support training and evaluation, we develop a data collection pipeline that produces informative, densely annotated trajectories at scale without human or agent supervision, together with a benchmark measuring generation quality, memory, and controllability. On this benchmark, our approach improves surpasses the state-of-the-art baselines. We demonstrate that StateCraft maintains high self-consistency and strong generation quality over long trajectories. Because the abstract state is explicit, it can be edited directly, and generation follows edits to states never observed in the data. Finally, a proof-of-concept 3D experiment, in which actions change the illumination of an unobserved room, shows that the paradigm can also be applied to spatial environments. Project page: https://insait-institute.github.io/StateCraft/.

Lorenzo Venturoli, Nedko Savov, Xi Wang +2

INSAIT, Sofia University "St. Kliment Ohridski" · ETH Zurich · University of Ulm +3

Video World ModelsLong-Horizon Video GenerationMemory-Augmented Video GenerationAction-Conditioned Video GenerationVideo Diffusion Models
AskPDF
Mar 13, 2026

Out of Sight, Out of Mind? Evaluating State Evolution in Video World Models

Evolutions in the world, such as water pouring or ice melting, happen regardless of being observed. Video world models generate "worlds" via 2D frame observations. Can these generated "worlds" evolve regardless of observation? To probe this question, we design a benchmark to evaluate whether video world models can decouple state evolution from observation. Our benchmark, STEVO-Bench, applies observation control to evolving processes via instructions of occluder insertion, turning off the light, or specifying camera "lookaway" trajectories. By evaluating video models with and without camera control for a diverse set of naturally-occurring evolutions, we expose their limitations in decoupling state evolution from observation. STEVO-Bench proposes an evaluation protocol to automatically detect and disentangle failure modes of video world models across key aspects of natural state evolution. Analysis of STEVO-Bench results provide new insight into potential data and architecture bias of present-day video world models. Project website: https://glab-caltech.github.io/STEVOBench/. Blog: https://ziqi-ma.github.io/blog/2026/outofsight/

Ziqi Ma, Mengzhan Liufu, Georgia Gkioxari

California Institute of Technology

Video World ModelsWorld Model EvaluationPhysical Consistency in Video GenerationCamera-Controlled Video Generation
AskPDF
May 31, 2026

Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends

With rapid development of large language models and diffusion-based content generation, world modeling has attracted increasing research attention, benefiting various downstream domains such as game engines, embodied AI, autonomous driving, etc. Through explicitly incorporating user actions into world state transition, recent literature empowers world modeling with interactivity in an action-conditioned video or 3D generation paradigm, further enhancing controllability over world evolutions and facilitating users to freely traverse, manipulate, navigate, and personalize the state evolution. In this paper, we aim to systematically review recent research trends, technical developments, evaluation benchmarks, and also propose future potential directions in interactive world modeling. Specifically, we first summarize recent efforts and trends in terms of application scenarios, world state evolution, and scene modality. Afterwards, we delve into three crucial technical challenges, including action-conditioned controllability, long-horizon interactions and memory, and action-following responsiveness for real-time interactivity. Furthermore, we also thoroughly compare existing benchmarks and metrics in four specific application fields: open-world exploration, game engine, autonomous driving, and robotics. Finally, we discuss several promising future directions in achieving next-generation interactive world modeling. The corresponding repository is publicly available at: https://github.com/liujiuming123/Awesome-Interactive-World-Model.

Jiuming Liu, Chaojun Ni, Mengmeng Liu +7

Department of Engineering, University of Cambridge, U.K. · Peking University, China · University of Twente, Netherlands +4

Action-Conditioned Video GenerationLong-Horizon Video GenerationAction-Conditioned World ModelsWorld Model EvaluationInteractive World ModelsVideo World Models
AskPDF