WorldCast: Exploring Intelligent Visual Generation in World State Prediction

Wenhao Yan; Haodong Li; Xiaotong Li; Yaxin Luo; Zhengrong Yue; Jialv Zou; Xu Huang; Haojin Yang

cs.CVOct 14, 2026

WorldCast: Exploring Intelligent Visual Generation in World State Prediction

Authors: Wenhao Yan, Haodong Li, Xiaotong Li, Yaxin Luo, Zhengrong Yue, Jialv Zou, Xu Huang, Haojin Yang,

Organizations: THU · CUHK · Meituan LongCat · MBZUAI ·

Abstract

Humans interpret multimodal observations of the world and forecast how it will evolve. Recent multimodal generative systems have demonstrated similar capabilities: they can integrate multimodal context and synthesize visual outputs that depict changes in the world. To characterize this emerging trajectory, we organize model capabilities into a four-level hierarchy and discover that higher-level tasks remain relatively underexplored despite model progress. To bridge this gap, we explore model performance on multi-step world state prediction, a high-level task that requires models to forecast successive future world states from multimodal observations and diverse references. Such a task requires models to possess entangled understanding and generation capabilities, as well as extensive world knowledge to translate human action sequences, physical transformations, and abstract visual reasoning into coherent visual outputs. We introduce WorldCast, a benchmark comprising 320 manually designed cases across eight categories, and evaluate 12 representative models. The conducted experiments yield three key insights: (a) advances in frontier models are rapidly pushing the boundaries of intelligent world modeling, with the GPT-Image-2.5 System scoring 72.8 and demonstrating early-stage visual state prediction capabilities on real-world tasks; (b) WorldCast reveals substantial gaps between models: even Nano Banana Pro reaches only 42.8 and the best open-source model scores 20.1, with deficits in handling complex context, entangled references, and reasoning-heavy tasks; (c) strong generative models effectively draw on the capabilities of their underlying language models. These findings highlight the need for multimodal generative models that are capable of dynamic planning and reasoning while also being deeply unified to faithfully simulate real-world states.

Figures & tables

Figure 1: Our generative capability tree and the WorldCast benchmark. (a) The proposed visual generative capability tree organizes intelligent visual generation into a four-layer hierarchy (L1-L4); (b) WorldCast targets L3-L4 level capabilities of dynamically transforming real-world context into successive future world states through unified multimodal understanding and generation.

Figure 2: Evaluation scope and question example of WorldCast . The benchmark requires models to combine understanding and generation to dynamically produce evolving visual states. In this assembling task, GPT-Image-2.5 achieves key milestones while Wan 3.0 struggles in figuring out the correct order.

Figure 3: Representative examples of the eight WorldCast categories. The benchmark comprises 320 manually designed cases in total, with 1,352 requested visual states.

Figure 4: Qualitative examples on WorldCast . Models exhibit distinct failure patterns across the three evaluation dimensions; all scores are reported on a 0–3 scale.

Panel of Table 1 — Table 1: Overview of WorldCast performance across twelve image and video generation models and eight benchmark categories. Blue cells mark the best result within each model group, while orange cells mark the best result across all models. Icons indicate the generation format: Interleaved Generation, Grid Generation, Video Generation, Plan & Iterate.

Figure 5: Performance comparison on WorldCast . (a) Category-wise scores across L3 and L4. (b) L4 scores, with dashed extensions and labels showing the absolute gap from L3.

Panel of Table 2 — Table 2: Interleaved versus grid generation. Switching to grid generation reduces overall performance for both models, with a much larger drop for GPT-Image-2.5.

Figure 8: Visual coding as procedural guidance. Left: Result comparisons with different model harnesses. Right: code-generated intermediate states guide refinements of a towel-folding sequence.

Model Paired with GPT-6 Astra	Harness Enhancement	Observation- Conditioned Transitions	Physical Simulation	Adaptive Task Completion	Physical Properties	Visual Cues	In-Context Procedural Learning	Logic and Rules	Visual Goal Realization	Overall
GPT-Image-2.5	- (GPT-6 Responses, baseline)	85.6	69.6	66.1	73.6	76.7	71.6	74.3	61.3	72.8
	Inspection loop	92.4 ( $+6.8$ )	86.2 ( $+16.6$ )	73.2 ( $+7.1$ )	84.4 ( $+10.8$ )	69.5 ( $-7.2$ )	72.9 ( $+1.3$ )	81.1 ( $+6.8$ )	69.2 ( $+7.9$ )	80.2 ( $+7.4$ )
	Coding-first inspection loop	90.7 ( $+5.1$ )	88.6 ( $+19.0$ )	74.9 ( $+8.8$ )	80.1 ( $+6.5$ )	74.5 ( $-2.2$ )	74.3 ( $+2.7$ )	85.7 ( $+11.4$ )	81.3 ( $+20.0$ )	82.4 ( $+9.6$ )
Nano Banana Pro	- (GPT-6 Responses, baseline)	58.9	68.1	56.7	64.0	60.1	63.9	66.9	48.3	61.2
	Inspection loop	74.7 ( $+15.8$ )	82.1 ( $+14.0$ )	65.3 ( $+8.6$ )	77.2 ( $+13.2$ )	68.6 ( $+8.5$ )	69.8 ( $+5.9$ )	74.3 ( $+7.4$ )	60.6 ( $+12.3$ )	72.5 ( $+11.3$ )
	Coding-first inspection loop	78.1 ( $+19.2$ )	82.0 ( $+13.9$ )	65.8 ( $+9.1$ )	72.1 ( $+8.1$ )	63.3 ( $+3.2$ )	66.7 ( $+2.8$ )	81.9 ( $+15.0$ )	62.3 ( $+14.0$ )	72.9 ( $+11.7$ )

Table 3: Comparisons of harnesses for agentic visual generation using GPT-6 Astra as the language model. The coding-first inspection loop yields the best overall performance.

Figure 9: Validation of the automatic evaluation pipeline. (a) Correlation between human and GPT-5.5 win rates. Scores (win = 1, tie = 0.5, loss = 0) are averaged across pairwise comparisons for each model. (b) Correlation between evaluator scores from agent-selected and human-selected keyframes.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Data distribution across the eight WorldCast categories. Labels report cases/input images/requested visual states.

Figure 11: Performance across TC, CP, and VC on WorldCast .

Figure 12: Overall model scores assigned by three LMM judges.

Method	L3 Tasks			L4 Tasks					Overall
	Observation- Conditioned Transitions	Physical Simulation	Adaptive Task Completion	Physical Properties	Visual Cues	In-Context Procedural Learning	Logic and Rules	Visual Goal Realization
SenseNova-U1	24.1 $\pm$ $0.73$	24.6 $\pm$ $0.43$	24.7 $\pm$ $0.39$	19.9 $\pm$ $0.84$	19.1 $\pm$ $1.21$	20.1 $\pm$ $0.45$	5.9 $\pm$ $0.34$	12.2 $\pm$ $0.21$	20.1 $\pm$ $0.30$
Emu3.5	17.4 $\pm$ $0.43$	13.8 $\pm$ $0.29$	18.5 $\pm$ $0.84$	6.6 $\pm$ $0.49$	10.3 $\pm$ $0.71$	8.1 $\pm$ $0.54$	4.8 $\pm$ $0.50$	7.6 $\pm$ $0.56$	12.3 $\pm$ $0.16$
BAGEL	4.9 $\pm$ $0.43$	7.0 $\pm$ $0.42$	8.3 $\pm$ $0.28$	8.5 $\pm$ $0.48$	12.5 $\pm$ $0.11$	7.0 $\pm$ $1.07$	2.9 $\pm$ $0.32$	4.0 $\pm$ $0.67$	6.8 $\pm$ $0.13$
Qwen-Image	7.8 $\pm$ $0.29$	6.8 $\pm$ $0.16$	6.2 $\pm$ $0.63$	4.0 $\pm$ $0.56$	6.9 $\pm$ $0.31$	6.1 $\pm$ $0.49$	0.6 $\pm$ $0.16$	3.4 $\pm$ $0.21$	5.6 $\pm$ $0.12$
GPT-Image-2.5	85.6 $\pm$ $1.12$	69.6 $\pm$ $0.79$	66.1 $\pm$ $1.01$	73.6 $\pm$ $1.84$	76.7 $\pm$ $1.08$	71.6 $\pm$ $0.92$	74.3 $\pm$ $1.13$	61.3 $\pm$ $0.74$	72.8 $\pm$ $0.39$

Appendix

Table 4: Model stability across repeated generations. Scores are measured by GPT-5.5 and reported as mean

\pm

standard deviation over three generation runs. Model names and ordering follow Table 1 .

Figure 13: Usage patterns of coding artifacts under GPT-6 Astra with Codex. Top: the role and category distribution of coding artifacts across 320 cases. Bottom: representative examples using guiding overlays and simulated intermediate states as direct visual inputs.

Figure 14: Detailed Qualitative Examples, Model Outputs and Error Patterns.

World Models

World models represent and predict how environments change, helping agents plan, act, and reason about possible futures.

See all 363 papers

World Models Meet Language Models: On the Complementarity of Concrete and Abstract Reasoning
Jun 2, 2026Yucheng Zhou, Wei Tao, Yiwen Guo +1On-Policy Self-Distillation World Models
University of Macau · LIGHTSPEED · Independent Researcher
FutureWorlds: Learning Robotic World Models from Alternative Futures
Oct 1, 2026Hao Wu, Shengju Qian, Weiyan Wang +5Action-Conditioned Video Generation World Model Learning
HKUST (GZ) · CUHK · Tencent +3
Orca: The World is in Your Mind
Jun 29, 2026Yihao Wang, Yuheng Ji, Mingyu Cao +54World Model Learning Multimodal Pretraining

Jun 2, 2026J/K move · Enter open · S save

World Models Meet Language Models: On the Complementarity of Concrete and Abstract Reasoning

Yucheng Zhou, Wei Tao, Yiwen Guo, Jianbing Shen

University of Macau · LIGHTSPEED · Independent Researcher

On-Policy Self-Distillation World Models Multimodal Reasoning VLM Reasoning Visually Grounded Reasoning

Open paper AskPDF arXiv

World models and multimodal large language models (MLLMs) provide complementary capabilities for predicting future outcomes from static visual observations. World models can generate concrete visual rollouts of possible futures, while MLLMs can reason abstractly over questions, goals, and rules. However, generated rollouts are stochastic and may be visually plausible but task-incorrect, making it necessary to determine when visual simulation is useful, whether a rollout is credible, and how it should influence the final answer. We formulate this problem as controlled concrete reasoning, where a model learns to invoke, verify, and integrate visual future simulation alongside abstract reasoning. To study this setting, we construct two human-verified benchmarks, VRQABench for controllable spatial lookahead and OpenWorldQA for open-domain physical prediction, and propose Privileged-Future On-Policy Self-Distillation (PF-OPSD). During training, PF-OPSD uses ground-truth future videos and answers only as teacher-side privileged context to evaluate on-policy concrete-reasoning trajectories, while the deployable student never observes true futures at test time. Experimental results show that PF-OPSD outperforms baseline by 10.6% and 10.9% on VRQABench and OpenWorldQA, respectively, while increasing robustness to noisy or conflicting rollouts. Our code and dataset are available at https://github.com/yczhou001/PF-OPSD.

Abstract

World Models

Explore similar work

World Models Meet Language Models: On the Complementarity of Concrete and Abstract Reasoning

FutureWorlds: Learning Robotic World Models from Alternative Futures

Orca: The World is in Your Mind