Code Plans, Diffusion Renders: Open-Ended Generative World Modeling

Zixun Fang; Yawen Shao; Kai Zhu; Jie Xiao; Shihan Chen; Yu Liu; Xueyang Fu; Yang Cao

cs.CVSep 22, 2026

Code Plans, Diffusion Renders: Open-Ended Generative World Modeling

Authors: Zixun Fang, Yawen Shao, Kai Zhu, Jie Xiao, Shihan Chen, Yu Liu, Xueyang Fu, Yang Cao,

Organizations: USTC · TongYi Lab

Abstract

We introduce CoDeR, a new paradigm for world modeling. Unlike existing video world models that implicitly represent world dynamics through visual observations, our system explicitly constructs an executable world with code and employs video generation models for visual realization. Specifically, we coordinate five complementary roles to translate high-level concepts into structured world rules, executable dynamics, and perceptual observations. This design enables long-term memory, open-ended interactions, autonomous world evolution, and multi-agent scenarios, where multiple entities can act, interact, and evolve persistently beyond the current observation. Extensive experiments demonstrate that our framework substantially extends the capabilities of existing world models, enabling long-term memory, open-ended interactions, autonomous evolution, and persistent multi-agent dynamics, while achieving state-of-the-art performance across multiple evaluation settings. Code and model weights will be made publicly available. Project Page: CoDeR.

Figures & tables

Figure 1: Our conception of a world: the Creator establishes its rules, the Executors build the world accordingly, and the Artist brings it to life.

Figure 2: Method Overview. Left: The pipeline of our CoDeR. The God of Concepts conveys its intent to the Creator. The Creator (LLM) orchestrates the entire world, while the Executors (coding agents) implement the whitebox world. Finally, the Artists (video generation models) render observations along the trajectories sampled by the Travelers. Right: The Observation as World Registration paradigm. Once an observation is generated, it is registered back into the world using its depth map.

Figure 3: We sample a rotational trajectory to examine whether the “Coding World System” mark is consistently maintained on the wall. The results show that ABot-World exhibits severe quality degradation, while Matrix-Game 3.0 fails to preserve the mark. LingBot-World fails to accurately respond to the control signals, resulting in duplicated frames. In contrast, our method successfully follows the rotational trajectory while consistently preserving the mark on the wall.

Figure 4: Left: We support open-ended interactions ranging from playing the piano to riding a dragon. Right: Comparison with EgoSim, which fails to open the door, while our method successfully opens it and reveals the new environment.

Figure 5: We demonstrate the world evolution mechanism of our method in comparison with other approaches. In the first frame, a tap is pouring water into a glass cup. The camera then moves away from the cup and later returns to examine whether the water level has continued to rise. HyDRA produces a static water column, while LiveWorld exhibits obvious visual artifacts. In contrast, our method continuously updates the water level even when the cup is out of sight.

Figure 6: We further explore multi-agent scenarios in our system. When Agent 1 takes down a target, the event is simultaneously observed by Agent 2. Likewise, when Agent 2 moves forward and shoots another target, the event is also observed by Agent 1.

Methods	Camera Ctrl	Object Ctrl	Content Align	3D Consist	Photo Consist	Style Consist	Subjective Qual	Motion Acc	Motion Mag	Motion Smooth	Average
ABot-World ( Jiang et al., 2026 )	92.47	84.23	76.94	80.60	83.56	83.22	39.95	50.33	26.89	74.21	69.24
Matrix-Game 3.0 ( Wang et al., 2026e )	96.72	84.61	83.12	81.10	86.96	83.42	59.04	60.79	24.75	81.50	74.20
LingBot-World ( Robbyant Team et al., 2026 )	90.50	87.73	75.29	86.28	90.03	85.39	65.71	59.48	27.93	79.60	74.79
Ours	98.91	92.25	98.44	88.65	91.10	89.40	69.08	82.76	84.92	83.31	87.88

Table 1: Quantitative Comparison. Our method outperforms all competing methods across all metrics.

Methods	WorldScore			VBench
Methods	Content Align	Photo Consist	Subjective Qual	Imaging Quality	Aesthetic Quality	Subject Consistency	Dynamic Degree
w/o Vis. Cue Hack.	82.48	89.21	49.05	63.52	66.18	87.54	97.29
w/o Obs. as WR.	89.06	84.85	53.98	69.37	70.44	84.63	98.02
Full Method	98.44	91.10	69.08	70.77	72.29	96.31	98.16

Table 2: Ablation Study. Both Visual Cue Hacking and Observation as World Registration contribute to improved visual quality across WorldScore and VBench metrics.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: User Study. Human evaluation across Visual Quality, Interaction Consistency, Memory and Persistence, World Evolution, and Overall Preference. Our method consistently achieves the highest scores across all evaluated dimensions.

Figure 8: Effect of Visual Cue Hacking. Directly conditioning on whitebox renderings produces monotonous, render-like videos with rigid edges, whereas our Visual Cue Hacking strategy encourages the model to use the whitebox input as a structural cue, resulting in richer appearance and higher visual quality.

Figure 9: Given the first frame, we obtain its segmentation map and depth map, and then reconstruct a corresponding whitebox world.

Figure 10: Limitation of Long-Horizon Video Generation. Our Artist model is trained with a fixed 124-frame window and generates long videos through chunk-based rollout. Noticeable visual degradation emerges due to accumulated generation errors.

Figure 11: Limitations of Whitebox Modeling. Our current whitebox world may exhibit failure cases in complex hand-object interactions and 3D geometry, including severely distorted hand poses and object interpenetration.

Figure 12: Towards Disentangled Visual Generation. Future Artist models could decompose RGB observations into structured visual representations, including Albedo, Normal, Roughness, and Irradiance, enabling more explicit scene registration, editing, and re-rendering within the CoDeR.

World Models

World models represent and predict how environments change, helping agents plan, act, and reason about possible futures.

See all 358 papers

Code World Model: Coding Agent as World Brain
Aug 26, 2026Yiwen Chen, Guosheng Lin, Chi ZhangWorld Models Code World Models
AGI Lab, Westlake University · Nanyang Technological University
Programmable World Model
Sep 9, 2026Zheng-Hui Huang, Guixu Lin, Jiacheng Lin +8World Models Controllable Video Generation
Alaya Lab
StateCraft: Augmenting Video World Models with Abstract-State Reasoning
Oct 14, 2026Lorenzo Venturoli, Nedko Savov, Xi Wang +2Video World Models Long-Horizon Video Generation
INSAIT, Sofia University "St. Kliment Ohridski" · ETH Zurich · University of Ulm +3

Aug 26, 2026J/K move · Enter open · S save

Code World Model: Coding Agent as World Brain

Yiwen Chen, Guosheng Lin, Chi Zhang

AGI Lab, Westlake University · Nanyang Technological University

World Models Code World Models World Model Learning Video World Models Action-Conditioned World Models

Open paper AskPDF arXiv

World models aim to simulate how complex environments evolve under actions and events, yet existing video-based world models primarily learn dynamics from visual observations, which reveal outcomes rather than the underlying knowledge, rules, and mechanisms governing world evolution. This makes it difficult to maintain persistent consequences and support coherent, open-ended evolution. We introduce Code World Model, a framework that separates world evolution from visual realization by combining the reasoning and coding capabilities of language models with the generative priors of video models. A coding agent serves as the world brain, reasoning about events and their consequences and generating executable code to maintain persistent world state and perform rule-consistent evolution. To connect executable state with visual generation, we introduce a proxy representation that encodes frame-wise spatiotemporal constraints and is compiled into a proxy video, which conditions a video model to render high-fidelity visual observations. We further develop data pipelines for constructing aligned proxy-observation pairs from gameplay and real-world videos. After fine-tuning on paired gameplay data, MiniMax-H3 follows proxy-based spatiotemporal specifications from simple interactive worlds built by the coding agent while preserving rich visual details and dynamics. These results demonstrate the potential of combining code for persistent world evolution with video models for flexible visual realization, providing a new path toward open-ended world models.

Abstract

World Models

Explore similar work

Code World Model: Coding Agent as World Brain

Programmable World Model

StateCraft: Augmenting Video World Models with Abstract-State Reasoning