UniWorld: A Unified World Model for Real-Time Visual World Simulation

Xin Zhang; Yabo Chen; Shuaiwei Wang; Zixuan Duan; Haibin Huang; Feng Xu; Xuelong Li

cs.CVOct 23, 2026

UniWorld: A Unified World Model for Real-Time Visual World Simulation

Authors: Xin Zhang, Yabo Chen, Shuaiwei Wang, Zixuan Duan, Haibin Huang, Feng Xu, Xuelong Li

Organizations: Fudan University · Institute of Artificial Intelligence (TeleAI), China Telecom · Jilin University · Nanjing University

Abstract

Continuous visual world simulation requires coupling evolving scene geometry with temporally persistent appearance. We present UniWorld, a unified framework that integrates depth trajectories and reference appearance within a shared video generation model. The central design maintains visual and geometric histories on a common timeline. Noise-aligned overlapping continuation places generated history at the prescribed sampling noise level and commits only newly generated latents. Rolling depth latent state preserves the corresponding geometric controls, incrementally reads newly available geometry, and keeps the conditioning fixed within each sampling window. Together, these mechanisms ensure exact preservation of latent history while bounding the active depth state by the window size. We evaluate continuous generation, paired history interventions, and long-horizon behavior across diverse scene categories. The same generator supports estimated, native, and on-demand simulator depth, while cached and incremental execution paths produce bitwise-identical outputs. Appearance changes and geometric control branches operate through the same interface without modifying committed history. The measured steady-state DiT execution path achieves real-time generation throughput, demonstrating the efficiency of UniWorld for continuous visual world simulation. Project page: https://zephipage.github.io/UniWorld.

Figures & tables

Figure 1: UniWorld: from world geometry to continuous visual observations. Conceptual overview: depth trajectories describe geometric evolution, while reference RGB specifies appearance for a shared video renderer.

Figure 2: UniWorld: a unified renderer for heterogeneous world sources. Driving, human-activity, simulation, and synthetic sources are converted into a common rolling depth state. Each continuation window packs video history, depth/control, reference, and separator tokens into a shared DiT sequence. Overlapping positions are retained, only the newly required depth suffix is read, and only the newly generated video suffix is committed to the output stream.

Figure 3: Aligned visual and geometric state. Hatched intervals denote inherited history. Depth is read exactly once over

[0,102)

[102,192)

, and

[192,217)

. Dashed pixel lines mark the nominal latent-commitment boundaries, while stable incremental pixel output retains an additional five-frame blend tail.

Input: $I_{\mathrm{ref}},p$ , depth source $\mathcal{D}$ , windows $\{\Omega_{i}\}$ , global noise $\epsilon$ , grid $\{t_{k}\}_{k=0}^{K}$
$c_{\mathrm{g}}\leftarrow E_{\mathrm{Qwen}}(I_{\mathrm{ref}},p)$ ; $I_{0}\leftarrow I_{\mathrm{ref}}$ ; $Z\leftarrow[\,]$ ; depth state $\leftarrow[\,]$
for each $\Omega_{i}=[a_{i},b_{i})$ do
	$C_{i}\leftarrow\operatorname{RollDepth}(\mathcal{D},\Omega_{i})$
	$H_{i}\leftarrow Z[a_{i}:a_{i}+h_{i}]$ ; $\epsilon_{i}\leftarrow\epsilon[a_{i}:b_{i}]$
	$\mathcal{C}_{i}\leftarrow\bigl(C_{i},\operatorname{EncodeBoundary}(I_{i}),c_{\mathrm{g}}\bigr)$
	$z_{i}\leftarrow\mathcal{P}_{i,t_{0}}(\epsilon_{i})$

Algorithm 1 UniWorld continuous generation

Figure 4: One model across four scene categories. Actual reference, GT, and generated frames at the first continuation boundary and final frame. Rows use the first frozen ablation identity per category; selection is independent of quality scores.

Figure 5: Continuous generation over 60–120 seconds. Ten-second aggregates with category-stratified source-group 95% bootstrap bands. The duration groups use different sources; the 120-second group is a training-split stress test.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Clips	Collection
Driving	25	Nexar
Human activity	25	DanceTrack (10), BEHAVE (4), local originals (11)
Simulation	25	PointOdyssey v1.2 (15), TartanAir V2 (10)
Synthetic	25	MIND (25)

Appendix

Table S2: Primary 30-second sources. Every category initially uses estimated VDA depth. Native-depth inputs are a paired intervention on the same 25 simulation sources.

Condition	Intervention
Full	Project all 12 history latents at every sampling step; canonical rolling depth.
Boundary RGB only	Retain independent boundary-image conditioning; no target-history projection.
Endpoint only	Project the full overlap only at $t=1$ , after all model evaluations.
Fresh history noise	Redraw overlap noise once, then keep it fixed across steps; suffix noise is unchanged.
Last 6 / last 2 latents	Project only $[h_{i}-q,h_{i})$ with $q=6$ or 2, still discarding all $h_{i}$ overlap latents.
Direct global slice	Read canonical $C[a_{i}:b_{i}]$ independently at each advance.

Appendix

Table S5: Nine completed conditions, each applied to the same 24 sources. Geometry and commitment boundaries remain fixed.

Figure S1: Paired effect sizes and marginal 95% intervals for non-equivalent interventions, each minus full history projection. Endpoint-only is identical to boundary-only; direct slicing and aligned re-encoding are exact zero-effect controls.

Property	Result
Reconstruct canonical generation	24/24 full-method outputs exactly match
Incremental geometric reads	24/24 rolling/direct outputs bitwise equal
Canonical encoding phase	24/24 aligned re-encoded outputs bitwise equal
On-demand source equivalence	3/3 online/cached outputs bitwise equal
Committed control-branch history	Initial-window latents bitwise equal

Appendix

Table S9: Temporal state consistency in completed model runs. Equality is checked on latent tensors or complete video outputs.

Figure S2: Logical depth reads and retained state plus consumer copy at

960\times 544

. The resident global source is additional and grows with duration. Rolling reduces read positions but retains an extra owned state tensor.

Figure S3: One shared renderer for appearance and geometric control. Engine RGB and generated frames for warm/cool references and a control branch, at fixed times. Warm/cool conditions share depth; control changes begin at frame 357 and the camera revisits at 20 seconds.

Video World Models

Video world models generate and predict action-conditioned visual futures to simulate changing scenes for embodied agents.

See all 257 papers

WorldDirector: Building Controllable World Simulators with Persistent Dynamic Memory
Jul 2, 2026Hanlin Wang, Hao Ouyang, Qiuyu Wang +10Physical Consistency in Video Generation Camera-Controlled Video Generation
1HKUST · 2Ant Group · 3ZJU +1
WorldWeave: Growing Persistent Geometric Worlds for Video Generation
Sep 28, 2026Yifan Huang, Lifan Jiang, Qingyue Hao +5Memory-Augmented Video Generation 3D Scene Generation
Zhejiang University · Daerwen AI
DreamX-World 1.0: A General-Purpose Interactive World Model
Jun 15, 2026DreamX Team, Yancheng Bai, Rui Chen +20Camera-Controlled Video Generation Real-Time Interactive Video Generation
DreamX Team

Jul 2, 2026J/K move · Enter open · S save

WorldDirector: Building Controllable World Simulators with Persistent Dynamic Memory

Hanlin Wang, Hao Ouyang, Qiuyu Wang, Wen Wang+9

1HKUST · 2Ant Group · 3ZJU+1

Physical Consistency in Video Generation Camera-Controlled Video Generation Memory-Augmented Video Generation Controllable Video Generation Video World Models

Open paper AskPDF arXiv

We present WorldDirector, a highly controllable video world model framework designed for persistent dynamic object memory and unrestricted viewpoint exploration. Unlike existing world models that entangle physical dynamics with pixel rendering and rely on continuous visual observation to sustain motion, our framework explicitly decouples semantic motion orchestration from visual generation. By leveraging an LLM to coordinate 3D trajectories with camera movements and subsequently employing these orchestrated trajectories as control signals for video generation, our approach ensures strict physical logic and appearance stability, successfully preserving the exact visual identities of dynamic entities even when they re-enter the scene after prolonged periods out of view. Experimental results demonstrate that our method supports the synthesis of complex and extended events with unprecedented controllability and persistent dynamic object memory. Project Page: https://worlddirector.github.io/

Abstract

Video World Models

Explore similar work

WorldDirector: Building Controllable World Simulators with Persistent Dynamic Memory

WorldWeave: Growing Persistent Geometric Worlds for Video Generation

DreamX-World 1.0: A General-Purpose Interactive World Model