AEON: Unifying Video and 3D World Models

Weiqi Zhang; Wenyuan Zhang; Yu-Shen Liu; Junsheng Zhou

cs.CVOct 20, 2026

AEON: Unifying Video and 3D World Models

Authors: Weiqi Zhang, Wenyuan Zhang, Yu-Shen Liu, Junsheng Zhou

Organizations: School of Software, Tsinghua University, Beijing, China

Abstract

Current world models for explicit world simulation typically follow two distinct pathways: (1) video generative models (e.g., Sora), which simulate realistic appearances but fail to preserve 3D consistency; and (2) 3D generative models (e.g., Marble), which generate geometrically consistent scenes but often yield static, less realistic observations. We present AEON, a unified framework that bridges video-based and 3D-based world models and combines the benefits of both. The key insight of AEON lies in a generative model that operates within the text- or image-conditioned video latent space while producing renderable 4D graphics primitives---specifically, 4D Gaussian Splatting (4DGS)---as the explicit world representation. We train a 4D reconstruction model to predict both 4DGS and camera poses from monocular videos, serving as our initialized 4D decoder. Furthermore, we design a domain adapter that translates generative video latents into reconstructive 4D latents through a distillation process on the video VAE. AEON naturally supports video-based world modeling by rendering 4DGS at predicted camera trajectories, and constructs a persistent 4D environment with scene dynamics for realistic world simulation. Consequently, AEON empowers a diverse range of world modeling applications and achieves state-of-the-art performance on standard benchmarks. Project Page: https://weiqi-zhang.github.io/AEON

Figures & tables

Paradigm	Realistic	3D Consistent	Dynamic	Persistence	Real-time Exp.
Video WM (e.g. Sora)	✓	✗	✓	✗	✗
3D WM (e.g. Marble)	✗	✓	✗	✓	✓
AEON	✓	✓	✓	✓	✓

Table 1: Comparison of different world modeling paradigms across fundamental characteristics.

Figure 2: Overview of AEON. Top (1). The 4D Reconstruction Model (4DRM) decodes a monocular video into a 4DGS world through four task-specific heads that respectively predict depth, Gaussian appearance, dynamic evolution, and camera poses. The resulting primitives are rasterized into RGB, depth, and normal maps for end-to-end supervision. Bottom (2). We train AEON for generative world modeling by distilling the world generative capability of a frozen video diffusion model (e.g., Wan 2.2 [ 38 ] ) and the 4D reconstructive capability of a teacher 4DRM via a Domain Adapter and a learnable student 4DRM. The generated explicit 4DGS world enables versatile world modeling applications.

Figure 3: Qualitative reconstruction comparison on Dycheck and DaVis.

Method	I2V_Subject $\uparrow$	Subject_Consistency $\uparrow$	Motion_Smoothness $\uparrow$	Aesthetic_Quality $\uparrow$
4DNex [ 7 ]	0.796	0.790	0.968	0.342
Lyra [ 3 ]	0.865	0.864	0.974	0.459
AEON (Wan 2.1)	0.912	0.864	0.976	0.519
AEON (Wan 2.2)	0.913	0.875	0.980	0.531

Table 2: Quantitative comparison on 4D generation. All outputs are rendered along shared preset camera trajectories and scored with VBench [ 13 ] metrics.

Method	Dycheck			DaVis
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
L4GM [ 31 ]	$8.673$	$0.354$	$0.652$	$20.589$	$0.667$	$0.311$
4DGT [ 48 ]	$12.267$	$0.413$	$0.542$	$20.362$	$0.667$	$0.313$
MoVies [ 22 ]	$11.840$	$0.341$	$0.545$	$20.736$	$0.688$	$0.307$
NeoVerse [ 50 ]	$14.269$	$0.363$	$0.501$	$23.929$	$0.735$	$0.283$
StreamSplat [ 46 ]	$11.876$	$0.375$	$0.532$	$23.748$	$0.712$	$0.273$

Table 3: Video-to-4D reconstruction comparison.

Method	Subj. Consist. $\uparrow$	Back. Consist. $\uparrow$	Temp. Flick. $\uparrow$	Motion Smooth. $\uparrow$	Aesth. Quality $\uparrow$	Imag. Quality $\uparrow$
TrajectoryCrafter [ 54 ]	94.48	96.57	95.49	98.79	64.81	58.53
RecamMaster [ 4 ]	94.49	95.58	97.64	97.79	61.26	56.34
NeoVerse [ 50 ]	90.24	95.21	95.01	97.39	59.17	57.98
AEON	94.64	96.77	96.69	98.11	69.02	58.65

Table 4: Novel trajectory rendering comparison under VBench [ 13 ] metrics.

Variant	Dycheck			DaVis
Variant	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Conv3D projection	12.086	0.314	0.606	12.813	0.276	0.701
W/o Distill	12.390	0.349	0.537	16.718	0.503	0.489
W/o Duration	12.500	0.372	0.535	22.057	0.718	0.328
Full	14.487	0.421	0.469	24.661	0.771	0.243

Table 5: Ablation study on Dycheck and DaVis.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Dynamic	Pose	Depth	Real	#Scenes	Resample
PointOdyssey [ 61 ]	✓	✓	✓	✗	0.1K	100 $\times$
TartanAir [ 40 ]	✓	✓	✓	✗	0.5K	50 $\times$
DynamicReplica [ 16 ]	✓	✓	✓	✗	0.5K	20 $\times$
OmniWorld-Game [ 65 ]	✓	✓	✓	✗	0.06K	100 $\times$
SAIL-VOS 3D [ 12 ]	✓	✓	✓	✗	0.16K	50 $\times$
Spring [ 23 ]	✓	✓	✓	✗	0.03K	1000 $\times$

Appendix

Table 6: Training datasets. Resample is the factor by which each dataset is repeated during training.

	Variant	Dycheck			DaVis
	Variant	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Adapter	Conv3D projection	12.086	0.314	0.606	12.813	0.276	0.701
	Conv branch only	12.332	0.333	0.533	15.409	0.465	0.533
	Perceiver branch only	13.086	0.389	0.596	14.262	0.442	0.552
Training	W/o Stage 3	13.670	0.361	0.517	22.953	0.705	0.299
4DRM	W/o geometric prior	12.341	0.358	0.543	22.268	0.695	0.349

Appendix

Table 7: Additional ablation studies on Dycheck and DaVis.

Figure 4: Additional qualitative reconstruction comparisons on DaVis. AEON produces more coherent geometry and finer details in dynamic regions.

Video World Models

Video world models generate and predict action-conditioned visual futures to simulate changing scenes for embodied agents.

See all 248 papers

WorldAlign: Decoupled 4D Reward for World-Consistent Video Generation
Oct 8, 2026Jing He, Kaixin Ding, Xingye Tian +5Video Generation Temporal Consistency in Video Generation
The Hong Kong University of Science and Technology (Guangzhou) · KlingAI · The University of Hong Kong +1
DreamWorld: Unified World Modeling in Video Generation
Feb 28, 2026Boming Tan, Xiangdong Zhang, Ning Liao +73D-Aware Video Generation Video World Models
University of Science and Technology of China, Hefei, China · Shanghai Jiao Tong University, Shanghai, China
OuroWorld: Bringing Any 3D World Alive as Diverse, Endlessly Looping 3D Cinemagraphs
Oct 8, 2026You-Zhe Xie, Ting-Wei Chou, Yu-Hsuan Li +33D-Aware Video Generation Dynamic Scene Reconstruction
National Yang Ming Chiao Tung University · Alaya Lab

Oct 8, 2026J/K move · Enter open · S save

WorldAlign: Decoupled 4D Reward for World-Consistent Video Generation

Jing He, Kaixin Ding, Xingye Tian, Guibao Shen+4

The Hong Kong University of Science and Technology (Guangzhou) · KlingAI · The University of Hong Kong+1

Video Generation Temporal Consistency in Video Generation Video Diffusion Models Geometric Consistency in Video Generation Physical Consistency in Video Generation RL for Video Generation

Open paper AskPDF arXiv

Faithful visual world simulation requires generated videos to maintain 4D world consistency, encompassing both static and dynamic consistency. Static consistency requires coherent 3D structure in static environments across viewpoints, while dynamic consistency requires plausible subject motion and consistent appearance over time. Geometry-aware post-training offers a promising way to improve world consistency. However, existing methods often rely on a static-scene assumption. Even those that accommodate dynamic scenes struggle to provide reliable static-consistency feedback, while dynamic consistency is often overlooked or inadequately assessed. To address these limitations, we introduce WorldAlign, a decoupled 4D reward framework that semantically separates static regions and dynamic subjects and provides feedback by aligning each with a world prior suited to its assumptions. For static regions, WorldAlign aligns static geometry with a geometric world prior through semantically guided masked reprojection, enabling more reliable static-consistency evaluation; an auxiliary camera-motion reward discourages nearly static solutions. For dynamic subjects, WorldAlign uses a strong vision-language model (VLM) as a dynamic world prior and constructs a VLM-as-a-judge reward based on sample-specific checklists that assess dynamicity, physical plausibility, shape, and texture consistency. This decoupled design enables more effective online post-training without requiring human preference annotations. Across two pretrained image-to-video generators, Wan2.1 and Wan2.2, WorldAlign jointly improves static and dynamic consistency over existing methods without suppressing overall or subject motion. These results support decoupled world-prior alignment for more faithful visual world simulation. Project page: https://worldalign.github.io/.

Abstract

Video World Models

Explore similar work

WorldAlign: Decoupled 4D Reward for World-Consistent Video Generation

DreamWorld: Unified World Modeling in Video Generation

OuroWorld: Bringing Any 3D World Alive as Diverse, Endlessly Looping 3D Cinemagraphs