AEON: Unifying Video and 3D World Models
Organizations: School of Software, Tsinghua University, Beijing, China
Abstract
Current world models for explicit world simulation typically follow two distinct pathways: (1) video generative models (e.g., Sora), which simulate realistic appearances but fail to preserve 3D consistency; and (2) 3D generative models (e.g., Marble), which generate geometrically consistent scenes but often yield static, less realistic observations. We present AEON, a unified framework that bridges video-based and 3D-based world models and combines the benefits of both. The key insight of AEON lies in a generative model that operates within the text- or image-conditioned video latent space while producing renderable 4D graphics primitives---specifically, 4D Gaussian Splatting (4DGS)---as the explicit world representation. We train a 4D reconstruction model to predict both 4DGS and camera poses from monocular videos, serving as our initialized 4D decoder. Furthermore, we design a domain adapter that translates generative video latents into reconstructive 4D latents through a distillation process on the video VAE. AEON naturally supports video-based world modeling by rendering 4DGS at predicted camera trajectories, and constructs a persistent 4D environment with scene dynamics for realistic world simulation. Consequently, AEON empowers a diverse range of world modeling applications and achieves state-of-the-art performance on standard benchmarks. Project Page: https://weiqi-zhang.github.io/AEON
Figures & tables
| Paradigm | Realistic | 3D Consistent | Dynamic | Persistence | Real-time Exp. |
|---|---|---|---|---|---|
| Video WM (e.g. Sora) | ✓ | ✗ | ✓ | ✗ | ✗ |
| 3D WM (e.g. Marble) | ✗ | ✓ | ✗ | ✓ | ✓ |
| AEON | ✓ | ✓ | ✓ | ✓ | ✓ |
| Method | I2V_Subject | Subject_Consistency | Motion_Smoothness | Aesthetic_Quality |
|---|---|---|---|---|
| 4DNex [ 7 ] | 0.796 | 0.790 | 0.968 | 0.342 |
| Lyra [ 3 ] | 0.865 | 0.864 | 0.974 | 0.459 |
| AEON (Wan 2.1) | 0.912 | 0.864 | 0.976 | 0.519 |
| AEON (Wan 2.2) | 0.913 | 0.875 | 0.980 | 0.531 |
| Method | Dycheck | DaVis | ||||
|---|---|---|---|---|---|---|
| PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | |
| L4GM [ 31 ] | ||||||
| 4DGT [ 48 ] | ||||||
| MoVies [ 22 ] | ||||||
| NeoVerse [ 50 ] | ||||||
| StreamSplat [ 46 ] | ||||||
| Method | Subj. Consist. | Back. Consist. | Temp. Flick. | Motion Smooth. | Aesth. Quality | Imag. Quality |
|---|---|---|---|---|---|---|
| TrajectoryCrafter [ 54 ] | 94.48 | 96.57 | 95.49 | 98.79 | 64.81 | 58.53 |
| RecamMaster [ 4 ] | 94.49 | 95.58 | 97.64 | 97.79 | 61.26 | 56.34 |
| NeoVerse [ 50 ] | 90.24 | 95.21 | 95.01 | 97.39 | 59.17 | 57.98 |
| AEON | 94.64 | 96.77 | 96.69 | 98.11 | 69.02 | 58.65 |
| Variant | Dycheck | DaVis | ||||
|---|---|---|---|---|---|---|
| PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | |
| Conv3D projection | 12.086 | 0.314 | 0.606 | 12.813 | 0.276 | 0.701 |
| W/o Distill | 12.390 | 0.349 | 0.537 | 16.718 | 0.503 | 0.489 |
| W/o Duration | 12.500 | 0.372 | 0.535 | 22.057 | 0.718 | 0.328 |
| Full | 14.487 | 0.421 | 0.469 | 24.661 | 0.771 | 0.243 |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Dynamic | Pose | Depth | Real | #Scenes | Resample |
|---|---|---|---|---|---|---|
| PointOdyssey [ 61 ] | ✓ | ✓ | ✓ | ✗ | 0.1K | 100 |
| TartanAir [ 40 ] | ✓ | ✓ | ✓ | ✗ | 0.5K | 50 |
| DynamicReplica [ 16 ] | ✓ | ✓ | ✓ | ✗ | 0.5K | 20 |
| OmniWorld-Game [ 65 ] | ✓ | ✓ | ✓ | ✗ | 0.06K | 100 |
| SAIL-VOS 3D [ 12 ] | ✓ | ✓ | ✓ | ✗ | 0.16K | 50 |
| Spring [ 23 ] | ✓ | ✓ | ✓ | ✗ | 0.03K | 1000 |
| Variant | Dycheck | DaVis | |||||
|---|---|---|---|---|---|---|---|
| PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | ||
| Adapter | Conv3D projection | 12.086 | 0.314 | 0.606 | 12.813 | 0.276 | 0.701 |
| Conv branch only | 12.332 | 0.333 | 0.533 | 15.409 | 0.465 | 0.533 | |
| Perceiver branch only | 13.086 | 0.389 | 0.596 | 14.262 | 0.442 | 0.552 | |
| Training | W/o Stage 3 | 13.670 | 0.361 | 0.517 | 22.953 | 0.705 | 0.299 |
| 4DRM | W/o geometric prior | 12.341 | 0.358 | 0.543 | 22.268 | 0.695 | 0.349 |
Video World Models
Video world models generate and predict action-conditioned visual futures to simulate changing scenes for embodied agents.
See all 248 papers