Humans possess a remarkable ability to mentally explore and replay 3D environments they have previously experienced. Inspired by this mental process, we present EvoWorld: a world model that bridges panoramic video generation with evolving 3D memory to enable spatially consistent long-horizon exploration. Given a single panoramic image as input, EvoWorld first generates future video frames by leveraging a video generator with fine-grained view control, then evolves the scene's 3D reconstruction using a feedforward plug-and-play transformer, and finally synthesizes futures by conditioning on geometric reprojections from this evolving explicit 3D memory. Unlike prior state-of-the-arts that synthesize videos only, our key insight lies in exploiting this evolving 3D reconstruction as explicit spatial guidance for the video generation process, projecting the reconstructed geometry onto target viewpoints to provide rich spatial cues that significantly enhance both visual realism and geometric consistency. To evaluate long-range exploration capabilities, we introduce the first comprehensive benchmark spanning synthetic outdoor environments, Habitat indoor scenes, and challenging real-world scenarios, with particular emphasis on loop-closure detection and spatial coherence over extended trajectories. Extensive experiments demonstrate that our evolving 3D memory substantially improves visual fidelity and maintains spatial scene coherence compared to existing approaches, representing a significant advance toward long-horizon spatially consistent world modeling.
Figures & tables
Figure 1: Panoramic world generation with explicit 3D memory. EvoWorld maintains an evolving 3D memory (e.g., via VGGT [ 1 ] ) based on previously generated frames, and leverages it to guide spatially consistent video generation. The vanilla generator without memory [ 2 ] exhibits spatial inconsistency.
Figure 2: Overview of EvoWorld. Starting from a single panoramic frame and view control, EvoWorld generates spatially consistent videos by iteratively alternating between 3D reconstruction and video generation, where the video generation is conditioned on reprojections from the evolving 3D memory.
Algorithm 1 Generating an Explorable World with Explicit 3D Memory p(x0:T∣x0,a0:T)
Model
2D Metrics
3D Metrics
FVD ↓
LMSE ↓
LPIPS ↓
PSNR ↑
SSIM ↑
MEt3R ↓
AUC@30 ↑
LTX-Video-2B [ 13 ]
317.54
0.137
0.456
17.18
0.733
0.1146
0.6431
Wan-2.1 [ 14 ]
228.59
0.115
0.467
16.00
0.697
0.1137
0.4192
CogVideoX-1.5-5B [ 15 ]
205.57
0.109
0.430
15.78
0.716
0.1076
0.6527
CogVideoX-1.5-5B + Our SpherePlücker
148.94
0.071
0.259
19.02
0.778
0.1066
0.8125
ViewCrafter [ 16 ]
168.27
0.097
0.353
18.04
0.758
0.0985
0.7273
Table 1: Quantitative results on 25-frame panoramic video generation on Unity dataset. We report 2D metrics (FVD, LMSE, LPIPS, PSNR, SSIM) and 3D metrics (MEt3R [ 18 ] and AUC@30 [ 1 ] ). Arrows indicate whether lower ( ↓ ) or higher ( ↑ ) is better. All models are fine-tuned on the same panoramic dataset. EvoWorld achieves the best across all metrics.
Figure 3: Qualitative comparison of long-horizon video generation. EvoWorld produces more spatially consistent and geometrically coherent results than GenEx. In this example, EvoWorld accurately follows the conditional path and preserves building structure, while GenEx struggles to maintain layout consistency due to the absence of 3D memory and precise camera control.
FVD ↓
Loop LMSE ↓
SSIM ↑
PSNR ↑
LPIPS ↓
LMSE ↓
GenEx
491.71
0.192
0.714
14.558
0.517
0.184
Unity
EvoWorld
442.79
0.187
0.730
15.495
0.494
0.173
GenEx
516.85
0.199
0.420
12.042
0.594
0.192
UE5
EvoWorld
431.37
0.151
0.500
16.630
0.416
0.148
GenEx
649.29
0.228
0.545
12.164
0.665
0.218
Indoor
EvoWorld
570.44
0.235
0.559
12.990
0.624
0.218
Table 2: Quantitative comparisons on long-horizon video generation. We evaluate recursive generation of three clips from a single panoramic frame, with each segment conditioned on the last frame of the previous one. Overall, EvoWorld outperforms GenEx across metrics on four domains (Unity, UE5, Indoor, and Real-World), showing clear gains in fidelity and perceptual quality.
Model
2D Metrics
3D Metrics
FVD ↓
LMSE ↓
LPIPS ↓
PSNR ↑
SSIM ↑
MEt3R ↓
AUC@30 ↑
GenEx [ 2 ]
199.76
0.113
0.400
17.11
0.743
0.1117
0.6408
Baseline + GCD [ 7 ]
178.75
0.109
0.389
17.33
0.745
0.1127
0.6700
Baseline + SpherePlücker
162.96
0.090
0.275
19.54
0.785
0.1070
0.7958
EvoWorld
106.81
0.065
0.167
22.03
0.826
0.0954
0.8846
Table 3: Ablation study on the Unity dataset evaluating the impact of 3D memory and camera pose representations. “GCD”: Generative Camera Dolly [ 7 ] ; “SpherePlücker”: our Spherical Plücker representation for panoramic videos. Combining SpherePlücker embeddings with 3D memory (EvoWorld) yields the best performance across all metrics.
Algorithm
Target reaching
Frame retrieval
Average
GPT-4o
45.0%
N/A
N/A
GPT-4o + GenEx
83.5%
50.5%
67.0%
GPT-4o + EvoWorld
93.3%
68.8%
81.1 %
Table 4: Downstream task performance evaluating spatial consistency with GPT-4o [ 24 ] as the evaluator. We report target-reaching accuracy and spatial frame-retrieval accuracy. EvoWorld outperforms GenEx on both tasks, demonstrating stronger spatial grounding and controllability in panoramic world generation.
Figure 4: Qualitative comparison of 3D reconstructions using four ground truth (GT) images alone, GT with GenEx-generated frames, and with EvoWorld (in the same scale). GT-only reconstructions are incomplete with holes; adding GenEx frames improves coverage but introduces noise. EvoWorld yields more complete and cleaner reconstructions, demonstrating better spatial consistency.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Polyline Paths
Curved Paths
FVD ↓
MSE ↓
LPIPS ↓
PSNR ↑
SSIM ↑
FVD ↓
MSE ↓
LPIPS ↓
PSNR ↑
SSIM ↑
→ Direct test
SVD [ 5 ]
646.72
0.169
0.503
14.63
0.691
612.25
0.164
0.488
14.93
0.694
CogVideoX-1.5 [ 15 ]
800.29
0.159
0.469
15.52
0.686
820.13
0.163
0.486
15.28
0.679
COSMOS [ 6 ]
1058.40
0.195
0.582
14.78
0.638
1035.93
0.196
0.586
14.76
0.639
Wan-2.1 [ 14 ]
595.57
0.138
0.448
16.62
0.692
506.44
0.139
0.513
15.45
0.668
Appendix
Table S1: Quantitative comparison of single-clip (25-frame) panoramic video generation on the Unity dataset under two types of camera trajectories: polyline paths and curved paths. Models are evaluated using FVD, MSE, LPIPS, PSNR, and SSIM, where ↓ indicates lower is better and ↑ indicates higher is better. Our method, EvoWorld, achieves the best performance across all metrics in both trajectory settings.
Figure S1: Qualitative results of panoramic video generation in the Unity environment. Five key frames are shown from a longer video, with intermediate frames omitted for brevity. GenEx exhibits spatial drift and structural artifacts, while our method (EvoWorld) maintains spatial consistency throughout.
Figure S2: Qualitative results of panoramic video generation in the Unreal Engine 5 (UE5) environment. Five key frames are shown from a longer video, with intermediate frames omitted for brevity. GenEx exhibits spatial drift and missing object, while our method (EvoWorld) maintains spatial consistency throughout.
Figure S3: Qualitative results of panoramic video generation in the indoor environment. Five key frames are shown, with intermediate frames omitted for brevity. GenEx exhibits inconsistent object shapes and numbers, while our method (EvoWorld) maintains spatial consistency throughout.
Figure S4: Qualitative results of panoramic video generation in the real-world environment. Five key frames are shown from a longer video, with intermediate frames omitted for brevity. GenEx exhibits spatial drift and structural artifacts, while our method (EvoWorld) maintains spatial consistency throughout.
Environment
# Videos
# Frames
Distance (m)
Time (s)
Navigation Trajectory
Unity
7,200+
680,000+
260,000+
97,000+
Polyline / Curve
UE5
10,000+
1,000,000+
400,000+
140,000+
Curve
Indoor
34,000+
540,000+
210,000+
77,000+
Step-wise Forward or 22.5∘ Rotation
Real-world
7,200+
540,000+
5,400+
9,000+
Natural Human Walk (Curve)
Appendix
Table S2: Statistics of the four subsets in the Spatial360 dataset.
Figure S5: Examples from Unity Dataset. Each row is a case in the Unity dataset. The first column is camera trajectories visualized in a 3D coordinate system. The remaining five are evenly sampled five frames from the raw sequence.
Figure S6: Examples from UE5 Dataset. Each row is a case in the UE5 dataset. The first column is camera trajectories visualized in a 3D coordinate system. The remaining five are evenly sampled five frames from the raw sequence.
Figure S7: Examples from Indoor Dataset.
Figure S8: Examples from Real-world Dataset.
Figure S9: Comparison of (a) target image, (b) generated result, and (c) reprojection visualization. During memory construction, a relatively high threshold filters out the moving person and parts of the trees swaying in the wind. However, in the generation stage, the model still generates corresponding human and tree structures based on the input image elements.
Model
Function
FPS
CogVideoX-1.5
Generation
0.12
GenEx
Generation
0.33
EvoWorld
Generation
0.32
EvoWorld
Mem update
4.20
EvoWorld
Generation + Mem update
0.30
Appendix
Table S3: Comparison of generation and memory update speed (FPS).
Figure S10: Text-image prompt template for target reaching task
Figure S11: Image examples for target reaching task
Figure S12: Text-image prompt template for spatially-aware frame retrieval task
Figure S13: Image examples for spatially-aware frame retrieval task
Figure S14: Left: Pixel coordinate and Spherical Polar coordinate systems; Middle: rotation in Spherical coordinates corresponds to rotation in 2D image; Right: expansion from panorama to cubemap or composition in reverse. The figures are borrowed from [ 2 ] for explanation.
Polyline Paths
Curved Paths
LMSE ↓
LPIPS ↓
PSNR ↑
SSIM ↑
LMSE ↓
LPIPS ↓
PSNR ↑
SSIM ↑
GenEx
0.046 0.118
0.123 0.049
24.190 2.843
0.865 0.035
0.113 0.262
0.400 0.093
17.110 2.165
0.743 0.046
EvoWorld
0.041 0.106
0.108 0.047
24.357 2.622
0.869 0.033
0.065 0.161
0.167 0.065
22.026 2.218
0.826 0.038
Appendix
Table S4: Quantitative comparison for different camera trajectories in Unity. Metrics are reported as mean std for latent MSE, LPIPS, PSNR, and SSIM under Polylines and Curved Paths. Except for PSNR in the Curved Path setting, EvoWorld shows lower standard deviation than GenEx.
LMSE ↓
LPIPS ↓
PSNR ↑
SSIM ↑
mean std
mean std
mean std
mean std
Unity
GenEx
0.184 0.401
0.517 0.081
14.558 1.270
0.714 0.037
EvoWorld
0.173 0.384
0.494 0.086
15.495 1.456
0.730 0.038
UE
GenEx
0.192 0.476
0.594 0.048
12.043 1.598
0.420 0.045
Appendix
Table S5: Quantitative comparison with mean std for MSE, LPIPS, PSNR, and SSIM across different domains and models. For MSE, EvoWorld has a lower standard deviation in 3 out of 4 domains. EvoWorld exhibits higher standard deviation in 10 of the 12 LPIPS, PSNR, and SSIM comparisons across the four domains.
Recent advances in video generation enable a new paradigm for 3D scene creation: generating camera-controlled videos that simulate scene walkthroughs, then lifting them to 3D via feed-forward reconstruction techniques. This generative reconstruction approach combines the visual fidelity and creative capacity of video models with 3D outputs ready for real-time rendering and simulation. Scaling to large, complex environments requires 3D-consistent video generation over long camera trajectories with large viewpoint changes and location revisits, a setting where current video models degrade quickly. Existing methods for long-horizon generation are fundamentally limited by two forms of degradation: spatial forgetting and temporal drifting. As exploration proceeds, previously observed regions fall outside the model's temporal context, forcing the model to hallucinate structures when revisited. Meanwhile, autoregressive generation accumulates small synthesis errors over time, gradually distorting scene appearance and geometry. We present Lyra 2.0, a framework for generating persistent, explorable 3D worlds at scale. To address spatial forgetting, we maintain per-frame 3D geometry and use it solely for information routing -- retrieving relevant past frames and establishing dense correspondences with the target viewpoints -- while relying on the generative prior for appearance synthesis. To address temporal drifting, we train with self-augmented histories that expose the model to its own degraded outputs, teaching it to correct drift rather than propagate it. Together, these enable substantially longer and 3D-consistent video trajectories, which we leverage to fine-tune feed-forward reconstruction models that reliably recover high-quality 3D scenes.
Interactive world models continually generate video by responding to a user's actions, enabling open-ended generation capabilities. However, existing models typically lack a 3D representation of the environment, meaning 3D consistency must be implicitly learned from data, and spatial memory is restricted to limited temporal context windows. This results in an unrealistic user experience and presents significant obstacles to downstream tasks such as training agents. To address this, we present PERSIST, a new paradigm of world model which simulates the evolution of a latent 3D scene: environment, camera, and renderer. This allows us to synthesise new frames with persistent spatial memory and consistent geometry. Both quantitative metrics and a qualitative user study show substantial improvements in spatial memory, 3D consistency, and long-horizon stability over existing methods, enabling coherent, evolving 3D worlds. We further demonstrate novel capabilities, including synthesising diverse 3D environments from a single image, as well as enabling fine-grained, geometry-aware control over generated experiences by supporting environment editing and specification directly in 3D space. Project page: https://francelico.github.io/persist.github.io
We present OctWorld, a video diffusion framework with persistent 3D memory for generating explorable, world-consistent, and high-fidelity visual scenes. Given a single image, OctWorld performs stable autoregressive world generation along user-specified camera trajectories. We focus on long-range generation, characterized by extended camera paths and wide viewpoint coverage, where preserving spatial consistency is particularly challenging when previously generated regions are revisited. To address this problem, we introduce OctMap, an extensible and spatially adaptive 3D memory that progressively fuses generated visual observations and their corresponding depth maps into a global representation. OctMap employs TSDF fusion within a dynamic sparse octree whose spatial resolution adapts to image evidence. This design preserves geometric and appearance details across diverse scene scales while maintaining low memory overhead. Experiments demonstrate that OctWorld generates long-range, spatially consistent videos and outperforms prior methods on both existing benchmarks and challenging long-range generation settings. OctMap also provides clear advantages over point-based caches and fixed-resolution TSDF volumes. Project page: https://github.com/microsoft/OctWorld
Zelong Lv, Sicheng Xu, Jianfeng Xiang +5
University of Science and Technology of China · Microsoft Research Asia · Tsinghua University