EvoWorld: Evolving Panoramic World Generation with Explicit 3D Memory

Jiahao Wang; Luoxin Ye; TaiMing Lu; Junfei Xiao; Jiahan Zhang; Yuxiang Guo; Xijun Liu; Rama Chellappa

cs.CVOct 1, 2025

EvoWorld: Evolving Panoramic World Generation with Explicit 3D Memory

Authors: Jiahao Wang, Luoxin Ye, TaiMing Lu, Junfei Xiao, Jiahan Zhang, Yuxiang Guo, Xijun Liu, Rama Chellappa,

Organizations: Johns Hopkins University

Abstract

Humans possess a remarkable ability to mentally explore and replay 3D environments they have previously experienced. Inspired by this mental process, we present EvoWorld: a world model that bridges panoramic video generation with evolving 3D memory to enable spatially consistent long-horizon exploration. Given a single panoramic image as input, EvoWorld first generates future video frames by leveraging a video generator with fine-grained view control, then evolves the scene's 3D reconstruction using a feedforward plug-and-play transformer, and finally synthesizes futures by conditioning on geometric reprojections from this evolving explicit 3D memory. Unlike prior state-of-the-arts that synthesize videos only, our key insight lies in exploiting this evolving 3D reconstruction as explicit spatial guidance for the video generation process, projecting the reconstructed geometry onto target viewpoints to provide rich spatial cues that significantly enhance both visual realism and geometric consistency. To evaluate long-range exploration capabilities, we introduce the first comprehensive benchmark spanning synthetic outdoor environments, Habitat indoor scenes, and challenging real-world scenarios, with particular emphasis on loop-closure detection and spatial coherence over extended trajectories. Extensive experiments demonstrate that our evolving 3D memory substantially improves visual fidelity and maintains spatial scene coherence compared to existing approaches, representing a significant advance toward long-horizon spatially consistent world modeling.

Figures & tables

Figure 1: Panoramic world generation with explicit 3D memory. EvoWorld maintains an evolving 3D memory (e.g., via VGGT [ 1 ] ) based on previously generated frames, and leverages it to guide spatially consistent video generation. The vanilla generator without memory [ 2 ] exhibits spatial inconsistency.

Figure 2: Overview of EvoWorld. Starting from a single panoramic frame and view control, EvoWorld generates spatially consistent videos by iteratively alternating between 3D reconstruction and video generation, where the video generation is conditioned on reprojections from the evolving 3D memory.

\mathbf{x}_{t}\sim p_{\theta}(\mathbf{x}\mid x_{t-1}^{S},\text{\boldmath

\delta

}_{t},\text{\boldmath

\alpha

}_{t},\mathbf{r}_{t})

Algorithm 1 Generating an Explorable World with Explicit 3D Memory

p(\textbf{x}_{0:T}\mid x_{0},\textbf{a}_{0:T})

Model	2D Metrics					3D Metrics
Model	FVD $\downarrow$	LMSE $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	MEt3R $\downarrow$	AUC@30 $\uparrow$
LTX-Video-2B [ 13 ]	317.54	0.137	0.456	17.18	0.733	0.1146	0.6431
Wan-2.1 [ 14 ]	228.59	0.115	0.467	16.00	0.697	0.1137	0.4192
CogVideoX-1.5-5B [ 15 ]	205.57	0.109	0.430	15.78	0.716	0.1076	0.6527
CogVideoX-1.5-5B + Our SpherePlücker	148.94	0.071	0.259	19.02	0.778	0.1066	0.8125
ViewCrafter [ 16 ]	168.27	0.097	0.353	18.04	0.758	0.0985	0.7273

Table 1: Quantitative results on 25-frame panoramic video generation on Unity dataset. We report 2D metrics (FVD, LMSE, LPIPS, PSNR, SSIM) and 3D metrics (MEt3R [ 18 ] and AUC@30 [ 1 ] ). Arrows indicate whether lower (

\downarrow

) or higher (

\uparrow

) is better. All models are fine-tuned on the same panoramic dataset. EvoWorld achieves the best across all metrics.

Figure 3: Qualitative comparison of long-horizon video generation. EvoWorld produces more spatially consistent and geometrically coherent results than GenEx. In this example, EvoWorld accurately follows the conditional path and preserves building structure, while GenEx struggles to maintain layout consistency due to the absence of 3D memory and precise camera control.

		FVD $\downarrow$	Loop LMSE $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	LMSE $\downarrow$
	GenEx	491.71	0.192	0.714	14.558	0.517	0.184
Unity	EvoWorld	442.79	0.187	0.730	15.495	0.494	0.173
	GenEx	516.85	0.199	0.420	12.042	0.594	0.192
UE5	EvoWorld	431.37	0.151	0.500	16.630	0.416	0.148
	GenEx	649.29	0.228	0.545	12.164	0.665	0.218
Indoor	EvoWorld	570.44	0.235	0.559	12.990	0.624	0.218

Table 2: Quantitative comparisons on long-horizon video generation. We evaluate recursive generation of three clips from a single panoramic frame, with each segment conditioned on the last frame of the previous one. Overall, EvoWorld outperforms GenEx across metrics on four domains (Unity, UE5, Indoor, and Real-World), showing clear gains in fidelity and perceptual quality.

Model	2D Metrics					3D Metrics
Model	FVD $\downarrow$	LMSE $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	MEt3R $\downarrow$	AUC@30 $\uparrow$
GenEx [ 2 ]	199.76	0.113	0.400	17.11	0.743	0.1117	0.6408
Baseline + GCD [ 7 ]	178.75	0.109	0.389	17.33	0.745	0.1127	0.6700
Baseline + SpherePlücker	162.96	0.090	0.275	19.54	0.785	0.1070	0.7958
EvoWorld	106.81	0.065	0.167	22.03	0.826	0.0954	0.8846

Table 3: Ablation study on the Unity dataset evaluating the impact of 3D memory and camera pose representations. “GCD”: Generative Camera Dolly [ 7 ] ; “SpherePlücker”: our Spherical Plücker representation for panoramic videos. Combining SpherePlücker embeddings with 3D memory (EvoWorld) yields the best performance across all metrics.

Algorithm	Target reaching	Frame retrieval	Average
GPT-4o	45.0%	N/A	N/A
GPT-4o + GenEx	83.5%	50.5%	67.0%
GPT-4o + EvoWorld	93.3%	68.8%	81.1 %

Table 4: Downstream task performance evaluating spatial consistency with GPT-4o [ 24 ] as the evaluator. We report target-reaching accuracy and spatial frame-retrieval accuracy. EvoWorld outperforms GenEx on both tasks, demonstrating stronger spatial grounding and controllability in panoramic world generation.

Figure 4: Qualitative comparison of 3D reconstructions using four ground truth (GT) images alone, GT with GenEx-generated frames, and with EvoWorld (in the same scale). GT-only reconstructions are incomplete with holes; adding GenEx frames improves coverage but introduces noise. EvoWorld yields more complete and cleaner reconstructions, demonstrating better spatial consistency.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

$\rightarrow$ Direct test
Model	Polyline Paths					Curved Paths
Model	FVD $\downarrow$	MSE $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	FVD $\downarrow$	MSE $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$
SVD [ 5 ]	646.72	0.169	0.503	14.63	0.691	612.25	0.164	0.488	14.93	0.694
CogVideoX-1.5 [ 15 ]	800.29	0.159	0.469	15.52	0.686	820.13	0.163	0.486	15.28	0.679
COSMOS [ 6 ]	1058.40	0.195	0.582	14.78	0.638	1035.93	0.196	0.586	14.76	0.639
Wan-2.1 [ 14 ]	595.57	0.138	0.448	16.62	0.692	506.44	0.139	0.513	15.45	0.668

Appendix

Table S1: Quantitative comparison of single-clip (25-frame) panoramic video generation on the Unity dataset under two types of camera trajectories: polyline paths and curved paths. Models are evaluated using FVD, MSE, LPIPS, PSNR, and SSIM, where

\downarrow

indicates lower is better and

\uparrow

indicates higher is better. Our method, EvoWorld, achieves the best performance across all metrics in both trajectory settings.

Figure S1: Qualitative results of panoramic video generation in the Unity environment. Five key frames are shown from a longer video, with intermediate frames omitted for brevity. GenEx exhibits spatial drift and structural artifacts, while our method (EvoWorld) maintains spatial consistency throughout.

Figure S2: Qualitative results of panoramic video generation in the Unreal Engine 5 (UE5) environment. Five key frames are shown from a longer video, with intermediate frames omitted for brevity. GenEx exhibits spatial drift and missing object, while our method (EvoWorld) maintains spatial consistency throughout.

Figure S3: Qualitative results of panoramic video generation in the indoor environment. Five key frames are shown, with intermediate frames omitted for brevity. GenEx exhibits inconsistent object shapes and numbers, while our method (EvoWorld) maintains spatial consistency throughout.

Figure S4: Qualitative results of panoramic video generation in the real-world environment. Five key frames are shown from a longer video, with intermediate frames omitted for brevity. GenEx exhibits spatial drift and structural artifacts, while our method (EvoWorld) maintains spatial consistency throughout.

Environment	# Videos	# Frames	Distance (m)	Time (s)	Navigation Trajectory
Unity	7,200+	680,000+	260,000+	97,000+	Polyline / Curve
UE5	10,000+	1,000,000+	400,000+	140,000+	Curve
Indoor	34,000+	540,000+	210,000+	77,000+	Step-wise Forward or $22.5^{\circ}$ Rotation
Real-world	7,200+	540,000+	5,400+	9,000+	Natural Human Walk (Curve)

Appendix

Table S2: Statistics of the four subsets in the Spatial360 dataset.

Figure S5: Examples from Unity Dataset. Each row is a case in the Unity dataset. The first column is camera trajectories visualized in a 3D coordinate system. The remaining five are evenly sampled five frames from the raw sequence.

Figure S6: Examples from UE5 Dataset. Each row is a case in the UE5 dataset. The first column is camera trajectories visualized in a 3D coordinate system. The remaining five are evenly sampled five frames from the raw sequence.

Figure S7: Examples from Indoor Dataset.

Figure S8: Examples from Real-world Dataset.

Figure S9: Comparison of (a) target image, (b) generated result, and (c) reprojection visualization. During memory construction, a relatively high threshold filters out the moving person and parts of the trees swaying in the wind. However, in the generation stage, the model still generates corresponding human and tree structures based on the input image elements.

Model	Function	FPS
CogVideoX-1.5	Generation	0.12
GenEx	Generation	0.33
EvoWorld	Generation	0.32
EvoWorld	Mem update	4.20
EvoWorld	Generation + Mem update	0.30

Appendix

Table S3: Comparison of generation and memory update speed (FPS).

Figure S10: Text-image prompt template for target reaching task

Figure S11: Image examples for target reaching task

Figure S12: Text-image prompt template for spatially-aware frame retrieval task

Figure S13: Image examples for spatially-aware frame retrieval task

Figure S14: Left: Pixel coordinate and Spherical Polar coordinate systems; Middle: rotation in Spherical coordinates corresponds to rotation in 2D image; Right: expansion from panorama to cubemap or composition in reverse. The figures are borrowed from [ 2 ] for explanation.

	Polyline Paths				Curved Paths
	LMSE $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LMSE $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$
GenEx	0.046 0.118	0.123 0.049	24.190 2.843	0.865 0.035	0.113 0.262	0.400 0.093	17.110 2.165	0.743 0.046
EvoWorld	0.041 0.106	0.108 0.047	24.357 2.622	0.869 0.033	0.065 0.161	0.167 0.065	22.026 2.218	0.826 0.038

Appendix

Table S4: Quantitative comparison for different camera trajectories in Unity. Metrics are reported as mean std for latent MSE, LPIPS, PSNR, and SSIM under Polylines and Curved Paths. Except for PSNR in the Curved Path setting, EvoWorld shows lower standard deviation than GenEx.

Unity
	LMSE $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$
	mean std	mean std	mean std	mean std
GenEx	0.184 0.401	0.517 0.081	14.558 1.270	0.714 0.037
EvoWorld	0.173 0.384	0.494 0.086	15.495 1.456	0.730 0.038
UE
GenEx	0.192 0.476	0.594 0.048	12.043 1.598	0.420 0.045

Appendix

Table S5: Quantitative comparison with mean std for MSE, LPIPS, PSNR, and SSIM across different domains and models. For MSE, EvoWorld has a lower standard deviation in 3 out of 4 domains. EvoWorld exhibits higher standard deviation in 10 of the 12 LPIPS, PSNR, and SSIM comparisons across the four domains.

Video World Models

Video world models generate and predict action-conditioned visual futures to simulate changing scenes for embodied agents.

See all 276 papers

Lyra 2.0: Explorable Generative 3D Worlds
Apr 14, 2026Tianchang Shen, Sherwin Bahmani, Kai He +123D Scene Generation 3D-Aware Video Generation
NVIDIA
Beyond Pixel Histories: World Models with Persistent 3D State
Mar 3, 2026Samuel Garcin, Thomas Walker, Steven McDonagh +5World Models Interactive World Models
University of Edinburgh · Microsoft Research
OctWorld: Long-Range World-Consistent Video Generation with Octree-Based 3D Mapping
Sep 3, 2026Zelong Lv, Sicheng Xu, Jianfeng Xiang +53D Scene Representation Camera-Controlled Video Generation
University of Science and Technology of China · Microsoft Research Asia · Tsinghua University

Apr 14, 2026J/K move · Enter open · S save

Lyra 2.0: Explorable Generative 3D Worlds

Tianchang Shen, Sherwin Bahmani, Kai He, Sangeetha Grama Srinivasan+11

NVIDIA

3D Scene Generation 3D-Aware Video Generation Long-Horizon Video Generation Temporal Consistency in Video Generation Camera-Controlled Video Generation Memory-Augmented Video Generation

Open paper AskPDF arXiv

Recent advances in video generation enable a new paradigm for 3D scene creation: generating camera-controlled videos that simulate scene walkthroughs, then lifting them to 3D via feed-forward reconstruction techniques. This generative reconstruction approach combines the visual fidelity and creative capacity of video models with 3D outputs ready for real-time rendering and simulation. Scaling to large, complex environments requires 3D-consistent video generation over long camera trajectories with large viewpoint changes and location revisits, a setting where current video models degrade quickly. Existing methods for long-horizon generation are fundamentally limited by two forms of degradation: spatial forgetting and temporal drifting. As exploration proceeds, previously observed regions fall outside the model's temporal context, forcing the model to hallucinate structures when revisited. Meanwhile, autoregressive generation accumulates small synthesis errors over time, gradually distorting scene appearance and geometry. We present Lyra 2.0, a framework for generating persistent, explorable 3D worlds at scale. To address spatial forgetting, we maintain per-frame 3D geometry and use it solely for information routing -- retrieving relevant past frames and establishing dense correspondences with the target viewpoints -- while relying on the generative prior for appearance synthesis. To address temporal drifting, we train with self-augmented histories that expose the model to its own degraded outputs, teaching it to correct drift rather than propagate it. Together, these enable substantially longer and 3D-consistent video trajectories, which we leverage to fine-tune feed-forward reconstruction models that reliably recover high-quality 3D scenes.

Abstract

Video World Models

Explore similar work

Lyra 2.0: Explorable Generative 3D Worlds

Beyond Pixel Histories: World Models with Persistent 3D State

OctWorld: Long-Range World-Consistent Video Generation with Octree-Based 3D Mapping