Recent advances in video generation enable a new paradigm for 3D scene creation: generating camera-controlled videos that simulate scene walkthroughs, then lifting them to 3D via feed-forward reconstruction techniques. This generative reconstruction approach combines the visual fidelity and creative capacity of video models with 3D outputs ready for real-time rendering and simulation. Scaling to large, complex environments requires 3D-consistent video generation over long camera trajectories with large viewpoint changes and location revisits, a setting where current video models degrade quickly. Existing methods for long-horizon generation are fundamentally limited by two forms of degradation: spatial forgetting and temporal drifting. As exploration proceeds, previously observed regions fall outside the model's temporal context, forcing the model to hallucinate structures when revisited. Meanwhile, autoregressive generation accumulates small synthesis errors over time, gradually distorting scene appearance and geometry. We present Lyra 2.0, a framework for generating persistent, explorable 3D worlds at scale. To address spatial forgetting, we maintain per-frame 3D geometry and use it solely for information routing -- retrieving relevant past frames and establishing dense correspondences with the target viewpoints -- while relying on the generative prior for appearance synthesis. To address temporal drifting, we train with self-augmented histories that expose the model to its own degraded outputs, teaching it to correct drift rather than propagate it. Together, these enable substantially longer and 3D-consistent video trajectories, which we leverage to fine-tune feed-forward reconstruction models that reliably recover high-quality 3D scenes.
Figures & tables
Figure 1: Lyra 2.0 enables long-horizon 3D-consistent scene generation from a single image. Starting from an input image, users iteratively define camera motion to explore the scene, while Lyra 2.0 synthesizes spatially persistent video outputs that progressively expand the environment. These videos can be directly reconstructed into high-fidelity 3D Gaussians and surface meshes, yielding 3D assets deployable in simulation engines and interactive viewers.
Figure 2: Method overview. (Left) Given an input image, Lyra 2.0 iteratively generates video segments guided by a user-defined camera trajectory from an interactive 3D explorer and an optional text prompt, lifting each segment into 3D point clouds fed back for continued navigation. Generated video frames are finally reconstructed and exported as 3D Gaussians or meshes. (Right) At each step, history frames with maximal visibility of the target views are retrieved from the spatial memory. Their canonical coordinates are warped to establish dense 3D correspondences and injected into DiT via attention, together with compressed temporal history.
Method
DL3DV
Tanks-and-Temples
SSIM ↑
LPIPS ↓
FID ↓
Subjective Qual. ↑
Style Consist. ↑
Camera Ctrl. ↑
Reproj. Err. ↓
SSIM ↑
LPIPS ↓
FID ↓
Subjective Qual. ↑
Style Consist. ↑
Camera Ctrl. ↑
Reproj. Err. ↓
GEN3C [ 81 ]
0.346
0.535
58.96
24.60
76.77
69.54
0.068
0.350
0.589
79.07
21.75
75.54
70.91
0.054
Yume1.5 [ 70 ]
0.342
0.719
84.84
22.80
66.73
–
0.095
0.348
0.702
89.69
28.68
78.63
–
0.083
CaM [ 128 ]
0.370
0.562
50.43
35.19
82.63
42.71
0.069
0.367
0.605
59.20
34.22
82.83
31.86
0.056
VMem [ 51 ]
0.331
0.744
120.59
18.54
76.14
0.68
0.268
0.338
0.767
136.48
16.21
70.54
0.00
0.263
SPMem [ 117 ]
0.383
0.522
53.77
38.32
82.79
62.05
0.074
0.383
0.571
60.11
34.41
79.68
45.07
0.059
Table 1: Quantitative comparison on single-view to long video generation. Best results are shown in bold and second best are underlined .
Figure 3: Video generation comparisons. Given a single input image from Tanks and Temples, we compare long-horizon generations ( ∼ frame 800+) from all evaluated video models. Baselines exhibit severe quality degradation, geometric distortions, or content drifting at long horizons, while our method maintains realistic structures and appearances.
Method
DL3DV
Tanks-and-Temples
LPIPS-P ↓
LPIPS-G ↓
FID ↓
Subj. Qual. ↑
LPIPS-P ↓
LPIPS-G ↓
FID ↓
Subj. Qual. ↑
GEN3C [ 81 ] + DAv3
0.504
0.649
99.83
11.00
0.511
0.694
125.19
5.38
Yume1.5 [ 70 ] + DAv3
0.598
0.806
121.61
0.22
0.575
0.794
113.25
0.79
CaM [ 128 ] + DAv3
0.433
0.668
94.04
12.16
0.423
0.693
94.02
9.79
VMem [ 51 ] + DAv3
0.593
0.836
206.88
2.00
0.597
0.832
211.72
3.76
SPMem [ 117 ] + DAv3
0.419
0.625
93.56
13.72
0.412
0.666
94.11
9.95
Table 2: Quantitative comparison on 3D scene generation. Best results are shown in bold and second best are underlined .
Figure 4: 3DGS comparisons. We compare renderings from 3DGS scenes reconstructed from video diffusion model outputs, starting from a single input image from Tanks and Temples.
Figure 5: Qualitative comparison with Lyra and FantasyWorld. We show 3DGS renderings (Lyra and Ours) and point cloud renderings (FantasyWorld) in bird’s-eye view. Red bounding boxes highlight approximately the same spatial region across methods. Our interactive exploration framework produces scenes of significantly greater scale and complexity.
Method
SSIM ↑
LPIPS ↓
FID ↓
Subjective Qual. ↑
Style Consist. ↑
Camera Ctrl. ↑
Reproj. Err. ↓
Ours
0.384
0.552
51.33
43.35
85.07
63.87
0.069
w/ Global Point Cloud
0.368
0.562
52.54
44.58
82.42
49.86
0.067
w/ Explicit Corr. Fusion
0.370
0.554
49.13
45.71
83.28
57.29
0.071
w/o FramePack
0.362
0.549
50.98
45.27
80.61
62.62
0.079
w/o Self-Augmentation
0.363
0.568
55.15
47.88
77.98
53.92
0.066
Table 3: Ablation study on Tanks and Temples. We ablate key design choices of our framework. Best results are shown in bold .
Figure 6: Qualitative ablation study. Given a single input image, we compare generations from our full model and ablated variants on Tanks and Temples scenes.
Figure 7: Applications. Our interactive interface allows users to specify camera trajectories within the 3D cache to easily generate novel viewpoints. Moreover, the reconstructed 3DGS scenes can be converted into surface meshes and integrated into embodied AI simulators such as NVIDIA Isaac Sim for robot simulation.
Figure 8: In-the-Wild Scene Generation. We show video generations and 3DGS reconstructions for challenging in-the-wild input images that go beyond the training data distribution. Our approach supports flexible camera trajectories specified in the GUI for world exploration, including combining multiple trajectories from the same starting point (see second example).
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Target-frame coverage vs. number of retrieved spatial memory frames. We evaluate on training videos by treating the latter half as the target generation segment. Coverage is computed by forward-warping each retrieved frame to every target viewpoint using ground-truth depth; a target pixel is counted as covered only when the depth discrepancy between the warped point and the target ground-truth depth falls below a threshold. Ns=5 offers a favorable balance between spatial coverage and computational efficiency.
We present OctWorld, a video diffusion framework with persistent 3D memory for generating explorable, world-consistent, and high-fidelity visual scenes. Given a single image, OctWorld performs stable autoregressive world generation along user-specified camera trajectories. We focus on long-range generation, characterized by extended camera paths and wide viewpoint coverage, where preserving spatial consistency is particularly challenging when previously generated regions are revisited. To address this problem, we introduce OctMap, an extensible and spatially adaptive 3D memory that progressively fuses generated visual observations and their corresponding depth maps into a global representation. OctMap employs TSDF fusion within a dynamic sparse octree whose spatial resolution adapts to image evidence. This design preserves geometric and appearance details across diverse scene scales while maintaining low memory overhead. Experiments demonstrate that OctWorld generates long-range, spatially consistent videos and outperforms prior methods on both existing benchmarks and challenging long-range generation settings. OctMap also provides clear advantages over point-based caches and fixed-resolution TSDF volumes. Project page: https://maxtirerror.github.io/octworldpage/
Zelong Lv, Sicheng Xu, Jianfeng Xiang +5
University of Science and Technology of China · Microsoft Research Asia · Tsinghua University
Scene-consistent video generation aims to create videos that explore 3D scenes based on a camera trajectory. Previous methods rely on video generation models with external memory for consistency, or iterative 3D reconstruction and inpainting, which accumulate errors during inference due to incorrect intermediary outputs, non-differentiable processes, and separate models. To overcome these limitations, we introduce ``geometry-as-context". It iteratively completes the following steps using an autoregressive camera-controlled video generation model: (1) estimates the geometry of the current view necessary for 3D reconstruction, and (2) simulates and restores novel view images rendered by the 3D scene. Under this multi-task framework, we develop the camera gated attention module to enhance the model's capability to effectively leverage camera poses. During the training phase, text contexts are utilized to ascertain whether geometric or RGB images should be generated. To ensure that the model can generate RGB-only outputs during inference, the geometry context is randomly dropped from the interleaved text-image-geometry training sequence. The method has been tested on scene video generation with one-direction and forth-and-back trajectories. The results show its superiority over previous approaches in maintaining scene consistency and camera control.
JiaKui Hu, Jialun Liu, Liying Yang +7
Institute of Medical Technology, Peking University · TeleAI · MUST
Video diffusion models generate high-quality and diverse worlds; however, individual frames often lack 3D consistency across the output sequence, which makes the reconstruction of 3D worlds difficult. To this end, we propose a new method that handles these inconsistencies by non-rigidly aligning the video frames into a globally-consistent coordinate frame that produces sharp and detailed pointcloud reconstructions. First, a geometric foundation model lifts each frame into a pixel-wise 3D pointcloud, which contains unaligned surfaces due to these inconsistencies. We then propose a tailored non-rigid iterative frame-to-model ICP to obtain an initial alignment across all frames, followed by a global optimization that further sharpens the pointcloud. Finally, we leverage this pointcloud as initialization for 3D reconstruction and propose a novel inverse deformation rendering loss to create high quality and explorable 3D environments from inconsistent views. We demonstrate that our 3D scenes achieve higher quality than baselines, effectively turning video models into 3D-consistent world generators.