Video diffusion models generate high-quality and diverse worlds; however, individual frames often lack 3D consistency across the output sequence, which makes the reconstruction of 3D worlds difficult. To this end, we propose a new method that handles these inconsistencies by non-rigidly aligning the video frames into a globally-consistent coordinate frame that produces sharp and detailed pointcloud reconstructions. First, a geometric foundation model lifts each frame into a pixel-wise 3D pointcloud, which contains unaligned surfaces due to these inconsistencies. We then propose a tailored non-rigid iterative frame-to-model ICP to obtain an initial alignment across all frames, followed by a global optimization that further sharpens the pointcloud. Finally, we leverage this pointcloud as initialization for 3D reconstruction and propose a novel inverse deformation rendering loss to create high quality and explorable 3D environments from inconsistent views. We demonstrate that our 3D scenes achieve higher quality than baselines, effectively turning video models into 3D-consistent world generators.
Figures & tables
Figure 2: Method overview. We propose a three stage method that reconstructs a 2DGS [ 25 ] scene from generated videos. First, we estimate multi-view depth and cameras with a geometric foundation model [ 43 ] . The resulting dense scene initialization is unaligned (multiple non-overlapping surfaces) due to the inconsistent input frames. We propose a tailored non-rigid geometry alignment that leverages iterative frame-to-model ICP [ 31 , 6 ] and sparse correspondences, followed by global optimization, to create thin surfaces with detailed textures. Then, we leverage the alignment in a novel non-rigid aware 2DGS [ 25 ] optimization to obtain high-quality, consistent 3D worlds.
Method
Consistency
Fidelity
3D
Photometric
CLIP-IQA+
CLIP Aesthetic
DA3 [ 43 ]
69.53
71.62
31.64
36.64
3DGS-MCMC [ 35 ]
67.34
69.58
35.56
35.06
VGGT-X [ 44 ]
65.73
65.58
38.21
37.24
VGGT-X †
69.66
67.05
41.44
37.53
Ours
79.29
86.59
46.56
37.61
Table 1: Quantitative comparison. We compare against baselines on the single video 3D reconstruction task and report averaged results across all source video models. Our method achieves the highest 3D and photometric consistency, which underlines its ability to reconstruct worlds from inconsistent frames. We also improve in terms of image fidelity, which signals our images are of higher sharpness and texture detail.
Figure 3: Single video 3D reconstruction. We generate videos with HY-WorldPlay [ 71 ] (top), Genie3 [ 5 ] (mid), ViewCrafter [ 88 ] (bottom) and 3D reconstruct them. Our method optimizes consistent worlds from inconsistent generated frames. Compared to baselines, the renderings are of higher visual fidelity from both input and novel views.
Figure 4: Single video 3D reconstruction. We generate videos with SEVA [ 91 ] (top), Gen3C [ 56 ] (mid), Wan [ 76 ] (bottom) and 3D reconstruct these frames. Inconsistencies in the generations lead to blurry textures for the baselines compared to the corresponding video, and to severe floating artifacts from novel views. In contrast, our method creates 3D consistent worlds with high fidelity beyond the generated perspectives.
Figure 5: Pointcloud reconstructions. We compare the quality of the reconstructed pointclouds that are used as initialization for Gaussian Splatting [ 34 ] optimization in the subsequent stages for each method. Our approach achieves the highest alignment and compelling textures for individual objects with no overlap of multiple surfaces.
Method
Consistency
Fidelity
3D
Photometric
CLIP-IQA+
CLIP Aesthetic
only rigid
76.98
64.93
11.43
49.12
no inv
78.28
55.16
15.81
46.39
no filt
72.58
64.88
55.17
51.05
no corr
65.62
74.76
44.81
46.27
no global
63.90
77.85
49.69
51.61
Table 2: Ablation study. 3D reconstruction from only rigid alignment, without correspondences, or without backward deformation (“no inv”) negatively impacts 3D consistency and image quality. No point filter or global optimization decrease image fidelity and consistency. Our full method achieves the highest consistency and fidelity.
Figure 6: Ablation. We compare our full method on rendering quality of input views (top) and novel views (mid), as well as on pointcloud reconstruction quality (bottom). The DA3 [ 43 ] prediction contains misalignments that cannot be resolved by only rigid alignment or without correspondences. Removing the color filtering and global optimization steps adversely impact texture quality. Removing our novel inverse deformation loss reduces sharpness. In contrast, our method obtains flat/sharp surfaces which better constrain the 3D reconstruction, yielding high quality rendering results.
Figure 7: Large-scale 3D reconstructions. We compare rendering quality on input poses and novel views for entire 360 degree explorable scenes. Our method creates 3D consistent worlds with a high rendering fidelity far beyond the training views. Please see the supplementary material for animated flythroughs.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1 : Comparison against dynamic reconstruction. We compare against “Deformable 3DGS” [ 87 ] on reconstructing inconsistent generated video sequences. Since they do not model an aligned canonical space, their renderings both with deformation (dyn) and without (static) suffer from floating artifacts and decreased image quality.
Figure A2 : Single video 3D reconstruction. We generate a video with Gen3C [ 56 ] and 3D reconstruct the frames. The feedforward 3DGS prediction from DA3 [ 43 ] suffers from decreased quality in comparison to optimizing 3DGS from scratch from the DA3 predicted depth/cams. Combining SHARP [ 47 ] and DA3 to obtain feedforward 3DGS predictions from multiple cameras suffers from the misaligned world space of the DA3 predictions (even though we utilize our optimized poses). In contrast, our method obtains high-quality renderings from inconsistent views by lifting the corresponding geometry into an aligned space.
Figure A3 : Voxel-based confidence filtering. We propose a point pruning strategy of DA3 [ 43 ] predictions to retain reliable, but spatially dense geometry, based on the predicted confidence scores. Naive global filtering schemes can prune too few points (global 0; noticeable stretch artifacts), which makes it hard to obtain high quality surface alignments. Global filtering with too high thresholds (global 10-40) can remove important scene content (e.g., the lamp or windows), which requires additional densification in the reconstruction stage. In contrast, our voxel-based filtering obtains dense surfaces that can directly be aligned in the subsequent stages and require no further densification to model complete scenes.
Figure A4 : Limitations. Our method reconstructs 3D consistent worlds from inconsistent video sequences through geometry alignment and non-rigid Gaussian Splatting optimization. However, video diffusion models can also hallucinate novel content when revisiting areas (e.g., the street lamp appears suddenly). We can still align multiple frames that show such hallucinations, but they become a part of the 3D-consistent static geometry. Additionally, 3D reconstruction optimizes for the average explanation of all input views and as such, these inconsistencies stay visible (e.g., the street lamp is visible in both viewpoints when rendering from our 3D scene).
Figure A5 : Single video 3D reconstruction. We generate videos with ViewCrafter [ 88 ] (top), Voyager [ 27 ] (mid), Genie3 [ 5 ] (bottom) and 3D reconstruct these frames. Inconsistencies in the generations lead to blurry textures for the baselines compared to the corresponding video, and to floating artifacts from novel views. In contrast, our method creates 3D consistent worlds with high fidelity beyond generated perspectives.
Figure A6 : Single video 3D reconstruction. We generate videos with SEVA [ 91 ] (top), Gen3C [ 56 ] (mid), Wan [ 76 ] (bottom) and 3D reconstruct these frames. Inconsistencies in the generations lead to blurry textures for the baselines compared to the corresponding video, and to floating artifacts from novel views. In contrast, our method creates 3D consistent worlds with high fidelity beyond generated perspectives.
Figure A7 : Single video 3D reconstruction. We generate videos with Voyager [ 27 ] and 3D reconstruct these frames. Our method optimizes consistent worlds from inconsistent generated frames. Compared to baselines, the renderings are of higher visual fidelity from both input and novel views. The pointcloud visualizations demonstrate that our method obtains aligned surfaces with compelling textures, whereas the baselines produce sparser or less aligned geometry.
Figure A8 : Large-scale 3D reconstructions. We compare rendering quality on input poses and novel views for entire 360 degree explorable scenes. Our method creates 3D consistent worlds with a high rendering fidelity far beyond the training views. Please see the supplementary video for animated flythroughs.
Figure A9 : Large-scale 3D reconstructions. We compare rendering quality on input poses and novel views for entire 360 degree explorable scenes. Our method creates 3D consistent worlds with a high rendering fidelity far beyond the training views. Please see the supplementary video for animated flythroughs.
Figure A10 : Large-scale 3D reconstructions. We compare rendering quality on input poses and novel views for entire 360 degree explorable scenes. Our method creates 3D consistent worlds with a high rendering fidelity far beyond the training views. Please see the supplementary video for animated flythroughs.
Recent advances in video generation enable a new paradigm for 3D scene creation: generating camera-controlled videos that simulate scene walkthroughs, then lifting them to 3D via feed-forward reconstruction techniques. This generative reconstruction approach combines the visual fidelity and creative capacity of video models with 3D outputs ready for real-time rendering and simulation. Scaling to large, complex environments requires 3D-consistent video generation over long camera trajectories with large viewpoint changes and location revisits, a setting where current video models degrade quickly. Existing methods for long-horizon generation are fundamentally limited by two forms of degradation: spatial forgetting and temporal drifting. As exploration proceeds, previously observed regions fall outside the model's temporal context, forcing the model to hallucinate structures when revisited. Meanwhile, autoregressive generation accumulates small synthesis errors over time, gradually distorting scene appearance and geometry. We present Lyra 2.0, a framework for generating persistent, explorable 3D worlds at scale. To address spatial forgetting, we maintain per-frame 3D geometry and use it solely for information routing -- retrieving relevant past frames and establishing dense correspondences with the target viewpoints -- while relying on the generative prior for appearance synthesis. To address temporal drifting, we train with self-augmented histories that expose the model to its own degraded outputs, teaching it to correct drift rather than propagate it. Together, these enable substantially longer and 3D-consistent video trajectories, which we leverage to fine-tune feed-forward reconstruction models that reliably recover high-quality 3D scenes.
Recent advances in foundational Video Diffusion Models (VDMs) have yielded significant progress. Yet, despite the remarkable visual quality of generated videos, reconstructing consistent 3D scenes from these outputs remains challenging, due to limited camera controllability and inconsistent generated content when viewed from distinct camera trajectories. In this paper, we propose WorldStereo, a novel framework that bridges camera-guided video generation and 3D reconstruction via two dedicated geometric memory modules. Formally, the global-geometric memory enables precise camera control while injecting coarse structural priors through incrementally updated point clouds. Moreover, the spatial-stereo memory constrains the model's attention receptive fields with 3D correspondence to focus on fine-grained details from the memory bank. These components enable WorldStereo to generate multi-view-consistent videos under precise camera control, facilitating high-quality 3D reconstruction. Furthermore, the flexible control branch-based WorldStereo shows impressive efficiency, benefiting from the distribution matching distilled VDM backbone without joint training. Extensive experiments across both camera-guided video generation and 3D reconstruction benchmarks demonstrate the effectiveness of our approach. Notably, we show that WorldStereo acts as a powerful world model, tackling diverse scene generation tasks (whether starting from perspective or panoramic images) with high-fidelity 3D results. Models will be released.
Previous works that leverage video models for image-to-3D scene generation often suffer from geometric distortions and blurry content. Using video generation models to implicitly maintain geometric consistency according to a single-frame input is ineffective. In this paper, we present a two-stage method, named GeoWorld, that renovates the image-to-3D scene generation pipeline by providing full-frame geometry features. The first-stage video generation model, followed by a multi-view geometry model, produces full-frame geometry features, which are then used as a mental draft of geometric conditions to aid the second-stage video-generation model. A geometric loss is proposed to impose real-world geometric constraints, and a geometry adaptation module is introduced to ensure the effective utilization of geometry features. Thanks to full-frame geometric modeling, the two smaller video models in our two-stage method can generate higher-fidelity 3D scenes than SOTA methods, while being even faster, e.g. 7.5× faster than Hunyuan-Voyager. Project page: https://peaes.github.io/GeoWorld.
Yuhao Wan, Lijuan Liu, Jingzhi Zhou +6
1VCIP & AAIS, Nankai University · 2ByteDance Inc. · 3Renmin University of China +2