cs.CVMar 17, 2026
SaveWorld Reconstruction From Inconsistent Views
Organizations: Technical University of Munich, Germany
Abstract
Video diffusion models generate high-quality and diverse worlds; however, individual frames often lack 3D consistency across the output sequence, which makes the reconstruction of 3D worlds difficult. To this end, we propose a new method that handles these inconsistencies by non-rigidly aligning the video frames into a globally-consistent coordinate frame that produces sharp and detailed pointcloud reconstructions. First, a geometric foundation model lifts each frame into a pixel-wise 3D pointcloud, which contains unaligned surfaces due to these inconsistencies. We then propose a tailored non-rigid iterative frame-to-model ICP to obtain an initial alignment across all frames, followed by a global optimization that further sharpens the pointcloud. Finally, we leverage this pointcloud as initialization for 3D reconstruction and propose a novel inverse deformation rendering loss to create high quality and explorable 3D environments from inconsistent views. We demonstrate that our 3D scenes achieve higher quality than baselines, effectively turning video models into 3D-consistent world generators.
Figures & tables
Figure 2: Method overview. We propose a three stage method that reconstructs a 2DGS [ 25 ] scene from generated videos. First, we estimate multi-view depth and cameras with a geometric foundation model [ 43 ] . The resulting dense scene initialization is unaligned (multiple non-overlapping surfaces) due to the inconsistent input frames. We propose a tailored non-rigid geometry alignment that leverages iterative frame-to-model ICP [ 31 , 6 ] and sparse correspondences, followed by global optimization, to create thin surfaces with detailed textures. Then, we leverage the alignment in a novel non-rigid aware 2DGS [ 25 ] optimization to obtain high-quality, consistent 3D worlds.
| Method | Consistency | Fidelity | ||
|---|---|---|---|---|
| 3D | Photometric | CLIP-IQA+ | CLIP Aesthetic | |
| DA3 [ 43 ] | 69.53 | 71.62 | 31.64 | 36.64 |
| 3DGS-MCMC [ 35 ] | 67.34 | 69.58 | 35.56 | 35.06 |
| VGGT-X [ 44 ] | 65.73 | 65.58 | 38.21 | 37.24 |
| VGGT-X † | 69.66 | 67.05 | 41.44 | 37.53 |
| Ours | 79.29 | 86.59 | 46.56 | 37.61 |
Table 1: Quantitative comparison. We compare against baselines on the single video 3D reconstruction task and report averaged results across all source video models. Our method achieves the highest 3D and photometric consistency, which underlines its ability to reconstruct worlds from inconsistent frames. We also improve in terms of image fidelity, which signals our images are of higher sharpness and texture detail.
Figure 3: Single video 3D reconstruction. We generate videos with HY-WorldPlay [ 71 ] (top), Genie3 [ 5 ] (mid), ViewCrafter [ 88 ] (bottom) and 3D reconstruct them. Our method optimizes consistent worlds from inconsistent generated frames. Compared to baselines, the renderings are of higher visual fidelity from both input and novel views.
Figure 4: Single video 3D reconstruction. We generate videos with SEVA [ 91 ] (top), Gen3C [ 56 ] (mid), Wan [ 76 ] (bottom) and 3D reconstruct these frames. Inconsistencies in the generations lead to blurry textures for the baselines compared to the corresponding video, and to severe floating artifacts from novel views. In contrast, our method creates 3D consistent worlds with high fidelity beyond the generated perspectives.
Figure 5: Pointcloud reconstructions. We compare the quality of the reconstructed pointclouds that are used as initialization for Gaussian Splatting [ 34 ] optimization in the subsequent stages for each method. Our approach achieves the highest alignment and compelling textures for individual objects with no overlap of multiple surfaces.
| Method | Consistency | Fidelity | ||
|---|---|---|---|---|
| 3D | Photometric | CLIP-IQA+ | CLIP Aesthetic | |
| only rigid | 76.98 | 64.93 | 11.43 | 49.12 |
| no inv | 78.28 | 55.16 | 15.81 | 46.39 |
| no filt | 72.58 | 64.88 | 55.17 | 51.05 |
| no corr | 65.62 | 74.76 | 44.81 | 46.27 |
| no global | 63.90 | 77.85 | 49.69 | 51.61 |
Table 2: Ablation study. 3D reconstruction from only rigid alignment, without correspondences, or without backward deformation (“no inv”) negatively impacts 3D consistency and image quality. No point filter or global optimization decrease image fidelity and consistency. Our full method achieves the highest consistency and fidelity.
Figure 6: Ablation. We compare our full method on rendering quality of input views (top) and novel views (mid), as well as on pointcloud reconstruction quality (bottom). The DA3 [ 43 ] prediction contains misalignments that cannot be resolved by only rigid alignment or without correspondences. Removing the color filtering and global optimization steps adversely impact texture quality. Removing our novel inverse deformation loss reduces sharpness. In contrast, our method obtains flat/sharp surfaces which better constrain the 3D reconstruction, yielding high quality rendering results.
Figure 7: Large-scale 3D reconstructions. We compare rendering quality on input poses and novel views for entire 360 degree explorable scenes. Our method creates 3D consistent worlds with a high rendering fidelity far beyond the training views. Please see the supplementary material for animated flythroughs.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1 : Comparison against dynamic reconstruction. We compare against “Deformable 3DGS” [ 87 ] on reconstructing inconsistent generated video sequences. Since they do not model an aligned canonical space, their renderings both with deformation (dyn) and without (static) suffer from floating artifacts and decreased image quality.
Figure A2 : Single video 3D reconstruction. We generate a video with Gen3C [ 56 ] and 3D reconstruct the frames. The feedforward 3DGS prediction from DA3 [ 43 ] suffers from decreased quality in comparison to optimizing 3DGS from scratch from the DA3 predicted depth/cams. Combining SHARP [ 47 ] and DA3 to obtain feedforward 3DGS predictions from multiple cameras suffers from the misaligned world space of the DA3 predictions (even though we utilize our optimized poses). In contrast, our method obtains high-quality renderings from inconsistent views by lifting the corresponding geometry into an aligned space.
Figure A3 : Voxel-based confidence filtering. We propose a point pruning strategy of DA3 [ 43 ] predictions to retain reliable, but spatially dense geometry, based on the predicted confidence scores. Naive global filtering schemes can prune too few points (global 0; noticeable stretch artifacts), which makes it hard to obtain high quality surface alignments. Global filtering with too high thresholds (global 10-40) can remove important scene content (e.g., the lamp or windows), which requires additional densification in the reconstruction stage. In contrast, our voxel-based filtering obtains dense surfaces that can directly be aligned in the subsequent stages and require no further densification to model complete scenes.
Figure A4 : Limitations. Our method reconstructs 3D consistent worlds from inconsistent video sequences through geometry alignment and non-rigid Gaussian Splatting optimization. However, video diffusion models can also hallucinate novel content when revisiting areas (e.g., the street lamp appears suddenly). We can still align multiple frames that show such hallucinations, but they become a part of the 3D-consistent static geometry. Additionally, 3D reconstruction optimizes for the average explanation of all input views and as such, these inconsistencies stay visible (e.g., the street lamp is visible in both viewpoints when rendering from our 3D scene).
Figure A5 : Single video 3D reconstruction. We generate videos with ViewCrafter [ 88 ] (top), Voyager [ 27 ] (mid), Genie3 [ 5 ] (bottom) and 3D reconstruct these frames. Inconsistencies in the generations lead to blurry textures for the baselines compared to the corresponding video, and to floating artifacts from novel views. In contrast, our method creates 3D consistent worlds with high fidelity beyond generated perspectives.
Figure A6 : Single video 3D reconstruction. We generate videos with SEVA [ 91 ] (top), Gen3C [ 56 ] (mid), Wan [ 76 ] (bottom) and 3D reconstruct these frames. Inconsistencies in the generations lead to blurry textures for the baselines compared to the corresponding video, and to floating artifacts from novel views. In contrast, our method creates 3D consistent worlds with high fidelity beyond generated perspectives.
Figure A7 : Single video 3D reconstruction. We generate videos with Voyager [ 27 ] and 3D reconstruct these frames. Our method optimizes consistent worlds from inconsistent generated frames. Compared to baselines, the renderings are of higher visual fidelity from both input and novel views. The pointcloud visualizations demonstrate that our method obtains aligned surfaces with compelling textures, whereas the baselines produce sparser or less aligned geometry.
Figure A8 : Large-scale 3D reconstructions. We compare rendering quality on input poses and novel views for entire 360 degree explorable scenes. Our method creates 3D consistent worlds with a high rendering fidelity far beyond the training views. Please see the supplementary video for animated flythroughs.
Figure A9 : Large-scale 3D reconstructions. We compare rendering quality on input poses and novel views for entire 360 degree explorable scenes. Our method creates 3D consistent worlds with a high rendering fidelity far beyond the training views. Please see the supplementary video for animated flythroughs.
Figure A10 : Large-scale 3D reconstructions. We compare rendering quality on input poses and novel views for entire 360 degree explorable scenes. Our method creates 3D consistent worlds with a high rendering fidelity far beyond the training views. Please see the supplementary video for animated flythroughs.