The growing demand for Embodied AI and VR applications has highlighted the need for synthesizing high-quality 3D indoor scenes from sparse inputs. However, existing approaches struggle to infer massive amounts of missing geometry in large unseen areas while maintaining global consistency, often producing locally plausible but globally inconsistent reconstructions. We present Rein3D, a framework that reconstructs full 360-degree indoor environments by coupling explicit 3D Gaussian Splatting (3DGS) with temporally coherent priors from video diffusion models. Our approach follows a "restore-and-refine" paradigm: we employ a radial exploration strategy to render imperfect panoramic videos along trajectories starting from the origin, effectively uncovering occluded regions from a coarse 3DGS initialization. These sequences are restored by a panoramic video-to-video diffusion model and further enhanced via video super-resolution to synthesize high-fidelity geometry and textures. Finally, these refined videos serve as pseudo-ground truths to update the global 3D Gaussian field. To support this task, we construct PanoV2V-15K, a dataset of over 15K paired clean and degraded panoramic videos for diffusion-based scene restoration. Experiments demonstrate that Rein3D produces photorealistic and globally consistent 3D scenes and significantly improves long-range camera exploration compared with existing baselines.
Figures & tables
Figure 1: Overview of Rein3D. Starting from a single RGB-D panorama, Rein3D initializes a coarse 3DGS scene and renders degraded panoramic videos along radial trajectories to expose unseen regions. A panoramic video diffusion model restores these videos with temporally consistent priors, and the restored views are fused to refine the global 3DGS. This restore-and-refine paradigm enables photorealistic and globally consistent indoor scene reconstruction under large viewpoint changes.
Figure 2: Illustration of PanoV2V-15K construction. For each scene, we render a clean panoramic video along the longest traversable trajectory and generate its paired degraded RGB-alpha video by rendering a coarse 3DGS initialized from the first panoramic RGB-D observation.
Figure 3: Overview of the Rein3D pipeline. (a) Given a single panoramic RGB-D observation, we lift it into an initial 3D point cloud / coarse 3DGS representation. Rendering this coarse representation from novel viewpoints exposes holes, missing geometry, and opacity artifacts caused by single-view initialization. (b) The resulting low-quality RGB-alpha panoramic videos are restored by our panoramic video diffusion model. The RGB-alpha inputs are decomposed into observed and missing-region streams, combined with a context anchor, and processed with latitude-aware noise sampling to produce temporally consistent restored panoramas. (c) The restored videos are further enhanced by super-resolution and fused back as pseudo observations to refine the global 3DGS, yielding a more complete and view-consistent indoor scene.
Dataset
Method
PSNR ↑
SSIM ↑
WS-PSNR ↑
WS-SSIM ↑
FVD ↓
PanoV2V-15K
ProPainter
18.46
0.760
17.79
0.740
14.56
VACE
21.17
0.787
20.80
0.771
11.67
VACE-FT
22.18
0.798
21.89
0.790
9.27
Ours
23.77
0.823
23.45
0.820
5.26
3D-FRONT
ProPainter
19.12
0.674
18.68
0.656
29.27
VACE
21.91
0.701
21.00
0.702
25.50
Table 1: Quantitative comparison on panoramic video restoration. VACE-FT denotes VACE fine-tuned on PanoV2V-15K.
Figure 4: Qualitative comparison of rendered perspective views on Structured3D. The top row shows the input panoramas, and the following rows show novel views rendered from reconstructed 3D scenes. Rein3D maintains more stable geometry and fewer artifacts under large viewpoint changes.
Method
Q-Align ↑
CLIP ↑
NIQE ↓
BRISQUE ↓
EmbodiedGen
2.2914
0.2216
7.9231
58.3813
DreamScene360
1.8342
0.2356
8.5295
61.4567
Ours
3.1033
0.2609
6.4925
46.5411
Table 2: Quantitative comparison on single-panorama 3D scene reconstruction on 50 Structured3D test scenes. We report no-reference visual quality and semantic consistency metrics on novel views.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
EmbodiedGen
15.05
0.517
0.678
Ours
17.64
0.624
0.540
Table 3: Novel view synthesis comparison on 30 3D-FRONT scenes. Each method takes a single panoramic RGB-D image as input and is evaluated on 100 held-out target views per scene.
Setting
PSNR ↑
SSIM ↑
WS-PSNR ↑
WS-SSIM ↑
FVD ↓
w/o Context Anchor
22.84
0.810
22.40
0.802
7.75
w/o Latitude-Decay Loss
23.28
0.819
22.97
0.816
5.66
w/o Latitude-Aware Noise Sampling
22.96
0.816
22.65
0.812
6.24
Full
23.77
0.823
23.45
0.820
5.26
Table 4: Ablation study of key components in the panoramic video restoration model. Context Anchor stabilizes long-range restoration, while latitude-aware sampling and latitude-decay loss adapt the model to equirectangular geometry.
Setting
PSNR ↑
SSIM ↑
WS-PSNR ↑
WS-SSIM ↑
FVD ↓
λ=1.0
23.28
0.819
22.97
0.816
5.66
λ=0.5
23.80
0.822
23.38
0.818
5.66
λ=0.3
23.42
0.819
23.05
0.816
5.45
λ=0
23.44
0.824
22.99
0.819
5.62
λ=0.1 (Full)
23.77
0.823
23.45
0.820
5.26
Table 5: Sensitivity analysis of the latitude-decay coefficient λ . The default setting λ=0.1 provides the best overall balance between spherical fidelity and temporal consistency.
Setting
PSNR ↑
SSIM ↑
LPIPS ↓
w/o Radial Exploration
14.84
0.448
0.672
w/o VSR
17.42
0.601
0.549
Full ( τ=4 )
17.36
0.595
0.553
Full ( τ=8 )
17.64
0.624
0.540
Table 6: Ablation study on 3D-FRONT. w/o Radial Exploration uses local exploration around the input panorama center. w/o VSR removes super-resolution before 3DGS refinement. τ denotes the number of radial trajectories.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Overview of the PanoV2V-15K dataset. (Left) A pie chart illustrating the distribution of the 15,500 indoor scenes across different room types. (Right) Consecutive frames extracted from a high-quality panoramic video within the dataset, accompanied by its corresponding highly descriptive text prompt generated by Qwen2.5-VL.
Setting
Value
Batch size
4
Training steps
15,000
Evaluation / save steps
5,000 / 10,000 / 15,000
Test split
Every 16th view
SH degree
3
Near / far plane
0.01 / 100.0
Appendix
Table 7: Key hyperparameters for final 3DGS refinement.
Figure 7: Visualizations of our camera exploration and evaluation sampling strategies. (Top) The depth-optimized radial trajectory generation, illustrating the safe profile, the cross-score optimization, and the selected top-down trajectories. (Bottom) The Farthest Point Sampling (FPS) process for evaluation, showing the safe navigable area, the Monte Carlo candidate pool, and the selected 5 globally distributed camera locations.
Figure 8: Qualitative results of panoramic video restoration. The odd-numbered rows (1st, 3rd, and 5th) show sequences of degraded panoramic frames rendered from the initial coarse 3DGS, which suffer from severe missing regions (black holes) and structural artifacts. Directly below them, the even-numbered rows (2nd, 4th, and 6th) display the corresponding high-fidelity frames restored by our panoramic video diffusion model, demonstrating remarkable spatial completeness and temporal coherence across the trajectory.
Figure 9: Additional qualitative comparisons on novel view synthesis against EmbodiedGen and DreamScene360. The views are rendered from camera locations far from the initialization point to evaluate long-range exploration consistency. While baselines exhibit severe distortions, floating artifacts, and missing structures, our method preserves global geometry and fine textural details.