3D Gaussian Splatting achieves photorealistic reconstruction within training view distribution, yet it degrades on out-of-distribution novel views, exhibiting holes in unobserved regions and artifacts in observable areas. Recent works formulate this task as extrapolation and interpolation and try to address it with generative models, but remain limited in extrapolation scale and quality. They repeat a generate-reconstruct-shift cycle to progressively build a scene, which introduces accumulated errors with every step conditioning on previous outcomes. In this work, we propose a holistic framework for extrapolation and interpolation. We devise an independent camera view detection mechanism to enable parallel conflict-free extrapolation, circumventing the reliance on the aforementioned error-prone cycle. Building upon this, we design a hierarchical pipeline that extrapolates independent and dependent camera views separately. Additionally, previous methods overlook inconsistency between generated and original images, resulting in compromising well-reconstructed areas. We propose a plug-and-play Quality-Aware Mask (QA-Mask) module, enabling selective utilization of generated data. By calibrating learning weights with pixel-wise rendering quality, it prevents generation-induced degradations on well-constructed areas. Extensive experiments demonstrate the superior performance of our framework, with QA-Mask generalizing to multiple generative reconstruction models.
Figures & tables
Figure 1. Our framework excels at large-scale, geometrically coherent 3D scene extrapolation and interpolation on extreme out-of-distribution novel views.
Figure 2. Independent Camera View Detection. We construct a mesh-based view frustum enclosing the geometry of the inpainted region for each candidate view Vi . Collision detection is performed for frustums to determine whether their camera views are independent. In this example, Vi-1 and Vi are independent, while Vi and Vi+1 collide with each other and are therefore not independent.
Figure 3. An overview of our main pipeline. It consists of extrapolations for independent and dependent camera views. The former achieves extrapolation on the original scene over large areas in a single pass. The latter performs a second-stage residual completion on the remaining un-extrapolated regions, enabling seamless scene extrapolation. DIBR-based video depth alignment provides robust depth priors throughout the process. Blue indicates the generated data used in each stage.
Figure 4. Under-reconstructed areas, including holes and low-quality regions , should be updated using generated images. In contrast, well-reconstructed areas within the original scene should remain unchanged to avoid compromising the overall quality due to potential generative inconsistencies .
Figure 5. Visualizations of rendered RGB images and QA-Mask from extrapolation views on four scenes in Mip-NeRF 360. QA-Mask identified holes and low rendering quality regions (dashed yellow) that align well with human perception. However, Bayes’ Rays and Nerfbusters fail to clearly distinguish between low-quality and well-reconstructed areas, while opacity maps and NVF are unable to accurately identify the low-quality areas within the original scene or maintain high (uncertainty) values across the hole areas.
Figure 6. (a) Rendering degradations on out-of-distribution views. 3D Gaussians overfit on training views but underfit on out-of-distribution (OOD) views which results in degraded rendering. (b) Quality-Aware Mask rendering method. We first prepare a pretrained 3DGS model by setting the SH base color to red, view-dependent color coefficients al,m to 0 and converting all original training images to black. We fine-tune this model with only al,m receiving gradients. After fine-tuning, within the training view distribution with high rendering quality, the base color is canceled out by al,m with the final color being black. While in OOD views with low rendering quality, the lack of black supervisions causes Gaussians to remain the base color.
Figure 7. Qualitative comparisons of extrapolation on Mip-NeRF 360. The first row shows the inpainted images on independent camera views VI . The following rows show the images rendered on the camera views between each pair of VIi,VIi+1 . Our framework achieves high rendering quality on extrapolated areas while also leaves well-reconstructed original areas untouched.
Methods
C-IQA ↑
QUALI-C ↑
MVC ↑
SemSim ↑
Time
Few-Shot GS
0.278
0.352
1792
0.707
24min
DiFix3D+
0.354
0.451
1755
0.687
53min
GenFusion
0.340
0.378
1157
0.712
44min
Guidedvd-3dgs
0.230
0.298
1845
0.691
306min
Ours
0.404
0.498
1972
0.737
77min
Table 1. Non-reference metrics and runtime on Mip-NeRF 360. C-IQA denotes CLIP-IQA. QUALI-C denotes QUALI-CLIP.
Methods
PSNR ↑
SSIM ↑
LPIPS ↓
Few-Shot GS
18.92
0.80
0.3438
DiFix3D+
20.93
0.84
0.2554
GenFusion
21.93
0.85
0.2502
Guidedvd-3dgs
22.10
0.84
0.2662
Ours
23.79
0.88
0.2300
Table 2. Comparisons of reference metrics on ScanNet++. Our method achieves the best performance averaged on 10 scenes.
Methods
PSNR ↑
LPIPS ↓
Exst.
Extrapltd.
Exst.
Extrapltd.
DiFix3D+
28.11
10.23
0.1187
0.4654
w/ Bayes’ Rays
27.87
10.20
0.1235
0.4701
w/ Nerfbusters
27.73
10.09
0.1193
0.4659
w/ NVF
28.19
10.02
0.1093
0.4700
w/ Opacity Maps
28.15
10.17
0.1109
0.4655
Table 3. Generalizability of QA-Mask on ScanNet++. Exst. denotes existing and Extrapltd. denotes extrapolated.
Figure 8. Comparison between w/ and w/o QA-Mask on GenFusion and DiFix3D+. QA-Mask prevents generated data from disrupting well-reconstructed areas in the original scene. Noticeable improvements can be observed for both models.
Mip-NeRF 360
CLIP-IQA ↑
MVC ↑
SemSim ↑
w/o QA-Mask
0.343
1673
0.719
w/o DIBR Propagation
0.316
1468
0.697
Ours
0.404
1972
0.737
ScanNet++
PSNR ↑
SSIM ↑
LPIPS ↓
w/o Idpdt Cam Detection
20.27
0.80
0.3161
w/o Depth Supervision
23.66
0.88
0.2302
Table 4. Ablation study for core components and hyperparameter selection on both Mip-NeRF 360 and ScanNet++. Idpdt Cam Detection denotes Independent Camera View Detection.
School of Artificial Intelligence, University of Chinese Academy of Sciences, China · Global Institute of Future Technology, Shanghai Jiao Tong University, China