Using accurate depth priors in 3D Gaussian Splatting helps mitigate artifacts caused by sparse training data and textureless surfaces. However, acquiring accurate depth maps requires specialized acquisition systems. Foundation monocular depth estimation models offer a cost-effective alternative, but they suffer from scale ambiguity, multi-view inconsistency, and local geometric inaccuracies, which can degrade rendering performance when applied naively. This paper addresses the challenge of reliably leveraging monocular depth priors for Gaussian Splatting (GS) rendering enhancement. To this end, we introduce a training framework integrating scale-ambiguous and noisy depth priors into geometric supervision. We highlight the importance of learning from weakly aligned depth variations. We introduce a method to isolate ill-posed geometry for selective monocular depth regularization, restricting the propagation of depth inaccuracies into well-reconstructed 3D structures. Extensive experiments across diverse datasets show consistent improvements in geometric accuracy, leading to more faithful depth estimation and higher rendering quality across different GS variants and monocular depth backbones tested.
Figures & tables
Figure 1
Figure 2: Impact of SfM point count in scale alignment on GS rendering quality for identical training images and point clouds. When using only Lsid , fewer observed points lead to degraded performance compared to the baseline (without using monocular depth supervision), highlighting the limitation of scale-invariant depth supervision in aligning monocular depth cues.
Figure 3: Overview of our proposed framework. During GS optimization, depth-inconsistent Gaussians are identified using a virtual stereo setup ( Sec. 4.1 ). These regions guide the learning of absolute depth corrections from monocular depth priors aligned with sparse SfM point clouds. Additionally, a relative depth loss ( Sec. 4.2 ) is applied to learn geometric cues under ambiguous scene scales.
Figure 4: Comparison of different relative depth supervision losses, shown on two zoomed-in scene crops. Gradient-based loss ( GAL ) achieves sharper rendering with fewer smearing artifacts compared to the patch-based Pearson correlation loss, as seen in the reduced distortion on photo-frame edges in the top row and a clearer shape around the orange triangular region in the bottom.
Low Data
Moderate Data
PSNR ↑
SSIM ↑
LPIPS ↓
AVG-E ↓
PSNR ↑
SSIM ↑
LPIPS ↓
AVG-E ↓
3DGS [ 19 ]
3DGS (w/o depth)
19.916
0.7206
0.2715
0.1156
23.240
0.8158
0.1920
0.0747
SparseGS [ 42 ]
20.152
0.7202
0.2719
0.1134
23.146
0.8091
0.2133
0.0783
DNGaussian [ 23 ]
19.740
0.6905
0.3717
0.1314
22.190
0.7634
0.3017
0.0976
FSGS [ 49 ]
20.307
0.7213
0.3069
0.1165
22.998
0.7949
0.254
0.085
Lsid
20.173
0.7208
0.2721
0.1132
23.318
0.8149
0.1941
0.0747
Table 2: Results on the TanksAndTemples. Best performance is bolded in red and the second is marked in orange .
Low Data
Moderate Data
PSNR ↑
SSIM ↑
LPIPS ↓
AVG-E ↓
PSNR ↑
SSIM ↑
LPIPS ↓
AVG-E ↓
3DGS [ 19 ]
3DGS (w/o depth)
21.785
0.6267
0.3409
0.1210
25.591
0.7576
0.2513
0.0751
SparseGS [ 42 ]
22.015
0.6302
0.3462
0.1192
25.519
0.7570
0.2590
0.0762
DNGaussian [ 23 ]
21.801
0.6054
0.4594
0.1314
24.245
0.6789
0.3962
0.1002
FSGS [ 49 ]
21.693
0.6120
0.4186
0.1293
24.811
0.6997
0.358
0.093
Lsid
21.799
0.6274
0.3399
0.1202
25.668
0.7586
0.2501
0.0745
Table 3: Results on the MipNeRF 360. Best performance is bolded in red and the second is marked in orange .
Figure 5: Qualitative results on 3DGS . ( Top ): low-data setting; ( Bottom ): moderate-data setting.
Figure 6: Qualitative comparisons of rendered images and depth maps on the ScanNet++ dataset for the baseline (without depth supervision), Lsid , and our method using MoGe-2 monocular depth supervision. Our method produces clearer, more accurate depth maps and yields sharper, less artifact-prone renderings than the baseline and Lsid . We encourage readers to zoom in for a clearer visual comparison.
Lsid
DIM
GAL
PSNR ↑
SSIM ↑
LPIPS ↓
AVG-E ↓
✓
21.42
0.620
0.348
0.127
✓
✓
21.66
0.624
0.346
0.123
✓
✓
✓
22.09
0.639
0.354
0.118
Table 4: Module ablation study on all scenes of the MipNeRF 360 dataset. The best performance is bolded .
Scene
Method
PSNR ↑
SSIM ↑
LPIPS ↓
AVG-E ↓
Train(min)
Outdoor
NCC w63
18.60
0.449
0.423
0.167
26.76
NCC w127
18.56
0.447
0.421
0.168
37.39
Lcosine
18.09
0.430
0.425
0.175
22.5
Ours
18.97
0.464
0.430
0.163
21.52
Indoor
NCC w63
25.47
0.851
0.263
0.066
30.59
NCC w127
25.98
0.858
0.254
0.063
47.23
Table 5: Comparisons of different relative depth loss choices on MipNeRF 360. The best performance is bolded .
3D Gaussian Splatting (3DGS) has emerged as a powerful technique for generating photorealistic renderings of a scene in real-time. However, the volumetric nature of 3DGS limits its ability to accurately capture surface geometry. To address this, 2D Gaussian Splatting (2DGS) was proposed to enable view-consistent and geometrically accurate surface reconstruction from multi-view images. However, 2DGS can be sensitive to the initialization of the Gaussian primitives. Reliance on Structure-from-Motion (SfM) initializations, which can produce poor estimates on challenging image sets, may lead to subpar results. In this work, we enhance 2DGS by incorporating monocular depth and normal priors to improve both geometric accuracy and robustness. We propose a depth-guided initialization strategy for Gaussians and introduce a clustering-based technique for pruning degenerate Gaussians. We evaluate our method on the DTU dataset, where it achieves state-of-the-art results in mesh reconstruction while preserving high-quality novel view synthesis.
Prajwal Gupta C. R., Divyam Sheth, Jinjoo Ha +2
TU Darmstadt · ELIZA · Max Planck Institute for Intelligent Systems
3D Gaussian Splatting (3DGS) is a state-of-the-art technique for 3D scene rendering, offering high efficiency and excellent visual quality. However, because 3DGS relies on an initial sparse point set from Structure-from-Motion (SfM) and view-dependent properties, it can suffer from geometric inaccuracies and visual artifacts, particularly in complex scenes. To address these challenges, we propose an improved 3DGS approach that regularizes the optimization process by integrating geometric priors, including surface normals and dense depth information. Surface normal regularization improves geometric consistency by aligning Gaussian covariance with local surface structures, while dense depth priors combined with an initial points from SfM enhance per-pixel depth estimation, increasing accuracy and reducing ambiguities. These enhancements enable robust handling of diverse and complex real-world scenarios, minimizing visual distortions and improving reconstruction quality across various environments. To validate our method, we evaluate it on challenging datasets, including street-view scenes and highly reflective environments, while testing it across multiple SfM pipelines. Our results demonstrate compatibility across diverse environments and highlight the robustness of our approach. Experimental findings further show that our method enhances geometric accuracy and visual quality, establishing a reliable solution for real-time 3D scene rendering in complex environments.
Sparse-view neural reconstruction is challenging in outdoor driving scenes, where cameras usually move along a narrow forward-facing trajectory and provide limited multi-view overlap. Although monocular depth estimators can provide dense geometric priors, their predictions are noisy, and not uniformly reliable across image regions. In this work, we study monocular depth supervision for sparse-view neural reconstruction. We use Depth Anything V2 as a dense monocular depth prior, align its predictions to metric depth using scale-shift fitting, and apply depth supervision selectively through photometric masks generated from an RGB-only baseline model. We evaluate this strategy on two representative scene representations: Mip-NeRF-360 and Splatfacto. On KITTISeq02 under an every2 sparse-view setting, masked monocular depth supervision gives only marginal rendering gains for Mip-NeRF-360 and does not improve metric geometry. In contrast, Splatfacto benefits more clearly, improving PSNR from 14.903 to 15.932 and reducing RMSE from 0.542 to 0.100. Additional KITTISeq05 experiments and matched-ratio mask ablations further show that the gains for Splatfacto come from selecting reliable low-error regions rather than simply reducing the number of depth-supervised pixels. Additional experiments on the Bicycle scene show that depth supervision can improve geometry while hurting RGB rendering quality when multi-view coverage is already strong. Overall, our results suggest that monocular depth priors are useful for under-constrained sparse-view reconstruction, but should be applied selectively and with moderate weighting.