Rendering 3D surfaces has been revolutionized within the modeling of radiance fields through either 3DGS or NeRF. Although 3DGS has shown advantages over NeRF in terms of rendering quality or speed, there is still room for improvement in recovering high fidelity surfaces through 3DGS. To resolve this issue, we propose a self-constrained prior to constrain the learning of 3D Gaussians, aiming for more accurate depth rendering. Our self-constrained prior is derived from a TSDF grid that is obtained by fusing the depth maps rendered with current 3D Gaussians. The prior measures a distance field around the estimated surface, offering a band centered at the surface for imposing more specific constraints on 3D Gaussians, such as removing Gaussians outside the band, moving Gaussians closer to the surface, and encouraging larger or smaller opacity in a geometry-aware manner. More importantly, our prior can be regularly updated by the most recent depth images which are usually more accurate and complete. In addition, the prior can also progressively narrow the band to tighten the imposed constraints. We justify our idea and report our superiority over the state-of-the-art methods in evaluations on widely used benchmarks.
Figures & tables
Figure 1: Overview of our method. Given 3D Gaussians g , we employ a distance field specified by a fused TSDF grid as our prior ft . With ft , we define a bandwidth by the surface and iteratively refine ft with updated depth renderings. We also apply Gaussian geometric constraints (GC) that are related to interpolated distance s , centers μ and gradients ∇ft for high fidelity surface reconstruction.
Figure 2: Illustration of bandwidth and the range of opacity control.
Figure 3: Visualization of periodical update on our prior.
Figure 4: Visualization of Gaussian centers with each constraint. The error map indicates the distance to the ground truth surface.
Figure 5: Visual comparison of reconstruction on NeRF-Synthetic dataset.
Figure 6: Error map comparison of rendering on NeRF-Synthetic.
Figure 7: Visual comparison of reconstruction on DTU dataset. The error map indicates the distance to the ground truth surface.
Figure 8: Visual comparison of reconstruction on TNT dataset, the color indicates the normal direction.
Figure 9: Visual comparison of reconstruction on Mip-NerF 360 dataset, the color indicates the normal direction.
Self-constrained Priors
Opacity Constraints
Supervision Refinement
Constraints on Gaussians
Bandwidth Scales
Methods
w ft
w/o ft
w LSCP
w/o LSCP
{f0t}
{f0t,f1t}
{f0t,f1t,f2t}
w/o fremovet
w/o fprojt
Full
1
0.5
0.25
Full
CD ↓
0.50
0.56
0.50
0.53
0.54
0.52
0.50
0.53
0.51
0.50
0.54
0.53
0.51
0.50
Table 5: Ablation studies on DTU dataset.
Figure 10: Effect of the self-constrained prior.
Figure 11: Effect of periodical update on our prior.
Figure 12: Effect of Gaussian Removal and Projection.
Figure 13: Effect of Opacity Constraint Threshold.
Figure 14: Visual Comparisons on NeRF-Synthetic Dataset.
Figure 15: Visual Comparisons on DTU Dataset.
Figure 16: Surface Reconstruction Visualization on TNT and Mip-NeRF 360 Dataset.
Figure 17: Rendering Visualization on Mip-NeRF 360 Dataset.
While 3D Gaussian Splatting (3DGS) has demonstrated impressive real-time rendering performance, its efficacy remains constrained by a reliance on heuristic density control. Despite numerous refinements to these handcrafted rules, such methods inherently lack the flexibility to adapt to diverse scenes with complex geometries. In this paper, we propose a paradigm shift for density control from rigid heuristics to fully learnable policies. Specifically, we introduce LeGS, a framework that reformulates density control as a parameterized policy network optimized via Reinforcement Learning (RL). Central to our approach is the tailored effective reward function grounded in sensitivity analysis, which precisely quantifies the marginal contribution of individual Gaussians to reconstruction quality. To maintain computational tractability, we derive a closed-form solution that reduces the complexity of reward calculation from O(N2) to O(N). Extensive experiments on the Mip-NeRF 360, Tanks & Temples, and Deep Blending datasets demonstrate that LeGS significantly outperforms state-of-the-art methods, striking a superior balance between reconstruction quality and efficiency. The code will be released at https://github.com/AaronNZH/LeGS
Zhenhua Ning, Xin Li, Jun Yu +3
Pengcheng Laboratory, Shenzhen · Harbin Institute of Technology, Shenzhen
3D Gaussian Splatting (3DGS) has recently enabled real-time novel view synthesis with impressive quality. However, it struggles to recover accurate surfaces under limited viewpoints and due to the inherent irregularity of Gaussian primitives. The resulting geometric errors are notoriously difficult to correct manually. To address these issues, we propose Gaussian Sculpting, a fully differentiable end-to-end framework for high-quality surface reconstruction. Our key insight is to anchor Gaussians onto an evolving differentiable surface, allowing them to guide signed distance field (SDF) optimization instead of extracting the surface only during post-processing. To enable stable gradient isolation during joint optimization, we design a bi-level training strategy in which the outer loop optimizes the geometry represented by the SDF, while the inner loop updates the Gaussians with the geometry fixed. We further impose constraints on Gaussian parameters to ensure consistency with the underlying surface, thereby improving both geometric and appearance fidelity during optimization. In addition, we introduce a multi-resolution subdivision scheme based on octree-like partitioning to preserve fine details while reducing memory consumption. Experiments on object-level scenes demonstrate that our method effectively removes redundant surfaces, recovers missing structures caused by limited viewpoints, and achieves strong reconstruction quality even at relatively low resolutions.
Ke Jiaxin, Juncheng Liu, Yi Wang +4
Dalian University of Technology, China · Massey University, New Zealand · Peking University, China +1
3D Gaussian Splatting (3DGS) is a state-of-the-art technique for 3D scene rendering, offering high efficiency and excellent visual quality. However, because 3DGS relies on an initial sparse point set from Structure-from-Motion (SfM) and view-dependent properties, it can suffer from geometric inaccuracies and visual artifacts, particularly in complex scenes. To address these challenges, we propose an improved 3DGS approach that regularizes the optimization process by integrating geometric priors, including surface normals and dense depth information. Surface normal regularization improves geometric consistency by aligning Gaussian covariance with local surface structures, while dense depth priors combined with an initial points from SfM enhance per-pixel depth estimation, increasing accuracy and reducing ambiguities. These enhancements enable robust handling of diverse and complex real-world scenarios, minimizing visual distortions and improving reconstruction quality across various environments. To validate our method, we evaluate it on challenging datasets, including street-view scenes and highly reflective environments, while testing it across multiple SfM pipelines. Our results demonstrate compatibility across diverse environments and highlight the robustness of our approach. Experimental findings further show that our method enhances geometric accuracy and visual quality, establishing a reliable solution for real-time 3D scene rendering in complex environments.