3D Gaussian Splatting has demonstrated superior performance in rendering efficiency and quality, yet the generation of 3D Gaussians still remains a challenge without proper geometric priors. Existing methods have explored predicting point maps as geometric references for inferring Gaussian primitives, while the unreliable estimated geometries may lead to poor generations. In this work, we introduce GaussianGrow, a novel approach that generates 3D Gaussians by learning to grow them from easily accessible 3D point clouds, naturally enforcing geometric accuracy in Gaussian generation. Specifically, we design a text-guided Gaussian growing scheme that leverages a multi-view diffusion model to synthesize consistent appearances from input point clouds for supervision. To mitigate artifacts caused by fusing neighboring views, we constrain novel views generated at non-preset camera poses identified in overlapping regions across different views. For completing the hard-to-observe regions, we propose to iteratively detect the camera pose by observing the largest un-grown regions in point clouds and inpainting them by inpainting the rendered view with a pretrained 2D diffusion model. The process continues until complete Gaussians are generated. We extensively evaluate GaussianGrow on text-guided Gaussian generation from synthetic and even real-scanned point clouds. Project Page: https://weiqi-zhang.github.io/GaussianGrow
Figures & tables
Figure 2: Overview of GaussianGrow. Stage 1. We leverage depth-aware ControlNet for primary view generation, with a geometry-aware diffusion model for multi-view synthesis. Additional views are generated for improving appearances in overlap regions by optimizing camera poses to observe overlap regions. Gaussians are optimized to grow with supervision from both cardinal and additional views. Stage 2. We iteratively inpaint Gaussians by optimizing camera poses to observe unseen regions, and inpaint them by inpainting the rendering view with a pretrained 2D diffusion model. The iteration continues until complete Gaussians are generated. A spatial Gaussian inpainting strategy is also used to diffuse appearance from optimized Gaussians to the hard-to-observe ones.
Figure 3: We obtain the additional camera poses by optimizing them to observe largest overlap regions.
Figure 4: The effect of Gaussian inpainting.
Figure 5: The effectiveness of processing overlap regions.
Figure 6: Visual comparison on the Objaverse dataset shows that GaussianGrow uses point clouds instead of meshes.
Method
FID ↓
KID ↓
CLIP ↑
User Study
Overall Quality ↑
Text Fidelity ↑
TexTure [ 38 ]
42.63
7.84
26.84
1.49
1.67
Text2Tex [ 5 ]
41.62
6.45
26.73
2.37
3.23
SyncMVD [ 25 ]
40.85
5.77
27.24
4.13
4.34
Paint3D [ 57 ]
41.08
5.81
26.73
3.37
3.45
GAP [ 60 ]
40.39
5.28
27.26
3.37
4.13
Table 1: Quantitative comparison on the Objaverse dataset.
Ours + Uni3D
Ours + LGM
DiffSplat [ 22 ]
GVGEN [ 10 ]
LN3Diff [ 20 ]
DIRECT-3D [ 24 ]
3DTopia [ 13 ]
LGM [ 42 ]
GRM [ 53 ]
↑ CLIP Sim. %
31.55
30.17
30.95
23.66
24.36
24.80
25.55
29.96
28.19
↑ CLIP R-Prec. %
82.00
81.00
81.00
23.25
27.25
30.75
34.50
78.00
64.75
↑ ImageReward
-0.316
-0.329
-0.491
-2.156
-2.008
-2.005
-1.998
-0.720
-1.337
Table 2: Quantitative evaluations on T3Bench benchmark for text-to-3D generation. Higher is better for all metrics.
Figure 7: Text-to-3D comparisons on T3Bench.
Figure 8: Visual comparison with DreamGaussian and TriplaneGaussian on the task of Point-to-Gaussian.
Figure 9: Diverse style Gaussian generations.
Method
FID ↓
KID ↓
CLIP ↑
Full Model
36.07
3.04
27.30
W/o Overlap Processing
40.48
4.81
26.73
W/o Inpaint
40.46
4.68
26.71
Table 3: Ablation results for key components of GaussianGrow.
Number of Views
FID ↓
KID ↓
CLIP ↑
K=6 (cardinal only)
40.48
4.81
26.73
K=10
36.07
3.04
27.30
K=12
36.57
2.88
26.48
Table 4: Impact of the number of views K on generation quality.
School of Future Technology, South China University of Technology, Guangzhou 511442, China · School of Engineering, Westlake University, Hangzhou 310030, China