Intrinsic Geometry-Appearance Consistency Optimization for Sparse-View Gaussian Splatting
Authors: Kaiqiang Xiong, Rui Peng, Jiahao Wu, Zhanke Wang, Jie Liang, Xiaoyun Zheng, Feng Gao, Ronggang Wang
Organizations: Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology, Shenzhen Graduate School, Peking University · Peng Cheng Laboratory · Alibaba Group · School of Arts, Peking University · Migu Culture Technology Co., Ltd
3D human reconstruction from a single image is a challenging problem and has been exclusively studied in the literature. Recently, some methods have resorted to diffusion models for guidance, optimizing a 3D representation via Score Distillation Sampling(SDS) or generating a back-view image for facilitating reconstruction. However, these methods tend to produce unsatisfactory artifacts (e.g. flattened human structure or over-smoothing results caused by inconsistent priors from multiple views) and struggle with real-world generalization in the wild. In this work, we present MVD-HuGaS, enabling free-view 3D human rendering from a single image via a multi-view human diffusion model. We first generate multi-view images from the single reference image with an enhanced multi-view diffusion model, which is well fine-tuned on high-quality 3D human datasets to incorporate 3D geometry priors and human structure priors. To infer accurate camera poses from the sparse generated multi-view images for reconstruction, an alignment module is introduced to facilitate joint optimization of 3D Gaussians and camera poses. Furthermore, we propose a depth-based Facial Distortion Mitigation module to refine the generated facial regions, thereby improving the overall fidelity of the reconstruction. Finally, leveraging the refined multi-view images, along with their accurate camera poses, MVD-HuGaS optimizes the 3D Gaussians of the target human for high-fidelity free-view renderings. Extensive experiments on Thuman2.0 and 2K2K datasets show that the proposed MVD-HuGaS achieves state-of-the-art performance on single-view 3D human rendering.
Figures & tables
Figure 2: Framework of ICO-GS. Given sparse input views, we initialize 3D Gaussians and extract deep features. Our method enforces intrinsic geometry-appearance consistency through two synergistic components: (1) Robust Geometric Regularization. Source features are warped to reference views via rendered depth, establishing occlusion-aware multi-view constraints through: (a) Robust Multi-view Photometric Consistency that employs pixel-wise top- k selection for occlusion handling, and (b) Edge-aware Depth Smoothness that preserves sharp geometric structures. (2) Geometry-Guided Appearance Optimization. We leverage geometrically reliable regions identified by (c) Cycle Consistency Depth Filtering to synthesize virtual views, then apply (d) Virtual-view Photometric Consistency between synthesized and rendered images to propagate geometric correctness into appearance learning.
Figure 3: Geometry-appearance discrepancy under sparse-view settings. From top to bottom: RGB on training views, depth on training views, and RGB on test views, rendered by 3D Gaussian Splatting [ 19 ] with varying training view densities. With decreasing views, training-view appearance (top) remains well-fitted, but depth quality (middle) collapses with noise and floaters due to insufficient multi-view constraints. This geometry-appearance discrepancy leads to severe artifacts in novel-view rendering (bottom).
Methods
PSNR↑
SSIM↑
LPIPS↓
3-view
6-view
9-view
3-view
6-view
9-view
3-view
6-view
9-view
DietNeRF [ 16 ]
14.94
21.75
24.28
0.370
0.717
0.801
0.496
0.248
0.183
RegNeRF [ 29 ]
19.08
23.10
24.86
0.587
0.760
0.820
0.336
0.206
0.161
FreeNeRF [ 42 ]
19.63
23.73
25.13
0.612
0.779
0.827
0.308
0.195
0.160
SparseNeRF [ 35 ]
19.86
23.26
24.27
0.714
0.741
0.781
0.243
0.235
0.228
3DGS [ 19 ]
15.52
19.45
21.13
0.405
0.627
0.715
0.408
0.268
0.214
Table 1: Quantitative comparisons on LLFF [ 26 ] dataset under sparse view settings.
Figure 4: Visual comparison on LLFF [ 26 ] dataset.
Methods
PSNR↑
SSIM↑
LPIPS↓
3-view
6-view
9-view
3-view
6-view
9-view
3-view
6-view
9-view
DietNeRF [ 16 ]
11.85
20.63
23.83
0.633
0.778
0.823
0.214
0.201
0.173
RegNeRF [ 29 ]
18.89
22.20
24.93
0.745
0.841
0.884
0.190
0.117
0.089
FreeNeRF [ 42 ]
19.52
23.25
25.38
0.787
0.844
0.888
0.173
0.131
0.102
SparseNeRF [ 35 ]
19.47
-
-
0.829
-
-
0.183
-
-
3DGS [ 19 ]
10.99
20.33
22.90
0.585
0.776
0.816
0.313
0.223
0.173
Table 2: Quantitative comparisons on DTU dataset under sparse view settings.
Figure 5: Visual comparison on DTU [ 18 ] dataset.
Methods
PSNR ↑
SSIM ↑
LPIPS ↓
DietNeRF [ 16 ]
22.50
0.823
0.124
RegNeRF [ 29 ]
23.86
0.852
0.105
FreeNeRF [ 42 ]
24.26
0.883
0.098
SparseNeRF [ 35 ]
22.41
0.861
0.199
3DGS [ 19 ]
22.23
0.858
0.114
FSGS [ 47 ]
22.76
0.829
0.157
Table 3: Quantitative comparison on Blender [ 27 ] for 8 views.
Figure 6: Visual comparison on Blender [ 27 ] dataset.
LmpcFea
Lsmooth
CCDF
Lapp
LLFF (3-view)
DTU (3-view)
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
×
×
×
×
21.44
0.751
0.168
20.71
0.862
0.111
×
✓
✓
✓
21.82
0.761
0.165
21.31
0.883
0.099
✓
×
✓
✓
22.16
0.775
0.159
21.67
0.879
0.093
✓
✓
×
✓
21.86
0.767
0.162
21.25
0.875
0.104
✓
✓
✓
×
21.79
0.763
0.164
21.20
0.870
0.105
Table 4: Ablation studies on LLFF [ 26 ] and DTU [ 18 ] .
Figure 7: Visualization of ablation study results using 3-views.