Feed-forward 3D reconstruction has revolutionized 3D vision, providing a powerful baseline for downstream tasks such as novel-view synthesis with 3D Gaussian Splatting. Previous works explore fixing the corrupted rendering results with a diffusion model. However, they lack geometric concern and fail at filling the missing area on the extrapolated view. In this work, we introduce Leveling3D, a novel pipeline that integrates feed-forward 3D reconstruction with geometrical-consistent generation to enable holistic simultaneous reconstruction and generation. We propose a geometry-aware leveling adapter, a lightweight technique that aligns internal knowledge in the diffusion model with the geometry prior from the feed-forward model. The leveling adapter enables generation on the artifact area of the extrapolated novel views caused by underconstrained regions of the 3D representation. Specifically, to learn a more diverse distributed generation, we introduce the palette filtering strategy for training, and a test-time masking refinement to prevent messy boundaries along the fixing regions. More importantly, the enhanced extrapolated novel views from Leveling3D could be used as the inputs for feed-forward 3DGS, leveling up the 3D reconstruction. We achieve SOTA performance on public datasets, including tasks such as novel-view synthesis and depth estimation.
Figures & tables
Figure 2: Overview of the Leveling3D pipeline. Our pipeline integrates a geometry-aware leveling adapter that fuses geometry tokens with diffusion control, enabling robust refinement of extrapolated views and leveling up the 3D reconstruction with extended geometric-consistent extrapolation areas.
Methods
Diffusion Type
MipNeRF360 [ 4 ] Dataset
VRNeRF [ 37 ] Dataset
Time(s)
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
Baseline [ 13 ]
N/A
15.60
0.318
0.314
15.89
0.532
0.347
N/A
ViewExtrapolator [ 20 ]
Video
14.85
0.324
0.606
16.716
0.591
0.518
2.59
GSFixer [ 40 ]
15.69
0.332
0.348
15.86
0.554
0.356
12.07
GSFix3D [ 34 ]
14.96
0.302
0.354
13.47
0.412
0.53
1.094
Difix3D+ [ 35 ]
Image
15.68
0.334
0.312
16.22
0.540
0.363
0.718
Table 1: Quantitative comparison on MipNeRF360 [ 4 ] and VRNeRF [ 37 ] datasets with sparse two-views input . The best result is in bold , and the second-best is underlined . All experiments are conducted with the same baseline [ 13 ] .
Figure 3: Qualitative result on MipNeRF360 [ 4 ] and VRNeRF [ 37 ] datasets. Previous refinement methods suffer from severe geometry collapse, texture corruption, and missing content. In contrast, our Leveling3D method robustly generate fine details and fills large extrapolated areas with plausible content.
Methods
TartanAir [ 31 ] Dataset
ScanNet [ 8 ] Dataset
Abs Rel ↓
RMSE ↓
δ<1.25↑
Met3R ↓
Abs Rel ↓
RMSE ↓
δ<1.25↑
Met3R ↓
Baseline [ 13 ]
0.915
20.748
0.311
0.0683
0.372
58.826
0.708
0.0472
ViewExtrapolator [ 20 ]
3.791
139.387
0.102
0.0633
3.794
120.666
0.323
0.0646
GSFix3D [ 34 ]
0.905
20.493
0.315
0.0675
0.29
33.398
0.791
0.0453
GSFixer [ 40 ]
0.892
20.366
0.315
0.0665
0.296
31.296
0.784
0.0462
Difix3D+ [ 35 ]
0.893
20.298
0.321
0.0625
0.286
0.358
0.787
0.0452
Table 2: Quantitative results of depth estimation on TartanAir [ 31 ] and ScanNet [ 8 ] datasets . We evaluate all methods with the same baseline [ 13 ] , the same sparse two-view input, and novel poses. Our method outperforms other baselines.
Figure 4: Qualitative result of depth estimation on TartanAir [ 31 ] and ScanNet [ 8 ] datasets. Our method refines geometry-consistent novel views within the extrapolated artifact areas, leveling up the 3DGS representation to achieve complete scene reconstruction with superior geometry detail recovery and boundary coherence.
Baseline [ 13 ]
Ours
Views
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
2-Views
9.506
0.176
0.804
12.296
0.392
0.776
3-Views
10.076
0.272
0.722
12.676
0.404
0.702
4-Views
10.794
0.288
0.724
13.272
0.412
0.704
5-Views
11.242
0.322
0.708
13.446
0.424
0.678
6-Views
11.638
0.344
0.686
14.041
0.442
0.664
Table 3: Quantitative evaluation of view-dependency on the ScanNet dataset. We compare our method against AnySplat across varying numbers of input views. Bold numbers indicate the best performance.
Table 7
Figure 5: Qualitative result by removing the test-time mask refinement. Our mask refinement robustly prevent corrupted generation near the boundary.
Figure 6: Qualitative result by removing geometry tokens and palette filtering. Our full pipeline generate sharp details and coherent geometry.
range
PSNR ↑
SSIM ↑
LPIPS ↓
FID ↓
Met3R ↓
Baseline [ 13 ]
15.892
0.532
0.354
153.39
0.0322
mean ± 0.5 std (0.38)
17.984
0.586
0.334
161.89
0.0254
mean ± 1 std (0.68)
18.362
0.610
0.314
141.15
0.0242
mean ± 1.5 std (0.86)
18.106
0.594
0.324
151.48
0.0266
Table S1 : Comparison of different palette threshold on VRNeRF [ 37 ] .
Figure S1 : Qualitative Results with Different Diffusion Condition Methods. Our method outperforms other diffusion condition methods.
Figure S2 : Qualitative results on nuScenes dataset. Our method outperforms other methods on novel view synthesis.
Figure S3 : Qualitative results on SCARED dataset. Leveling3D achieves the best results for cross-domain surgical-scene novel-view synthesis and depth estimation, demonstrating robust generalization.
Method
Num. Param. ↓
Inf. Time ↓
Naive Diffusion* [ 23 ]
865.83M
1.071s
T2I [ 21 ]
83.51M
1.186s
ControlNet [ 42 ]
369.34M
1.548s
Ours
68.74M
1.167s
Table S5 : Comparison of number of parameters (Num. Param.) and inference time (Inf. Time). * indicates diffusion model without control.
Figure S4 : Failure cases. The highlighted areas are the key to failure.