Single-view 3D reconstruction, also known as image-to-3D, is a persistently challenging task due to the extreme lack of information. Recently, diffusion models pre-trained on large-scale datasets served as 2D priors are used to solve the ill-posed task but suffer from color deviation and view inconsistency, which can be curbed by using diffusion models fine-tuned with 3D annotated data served as 3D priors. However, 3D priors lack high-frequency details, which cannot be solved by direct complementation with 2D priors in spatial domain for introducing erroneous low-frequency 2D prior guidance. In this paper, we revisit the characteristics of different diffusion priors from the frequency perspective. Based on our observations, we theoretically present a unified framework of hybrid optimization using multiple diffusion priors in frequency domain. Under this framework, we further propose Morpheus3D, a pipeline of 3D object generation from any single unposed image in the wild. Morpheus3D enhances 3D prior with high-pass image-prompt 2D prior guidance to reconstruct high-quality 3D objects while effectively suppressing view inconsistency, low-frequency color deviation, and high-frequency lacking problems. Both quantitative and qualitative experiments on the public and our collected datasets with complex textures show that our method exhibits significant improvements in generation quality.
Figures & tables
Figure 1: Morpheus3D results of 3D object generation from any single unposed image in the wild. See more results in https://icvteam.github.io/Morpheus3D.html . Code: https://github.com/iCVTEAM/Morpheus3D
Figure 2: Properties of different diffusion priors. Text-prompt 2D priors in previous works with textual inversion, like Stable Diffusion ( Rombach et al., 2022 ) , suffer from the Janus Problem due to the object-inconsistency guidance caused by high information entropy. Image-prompt 3D priors , like Zero-1-to-3 ( Liu et al., 2023 ) , take advantages of object-consistency but lack high-frequency details. Image-prompt 2D priors , like Prompt Free Diffusion ( Xu et al., 2023b ) , controlled by Canny edges of coarse rendered images, show the high-frequency guidance ability, but take the object-inconsistency of low-frequency color deviation. Our work aims to utilize the high-pass image-prompt 2D priors guidance to enhance 3D priors.
Figure 3: Power spectral density of Zero-1-to-3 and PFD , obtained by 100 images sampled uniformly in azimuth of each case on Realfusion15 ( Melas-Kyriazi et al., 2023 ) . Left: the rendered image guided by Zero-1-to-3 ( Liu et al., 2023 ) with SDS ( Poole et al., 2023 ) ; Middle: the generated image of Zero-1-to-3; Right: the generated image of PFD ( Xu et al., 2023b ) controlled by the Canny edge of the left image. In order to better display the high-frequency part, we renormalize the frequency f to 1−log(1−f) .
Figure 4: Base: the coarse rendered image guided by Zero-1-to-3 with SDS; PFD: the generated image of PFD controlled by the Canny edge of the coarse rendered image. Amplitude spectrum of Base and PFD (min-max normalization for better visualization), and the spectrum residual (times 3 for better visualization) to Base from the rendered image.
Figure 5: Hybrid optimization using multi-diffusion priors in frequency domain . For a distribution of the reference image, we can take advantage of and discard the disadvantage of different diffusion priors in frequency domain under our framework to reconstruct high-quality 3D objects matching the reference distribution.
Figure 6: Image-prompt 2D prior guiance . The Prompt Free Diffusion (PFD), which served as an image-prompt 2D prior, takes the Canny edge of the rendered image as the condition for guidance, which has object consistency but also low-frequency color deviation.
Figure 7: Overview of proposed pipeline. In the first stage , we optimize a NeRF and subsequently initialize a DMTet from it. Either of them is optimized using 3D prior without fine-tuning. In the second stage , we continue to optimize the inherited DMTet using 3D prior without fine-tuning to maintain low-frequency characteristics and high-pass 2D prior with fine-tuning to boost high-frequency details.
Feed-forward Methods
Method
Pub.
Views
Realfusion15
MorpheusObj30
MANIQA
CLIPIQA
CLIP-Sim.
PSNR
LPIPS
MANIQA
CLIPIQA
CLIP-Sim.
PSNR
LPIPS
Shap-E
-
100
0.410
0.560
0.633
-
-
0.433
0.539
0.621
-
-
Zero-1-to-3
ICCV23
100
0.166
0.565
0.825
18.53
0.117
0.154
0.531
0.805
25.12
0.041
OpenLRM
ICLR24
100
0.162
0.466
0.761
18.41
0.115
0.170
0.453
0.769
24.34
0.034
ImageDream
-
4
0.160
0.542
0.769
-
-
0.137
0.513
0.747
-
-
Table 1: Quantitative comparisons with other methods.
Figure 8: Qualitative comparisons with other methods.
Figure 9: Study of color deviation using full/high-pass 2D prior guidance . Using full-pass 2D prior guidance in spatial domain without high-passing shows heavier color deviation, which indicates the significance of optimization in frequency domain.
Figure 10: Ablation of high-passing 2D prior guidance . Using high-pass 2D prior guidance exhibits both improvements in semantic consistency and reconstruction quality for novel views.
Figure 11: Boosting high-frequency details using high-pass 2D prior guidance with fine-tuning 2D prior.
Figure 12: Ablation of fine-tuning 2D prior , which shows both improvements in semantic consistency and reconstruction quality.
Figure 13: Power Spectral Density (PSD) of Base and w/ High-pass 2D Prior Guidance (Ours) , obtained by 100 images sampled uniformly in azimuth of each case on Realfusion15 ( Melas-Kyriazi et al., 2023 ) . We visualize the high-frequency part of PSD. Both PSD and Frequency are scaled by log .
Figure 14: The average high-frequency PSD during the training process on Realfusion15 ( Melas-Kyriazi et al., 2023 ) .
Figure 15: Analysis of different prior weights.
Realfusion15
MorpheusObj30
Prior Weights
CLIP-Similarity ↑
MANIQA ↑
CLIPIQA ↑
CLIP-Similarity ↑
MANIQA ↑
CLIPIQA ↑
k2D=0.0 (w/o 2D Prior)
0.824
0.433
0.707
0.816
0.456
0.699
k2D=1.0
0.835
0.453
0.737
0.833
0.483
0.733
k2D=2.0
0.836
0.454
0.748
0.837
0.486
0.744
k2D=3.0 (Ours)
0.838
0.454
0.748
0.835
0.487
0.751
k2D=4.0
0.833
0.453
0.758
0.831
0.484
0.755
Table 2: Analysis of different prior weights.
Figure 16: Analysis of different frequency bounds.
Realfusion15
MorpheusObj30
Frequency Bound Rate
CLIP-Similarity ↑
MANIQA ↑
CLIPIQA ↑
CLIP-Similarity ↑
MANIQA ↑
CLIPIQA ↑
r=0.0 (Full Pass)
0.823
0.401
0.695
0.827
0.438
0.696
r=0.2
0.824
0.461
0.748
0.828
0.487
0.745
r=0.4
0.829
0.458
0.757
0.835
0.486
0.755
r=0.6 (Ours)
0.838
0.454
0.748
0.835
0.487
0.751
r=0.8
0.835
0.457
0.742
0.836
0.485
0.738
Table 3: Analysis of different frequency bounds.
Figure 17: Study of fine-tuning 3D prior on MorpheusObj30 . Fine-tuning 3D prior shows an overall trend of increasing high-frequency PSD, but decreasing reconstruction quality.
Figure 18: Artifacts caused by fine-tuning 3D prior. Improvements in high-frequency PSD are actually artifacts.
Figure 19: Study of refining from generated assets of the feed-forward method. Top: Quantitative comparison shows that our method provides overall improvements. Bottom: Qualitative comparison shows that our method further improves the texture details of the TRELLIS baseline.