Single-view 3D reconstruction, also known as image-to-3D, is a persistently challenging task due to the extreme lack of information. Recently, diffusion models pre-trained on large-scale datasets served as 2D priors are used to solve the ill-posed task but suffer from color deviation and view inconsistency, which can be curbed by using diffusion models fine-tuned with 3D annotated data served as 3D priors. However, 3D priors lack high-frequency details, which cannot be solved by direct complementation with 2D priors in spatial domain for introducing erroneous low-frequency 2D prior guidance. In this paper, we revisit the characteristics of different diffusion priors from the frequency perspective. Based on our observations, we theoretically present a unified framework of hybrid optimization using multiple diffusion priors in frequency domain. Under this framework, we further propose Morpheus3D, a pipeline of 3D object generation from any single unposed image in the wild. Morpheus3D enhances 3D prior with high-pass image-prompt 2D prior guidance to reconstruct high-quality 3D objects while effectively suppressing view inconsistency, low-frequency color deviation, and high-frequency lacking problems. Both quantitative and qualitative experiments on the public and our collected datasets with complex textures show that our method exhibits significant improvements in generation quality.
Figures & tables
Figure 1: Morpheus3D results of 3D object generation from any single unposed image in the wild. See more results in https://icvteam.github.io/Morpheus3D.html . Code: https://github.com/iCVTEAM/Morpheus3D
Figure 2: Properties of different diffusion priors. Text-prompt 2D priors in previous works with textual inversion, like Stable Diffusion ( Rombach et al., 2022 ) , suffer from the Janus Problem due to the object-inconsistency guidance caused by high information entropy. Image-prompt 3D priors , like Zero-1-to-3 ( Liu et al., 2023 ) , take advantages of object-consistency but lack high-frequency details. Image-prompt 2D priors , like Prompt Free Diffusion ( Xu et al., 2023b ) , controlled by Canny edges of coarse rendered images, show the high-frequency guidance ability, but take the object-inconsistency of low-frequency color deviation. Our work aims to utilize the high-pass image-prompt 2D priors guidance to enhance 3D priors.
Figure 3: Power spectral density of Zero-1-to-3 and PFD , obtained by 100 images sampled uniformly in azimuth of each case on Realfusion15 ( Melas-Kyriazi et al., 2023 ) . Left: the rendered image guided by Zero-1-to-3 ( Liu et al., 2023 ) with SDS ( Poole et al., 2023 ) ; Middle: the generated image of Zero-1-to-3; Right: the generated image of PFD ( Xu et al., 2023b ) controlled by the Canny edge of the left image. In order to better display the high-frequency part, we renormalize the frequency f to 1−log(1−f) .
Figure 4: Base: the coarse rendered image guided by Zero-1-to-3 with SDS; PFD: the generated image of PFD controlled by the Canny edge of the coarse rendered image. Amplitude spectrum of Base and PFD (min-max normalization for better visualization), and the spectrum residual (times 3 for better visualization) to Base from the rendered image.
Figure 5: Hybrid optimization using multi-diffusion priors in frequency domain . For a distribution of the reference image, we can take advantage of and discard the disadvantage of different diffusion priors in frequency domain under our framework to reconstruct high-quality 3D objects matching the reference distribution.
Figure 6: Image-prompt 2D prior guiance . The Prompt Free Diffusion (PFD), which served as an image-prompt 2D prior, takes the Canny edge of the rendered image as the condition for guidance, which has object consistency but also low-frequency color deviation.
Figure 7: Overview of proposed pipeline. In the first stage , we optimize a NeRF and subsequently initialize a DMTet from it. Either of them is optimized using 3D prior without fine-tuning. In the second stage , we continue to optimize the inherited DMTet using 3D prior without fine-tuning to maintain low-frequency characteristics and high-pass 2D prior with fine-tuning to boost high-frequency details.
Feed-forward Methods
Method
Pub.
Views
Realfusion15
MorpheusObj30
MANIQA
CLIPIQA
CLIP-Sim.
PSNR
LPIPS
MANIQA
CLIPIQA
CLIP-Sim.
PSNR
LPIPS
Shap-E
-
100
0.410
0.560
0.633
-
-
0.433
0.539
0.621
-
-
Zero-1-to-3
ICCV23
100
0.166
0.565
0.825
18.53
0.117
0.154
0.531
0.805
25.12
0.041
OpenLRM
ICLR24
100
0.162
0.466
0.761
18.41
0.115
0.170
0.453
0.769
24.34
0.034
ImageDream
-
4
0.160
0.542
0.769
-
-
0.137
0.513
0.747
-
-
Table 1: Quantitative comparisons with other methods.
Figure 8: Qualitative comparisons with other methods.
Figure 9: Study of color deviation using full/high-pass 2D prior guidance . Using full-pass 2D prior guidance in spatial domain without high-passing shows heavier color deviation, which indicates the significance of optimization in frequency domain.
Figure 10: Ablation of high-passing 2D prior guidance . Using high-pass 2D prior guidance exhibits both improvements in semantic consistency and reconstruction quality for novel views.
Figure 11: Boosting high-frequency details using high-pass 2D prior guidance with fine-tuning 2D prior.
Figure 12: Ablation of fine-tuning 2D prior , which shows both improvements in semantic consistency and reconstruction quality.
Figure 13: Power Spectral Density (PSD) of Base and w/ High-pass 2D Prior Guidance (Ours) , obtained by 100 images sampled uniformly in azimuth of each case on Realfusion15 ( Melas-Kyriazi et al., 2023 ) . We visualize the high-frequency part of PSD. Both PSD and Frequency are scaled by log .
Figure 14: The average high-frequency PSD during the training process on Realfusion15 ( Melas-Kyriazi et al., 2023 ) .
Figure 15: Analysis of different prior weights.
Realfusion15
MorpheusObj30
Prior Weights
CLIP-Similarity ↑
MANIQA ↑
CLIPIQA ↑
CLIP-Similarity ↑
MANIQA ↑
CLIPIQA ↑
k2D=0.0 (w/o 2D Prior)
0.824
0.433
0.707
0.816
0.456
0.699
k2D=1.0
0.835
0.453
0.737
0.833
0.483
0.733
k2D=2.0
0.836
0.454
0.748
0.837
0.486
0.744
k2D=3.0 (Ours)
0.838
0.454
0.748
0.835
0.487
0.751
k2D=4.0
0.833
0.453
0.758
0.831
0.484
0.755
Table 2: Analysis of different prior weights.
Figure 16: Analysis of different frequency bounds.
Realfusion15
MorpheusObj30
Frequency Bound Rate
CLIP-Similarity ↑
MANIQA ↑
CLIPIQA ↑
CLIP-Similarity ↑
MANIQA ↑
CLIPIQA ↑
r=0.0 (Full Pass)
0.823
0.401
0.695
0.827
0.438
0.696
r=0.2
0.824
0.461
0.748
0.828
0.487
0.745
r=0.4
0.829
0.458
0.757
0.835
0.486
0.755
r=0.6 (Ours)
0.838
0.454
0.748
0.835
0.487
0.751
r=0.8
0.835
0.457
0.742
0.836
0.485
0.738
Table 3: Analysis of different frequency bounds.
Figure 17: Study of fine-tuning 3D prior on MorpheusObj30 . Fine-tuning 3D prior shows an overall trend of increasing high-frequency PSD, but decreasing reconstruction quality.
Figure 18: Artifacts caused by fine-tuning 3D prior. Improvements in high-frequency PSD are actually artifacts.
Figure 19: Study of refining from generated assets of the feed-forward method. Top: Quantitative comparison shows that our method provides overall improvements. Bottom: Qualitative comparison shows that our method further improves the texture details of the TRELLIS baseline.
Per-scene optimization methods such as 3D Gaussian Splatting provide state-of-the-art novel view synthesis quality but extrapolate poorly to under-observed areas. Methods that leverage generative priors to correct artifacts in these areas hold promise but currently suffer from two shortcomings. The first is scalability, as existing methods use image diffusion models or bidirectional video models that are limited in the number of views they can generate in a single pass (and thus require a costly iterative distillation process for consistency). The second is quality itself, as generators used in prior work tend to produce outputs that are inconsistent with existing scene content and fail entirely in completely unobserved regions. To solve these, we propose a two-stage pipeline that leverages two key insights. First, we train a powerful bidirectional generative model with a novel opacity mixing strategy that encourages consistency with existing observations while retaining the model's ability to extrapolate novel content in unseen areas. Second, we distill it into a causal auto-regressive model that generates hundreds of frames in a single pass. This model can directly produce novel views or serve as pseudo-supervision to improve the underlying 3D representation in a simple and highly efficient manner. We evaluate our method extensively and demonstrate that it can generate plausible reconstructions in scenarios where existing approaches fail completely. When measured on commonly benchmarked datasets, we outperform all existing baselines by a wide margin, exceeding prior state-of-the-art methods by 1-3 dB PSNR.
Riccardo de Lutio, Tobias Fischer, Yen-Yu Chang +7
NVIDIA, USA · NVIDIA, Switzerland · ETHZ, Switzerland +4
Single-image-to-3D generative models can now produce high-quality geometry, yet conditioning on a single view inevitably introduces ambiguity about unseen regions. Multi-view conditioning can reduce this ambiguity, but existing methods either require fixed canonical viewpoints or rely on external reconstruction modules that impose heavy training costs and limit generation quality. We observe that pretrained single-view models already possess strong 2D-to-3D grounding that can be reused for multi-view conditioning. However, a closer analysis reveals that their conditioning mechanism entangles orientation control with geometry transfer, two functions that conflict when images from different viewpoints are naively combined. Based on this analysis, we propose ROAR-3D, a lightweight method that upgrades a pretrained single-view model to accept an arbitrary number of unposed images. A token-wise view router assigns each 3D latent token to its most relevant view, implicitly establishing 2D-to-3D correspondences without explicit pose input. A dual-stream attention design preserves the pretrained primary-view behavior while routing auxiliary views through a separate path dedicated to geometric enrichment. An orientation perturbation strategy ensures the auxiliary path learns orientation-independent geometry transfer. These components introduce minimal trainable parameters and add negligible inference overhead relative to the single-view baseline. ROAR-3D achieves state-of-the-art multi-view 3D generation quality and supports test-time view scaling from 1 to 12+ views with consistent improvements.
Hanxiao Sun, Mingxin Yang, Shuhui Yang +5
The Hong Kong University of Science and Technology · Tencent Hunyuan
Text-to-3D generation based on diffusion models often suffers from the Janus problem, leading to inconsistent geometry across viewpoints. This work identifies viewpoint bias in 2D diffusion priors as the main cause and proposes Structural Energy-Guided Sampling (SEGS), a training-free and plug-and-play framework to improve multi-view consistency. SEGS constructs a structural energy in the PCA subspace of U-Net features and injects its gradient into the denoising process. It can be easily integrated into SDS/VSD pipelines without retraining. Experiments show that SEGS reduces the Janus Rate by about 10% on average and improves View-CS scores across multiple baselines, including DreamFusion, Magic3D, and LucidDreamer. This method effectively alleviates viewpoint artifacts while preserving appearance fidelity, providing a flexible solution for high-quality text-to-3D content generation.
Qing Zhang, Jinguang Tong, Jing Zhang +2
Australian National University, Canberra, 2601, ACT, Australia. · CSIRO, Canberra, 2601, ACT, Australia. · The University of Hong Kong, Hong Kong SAR.