Recent diffusion models have achieved impressive progress in single image to multi-view generation by synthesizing consistent novel views efficiently. Existing methods usually explicitly or implicitly couple geometry inference and texture synthesis, making generation vulnerable to inaccurate geometry prediction. Although several geometry-guided methods have explored decoupling, they usually treat predicted geometry as a fixed condition and do not explicitly model whether the geometry is reliable. We therefore propose Reliability-Guided Multi-View Generation (RGMV), which comprises a parameter-free Self-Consistency Geometry Gate (SCG) and a Reliability-aware Geometry Injection (RGI) module. Specifically, a geometry proposal generator produces multiple geometry predictions for the same reference image and target viewpoints under different stochastic sampling configurations. SCG measures their spatial agreement to derive a reliability map, which RGI uses to modulate the injection strength of geometry-conditioned residuals in the RGB synthesis model. Moreover, RGMV can adapt to different geometry proposal generators. As a result, RGMV preserves stronger geometric guidance in stable regions while attenuating unreliable geometry cues before they propagate into multi-view RGB generation. Extensive experiments show that reliability-aware geometry injection improves the geometric quality of multi-view generation. Code: https://github.com/WISLab-GDUT/RGMV.
Figures & tables
Figure 1: Comparison of three paradigms for multi-view image generation. (a) Direct generation entangles geometric reasoning and texture synthesis in a single generator, making geometric errors difficult to isolate. (b) Geometry-guided generation separates geometry prediction from RGB synthesis, but applies predicted geometry with spatially uniform influence. (c) RGMV conditions RGB synthesis on an anchor geometry prediction and uses per-view spatial reliability maps estimated by SCG to modulate geometry residuals through RGI.
Figure 2: Overview of RGMV. Given a reference image and target camera poses, an off-the-shelf geometry proposal generator produces K stochastic sets of multi-view normal predictions. The first prediction set serves as the anchor normal condition to produce multi-level geometry residuals. In parallel, Self-Consistency Geometry Gate (SCG) in (a) groups normal predictions by target view, measures their per-pixel angular disagreement, and converts it into per-view spatial reliability maps. Reliability-aware Geometry Injection (RGI) module in (b) resizes and affinely calibrates the corresponding reliability map at each denoising level l to obtain Rv,l , which selectively modulates the geometry residual before it is injected into the multi-view diffusion backbone.
Method
Multi-view image generation
3D reconstruction
PSNR ↑
SSIM ↑
LPIPS ↓
CD ↓
IoU ↑
Zero-1-to-3
17.38
0.8479
0.1383
0.071
0.33
SyncDreamer
18.83
0.8212
0.2146
0.066
0.39
Wonder3D++
20.05
0.8575
0.1463
0.062
0.44
MV-Adapter
20.13
0.8664
0.1268
0.036
0.54
Era3D
20.77
0.8634
0.1253
0.038
0.53
Table 1: Comparison of RGB synthesis and 3D reconstruction on GSO. All RGMV variants share the same adapted RGB synthesis model; parentheses indicate the geometry proposal generator. CD denotes Chamfer Distance, and IoU denotes Volume IoU.
Figure 3: Qualitative comparison on rendered objects with ground-truth target views. Each row shows the multi-view results of one method under the same target viewpoints.
Figure 4: Qualitative comparison on real captured images without ground-truth target views. The examples are organized by representative failure modes commonly observed in multi-view generation. Each pair compares a baseline result with its corresponding RGMV result under the same input and target viewpoints.
Statistic
Result
Range
Pooled Spearman ρ
0.299
[−1,1]
Per-sample ρ ( μ/med/σ )
0.266 / 0.289 / 0.166
[−1,1]
AUROC ( e>20∘/30∘ )
0.729 / 0.710
[0,1]
Table 2: Association between SCG reliability and ground-truth normal error on Objaverse with K=3 .
Figure 5: Sensitivity of spatially uniform geometry injection on Objaverse with a frozen RGB synthesis model. The dashed line marks αuni=0.1 , which is selected for the uniform-injection baseline. Larger global scales progressively degrade performance.
Setting
Injection
RGB Model
PSNR ↑
SSIM ↑
LPIPS ↓
Pure i2mv
–
Frozen
24.22
0.8477
0.2069
Uniform Injection
Uniform
Frozen
24.43
0.8498
0.2073
SCG Injection
SCG
Frozen
24.63
0.8545
0.1935
Uniform Injection
Uniform
Adapted
25.14
0.8569
0.1948
RGMV
SCG
Adapted
25.31
0.8621
0.1823
Table 3: Controlled ablation of predicted geometry conditioning, reliability-aware injection, and multi-view RGB synthesis model adaptation. “Uniform” denotes spatially constant geometry residual injection with αuni=0.1 , selected by the sweep in Fig. 5 .
K
proposal generation (s)
SCG computation (s)
RGB synthesis (s)
Total (s)
1
3.84±0.02
0.053±0.001
4.95±0.01
8.85±0.02
2
7.71±0.02
0.089±0.002
4.96±0.01
12.76±0.02
3
11.56±0.02
0.124±0.003
4.97±0.01
16.65±0.02
4
15.41±0.02
0.163±0.002
4.96±0.01
20.54±0.03
Table 5: Inference time on an RTX 4090 GPU for different geometry proposal sets K .
We present GeoFace, a geometry-constrained multi-view diffusion framework for consistent face generation from a single input. % While recent multi-view diffusion models achieve photorealistic synthesis at the per-view level, they lack an explicit mechanism to enforce a shared 3D structure across views, often leading to inconsistent geometry across viewpoints. To address this, GeoFace proposes a unified dual-stream framework for joint generation of multi-view RGB images and 3D face geometry, where the appearance and geometry streams interact through shared attention layers. To encourage the two streams to mutually constrain each other, we introduce a geometry-guided attention alignment loss that supervises the cross-attention between appearance and geometry tokens with 3D-consistent correspondences, enabling the appearance stream to correctly reference pose-invariant geometric cues for robust alignment across viewpoints. Geometry is represented as a canonical UV position map, derived from a FLAME mesh fitted to multi-view observations, serving as a view-invariant shared constraint across all generated views. Experiments on RenderMe-360 and NeRSemble demonstrate that GeoFace consistently outperforms existing methods in both visual quality and cross-view geometric consistency, facilitating more efficient 3D reconstruction.
We present VGGT-Diff, a geometry-routed multi-view diffusion model for sparse-view novel view synthesis. Existing novel view synthesis (NVS) methods face a fundamental trade-off: reconstruction-based approaches preserve observed geometry but struggle to synthesize unseen regions, while diffusion-based methods provide strong generative priors yet rely on implicit source-to-query correspondence. VGGT-Diff bridges these regimes by routing visual geometry latents from VGGT-Ω into a pretrained video diffusion model. Each visual token is associated with a 3D point and confidence, then transformed into query-aligned latent conditions through a confidence-aware Visual Geometry Router (VGR) that preserves front and back surface evidence. These conditions guide joint target-view denoising, while Point-Track Residual Consistency (PTRC) regularizes predicted-clean residuals along reliable 3D tracks, improving multi-view stability. We further introduce robust geometry conditioning, combining training-time regularization with inference-time guidance for improved robustness. Experiments show competitive or state-of-the-art performance across interpolation and extrapolation under different viewpoint difficulties. Our code is available at https://github.com/chenkangjie1123/VGGT-Diff.
Kangjie Chen, Xiangyu Li, Dongbin Zhang +12
XPeng Motors · The Chinese University of Hong Kong · Tsinghua University
While recent advances in generative latent spaces have driven substantial progress in single-image generation, the optimal latent space for novel view synthesis (NVS) remains largely unexplored. In particular, NVS requires geometrically consistent generation across viewpoints, but existing approaches typically operate in a view-independent VAE latent space. In this paper, we propose Geometric Latent Diffusion (GLD), a framework that repurposes the geometrically consistent feature space of geometric foundation models as the latent space for multi-view diffusion. We show that these features not only support high-fidelity RGB reconstruction but also encode strong cross-view geometric correspondences, providing a well-suited latent space for NVS. Our experiments demonstrate that GLD outperforms both VAE and RAE on 2D image quality and 3D consistency metrics, while accelerating training by more than 4.4x compared to the VAE latent space. Notably, GLD remains competitive with state-of-the-art methods that leverage large-scale text-to-image pretraining, despite training its diffusion model from scratch without such generative pretraining.