Organizations: School of Computing, State University of New York at Binghamton · Department of Computer Science, Missouri University of Science and Technology
We propose a novel one-stage method, NVB-Face, for generating consistent Novel-View images directly from a single Blind Face image. Existing approaches to novel-view synthesis for objects or faces typically require a high-resolution RGB image as input. When dealing with degraded images, the conventional pipeline follows a two-stage process: first restoring the image to high resolution, then synthesizing novel views from the restored result. However, this approach is highly dependent on the quality of the restored image, often leading to inaccuracies and inconsistencies in the final output. To address this limitation, we extract single-view features directly from the blind face image and introduce a feature manipulator that transforms these features into 3D-aware, multi-view latent representations. Leveraging the powerful generative capacity of a diffusion model, our framework synthesizes high-quality, consistent novel-view face images. Experimental results show that our method significantly outperforms traditional two-stage approaches in both consistency and fidelity.
Figures & tables
Figure 1: We compare our method with typical two-stage pipelines, such as CodeFormer ( Zhou et al. 2022 ) + PanoHead-PTI ( An et al. 2023 ) , which first restore the degraded image and then synthesize novel views. It is evident that when the restoration stage fails to recover accurate details, these errors are further amplified during the novel view synthesis, leading to results that deviate significantly from the original identity and appearance. In contrast, our method generates novel views in a single stage directly from the low-quality input. This end-to-end design suppresses error accumulation, resulting in more reliable and faithful novel-view images.
Figure 2: An overview of the proposed NVB-Face architecture. (a) Our first training step focuses solely image restoration. (b) In the second training step, we update only the parameters of the newly introduced modules (highlighted in dark green), keeping the rest of the network frozen. After training, this two-step process forms our complete inference pipeline.
Figure 3: Qualitative comparisons on NeRSemble ( Kirschstein et al. 2023 ) dataset. As shown in our results, this end-to-end strategy achieves superior perceptual quality and preserves identity and expression information more effectively than two-stage methods, minimizing the loss of critical facial attributes.
Figure 4: Qualitative comparisons on LFW-Test ( Huang et al. 2008 ) dataset. Our method produces consistently stable results across varying levels of input degradation. Compared to other approaches, our generated images preserve the most information from the original input and exhibit higher visual realism, even under severe degradation.
Table 1: (a) Quantitative comparisons with other methods on NeRSemble dataset (novel-view synthesis) / CelebA-Test (single-view reconstruction). (b) Ablation study on NeRSemble dataset to assess the effectiveness of the feature loss in our framework. Specifically, we compare the quantitative performance of our model with and without feature loss.
Figure 5: Qualitative ablation study on LFW-Test dataset to compare our method with and without feature loss.
Figure 6: Feature correction of our training step 2.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Architecture of our Time-aware Image Encoder
Figure 8: Qualitative comparison among CodeFormer-based two-stage method, DiffBIR-based method and ours on LFW-Test dataset
PSNR ↑
SSIM ↑
LPIPS ↓
DISTS ↓
FID ↓
MUSIQ ↑
NIQE ↓
C-IQA ↑
M-IQA ↑
CodeFormer
20.98
0.6174
0.4750
0.2636
61.44
66.06
5.6992
0.5608
0.4775
DiffBIR
22.00
0.6000
0.4631
0.2340
32.54
75.61
6.2855
0.7895
0.6256
Ours
20.05
0.5574
0.4969
0.2133
18.43
74.71
5.2747
0.7431
0.6536
Appendix
Table 2: Comparison with other classical blind face restoration (BFR) methods. Bold indicates the best result, and underlined indicates the second-best.
SSIM ↑
LPIPS ↓
DISTS ↓
FID ↓
ID ↑
POSE ↓
PanoHead-PTI
0.63( 0.73 )
0.54( 0.48 )
0.27( 0.26 )
85.31( 83.19 )
0.30( 0.32 )
0.0484( 0.0446 )
GOAE
0.62( 0.73 )
0.50( 0.45 )
0.27( 0.25 )
108.65( 90.00 )
0.27 ( 0.27 )
0.0392 (0.0425)
TriplaneNet
0.70( 0.75 )
0.52( 0.51 )
0.29 (0.32)
136.21( 111.84 )
0.29( 0.30 )
0.0484( 0.0456 )
Ours
0.78
0.17
0.10
5.67
0.77
0.0084
Appendix
Table 3: Quantitative comparisons with DiffBIR-based two-stage methods on NeRSemble dataset. Values in parentheses indicate results based on the CodeFormer restoration. Underlined values represent the better result between the CodeFormer-based and DiffBIR-based variants.
Figure 9: Limitation of our model
Figure 10: More novel-view results from easy cases
Figure 11: More novel-view results from hard cases
Photorealistic novel view synthesis of people remains challenging at high spatial resolutions and across multiple target cameras, where preserving identity, fine appearance details, and geometric coherence is critical. We build on the next-scale autoregressive paradigm and adapt it for human-centric view synthesis by enabling higher image resolutions, multi-view outputs and stronger cross-view consistency in a single forward pass. We train on a synthetic dataset of human faces spanning diverse identities and apparel. Contrary to diffusion models, this paradigm does not need 2D pre-training and, thanks to its next-scale architecture, it benefits from lower-resolution, general-purpose pre-trainings, with the full-sized purpose-specific images being used only in the last training stages. This enables our architecture to converge with a smaller amount of purpose-specific training data, allowing us to use a smaller but more realistic training dataset. The resulting model produces sharp and realistic views, with the option to synthesize multiple novel viewpoints simultaneously for improved agreement across views. Empirically, we observe gains in perceptual fidelity and cross-view coherence on human subjects, demonstrating that next-scale autoregression is an effective backbone for scalable, multi-output human view synthesis. We also couple our pipeline with an existing transformer-based model for pixel-aligned 3D gaussian lifting from multi-view facial inputs, resulting in accurate and photorealistic 3D models of human faces.
Federico Stella, Fei Jiang, Zhongshi Jiang +4
École polytechnique fédérale de Lausanne, Switzerland · Meta
We study the challenging problem of novel view video synthesis from single images or monocular videos. Existing methods, which operate under the assumption that pre-trained video models lack native novel view synthesis capability and enforce view alignment via camera conditioning, task-specific fine-tuning, or stepwise hard denoising guidance, often suffer from artifacts and compromised global scene consistency. In this paper, we introduce NeoMap, a novel training-free framework designed to locate high-fidelity, view-consistent novel view solutions from general pre-trained video models. The key to our approach is the core insight that promising novel view solutions are inherently encoded within the natural video data manifold learned by pre-trained models, and the core challenge is simply to locate this optimal solution. We solve this via our core mechanism: convergent manifold alternating projection iterations that optimize the initial noise. Extensive experiments demonstrate that NeoMap significantly outperforms all existing methods across 3 standard novel view synthesis benchmarks, including the challenging Tanks-and-Temples, LLFF and DAVIS datasets, achieving state-of-the-art generation fidelity and top-tier view consistency.
Jinxi Li, Tianyi Zhang, Yafei Yang +4
Shenzhen Research Institute, The Hong Kong Polytechnic University · vLAR Group, The Hong Kong Polytechnic University
Novel View Synthesis (NVS) aims to generate unseen views of a 3D object given a limited number of known views. Existing methods often struggle to synthesize plausible views for unobserved regions, particularly under single-view input, and still face challenges in maintaining geometry- and appearance-consistency. To address these issues, we propose OrbitNVS, which reformulates NVS as an orbit video generation task. Through tailored model design and training strategies, we adapt a pre-trained video generation model to the NVS task, leveraging its rich visual priors to achieve high-quality view synthesis. Specifically, we incorporate camera adapters into the video model to enable accurate camera control. To enhance two key properties of 3D objects, geometry and appearance, we design a normal map generation branch and use normal map features to guide the synthesis of the target views via attention mechanism, thereby improving geometric consistency. Moreover, we apply a pixel-space supervision to alleviate blurry appearance caused by spatial compression in the latent space. Extensive experiments show that OrbitNVS significantly outperforms previous methods on the GSO and OmniObject3D benchmarks, especially in the challenging single-view setting (\eg, +2.9 dB and +2.4 dB PSNR).
Jinglin Liang, Zijian Zhou, Rui Huang +2
South China University of Technology · Agentic Labs, XGTech · King’s College London +1