Photorealistic novel view synthesis of people remains challenging at high spatial resolutions and across multiple target cameras, where preserving identity, fine appearance details, and geometric coherence is critical. We build on the next-scale autoregressive paradigm and adapt it for human-centric view synthesis by enabling higher image resolutions, multi-view outputs and stronger cross-view consistency in a single forward pass. We train on a synthetic dataset of human faces spanning diverse identities and apparel. Contrary to diffusion models, this paradigm does not need 2D pre-training and, thanks to its next-scale architecture, it benefits from lower-resolution, general-purpose pre-trainings, with the full-sized purpose-specific images being used only in the last training stages. This enables our architecture to converge with a smaller amount of purpose-specific training data, allowing us to use a smaller but more realistic training dataset. The resulting model produces sharp and realistic views, with the option to synthesize multiple novel viewpoints simultaneously for improved agreement across views. Empirically, we observe gains in perceptual fidelity and cross-view coherence on human subjects, demonstrating that next-scale autoregression is an effective backbone for scalable, multi-output human view synthesis. We also couple our pipeline with an existing transformer-based model for pixel-aligned 3D gaussian lifting from multi-view facial inputs, resulting in accurate and photorealistic 3D models of human faces.
Figures & tables
Figure 1: Face reconstruction from single view: our method achieves more realistic results than competing methods. † Variant of our method trained to handle multiple loosely-posed inputs, it achieves even better results thanks to the additional inputs.
Figure 2: Our architecture : input images are encoded into multi-scale features by the VQ-VAE encoder and used as inputs for the transformer. They are also embedded into a global conditioning along with the camera-pose RT matrix, used both in the AdaLN layers and the SOS tokens. The transformer outputs one scale at a time for every output view simultaneously, up to the final scale. The output features are decoded by the VQ-VAE decoder to produce the final RGB images, which can be lifted to 3D gaussians by an off-the-shelf GS-LRM model. During training, previous scales are teacher-forced, while during inference the model is auto-regressive. The architecture is the same for all our models, with the only difference being the number of input and output views.
Figure 3: Multi-scale VQ-VAE outputs for different resolutions and scale configurations. Notice that the small details appear at the last 2-3 scales, regardless of the resolution and the number of scales.
PSNR ↑
SSIM ↑
LPIPS ↓
DS ↓
ArcFace ↑
SI (PSGS) - I stage [ 41 ]
20.15
0.8640
0.2612
0.02606
0.6761
SI (PSGS) - II stage [ 41 ]
20.16
0.8614
0.2351
0.02297
0.7403
FaceLift-NVS [ 24 ]
17.76
0.7942
0.2785
0.04531
0.6554
FaceLift-NVS (PSGS) [ 24 ]
16.75
0.7900
0.3206
0.02262
0.6680
Ours
22.29
0.8806
0.1905
0.02191
0.6944
Ours MO
22.40
0.8752
0.1927
0.01729
0.7069
Table 1: Quantitative evaluation on PSGS [ 11 ] validation set for NVS models (6 canonical views). Best in bold, second best in italics.
Figure 4: Qualitative results on PSGS [ 11 ] validation set on the 6 canonical views.
Figure 5: Qualitative results on PSGS [ 11 ] validation set on the 32 turntable views.
PSNR ↑
SSIM ↑
LPIPS ↓
DS ↓
ArcFace ↑
SI (PSGS) - I stage [ 41 ]
18.46
0.8548
0.2755
0.03257
0.7108
SI (PSGS) - II stage [ 41 ]
18.32
0.8488
0.2549
0.03184
0.7596
FaceLift [ 24 ]
13.28
0.7791
0.3103
0.1008
0.7121
FaceLift (PSGS) [ 24 ]
15.41
0.8377
0.2668
0.07339
0.7357
Ours + [ 11 ]
20.78
0.8805
0.2248
0.02440
0.7520
Ours MO + [ 11 ]
20.82
0.8772
0.2249
0.02319
0.7608
Table 2: Quantitative evaluation on PSGS [ 11 ] validation set of the full image-to-3D pipelines (32 turntable views). Best in bold, second best in italics.
PSNR ↑
SSIM ↑
LPIPS ↓
DS ↓
ArcFace ↑
SI (PSGS) [ 41 ]
13.06
0.7736
0.2911
0.06242
0.6689
FaceLift [ 24 ]
14.49
0.8038
0.2546
0.03482
0.7699
FaceLift (PSGS) [ 24 ]
14.51
0.8146
0.2489
0.03584
0.5837
Ours + [ 11 ]
15.34
0.8201
0.2357
0.03616
0.6275
Ours MO + [ 11 ]
15.24
0.8197
0.2448
0.03385
0.6405
Ours MI-MO + [ 11 ]
15.67
0.8209
0.2350
0.04187
0.6251
Table 3: Quantitative evaluation on AVA-256, 10 subjects, 11 views, as defined in [ 24 ] . Best in bold, second best in italics.
Figure 6: Qualitative results on AVA-256 [ 25 ] for subject 20210901–0833–LAS440
Depth
Resolution
Checkpoint
1st stage
2nd stage
VQ-VAE
PSNR ↑
ArcFace ↑
d12
256
AV [ 48 ]
default
12.02
17.01
d12
256
AV [ 48 ]
✓
default
19.77
39.20
d12
256
Scratch
✓
default
19.68
34.87
d36
512
VAR [ 43 ]
✓
default
16.72
38.17
d12
512 K=12
AV [ 48 ]
✓
default
15.70
33.71
d12
512
AV [ 48 ]
✓
default
21.43
57.75
Table 4: Ablation study of the NVS-stage of our pipeline on PSGS [ 11 ] validation set (6 canonical views) for single-input single-output models. K=10 unless otherwise specified, and using around half the training steps of the final model.
Figure A.1 : Qualitative results of our method on the PSGS [ 11 ] dataset for human body reconstruction. We show the canonical views produced by our method, compared to the GT and the input views.
Figure A.3 : Example object views from the SS3D dataset.
Figure A.4 : Multi-scale VQ-VAE finetuning results. We show the reconstructions from the base and finetuned VQ-VAEs at both 256 and 512 resolutions, compared to the GT, showing small but noticeable improvements in reconstruction quality after finetuning.
Figure A.5 : Qualitative results on PSGS [ 11 ] validation set on the 6 canonical views for the NVS part of each model.
Figure A.6 : Qualitative results on AVA-256 [ 25 ] for subject 20220310–1128–ZSC414