We present a feed-forward framework for static Gaussian full-head reconstruction and novel-view rendering from a single unposed image. Unlike previous work that relies on time-consuming GAN inversion and test-time optimization, our framework can reconstruct the Gaussian full-head model given a single unposed image in a single forward pass. This enables fast reconstruction and rendering during inference. To mitigate the lack of large-scale 3D head assets, we propose a large-scale synthetic dataset from trained 3D GANs and train our framework using only synthetic data. For efficient high-fidelity generation, we introduce a coarse-to-fine Gaussian head generation pipeline, where sparse points from the FLAME model interact with the image features by transformer blocks for feature extraction and coarse shape reconstruction, which are then densified for high-fidelity reconstruction. To fully leverage the prior knowledge residing in pretrained 3D GANs for effective reconstruction, we propose a dual-branch framework that effectively aggregates the structured spherical triplane feature and unstructured point-based features for more effective Gaussian head reconstruction. Experimental results show the effectiveness of our framework compared to existing work. The dataset is available at https://github.com/PanoLAM/PanoLAM_Dataset.
Figures & tables
Fig. 1: PanoLAM creates high-fidelity Gaussian full-heads given an unposed single image in a single forward pass within a second, enabling efficient creation of high-quality 3D avatar assets.
Fig. 2: Overall Framework. Given an unposed head image as input, PanoLAM involves two branches to achieve single-pass 3D Gaussian head reconstruction: a point-based transformer for coarse-to-fine point shape reconstruction and point feature extraction, and a spherical triplane transformer to distill prior knowledge from a 3D GAN. Features from the two branches are concatenated for high-fidelity Gaussian head regression.
Fig. 3: The proposed feature aggregation mechanism samples and aggregates multi-layer features from the spherical triplane for each point.
Dataset
Sub.
Range
Hair
Dataset
Sub.
Range
Hair
BU-3DFE
100
front
✗
HeadSpace
1519
270∘
✗
BU-4DFE
101
front
✓
FaceScape
938
360∘
✗
BJUT-3D
500
front
✗
AVA-256
256
360∘
✓
Bosphorus
105
front
✗
FaceVerse ∗
128
360∘
✓
FaceWarhouse
150
front
✓
RenderMe360 ∗
500
360∘
✓
4DFAB
180
front
✗
SynHead100
100
360∘
✓
TABLE I: Comparisons of existing 3D head datasets. ∗ denotes 3D head with rough captured 3D hair shape.
Fig. 4: Visualization of reconstruction and novel view synthesis of different methods. The left part shows comparisons of different methods on synthesis data, and the right part shows visualization on in-the-wild real-world images, showing the out-of-domain generalizability of PanoLAM trained on our curated datasets.
Fig. 5: Comparison of reconstruction and novel view synthesis of different methods on the test set. Our framework can reconstruct 360-degree heads given a single input view, with fine-grained details and fewer artifacts compared to baselines.
Fig. 6: Qualitative comparisons with FaceLift [ 39 ] .
Method
Real-world
Synthesis
PSNR ↑
LPIPS ↓
SSIM ↑
PSNR ↑
LPIPS ↓
SSIM ↑
PanoHead-PTI
16.260
0.165
0.704
24.37
0.141
0.806
SphereHead-PTI
16.983
0.142
0.698
24.65
0.127
0.811
LGM
11.472
0.444
0.572
21.33
0.350
0.765
LGM-Finetuned
19.313
0.182
0.699
25.67
0.135
0.806
LAM
15.627
0.245
0.647
18.72
0.157
0.713
TABLE II: Quantitative evaluation on synthesis and real-world testsets.
Table 9
Fig. 7: Out-of-domain qualitative results of PanoLAM under challenging input conditions. Each triplet shows the input and two synthesized novel views.
Fig. 8: Effect of coarse-to-fine generation. Without the coarse-to-fine strategy, the dense Gaussians are distributed on multiple layers, leading to degradation of rendering quality. Zoom in for details.
Fig. 9: Comparison of our network trained on different numbers of subjects. The model trained on a small number of subjects cannot generalize well to unseen identities.
Fig. 10: Comparisons on different spherical triplane feature query and aggregation strategies. For each point, sampling only one sample of features from the spherical triplane cannot fully fetch all valuable features, resulting in a collapsed rendered image. Aggregating 32 points with Gaussian distribution weights gets better results, but still blurry. Our MLPs learn to aggregate features as the original NeRF and get quite better results.
Fig. 11: Comparison of different numbers of Gaussian points. A small number of Gaussian points cannot model texture details like hair strands and teeth. More high-fidelity Gaussian avatars are reconstructed with an increasing number of Gaussian points. Zoom in to see more details.
Method
PSNR ↑
LPIPS ↓
SSIM ↑
Infer.(s) ↓
Memory(G) ↓
10K
22.22
0.203
0.738
0.074
5.65
30K
26.42
0.134
0.835
0.082
7.22
100K
27.45
0.093
0.882
0.110
12.70
TABLE VI: Quantitative comparison of different numbers of Gaussian points on the test set.
Building one-shot 3D animatable head avatars is an important yet challenging problem. Existing methods generally collapse under large camera pose variations, compromising the realism of 3D avatars. In this work, we propose a new framework to tackle the novel setting of one-shot 3D full-head animatable avatar reconstruction in a single forward pass via inpainted UV-space Gaussian modeling, enabling 360∘ rendering views and real-time animation. To facilitate efficient animation control, we model 3D head avatars with Gaussian primitives embedded on the surface of a parametric face model within the UV space, and project the input image features to the UV space, resulting in incomplete local UV feature maps. To inpaint the missing regions, we obtain knowledge of full-head geometry and textures from rich 3D full-head priors within a pretrained 3D generative adversarial network (GAN) for global full-head feature extraction and multi-view supervision. Specifically, to enhance the fidelity of 3D reconstruction during inpainting, we take advantage of the symmetric nature of the UV space and human faces to fuse incomplete yet detailed local UV feature maps with the extracted global full-head textures, resulting in inpainted UV Gaussian attribute maps for avatar modeling. Extensive experiments demonstrate that our method is the first to achieve high-quality 3D full-head animatable avatar modeling, significantly improving side and back views while outperforming state-of-the-art animation approaches, thereby improving the realism of 3D animatable avatars.
Shuling Zhao, Dan Xu
The Hong Kong University of Science and Technology · Zeekr Automobile R&D Co., Ltd.
We propose HeadsUp, a scalable feed-forward method for reconstructing high-quality 3D Gaussian heads from large-scale multi-camera setups. Our method employs an efficient encoder-decoder architecture that compresses input views into a compact latent representation. This latent representation is then decoded into a set of UV-parameterized 3D Gaussians anchored to a neutral head template. This UV representation decouples the number of 3D Gaussians from the number and resolution of input images, enabling training with many high-resolution input views. We train and evaluate our model on an internal dataset with more than 10,000 subjects, which is an order of magnitude larger than existing multi-view human head datasets. HeadsUp achieves state-of-the-art reconstruction quality and generalizes to novel identities without test-time optimization. We extensively analyze the scaling behavior of our model across identities, views, and model capacity, revealing practical insights for quality-compute trade-offs. Finally, we highlight the strength of our latent space by showcasing two downstream applications: generating novel 3D identities and animating the 3D heads with expression blendshapes.
We propose FA-LAM, a Focus-Aware Large Avatar Model for one-shot animatable Gaussian head creation, while simultaneously enabling static 3D and dynamic 4D full-head recovery. The core of our method lies in a thorough analysis of the attention mechanisms and the entangled reconstruction and animation training pipeline adopted by prior state-of-the-art approaches. Our analysis identifies two main factors that compromise the quality of 3D full-head generation: (1) incorrect and noisy attention activations, and (2) conflicts between the tasks of reconstruction and animation. To address the first issue, we introduce a symmetric and semantic attention regularization strategy that leverages the inherent semantics and structural symmetry of human heads. To disentangle the objectives of reconstruction and animation, we develop a novel dual-phase training pipeline that separates the model's capabilities for large-view hallucination and animation into distinct modules. Moreover, we enhance our model to support multi-view and streaming 4D reconstruction in an efficient and memory-friendly manner through a core autoregressive modification with tailored visibility-aware token fusion. Collectively, these innovations enable FA-LAM to reconstruct animatable Gaussian full heads with superior quality, particularly in fine facial regions and large viewing angles.
Yingdong Hu, Yisheng He, Yiming Jiang +3
HKUST Hong Kong SAR, China · Tongyi Lab, Alibaba Group Hangzhou, China · Lingnan University Hong Kong SAR, China