The human face is central to communication. For immersive applications, the digital presence of a person should mirror the physical reality, capturing the users idiosyncrasies and detailed facial expressions. However, current 3D head avatar methods often struggle to faithfully reproduce the identity and facial expressions, despite having multi-view data or learned priors. Learning priors that capture the diversity of human appearances, especially, for regions with highly person-specific features, like the mouth and teeth region is challenging as the underlying training data is limited. In addition, many of the avatar methods are purely relying on 3D morphable model-based expression control which strongly limits expressivity. To address these challenges, we are introducing 3DRealHead, a few-shot head avatar reconstruction method with a novel expression control signal that is extracted from a monocular video stream of the subject. Specifically, the subject can take a few pictures of themselves, recover a 3D head avatar and drive it with a consumer-level webcam. The avatar reconstruction is enabled via a novel few-shot inversion process of a 3D human head prior which is represented as a Style U-Net that emits 3D Gaussian primitives which can be rendered under novel views. The prior is learned on the NeRSemble dataset. For animating the avatar, the U-Net is conditioned on 3DMM-based facial expression signals, as well as features of the mouth region extracted from the driving video. These additional mouth features allow us to recover facial expressions that cannot be represented by the 3DMM leading to a higher expressivity and closer resemblance to the physical reality.
Figures & tables
Fig. 2: Overview of 3DRealHead. Based on few-shot input data, an RGB texture is generated which serves as identity conditioning to the Style U-Net which predicts 3D Gaussian primitives in UV-space relative to a FLAME head mesh. To control detailed facial expressions from a driving input video, we extract 3DMM-based position maps [ 35 ] and image gradient features which are projected to the UV texture space. From this control signal, the model learns how to produce dynamically changing 3D Gaussian primitives which can be rendered using 3D Gaussian splatting. The Style U-Net is trained on the NeRSemble [ 24 ] dataset, such that it can act as a prior for few-shot inversion, where only the identity encoder EId is finetuned. During the training of the prior, we employ several reconstruction losses, as well as a multi-head discriminator D for adversarial training.
Studio Data
Method
LPIPS ↓
SSIM ↑
PSNR ↑
ID ↑
t-LPIPS ↓
Ours
0.036
0.944
33.152
0.992
0.073
GAGAvatar [ 9 ]
0.037
0.935
31.595
0.992
0.077
GPAvatar [ 10 ]
0.063
0.927
32.379
0.982
0.101
LAM [ 16 ]
0.081
0.899
25.035
0.974
0.098
In-the-Wild Data – Few-Shot
TABLE I: Quantitative comparison of our method against prior-based and SOTA monocular methods on Studio and In-the-Wild datasets (8 subjects). For studio data, we perform 1-shot inversion.
Fig. 3: We show qualitative results of our method along different views for cases with 1 and 3 target images.
Fig. 4: Qualitative comparison of our approach (single target) against one-shot prior-based methods. We observe that our method produces sharper results and faithfully reproduces the driving expression.
Fig. 5: Qualitative comparison to SynShot [ 61 ] and SOTA monocular approaches on In-the-Wild INSTA dataset [ 60 ] . It is important to note that SOTA monocular methods are trained per subject on minutes of video recording while SynShot and our method are fine-tuned on only 3 frames.
Method
LPIPS ↓
SSIM ↑
PSNR ↑
ID ↑
t-LPIPS ↓
Ours
0.064
0.937
28.016
0.991
0.079
Ours-RGB
0.063
0.937
28.070
0.991
0.078
w/o Adversar. Loss
0.070
0.939
28.137
0.990
0.084
w/o Mouth Cond.
0.065
0.934
27.730
0.989
0.079
Ours-Full
0.068
0.937
27.636
0.990
0.082
RGB-Full
0.061
0.941
29.632
0.991
0.074
TABLE II: Quantitative evaluation of our ablation studies on NeRSemble data.
Fig. 6: Ablation experiments where we replace our gradient-based conditioning with different other types of conditioning, and where we show the influence of the adversarial loss.
Method
LPIPS ↓
SSIM ↑
PSNR ↑
ID ↑
t-LPIPS ↓
Ours
0.068 ± 0.004
0.932 ± 0.002
28.156 ± 0.106
0.991 ± 0.0006
0.084 ± 0.003
DINO
0.071 ± 0.005
0.930 ± 0.001
28.037 ± 0.102
0.990 ± 0.0007
0.086 ± 0.004
HOG
0.079 ± 0.003
0.928 ± 0.002
27.941 ± 0.109
0.990 ± 0.0003
0.089 ± 0.004
TABLE III: Quantitative evaluation of different mouth feature conditionings on NeRSemble data.
Fig. 7: Comparison of different mouth feature conditionings.
Fig. 8: Robustness of our method w.r.t. identity and illumination changes. In the first row, the single enrollment image used is an edited image, from which we recover the 3D avatar which can be driven by a monocular sequence. In the bottom row, one can see that the 3D avatar of a person can be driven by a monocular input in different illumination settings.
Fig. 9: Interpolation between two test identities (leftmost and rightmost columns).
Fig. 10: Cross-reenactment: driving actor (left), output (middle), identity of target (right).
Fig. 11: Inversion Experiment for the No Fine-Tuning, 1-shot and 3-shot cases.
Fig. 12: Comparison against GAN-based baselines on In-the-Wild data.
Fig. 13: Comparison against Avat3r [ 25 ] for different subjects and driving frames rendered along different camera views.
Fig. 14: Novel View Synthesis on In-the-Wild data.
Fig. 15: Qualitative results of using a different UV map (shown on left side of each corresponding experiment) with different number and distribution of Gaussians in the mouth region.
Fig. 16: Qualitative results of using 3DMM-only based control versus using additional mouth gradients (3DMM-G) versus our method (expression + mouth gradients).
Fig. 17: Qualitative comparison of our method against GaussianAvatars [ 33 ] .
Fig. 18: Plots of performance versus number of frames during inversion for LPIPS, SSIM and PSNR metrics.
Method
LPIPS ↓
SSIM ↑
PSNR ↑
ID ↑
t-LPIPS ↓
Ours
0.068 ± 0.004
0.932 ± 0.002
28.156 ± 0.106
0.991 ± 0.001
0.084 ± 0.003
New UV (70 % less)
0.077
0.928
27.83
0.987
0.089
TABLE IV: Quantitative evaluation of the distribution of mouth Gaussians on NeRSemble data.
Method
LPIPS ↓
SSIM ↑
PSNR ↑
ID ↑
t-LPIPS ↓
Ours
0.068 ± 0.004
0.932 ± 0.002
28.156 ± 0.106
0.991 ± 0.001
0.084 ± 0.003
3DMM-only
0.073
0.925
27.102
0.990
0.089
3DMM-Grad.
0.068
0.932
27.551
0.991
0.087
TABLE V: Quantitative evaluation of our method versus 3DMM-only control versus having both 3DMM control with additional Gradient-based mouth maps on NeRSemble data.
Training Set Size
LPIPS ↓
SSIM ↑
PSNR ↑
ID ↑
t-LPIPS ↓
100 % (Ours)
0.049
0.939
30.035
0.990
0.073
50 %
0.049
0.936
30.032
0.989
0.075
25 %
0.055
0.930
29.780
0.982
0.078
5 %
0.072
0.914
27.893
0.967
0.084
TABLE VI: Quantitative evaluation of the size of training set on 8 In-the-Wild subjects.
No. of Frames
LPIPS ↓
SSIM ↑
PSNR ↑
ID ↑
t-LPIPS ↓
5 Frames
0.045
0.941
30.163
0.992
0.071
3 Frames
0.049
0.939
30.035
0.990
0.073
2 Frames
0.049
0.937
30.028
0.990
0.075
1 Frame
0.050
0.935
29.935
0.987
0.076
TABLE VII: Quantitative evaluation of the number of frames during enrollment on 8 In-the-Wild subjects.
Subject Category
No. of Subjects
LPIPS ↓
SSIM ↑
PSNR ↑
ID ↑
t-LPIPS ↓
Similar to training set
3
0.050
0.930
30.20
0.989
0.071
Different ethnicity
2
0.065
0.924
29.21
0.984
0.077
Extreme views
3
0.054
0.929
29.22
0.989
0.075
TABLE VIII: Category-wise quantitative evaluation of our method on 8 In-the-Wild subjects.
Fig. 19: Qualitative comparison of different training set sizes.
Fig. 20: Qualitative comparison between different number of frames used for inversion.