GSHeadRelight: Fast Relightability for 3D Gaussian Head Synthesis
Authors: Henglei Lv, Bailin Deng, Jianzhu Guo, Xiaoqiang Liu, Pengfei Wan, Di Zhang, Lin Gao
Organizations: Institute of Computing Technology, CAS and University of Chinese Academy of Sciences, China · Cardiff University, United Kingdom · Kuaishou Technology, China
Relighting and novel view synthesis of human portraits are essential in applications such as portrait photography, virtual reality (VR), and augmented reality (AR). Despite recent progress, 3D-aware portrait relighting remains challenging due to the demands for photorealistic rendering, real-time performance, and generalization to unseen subjects. Existing works either rely on supervision from limited and expensive light stage captured data or produce suboptimal results. Moreover, many works are based on generative NeRFs, which suffer from poor 3D consistency and low real-time performance. We resort to recent progress on generative 3D Gaussians and design a lighting model based on a unified neural radiance transfer representation, which responds linearly to incident light. Using only in-the-wild images, our method achieves state-of-the-art relighting results and a significantly faster rendering speed (×12) compared to previous 3D-aware portrait relighting research. Project page: http://www.geometrylearning.com/GSHeadRelight/.
Figures & tables
Figure 1. Our method incorporates an efficient yet effective lighting model into generative 3D Gaussian and can synthesize high-quality relightable 3D Gaussian heads that allow for novel view synthesis and relighting given any HDRI environment maps. Our method does not require expensive light stage data and achieves real-time rendering at 240 FPS, surpassing the previous 3D-aware portrait relighting research by at least 12 times. This is the teaser figure for the article.
Figure 2. We incorporate a lighting model based on unified radiance transfer into a generative 3D Gaussian framework. The generator G takes in Gaussian noise and camera pose as condition and generates for each Gaussian an embedding x , which is then linearly transformed into the albedo ρ and geometry attributes {g} including position, scale, rotation and opacity. A light-weight decoder conditioned on view direction transforms x into radiance transfer coefficients t , which are used to compute the unified light transport with the light condition. Color is obtained by multiplying the light transport component with the albedo. The image is then rendered by standard splatting and sent to the discriminator D, which takes both camera pose and light condition as input. Mapping modules are omitted for simplicity. Gaussian noise and camera pose are represented as icons, and light condition as a diffuse sphere.
Method
SR
FID ↓
KID ↓
Rendering Speed ↑
EG3D
✓
4.30
0.132
42 fps
GSGAN
×
5.60
0.161
245 fps
Volux-GAN
✓
59.79
4.124
21 fps
ShadeGAN
✓
9.92
0.453
0.5 fps
LumiGAN
✓
5.28
0.251
-
NeRFFaceLighting
✓
4.16
0.147
2.8 fps
Table 1. Comparison of generative quality and rendering speed. KID is reported by 100 × . SR denotes super resolution. For relighting methods, rendering speed is measured under varying illumination and view direction. All methods render at 512 resolution except for Volux-GAN at 256 on a single RTX A6000 GPU. We bold the best score, underline the second, and double-underline the third.
Method
Lighting Error ↓
Lighting Stability ↓
StyleFlow
0.7523
0.1530
DisCoFaceGAN
0.5860
0.1335
GAN-Control
0.6647
0.1485
ShadeGAN
1.0714
0.2149
NeRFFaceLighting
0.6377
0.1455
Ours
0.6213
0.1537
Table 2. Comparison of lighting quality.
Method
SR
PSNR ↑
SSIM ↑
Id. Sim. (view) ↑
EG3D
✓
38.13
0.9612
0.7501
GSGAN
×
35.85
0.9395
0.7416
ShadeGAN
✓
24.23
0.7316
0.5238
Volux-GAN
✓
31.64
0.8820
0.6882
NeRFFaceLighting
✓
37.98
0.9561
0.7303
Ours
×
35.57
0.9521
0.7538
Table 3. Comparison of spatial consistency. SR denotes super resolution.
Figure 3. Qualitative Comparison. We use the same environment maps as LumiGAN ( Deng et al., 2024 ) for comparison, since it is not publicly accessible.
Figure 4. White lighting samples of our method. Lighting conditions are provided at the lower left corner.
Figure 5. RGB lighting samples of our method.
Figure 6. Environment map relighting samples of our method. The first column presents albedo images and the rest columns present renderings under different environment maps and view directions. Environment maps are rotated for different rows.
Figure 7. Results of real portrait relighting and novel view synthesis. LC and VD stand for light condition and view direction, respectively. Our method shows better photorealism and identity consistency compared to NeRFFaceLighting ( Jiang et al., 2023 ) .
Figure 8. Visual comparison between training under RGB light condition (left) and white light condition (right), both trained for 3M images. We present albedo and relighting results under seen lighting and unseen lighting. Lighting conditions are provided at the lower left corner.
Figure 9. Visual demonstration of diffuse and specular decomposition. Color inconsistencies are marked by red dashed boxes.
Figure 10. Visual ablation of view condition. Albedo and left light renderings with and without view condition are presented. Zoom in for a better view.
Method
Lighting Error ↓
Lighting Stability ↓
w/o R.T.
1.2296
0.4288
w/o V.C.
0.7019
0.1723
Ours
0.6520
0.1582
Table 4. Quantitative ablation of unified radiance transfer. R.T. denotes radiance transfer and V.C. denotes view condition.
Figure 11. Failure cases. Albedo with highlight residuals and renderings under illumination from three directions are presented. Zoom in for a better view.