Improving Generative Adversarial Network Generalization for Facial Expression Synthesis
Authors: Arbish Akram, Nazar Khan, Arif Mahmood
Organizations: Department of Computer Science, University of the Punjab, Lahore, Pakistan · Department of Computer Science, Information Technology University, Lahore, Pakistan
Facial expression synthesis aims to generate realistic facial expressions while preserving identity. Existing conditional generative adversarial networks (GANs) achieve excellent image-to-image translation results, but their performance often degrades when test images differ from the training dataset. We present Regression GAN (RegGAN), a model that learns an intermediate representation to improve generalization beyond the training distribution. RegGAN consists of two components: a regression layer with local receptive fields that learns expression details by minimizing the reconstruction error through a ridge regression loss, and a refinement network trained adversarially to enhance the realism of generated images. We train RegGAN on the CFEE dataset and evaluate its generalization performance both on CFEE and challenging out-of-distribution images, including celebrity photos, portraits, statues, and avatar renderings. For evaluation, we employ four widely used metrics: Expression Classification Score (ECS) for expression quality, Face Similarity Score (FSS) for identity preservation, QualiCLIP for perceptual realism, and Fréchet Inception Distance (FID) for assessing both expression quality and realism. RegGAN outperforms six state-of-the-art models in ECS, FID, and QualiCLIP, while ranking second in FSS. Human evaluations indicate that RegGAN surpasses the best competing model by 25% in expression quality, 26% in identity preservation, and 30% in realism.
Figures & tables
Fig. 1: Given a neutral input image, the proposed RegGAN synthesizes photorealistic facial expressions on out-of-distribution testing images. Despite being trained only on real human faces from the CFEE dataset [ 8 ] , the proposed method introduces realistic expressions, preserves identity and retains facial details of input images.
Fig. 2: Left: Our method consists of two components: an expression layer and a refinement network. The expression layer takes an input image and generates a new image of the same person with a different facial expression. The refinement network then enhances the quality of the synthesized image, making it more realistic and sharper. Right: Architecture of EAB.
Fig. 3: Illustration of the input, intermediate, and final outputs synthesized by our proposed RegGAN. The expression layer GE generates an intermediate image that captures the target expression, while the refinement network GR transforms this intermediate result into a convincing photorealistic output.
Fig. 4: Results of RegGAN for facial expression synthesis on out-of-distribution images, including an impasto face (row 1), a celebrity face (row 2), a fantasy image (row 3), a portrait (row 4), as well as an avatar (row 5). Impasto and fantasy images were generated using stable diffusion [ 49 ] . The proposed method introduces convincing expressions while preserving the identity and facial details of the input image.
Fig. 5: Comparison of the proposed method, RegGAN, with six state-of-the-art facial expression synthesis models - StarGAN, GANimation, MR, DAI2I, SARGAN as well as US-GAN on out-of-distribution facial images. GANimation synthesizes realistic expression but introduces noticeable artifacts. StarGAN and DAI2I fail to preserve the input image’s color distribution in their output. SARGAN and US-GAN preserve facial details but are unable to induce happy expressions. MR generates realistic happy expressions, but the results are often blurry. In contrast, RegGAN produces sharper and more realistic expressions on out-of-distribution images. Moreover, RegGAN demonstrates consistent performance across diverse image types, including human portraits, status, and avatars collected from the Internet, despite being trained exclusively on CFEE dataset captured in a controlled environment with consistent lighting and background.
Fig. 6: Zoomed-in versions of smiles from our proposed method as well as other methods. Our method generates more realistic smiles and enhances the clarity of individual teeth, improving their differentiation and visibility compared to all other methods.
Method
ECS ↑
FSS ↑
RS ↑
FID ↓
StarGAN [ 11 ]
0.71
0.68
0.55
111.9
GANimation [ 19 ]
0.45
0.64
0.80
94.09
MR [ 18 ]
0.72
0.66
0.19
113.9
DAI2I [ 21 ]
0.54
0.64
0.58
140.2
SARGAN [ 23 ]
0.61
0.71
0.73
98.86
US-GAN [ 22 ]
0.66
0.71
0.69
98.92
TABLE I: RegGAN outperforms other methods in ECS and RS by synthesizing convincing, photo-realistic facial expressions. The lowest FID indicates that the generated images closely match the distribution of real images. In FSS, it attains the second-best value, as SARGAN and US-GAN use ultimate skip connections that directly copy input details to the output.
Fig. 7: RegGAN outperforms all existing baselines in synthesizing convincing and realistic expression, while preserving facial details.
Fig. 8: StyleGAN-based methods synthesize smiles but tend to lose fine facial details and introduce unwanted artifacts, such as glasses (third row, second column). The proposed approach better preserves identity details while synthesizing realistic facial expressions.
Method
ECS ↑
FSS ↑
RS ↑
FID ↓
No LABs
0.87
0.67
0.90
80.8
No AUs
0.86
0.68
0.88
89.9
RegGAN
0.85
0.68
0.90
74.3
TABLE II: Quantitative results of ablation studies. The inclusion of LABs and AUs helps achieve a better balance among expression, facial details, and realism.
Method
ECS ↑
FSS ↑
RS ↑
FID ↓
StyleRES [ 31 ]
0.91
0.58
0.28
83.5
SFE [ 32 ]
0.70
0.64
0.39
67.4
RegGAN
0.93
0.64
0.85
61.8
TABLE III: Quantitative results of RegGAN and StyleGAN-based methods. RegGAN achieves higher scores in expression quality, identity preservation, and realism.
Fig. 9: Results of the proposed method on diverse facial images: Row 1: faces with non-uniform, multi-colored illumination. Row 2: faces in the style of fantasy artist Peter Mohrbacher. Row 3: faces in the style of artist Tomer Hanuka. Row 4: impasto in the style of Gregory Manchess. Row 5: faces with makeup and occlusion. All input images were generated using Stable Diffusion [ 58 ] . Results demonstrate strong generalization of the proposed method, that synthesizes realistic facial expressions while preserving the artistic details and styles of the input images.
Fig. 10: RegGAN results on Gaussian-smoothed and motion-blurred images. While the expression layer successfully induces desired expressions on sharp as well as blurred inputs, the sharpening effect of our refinement network causes inputs and outputs to have different levels of blurriness. This can be beneficial in some contexts, but not so desirable in others.
Fig. 11: RegGAN produces human-like features, including eyes and facial contours, in cat and panda images. While cats naturally have vertical pupils, RegGAN tends to make them circular due to its training on human faces.