Improving Generative Adversarial Network Generalization for Facial Expression Synthesis
Authors: Arbish Akram, Nazar Khan, Arif Mahmood
Organizations: Department of Computer Science, University of the Punjab, Lahore, Pakistan · Department of Computer Science, Information Technology University, Lahore, Pakistan
Facial expression synthesis aims to generate realistic facial expressions while preserving identity. Existing conditional generative adversarial networks (GANs) achieve excellent image-to-image translation results, but their performance often degrades when test images differ from the training dataset. We present Regression GAN (RegGAN), a model that learns an intermediate representation to improve generalization beyond the training distribution. RegGAN consists of two components: a regression layer with local receptive fields that learns expression details by minimizing the reconstruction error through a ridge regression loss, and a refinement network trained adversarially to enhance the realism of generated images. We train RegGAN on the CFEE dataset and evaluate its generalization performance both on CFEE and challenging out-of-distribution images, including celebrity photos, portraits, statues, and avatar renderings. For evaluation, we employ four widely used metrics: Expression Classification Score (ECS) for expression quality, Face Similarity Score (FSS) for identity preservation, QualiCLIP for perceptual realism, and Fréchet Inception Distance (FID) for assessing both expression quality and realism. RegGAN outperforms six state-of-the-art models in ECS, FID, and QualiCLIP, while ranking second in FSS. Human evaluations indicate that RegGAN surpasses the best competing model by 25% in expression quality, 26% in identity preservation, and 30% in realism.
Figures & tables
Fig. 1: Given a neutral input image, the proposed RegGAN synthesizes photorealistic facial expressions on out-of-distribution testing images. Despite being trained only on real human faces from the CFEE dataset [ 8 ] , the proposed method introduces realistic expressions, preserves identity and retains facial details of input images.
Fig. 2: Left: Our method consists of two components: an expression layer and a refinement network. The expression layer takes an input image and generates a new image of the same person with a different facial expression. The refinement network then enhances the quality of the synthesized image, making it more realistic and sharper. Right: Architecture of EAB.
Fig. 3: Illustration of the input, intermediate, and final outputs synthesized by our proposed RegGAN. The expression layer GE generates an intermediate image that captures the target expression, while the refinement network GR transforms this intermediate result into a convincing photorealistic output.
Fig. 4: Results of RegGAN for facial expression synthesis on out-of-distribution images, including an impasto face (row 1), a celebrity face (row 2), a fantasy image (row 3), a portrait (row 4), as well as an avatar (row 5). Impasto and fantasy images were generated using stable diffusion [ 49 ] . The proposed method introduces convincing expressions while preserving the identity and facial details of the input image.
Fig. 5: Comparison of the proposed method, RegGAN, with six state-of-the-art facial expression synthesis models - StarGAN, GANimation, MR, DAI2I, SARGAN as well as US-GAN on out-of-distribution facial images. GANimation synthesizes realistic expression but introduces noticeable artifacts. StarGAN and DAI2I fail to preserve the input image’s color distribution in their output. SARGAN and US-GAN preserve facial details but are unable to induce happy expressions. MR generates realistic happy expressions, but the results are often blurry. In contrast, RegGAN produces sharper and more realistic expressions on out-of-distribution images. Moreover, RegGAN demonstrates consistent performance across diverse image types, including human portraits, status, and avatars collected from the Internet, despite being trained exclusively on CFEE dataset captured in a controlled environment with consistent lighting and background.
Fig. 6: Zoomed-in versions of smiles from our proposed method as well as other methods. Our method generates more realistic smiles and enhances the clarity of individual teeth, improving their differentiation and visibility compared to all other methods.
Method
ECS ↑
FSS ↑
RS ↑
FID ↓
StarGAN [ 11 ]
0.71
0.68
0.55
111.9
GANimation [ 19 ]
0.45
0.64
0.80
94.09
MR [ 18 ]
0.72
0.66
0.19
113.9
DAI2I [ 21 ]
0.54
0.64
0.58
140.2
SARGAN [ 23 ]
0.61
0.71
0.73
98.86
US-GAN [ 22 ]
0.66
0.71
0.69
98.92
TABLE I: RegGAN outperforms other methods in ECS and RS by synthesizing convincing, photo-realistic facial expressions. The lowest FID indicates that the generated images closely match the distribution of real images. In FSS, it attains the second-best value, as SARGAN and US-GAN use ultimate skip connections that directly copy input details to the output.
Fig. 7: RegGAN outperforms all existing baselines in synthesizing convincing and realistic expression, while preserving facial details.
Fig. 8: StyleGAN-based methods synthesize smiles but tend to lose fine facial details and introduce unwanted artifacts, such as glasses (third row, second column). The proposed approach better preserves identity details while synthesizing realistic facial expressions.
Method
ECS ↑
FSS ↑
RS ↑
FID ↓
No LABs
0.87
0.67
0.90
80.8
No AUs
0.86
0.68
0.88
89.9
RegGAN
0.85
0.68
0.90
74.3
TABLE II: Quantitative results of ablation studies. The inclusion of LABs and AUs helps achieve a better balance among expression, facial details, and realism.
Method
ECS ↑
FSS ↑
RS ↑
FID ↓
StyleRES [ 31 ]
0.91
0.58
0.28
83.5
SFE [ 32 ]
0.70
0.64
0.39
67.4
RegGAN
0.93
0.64
0.85
61.8
TABLE III: Quantitative results of RegGAN and StyleGAN-based methods. RegGAN achieves higher scores in expression quality, identity preservation, and realism.
Fig. 9: Results of the proposed method on diverse facial images: Row 1: faces with non-uniform, multi-colored illumination. Row 2: faces in the style of fantasy artist Peter Mohrbacher. Row 3: faces in the style of artist Tomer Hanuka. Row 4: impasto in the style of Gregory Manchess. Row 5: faces with makeup and occlusion. All input images were generated using Stable Diffusion [ 58 ] . Results demonstrate strong generalization of the proposed method, that synthesizes realistic facial expressions while preserving the artistic details and styles of the input images.
Fig. 10: RegGAN results on Gaussian-smoothed and motion-blurred images. While the expression layer successfully induces desired expressions on sharp as well as blurred inputs, the sharpening effect of our refinement network causes inputs and outputs to have different levels of blurriness. This can be beneficial in some contexts, but not so desirable in others.
Fig. 11: RegGAN produces human-like features, including eyes and facial contours, in cat and panda images. While cats naturally have vertical pupils, RegGAN tends to make them circular due to its training on human faces.
Facial micro-expressions are subtle and short-lived facial movements that provide important cues about genuine human emotions. However, modeling and generating them remains difficult because annotated micro-expression data is limited and the underlying facial motions are extremely weak. Existing micro-expression generation methods therefore often suffer from limited quality, weak robustness, and poor generalization. We propose MagPlus, a transferable micro-expression processing pipeline that connects micro-expression analysis with standard facial animation models. Instead of training a dedicated generator from scratch, MagPlus learns to magnify subtle facial motions into the range of regular facial expressions, transforming micro-expressions into signals that are compatible with existing facial expression processing models. The magnified sequence is then used by a standard facial expression model for tasks such as transfer and synthesis. A complementary DeMagPlus module then restores the generated motion back to realistic micro-expression intensity levels while preserving the synthesized dynamics. We evaluate the framework using four facial animation models: FOMM, FSRT, MetaPortrait, and EmoPortraits. None of these models are trained on micro-expression data. Experiments show that MagPlus-DeMagPlus enables pretrained macro-expression models to generate more realistic micro-expression motion without retraining the backbones.
Identity-preserving video generation aims to maintain a subject's identity while synthesizing realistic videos. Yet a single reference portrait captures the subject's appearance under only one facial configuration. As expressions change, facial appearance can vary in highly identity-specific ways, leaving the subject's appearance under unseen expressions underdetermined by the reference alone. This expression-dependent variation also complicates evaluation: similarity to a neutral reference may decrease under strong expressions even for real images of the same person. We investigate this limitation from both generation and evaluation perspectives. First, we quantify how face-recognition similarity varies with expression intensity using controlled photographs and MEAD videos. We then construct a compact yet expressive reference gallery that captures diverse expression-dependent facial configurations. Matching against this gallery provides a more robust measure of identity similarity under expressive motion. To further expose performance degradation with expression intensity, we report identity similarity separately for mild, intense, and extreme expressions. For generation, we extend Stand-In to condition on our expression-diverse reference sets and develop a data-curation pipeline that extracts consistent yet diverse face crops from training videos. In practical settings where only a single portrait is available, we construct the reference set by synthesizing additional expressions with a pretrained facial reenactment model. On our controlled benchmark, both real and synthesized reference sets outperform the evaluated baselines in identity similarity across all three expression-intensity regimes, with the largest improvements for extreme expressions.
Tianwen Fu, Wenbin Teng, Gonglin Chen +3
Institute for Creative Technologies, University of Southern California
Precise control of 3D facial expressions from text is crucial for virtual avatars, animation, and human-computer interaction, yet existing text-to-3D methods jointly generate identity, expression, and texture, making fine-grained expression control difficult. We instead formulate text-driven expression synthesis as a regression problem in the disentangled parameter space of a 3D Morphable Model (3DMM). This setting, however, requires paired data linking detailed language to precise expression parameters, which are missing from existing resources. To fill this gap, we introduce Txt2Emote, a benchmark of diverse 3D facial expressions with fine-grained textual annotations obtained from GPT-4o and a high-fidelity face tracker, providing both explicit descriptions detailing facial features and implicit descriptions referencing the situational context behind the expression. Leveraging this dataset, we present EmoteGPT, a text-to-3D expression framework based on a Multimodal Large Language Model (MLLM) with a dedicated <Expr> token to semantically ground expression representations, which are then decoded into 3DMM parameters. We further improve EmoteGPT by augmenting training with large-scale image-to-3DMM data, enabling it to surpass state-of-the-art text-to-3D face synthesis methods on emotion recognition metrics and in perceived expressiveness. Integrated into avatar pipelines, our method enables photorealistic and stylized 3D avatars, as well as expressive 3D-consistent 2D face synthesis from textual input.
Haoran Wang, Mohit Mendiratta, Christian Theobalt +1
Max Planck Institute for Informatics, Saarland Informatics Campus · CISPA Helmholtz Center for Information Security