Emotion Intensity Matters: Generating Realistic Expressions in Virtual Humans with CVAEs

Vitor Miguel Xavier Peres; Lara Volpato; Gabriel Ferri Scnheider; Soraia Raupp Musse

cs.CVAug 22, 2026

Emotion Intensity Matters: Generating Realistic Expressions in Virtual Humans with CVAEs

Authors: Vitor Miguel Xavier Peres, Lara Volpato, Gabriel Ferri Scnheider, Soraia Raupp Musse

Organizations: School of Technology Pontifical Catholic University of Rio Grande do Sul Porto Alegre, Brazil

Abstract

Generating expressive facial behavior in virtual humans (VHs) remains a central challenge in affective computing and character animation. This paper presents a novel approach based on Conditional Variational Autoencoders (CVAEs), trained on real human facial expression data, to synthesize controllable emotional expressions at varying intensities. Using a dataset comprising six basic emotions represented at two intensity levels (low and high), we train a CVAE model to generate synthetic facial expression data while preserving semantic consistency with real human expressions. Despite the limited amount of training data (only 7,680 facial expression samples), the proposed approach learns meaningful latent representations and generates coherent emotional variations. Our method enables control over emotional intensity, making it suitable for animating virtual characters without requiring actor performances or manual artistic intervention. Our research aimed to evaluate whether the method (CVAE) preserves the characteristics associated with the different intensity levels present in the dataset. Results show that the proposed model preserves key expressive characteristics across intensity levels while supporting generalization across emotional intensity levels, contributing to the creation of emotionally expressive virtual characters from relatively small datasets.

Figures & tables

Fig. 1: Method Overview: (1) Obtaining dataset; (2) Preparing to Capture Data (Live Link Hub); (3) Transferring the Capture Data to Metahuman and Prepare a dataset to (4) Training a CVAE and (5) the Generation of new facial expressions using the CVAE output, for each level of emotion intensity; Residual Check (6) from Generated Data and Residual Enhancement (7).

	Men				Women
	Low		High		Low		High
Emotion	p	$\sigma$	p	$\sigma$	p	$\sigma$	p	$\sigma$
Angry	0.370	0.320	0.364	0.311	0.302	0.323	0.308	0.318
Disgusted	0.386	0.318	0.353	0.321	0.278	0.315	0.275	0.314
Fear	0.367	0.326	0.360	0.310	0.282	0.308	0.313	0.314
Happy	0.361	0.316	0.296	0.325	0.320	0.312	0.308	0.318

TABLE I: Average p-values and standard deviations between real and synthetic data, for each emotion according to gender and intensity level.

	Men				Women
	Low		High		Low		High
Emotion	p	$\sigma$	p	$\sigma$	p	$\sigma$	p	$\sigma$
Angry	0.648	0.318	0.591	0.323	0.616	0.329	0.631	0.322
Disgusted	0.606	0.314	0.594	0.320	0.599	0.322	0.597	0.317
Fear	0.614	0.321	0.657	0.318	0.602	0.315	0.635	0.315
Happy	0.698	0.319	0.670	0.317	0.615	0.314	0.597	0.324

TABLE II: Average p-values and standard deviations between real and enhanced synthetic data, for each emotion according to gender and intensity level.

	Men				Women
	Low		High		Low		High
Emotion	p	$\sigma$	p	$\sigma$	p	$\sigma$	p	$\sigma$
Angry	0.655	0.265	0.758	0.233	0.632	0.244	0.633	0.241
Disgusted	0.674	0.267	0.660	0.219	0.581	0.022	0.615	0.230
Fear	0.653	0.241	0.646	0.253	0.721	0.242	0.686	0.236
Happy	0.704	0.241	0.612	0.253	0.747	0.232	0.797	0.233

TABLE III: Average p-values and standard deviations between synthetic and enhanced data, for each emotion according to gender and intensity level.

Fig. 2: Rendering of faces, based on imported data generated via CVAE. We provide a separation into 6 emotions and 2 intensities, and illustrate only 2 individuals. Side-by-side, we show the real face, the face rendered via CVAE, and Enhanced CVAE.

Emotion		Low			High
Emotion		Real	CVAE	Enh.	Real	CVAE	Enh.
Angry	$\mu$ $\sigma$	0.3653 0.5393	0.1857 0.3541	0.2346 0.4274	0.9462 1.2141	0.2906 0.518	0.3418 0.6026
Disgusted	$\mu$ $\sigma$	0.6406 0.7553	0.2857 0.4778	0.3402 0.5388	1.0029 1.3333	0.4297 0.6531	0.4882 0.7363
Fear	$\mu$ $\sigma$	0.5818 0.8543	0.2307 0.4449	0.2646 0.4886	0.8494 1.1438	0.3351 0.5949	0.4031 0.6529
Happy	$\mu$ $\sigma$	0.3774 0.5524	0.3591 0.5834	0.3644 0.6088	0.9106 1.2511	0.5105 0.8279	0.5353 0.8586
Sad	$\mu$ $\sigma$	0.5156 0.7146	0.1883 0.3739	0.1713 0.341	0.8815 0.9397	0.3596 0.5542	0.3722 0.554

TABLE IV: Mean and standard deviation of the emotion intensity scores estimated by the OpenFace [ 2 ] analysis tool for each emotion and intensity level. Results are reported for the Real dataset, the original CVAE-generated expressions and the Enhanced expressions. Higher mean values indicate stronger expression intensity, while the standard deviation reflects the variability across samples.

Explore similar work

EmoteGPT: 3D Human Facial Expressions from Natural Language Descriptions
Jul 2, 2026Haoran Wang, Mohit Mendiratta, Christian Theobalt +1Text-to-3D Generation
Max Planck Institute for Informatics, Saarland Informatics Campus · CISPA Helmholtz Center for Information Security
Generative Learning as a Tool to Improve Perception of Emotional Body Motion Expressions
Jun 27, 2026Huakun Liu, Miao Cheng, Xin Wei +6Human Motion Generation Conditional Generative Modeling
Nara Institute of Science and Technology · Tohoku University
ARGen: Affect-Reinforced Generative Augmentation towards Vision-based Dynamic Emotion Perception
Apr 14, 2026Huanzhen Wang, Ziheng Zhou, Jiaqi Song +4Synthetic Data Augmentation Facial Expression Recognition
Fudan University · East China Normal University

Jul 2, 2026J/K move · Enter open · S save

EmoteGPT: 3D Human Facial Expressions from Natural Language Descriptions

Haoran Wang, Mohit Mendiratta, Christian Theobalt, Adam Kortylewski

Max Planck Institute for Informatics, Saarland Informatics Campus · CISPA Helmholtz Center for Information Security

Text-to-3D Generation

Open paper AskPDF arXiv

Precise control of 3D facial expressions from text is crucial for virtual avatars, animation, and human-computer interaction, yet existing text-to-3D methods jointly generate identity, expression, and texture, making fine-grained expression control difficult. We instead formulate text-driven expression synthesis as a regression problem in the disentangled parameter space of a 3D Morphable Model (3DMM). This setting, however, requires paired data linking detailed language to precise expression parameters, which are missing from existing resources. To fill this gap, we introduce Txt2Emote, a benchmark of diverse 3D facial expressions with fine-grained textual annotations obtained from GPT-4o and a high-fidelity face tracker, providing both explicit descriptions detailing facial features and implicit descriptions referencing the situational context behind the expression. Leveraging this dataset, we present EmoteGPT, a text-to-3D expression framework based on a Multimodal Large Language Model (MLLM) with a dedicated <Expr> token to semantically ground expression representations, which are then decoded into 3DMM parameters. We further improve EmoteGPT by augmenting training with large-scale image-to-3DMM data, enabling it to surpass state-of-the-art text-to-3D face synthesis methods on emotion recognition metrics and in perceived expressiveness. Integrated into avatar pipelines, our method enables photorealistic and stylized 3D avatars, as well as expressive 3D-consistent 2D face synthesis from textual input.

Abstract

Explore similar work

EmoteGPT: 3D Human Facial Expressions from Natural Language Descriptions

Generative Learning as a Tool to Improve Perception of Emotional Body Motion Expressions

ARGen: Affect-Reinforced Generative Augmentation towards Vision-based Dynamic Emotion Perception