Capturing Dynamics: The 4D Facial Expression Intensity Dataset
Authors: Zesheng Wang, Alexandre Bruckert, Pierre Lebreton, Patrick Le Callet, Yante Li, Guoying Zhao
Organizations: École Centrale Nantes, CNRS, LS2N, UMR 6004 Nantes Université, France · Center for Machine Vision and Signal Analysis University of Oulu, Finland
The estimation and analysis of facial expression intensity play a crucial role in affective communication and human-computer interaction. Previous research has primarily focused on detecting and estimating facial expression intensity from frame-level 2D representations. However, this limitation restricts a comprehensive understanding of real-world facial expressions, as they are inherently 3D and temporally continuous. This paper investigates the perception of facial expression intensity by introducing the 4D Facial Expression Intensity Dataset (4DFEID). We employ a parametric face model and compile a total of 2,869 mesh sequences with controlled geometric variations, generating 4D data instances with diverse peak intensities and identity attributes. Using a Likert scale, we collect more than 90,000 subjective intensity perception ratings via a crowdsourcing platform. We explore various architectures and aggregation methods to establish baselines for episode intensity estimation on the new dataset, revealing that spatial-temporal graph models consistently outperform traditional frame-aggregation methods. In contrast to existing datasets that rely on 2D static imagery, the proposed 4D-FEID dataset provides the community with a unique and vital resource for investigating the perception of facial expression intensity through the use of dynamic 3D stimuli. By offering high-fidelity, spatio-temporally coherent facial data, 4D-FEID establishes a new foundation for research into more nuanced and naturalistic expression analysis, thereby addressing a gap in the current landscape of affective computing and human-computer interaction studies. The dataset is available at link.
Figures & tables
Dataset
Modality
Granularity
Intensity Type
MultiEmoVA [ 16 ]
2D
Frame-level
Arousal
HAPPEI [ 6 ]
2D
Frame-level
Happiness
DISFA [ 15 ]
2D
Frame-level
AUs-based
BP4D [ 25 ]
4D
Frame-level
AUs-based
4DME [ 13 ]
4D
Sequence-level
None (Categorical)
4D-FEID (Ours)
4D
Sequence-level
Global Expression
TABLE I: The previous expression-intensity and 4D-expression datasets
Fig. 1: Example of intensity-based pruning on a sample sequence from the dataset. Each colored curve corresponds to one of the preserved expression-parameter dimensions.
Fig. 2: Examples of rendering outputs, displaying peak-frame results of various facial expressions under progressively increasing intensities. The overlaid heatmaps qualitatively verify that the expression intensity strengthens from left to right.
Fig. 3: Interface of subjective experiment.
Session
Reliability(ICC)
Session
Reliability(ICC)
1
0.97
8
0.97
2
0.97
9
0.98
3
0.98
10
0.97
4
0.98
11
0.97
5
0.97
12
0.97
6
0.97
13
0.97
TABLE II: Inter-observer consistency measured by ICC for each session.
Model
MAE ↓
ICC ↑
PCC ↑
CCC ↑
FLAME-based
0.192
0.946
0.946
0.943
Video-CNN
0.289
0.960
0.960
0.911
4D-Vertex
0.139
0.975
0.975
0.974
TABLE III: Test-set performance of the baseline methods in the standard set. Symbols ↑ and ↓ indicate that higher and lower values are better, respectively.
Model
MAE ↓
ICC ↑
PCC ↑
CCC ↑
FLAME-based
0.204
0.942
0.942
0.939
Video-CNN
0.282
0.962
0.961
0.917
4D-Vertex
0.146
0.973
0.973
0.970
TABLE IV: Test-set performance of the baseline methods in the cross-identity set. Symbols ↑ and ↓ indicate that higher and lower values are better, respectively.
Generating expressive facial behavior in virtual humans (VHs) remains a central challenge in affective computing and character animation. This paper presents a novel approach based on Conditional Variational Autoencoders (CVAEs), trained on real human facial expression data, to synthesize controllable emotional expressions at varying intensities. Using a dataset comprising six basic emotions represented at two intensity levels (low and high), we train a CVAE model to generate synthetic facial expression data while preserving semantic consistency with real human expressions. Despite the limited amount of training data (only 7,680 facial expression samples), the proposed approach learns meaningful latent representations and generates coherent emotional variations. Our method enables control over emotional intensity, making it suitable for animating virtual characters without requiring actor performances or manual artistic intervention. Our research aimed to evaluate whether the method (CVAE) preserves the characteristics associated with the different intensity levels present in the dataset. Results show that the proposed model preserves key expressive characteristics across intensity levels while supporting generalization across emotional intensity levels, contributing to the creation of emotionally expressive virtual characters from relatively small datasets.
Vitor Miguel Xavier Peres, Lara Volpato, Gabriel Ferri Scnheider +1
School of Technology Pontifical Catholic University of Rio Grande do Sul Porto Alegre, Brazil
Do people perceive the same facial expression in the same way? Should we expect vision models to be flexible in how they perceive facial expressions? Facial expressions are nonverbal social signals used in human interaction, but facial expression recognition datasets often focus on a single deterministic annotation per sample. We introduce Chehre, an emoji-prompted video dataset with a wide range of dynamic facial expressions for exploring perceptual variation. In Chehre, 203 participants were prompted to express and record 40 facial emojis. Later, their facial motions were transferred onto synthetic faces to preserve privacy. A separate group annotated the videos, resulting in 2,111 videos annotated by 1,242 perceivers, with ~30 annotators per video. Chehre enables us to define a new task: "distributional expression recognition", which tests whether a model can reproduce the variation observed across annotator responses. We test a selection of video language models on our task. Interestingly, we find that persona prompting can act as a controllable way to shift model perception while helping models better capture the variation observed across human annotators. The dataset and code are available at https://chehre-dataset.github.io/.
Recent advances in 3D facial expression reconstruction have demonstrated remarkable performance in capturing macro-expressions, yet the reconstruction of micro-expressions remains unexplored. This novel task is particularly challenging due to the subtle, transient, and low-intensity nature of micro-expressions, which complicate the extraction of stable and discriminative features essential for accurate reconstruction. In this paper, we propose a fine-grained micro-expression reconstruction method that integrates a global dynamic feature capturing stable facial motion patterns with a locally-enriched feature incorporating multiple informative cues from 2D motions, facial priors and 3D facial geometry. Specifically, we devise a plug-and-play dynamic-encoded module to extract micro-expression feature for global facial action, allowing it to leverage prior knowledge from abundant macro-expression data to mitigate the scarcity of micro-expression data. Subsequently, a dynamic-guided mesh deformation module is designed for extracting aggregated local features from dense optical flow, sparse landmark cues and facial mesh geometry, which adaptively refines fine-grained facial micro-expression without compromising global 3D geometry. Extensive experiments on micro-expression datasets demonstrate that our method consistently outperforms state-of-the-art methods in both geometric accuracy and perceptual detail.