A Comprehensive Review of Multimodal Facial State Analysis: Tasks, Methods, and Resources
Authors: Xuri Ge, Tianshuo Zhang, Ruihan Li, Hui Ye, Kaiwen Zheng, Junchen Fu, Da Huo, Joemon M. Jose, Hu Han
Organizations: Shandong University, Qingdao Key Laboratory of Trustworthy Artificial Intelligence, Shandong, China · University of Glasgow, United Kingdom · Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
Facial state analysis plays a crucial role in understanding human expressions, psychological modeling, and human computer interaction. Traditional unimodal vision-based methods are often limited by environmental sensitivity and weak interpretability. Multimodal facial state analysis addresses these issues by integrating complementary cues from visual, audio, textual, physiological, and other related modalities. This survey emphasizes two key aspects: on one hand, multimodal learning enables contextual semantic understanding for improved facial state reasoning and leverages interpretable language generation to enhance model explainability; on the other hand, multi-task learning allows simultaneous analysis of expressions, action units (AUs), and face-based soft biometrics (e.g., age, gender), effectively capturing fine-grained expressions and improving cross-scene generalization. This survey reviews core tasks, representative methods, and datasets in multimodal facial state analysis, focusing on facial expression recognition, AU detection, and face-based soft biometric estimation, and emphasizing the unique value of language in providing contextual semantics, enhancing reasoning, and generating explanations. The survey aims to provide an up-to-date overview of the literature and to highlight future research directions for multimodal, interpretable, and multi-task adaptive facial state analysis.
Figures & tables
Fig. 1: Multimodal facial state analysis integrates visual information and other complementary modalities, providing a comprehensive assessment of facial action units (AU), expressions, and soft biometrics. By leveraging the intrinsic correlations among tasks, multi-task learning frameworks further strengthen feature robustness.
Fig. 2: Taxonomy of multimodal facial state analysis, covering three core tasks and multi-task joint learning, along with representative methods for each category.