cs.CVApr 12, 2026

Bidirectional Learning of Facial Action Units and Expressions via Structured Semantic Mapping across Heterogeneous Datasets

Authors: Jia Li, Yu Zhang, Yin Chen, Zhenzhen Hu, Yong Li, Richang Hong, Shiguang Shan, Meng Wang

Organizations: School of Computer Science and Information Engineering, Hefei University of Technology, Hefei 230601, China · School of Computer Science and Engineering, and the Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University, Nanjing 210096, China · Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China, and also with the University of Chinese Academy of Sciences, Beijing 100049, China

Abstract

Facial action unit (AU) detection and facial expression (FE) recognition can be jointly viewed as affective facial behavior tasks, representing fine-grained muscular activations and coarse-grained holistic affective states, respectively. Despite their inherent semantic correlation, existing studies predominantly focus on knowledge transfer from AUs to FEs, while bidirectional learning remains insufficiently explored. In practice, this challenge is further compounded by heterogeneous data conditions, where AU and FE datasets differ in annotation paradigms (frame-level vs.\ clip-level), label granularity, and data availability and diversity, hindering effective joint learning. To address these issues, we propose a Structured Semantic Mapping (SSM) framework for bidirectional AU--FE learning under different data domains and heterogeneous supervision. SSM consists of three key components: (1) a shared visual backbone that learns unified facial representations from dynamic AU and FE videos; (2) semantic mediation via a Textual Semantic Prototype (TSP) module, which constructs structured semantic prototypes from fixed textual descriptions with learnable context prompts for supervision and cross-task alignment in a shared semantic space; and (3) a Dynamic Prior Mapping (DPM) module that incorporates FACS-derived prior knowledge and learns data-adaptive bidirectional association matrices in the textual semantic space for explicit knowledge transfer. Extensive experiments on popular AU detection and FE recognition benchmarks show that SSM consistently outperforms its single-task and multi-task baselines and achieves competitive performance against task-specific methods. The FE-to-AU results further show that holistic expression semantics provides useful supervision for fine-grained AU learning across heterogeneous datasets.

Figures & tables

Explore similar work

CardsList
  1. Hierarchical Vision-Language Interaction for Facial Action Unit Detection

    Feb 16, 2026Yong Li, Yi Ren, Yizhe Zhang +5Facial Action Unit DetectionVision-Language Models

  2. AULLM++: Structured-Token-Conditioned Large Language Models for Micro-Expression Action Unit Detection

    Mar 9, 2026Zhishu Liu, Kaishen Yuan, Bo Zhao +2Facial Action Unit DetectionFacial Expression Recognition

  3. AUCH-Net: Action Unit-Based Consistency-Aware Hypergraph Network for Cross-Domain Few-Shot Facial Expression Recognition

    Jul 23, 2026Xinhan Qiu, Yan Yan, Rui Zhu +2Few-Shot LearningCross-Domain Transfer Learning