Organizations: School of Computer Science and Information Engineering, Hefei University of Technology, Hefei 230601, China · School of Computer Science and Engineering, and the Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University, Nanjing 210096, China · Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China, and also with the University of Chinese Academy of Sciences, Beijing 100049, China
Facial action unit (AU) detection and facial expression (FE) recognition can be jointly viewed as affective facial behavior tasks, representing fine-grained muscular activations and coarse-grained holistic affective states, respectively. Despite their inherent semantic correlation, existing studies predominantly focus on knowledge transfer from AUs to FEs, while bidirectional learning remains insufficiently explored. In practice, this challenge is further compounded by heterogeneous data conditions, where AU and FE datasets differ in annotation paradigms (frame-level vs.\ clip-level), label granularity, and data availability and diversity, hindering effective joint learning. To address these issues, we propose a Structured Semantic Mapping (SSM) framework for bidirectional AU--FE learning under different data domains and heterogeneous supervision. SSM consists of three key components: (1) a shared visual backbone that learns unified facial representations from dynamic AU and FE videos; (2) semantic mediation via a Textual Semantic Prototype (TSP) module, which constructs structured semantic prototypes from fixed textual descriptions with learnable context prompts for supervision and cross-task alignment in a shared semantic space; and (3) a Dynamic Prior Mapping (DPM) module that incorporates FACS-derived prior knowledge and learns data-adaptive bidirectional association matrices in the textual semantic space for explicit knowledge transfer. Extensive experiments on popular AU detection and FE recognition benchmarks show that SSM consistently outperforms its single-task and multi-task baselines and achieves competitive performance against task-specific methods. The FE-to-AU results further show that holistic expression semantics provides useful supervision for fine-grained AU learning across heterogeneous datasets.
Figures & tables
Fig. 1: Paradigm comparison of joint AU and FE learning. (a): conventional multi-task learning (MTL) on homogeneous datasets achieves bidirectional gains but suffers from high annotation cost and limited generalization. (b): predominant unidirectional transfer (AU → FE) on heterogeneous datasets, where mismatched annotation paradigms and semantic granularity hinder reverse learning and introduce distribution bias from lab-controlled AU data. (c): our method enables adaptive bidirectional learning by explicitly modeling AU ↔ FE relationships via textual mediation in a shared semantic space, improving scalability and generalization under heterogeneous data.
Fig. 2: Overview of the Structured Semantic Mapping (SSM) framework. SSM reformulates joint AU detection and DFER in a unified vision–text semantic space. The visual branch employs a shared CLIP vision encoder with MoE layers (i.e., our Baseline model) and two task-specific temporal models, producing a clip-level representation for DFER and a temporally enhanced center-frame representation for AU detection from heterogeneous datasets. In the textual branch, the Textual Semantic Prototype (TSP) module constructs AU prototypes from FACS-defined AU descriptions and composes expression prototypes from FACS-derived AU–expression configurations, which are then encoded by a shared CLIP text encoder. Built upon these representations, the Dynamic Prior Mapping (DPM) module performs prior-initialized bidirectional semantic mapping via two learnable association matrices, generating dynamically updated textual representations for cross-task knowledge transfer and contrastive supervision.
Fig. 3: Our Baseline model for learning dynamic AUs and FEs jointly. A shared CLIP vision encoder with MoE layers first extracts frame-wise features from FE or AU videos. Then, these features are fed into two Transformer-based temporal models, Φexp(⋅) and Φau(⋅) , for task-specific temporal modeling. The DFER branch aggregates sequential features into a clip-level representation Zexp . In contrast, the AU branch performs temporal interaction among the input frames and selects the temporally enhanced feature of the center frame as the AU representation Zau . Finally, Zexp and Zau are fed into independent linear heads for classification. This Baseline model corresponds to the visual backbone in Fig. 2 .
Fig. 4: The linear feature transformation in the DPM module is to multiply a learnable weight matrix with a task-specific semantic matrix. Composition transformation combines the semantics of multiple AUs into the semantics of a single FE. Hence, the AU semantic matrix is transformed into a new FE semantic matrix. Decomposition transformation decomposes the semantics of a single FE into the semantics of multiple AUs. Hence, the FE semantic matrix is transformed into a new AU semantic matrix. Matrices A , B , C , D , E , and F denote Wau→exp , Wexp→au , Tau , T~exp , Texp , and T~au , respectively.
Method
Backbone
DFEW
FERV39K
MAFW
UAR
WAR
UAR
WAR
UAR
WAR
Supervised learning models
Former-DFER [ 39 ]
Transformer
53.69
65.70
37.20
46.85
31.16
43.27
NR-DFERNet [ 75 ]
CNN-Transformer
54.21
68.19
33.99
45.97
-
-
EST [ 37 ]
ResNet-18
53.43
65.85
-
-
-
-
Freq-HD [ 76 ]
VGG13-LSTM
46.85
55.68
33.07
45.26
-
-
TABLE III: Comparison with state-of-the-art methods on DFEW, FERV39K, and MAFW. UAR: Unweighted Average Recall; WAR: Weighted Average Recall. Using joint learning based on the BP4D dataset. The best results are highlighted in bold, and the second-best underlined. STL refers to the single-task counterpart.
Baseline (MTL)
TSP
DPM
BP4D
DFEW
DISFA
DFEW
66.2
63.98/76.16
69.6
63.98/76.16
✓
67.2
65.25/76.97
70.4
65.93/77.53
✓
✓
67.7
66.75/77.35
70.6
66.03/77.78
✓
✓
✓
68.5
68.59 / 77.88
71.3
66.64 / 78.09
TABLE V: Investigation of the contributions of each component in the SSM framework. Multi-task Baseline: Baseline model in Fig. 3 trained jointly on heterogeneous datasets; TSP: Textual Semantic Prototype module; DPM: Dynamic Prior Mapping module. Note that the SSM framework is always built on top of the Baseline model. Metrics on BP4D and DISFA: F1 score. Metrics on DFEW: UAR/WAR.
R init.
P init.
Dual
BP4D
DFEW
DISFA
DFEW
✓
67.0
65.35/77.35
69.6
64.52/76.88
✓
✓
67.2
65.75/77.53
70.2
65.20/76.84
✓
68.1
67.00/77.01
70.9
66.37/77.23
✓
✓
68.5
68.59 / 77.88
71.3
66.64 / 78.09
TABLE VI: How to construct an effective DPM module. We further probe its working mechanism by constructing the DPM in different ways. R init.: random initialization; P init.: prior initialization; Dual: whether bidirectional independent learning is performed. Metrics on BP4D and DISFA: F1 score. Metrics on DFEW: UAR/WAR.
Setting
BP4D
DFEW
DISFA
DFEW
Linear
66.8
64.79/76.87
69.7
64.77/76.88
MLP
67.5
65.26/77.23
70.6
65.89/77.18
DPM (Random, Frozen)
66.9
65.12/76.61
69.7
64.73/76.84
DPM (Prior, Frozen)
67.3
66.62/77.05
70.2
65.47/77.53
DPM
68.5
68.59 / 77.88
71.3
66.64 / 78.09
TABLE VII: Effectiveness of the DPM module. We compare the full DPM with Linear, MLP, DPM (Random, Frozen), and DPM (Prior, Frozen). “Frozen” indicates that the DPM weight matrices are fixed during training. Metrics on BP4D and DISFA: F1 score. Metrics on DFEW: UAR/WAR.
Setting
BP4D
DFEW
DISFA
DFEW
Words
68.2
64.56/77.87
68.1
67.24 /77.40
Standalone
68.1
65.33/77.31
69.7
65.60/77.53
Compound
68.5
68.59 / 77.88
71.3
66.64/ 78.09
TABLE VIII: How to construct an effective textual semantic prototype module (TSP). Since DPM is always based on TSP, the effect of knowledge transfer also indirectly depends on the type of text. We use different textual descriptions to investigate their impact on the model. Words: word descriptions; Standalone: standalone descriptions; Compound: composite descriptions. BP4D and DISFA: F1 score. DFEW: UAR/WAR.
Prompt Count
BP4D
DFEW
DISFA
DFEW
0
67.9
63.70/77.23
70.2
63.84/76.67
4
66.8
67.49/77.05
68.6
64.95/77.06
8
68.5
68.59 / 77.88
71.3
66.64 / 78.09
12
67.0
66.15/77.48
69.6
66.50/77.19
16
66.6
65.96/77.53
69.5
64.82/77.05
TABLE IX: Analysis of the number of learnable tokens in text descriptions. BP4D and DISFA: F1 score. DFEW: UAR/WAR.
Fig. 5: The analysis of data scale, where 0% data scale indicates single-task training. In the left panel (Expression → AU), BP4D and DISFA denote joint learning with DFEW. In the right panel (AU → Expression), DFEW col1 denotes joint learning with BP4D, and DFEW col2 denotes joint learning with DISFA.
Fig. 6: Visualization of the input facial frame sequences. The left side shows samples from the DFER task. The right side shows samples from the AU detection task. Each subfigure presents an overlay of the original face frame and the attention heatmap. Warmer colors indicate regions with higher responses. The attention maps are computed from the multi-head self-attention of the CLIP-ViT-B/16 visual encoder using the attention rollout method [ 82 ] . From top to bottom, the attention overlays correspond to STL, Baseline, and SSM, respectively. Compared with STL, Baseline attends to more facial regions. Compared with Baseline, SSM shows denser and more spatially organized responses across frames.
Fig. 7: The upper part shows the weight matrix activation maps from AUs to facial expressions on DISFA and DFEW (fd5). From left to right, the three matrices correspond to the initial matrix, the matrix learned from random initialization, and the matrix learned from prior-based initialization. The lower part shows the corresponding weight matrices from facial expressions to AUs.
Train
Test
BP4D
DFEW
DISFA
FERV39K
STL
66.2
63.98/76.16
46.5
29.91/39.17
Baseline
67.2
65.25/76.97
59.4
31.55/41.98
SSM
68.5
68.59 / 77.88
67.1
32.10 / 43.52
TABLE X: Cross-dataset testing. We train on the BP4D+DFEW combination and perform zero-shot testing on the DISFA+FERV39K combination. We compare STL, Baseline, and SSM. For the DISFA test, we select the five AUs shared by BP4D and DISFA and report their average F1 score. BP4D and DISFA: F1 score. DFEW and FERV39K: UAR/WAR.
Fig. 8: Sensitivity analysis of the initial values of α and β in Eqn. 23 and Eqn. 24 . Left: performance on AU datasets (BP4D and DISFA), measured by F1 score. Right: performance on DFEW, measured by UAR and WAR. The dashed lines indicate the average performance across datasets. The results show that SSM maintains stable performance over a wide range of α=β . The performance shows a mild peak around 0.1. DFEW col1: jointly learned with BP4D. DFEW col2: jointly learned with DISFA.
FE data scaling
BP4D
DFEW
DISFA
DFEW
0%
66.2
-
69.6
-
20%
67.6
57.79/71.40
70.8
59.05/71.36
40%
67.8
60.76/73.50
70.2
62.71/74.19
60%
67.9
63.42/75.39
70.7
64.64/75.39
80%
68.2
65.23/77.31
71.0
65.12/76.33
100%
68.5
68.59 / 77.88
71.3
66.64 / 78.09
TABLE S1: Study of data scaling (Expression → AU). We examine how expression datasets of different scales affect AU detection. This analysis shows that the improvement of the framework on AU detection is not merely caused by the increased scale of expression data. BP4D and DISFA: F1 score. DFEW: UAR/WAR.
AU data scaling
BP4D
DFEW
DISFA
DFEW
0%
-
63.98/76.16
-
63.98/76.16
20%
67.4
66.95/77.57
70.5
66.69/ 78.52
40%
68.3
64.92/77.53
70.8
65.88/77.31
60%
67.9
67.06/77.23
70.5
67.55 /77.31
80%
68.0
67.83/ 78.52
71.1
65.50/77.91
100%
68.5
68.59 /77.88
71.3
66.64/78.09
TABLE S2: Study of data scaling (AU → Expression). We examine how AU datasets of different scales affect expression recognition. This analysis shows that the framework’s performance gains on expression recognition are not merely caused by the increased scale of AU data. BP4D and DISFA: F1 score. DFEW: UAR/WAR.
α , β
BP4D
DFEW
DISFA
DFEW
0.01
68.4
69.05 /77.83
71.2
65.74/77.74
0.05
68.5
67.49/77.57
71.1
65.72/77.70
0.1
68.5
68.59/ 77.88
71.3
66.64 / 78.09
0.5
68.4
67.64/77.83
70.9
65.50/77.83
1.0
68.1
67.63/77.44
70.6
66.04/77.87
TABLE S3: Analysis of the initial values of weighting factors, α and β , which directly regulate the strength of the DPM module. BP4D and DISFA: F1 score. DFEW: UAR/WAR.
Fig. S1: The weight matrix activation maps, from left to right, represent joint learning of DISFA with DFEW, FERV39K, and MAFW, respectively. The first row shows the weights from AUs to expressions, and the second row shows the weights from expressions to AUs. The two directions are not transposes of each other.
Fig. S2: The weight matrix activation maps, from left to right, represent joint learning of BP4D with DFEW, FERV39K, and MAFW, respectively. The first row shows the weights from AUs to expressions, and the second row shows the weights from expressions to AUs. The two directions are not transposes of each other.
Fig. S3: Proposed MoE in the CLIP visual encoder. Each Transformer block replaces the original FFN with an MoE layer containing one shared pretrained CLIP expert and multiple private experts.
λ
BP4D
DFEW
DISFA
DFEW
0.25
67.5
64.74/77.61
69.0
65.50/77.83
0.50
67.6
67.77/77.53
69.8
65.73/77.57
1.0
67.8
67.70/77.40
70.3
65.61/77.87
2.0
68.5
68.59 / 77.88
71.3
66.64 / 78.09
3.0
68.0
68.24/77.48
71.2
66.62/77.40
4.0
67.8
66.41/77.14
71.8
65.61/77.48
TABLE S5: Sensitivity to the task-loss weight λ . The two DFEW columns report joint training with BP4D and DISFA, respectively. BP4D and DISFA: F1 score. DFEW: UAR/WAR.
Transfer Setting
AU Detection (F1)
FE Data
FACS Prior
Dynamic
BP4D
DISFA
0%
66.2
69.6
0%
✓
66.1
69.7
100%
67.2
70.4
100%
✓
67.3
70.2
20%
✓
✓
67.6
70.8
TABLE S6: Comparison of auxiliary FE data, the FACS-informed prior knowledge, and dynamic updating for FE → AU transfer. FE data are from DFEW.
Transfer Setting
DFER on DFEW
AU Data
FACS Prior
Dynamic
UAR
WAR
0%
63.98
76.16
0%
✓
63.56
77.53
100%
65.25
76.97
100%
✓
66.62
77.05
20%
✓
✓
66.95
77.57
TABLE S7: Comparison of auxiliary AU data, the FACS-informed prior knowledge, and dynamic updating for AU → FE transfer. AU data are from BP4D.
Fig. S4: Failure analysis on DFEW fold 5. (a) SSM failure rate among Baseline-correct samples, computed within intervals of the Baseline confidence margin. The horizontal axis shows the confidence-margin intervals, and the vertical axis shows the proportion of samples misclassified by SSM in each interval. The confidence margin is the difference between Baseline’s top-1 and top-2 class probabilities; a smaller margin indicates greater ambiguity. (b) Most frequent Baseline-correct → SSM-wrong transitions.
Fig. S5: AU-level analysis on DISFA fold 1. (a) Bars show the AU-wise F1 change from Baseline to SSM (left vertical axis), and the black line shows test-set prevalence (right vertical axis). (b) Macro-F1 of Baseline and SSM at different levels of compatibility between the ground-truth active AU set and the FACS-informed AU combinations. Compatibility is the maximum Jaccard similarity to the six non-neutral expression prototypes; higher compatibility indicates less deviation from the FACS-informed prior. Low, Partial, and Exact denote Jaccard similarities <0.5 , [0.5,1.0) , and 1.0 , respectively.
Fig. S6: Examples correctly predicted by Baseline but incorrectly predicted by SSM (Set 1). The upper row shows two independent DFEW cases, and the lower row shows two independent DISFA cases. Orange outlines mark the ground-truth labels. In the AU panels, the dashed line marks the decision threshold of 0.5. “SSM w/o DPM” uses the same trained SSM checkpoint with DPM disabled at inference.
Fig. S7: Additional examples correctly predicted by Baseline but incorrectly predicted by SSM (Set 2). The upper row shows two independent DFEW cases, and the lower row shows two independent DISFA cases. Orange outlines mark the ground-truth labels. In the AU panels, the dashed line marks the decision threshold of 0.5. “SSM w/o DPM” uses the same trained SSM checkpoint with DPM disabled at inference.
Model
Frames
Total(M)
Train(M)
FLOPs(G)
Time(ms)
STL-DFER
192
88.308
88.308
3238.905
253.251
STL-AU
128
88.308
88.308
2159.253
169.090
Baseline
320
109.346
109.346
5994.240
487.113
Baseline + TSP
320
147.485
109.354
6037.918
494.001
Baseline + TSP + DPM (SSM)
320
147.485
109.354
6037.918
494.721
TABLE S8: Computational complexity under the heterogeneous mini-batch setting. DFER and AU batches contain 12 and 8 clips, respectively, with 16 frames per clip; thus joint models process 320 video frames per forward pass. “Frames” denotes the number of video frames processed in one forward pass. “Total” and “Train” denote total and trainable parameters. “Time” denotes the average inference time over 50 forward passes after 10 warm-up runs under the torch.no_grad() context manager.
Setting
MMD2
Cross/Intra
STL
0.2656
1.3046
Baseline
0.1764
1.1769
SSM
0.1459
1.1374
TABLE S9: Quantitative comparison of cross-dataset representation discrepancy among STL, Baseline, and SSM. MMD2 and Cross/Intra are computed based on the 512-dimensional encoder embeddings before the classification heads. An equal number of DFEW and DISFA samples are used, with the same sampled instances across all three settings. Lower values indicate a smaller representation discrepancy.
Fig. S8: Visualization of the DFEW–DISFA representation gap under different training settings. The density curves show the distributions of maximum mean discrepancy (MMD) witness scores computed with an RBF kernel from the 512-dimensional encoder embeddings. A witness score reflects an embedding’s relative similarity between the DFEW and DISFA distributions. Dashed lines indicate the domain-wise means. Compared with STL ( 0.2656 ) and the joint Baseline ( 0.1764 ), SSM reduces the value of MMD2 to 0.1459 , corresponding to a 17.3% relative reduction from Baseline and indicating a smaller cross-domain representation discrepancy.
Fig. S9: Comparison of the AU and FE label spaces in Baseline (left) and SSM (right), using the DFEW and DISFA datasets. A matrix entry gives the cosine similarity between two label representations. Baseline uses independent one-hot codes, whereas SSM uses the textual semantic codes.
Facial Action Unit (AU) detection seeks to recognize subtle facial muscle activations as defined by the Facial Action Coding System (FACS). A primary challenge w.r.t AU detection is the effective learning of discriminative and generalizable AU representations under conditions of limited annotated data. To address this, we propose a Hierarchical Vision-language Interaction for AU Understanding (HiVA) method, which leverages textual AU descriptions as semantic priors to guide and enhance AU detection. Specifically, HiVA employs a large language model to generate diverse and contextually rich AU descriptions to strengthen language-based representation learning. To capture both fine-grained and holistic vision-language associations, HiVA introduces an AU-aware dynamic graph module that facilitates the learning of AU-specific visual representations. These features are further integrated within a hierarchical cross-modal attention architecture comprising two complementary mechanisms: Disentangled Dual Cross-Attention (DDCA), which establishes fine-grained, AU-specific interactions between visual and textual features, and Contextual Dual Cross-Attention (CDCA), which models global inter-AU dependencies. This collaborative, cross-modal learning paradigm enables HiVA to leverage multi-grained vision-based AU features in conjunction with refined language-based AU details, culminating in robust and semantically enriched AU detection capabilities. Extensive experiments show that HiVA consistently surpasses state-of-the-art approaches. Besides, qualitative analyses reveal that HiVA produces semantically meaningful activation patterns, highlighting its efficacy in learning robust and interpretable cross-modal correspondences for comprehensive facial behavior analysis.
Yong Li, Yi Ren, Yizhe Zhang +5
the School of Computer Science and Engineering, and the Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University, Nanjing 210096, China. · the School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, 210094, China. · Key Laboratory of Child Development and Learning Science (Ministry of Education), School of Biological Sciences and Medical Engineering, Southeast University, Nanjing, China. +1
Micro-expression Action Unit (AU) detection identifies localized AUs from subtle facial muscle activations, providing a foundation for decoding affective cues. Previous methods face three key limitations: (1) heavy reliance on low-density visual information, rendering discriminative evidence vulnerable to background noise; (2) coarse-grained feature processing that misaligns with the demand for fine-grained representations; and (3) neglect of inter-AU correlations, restricting the parsing of complex expression patterns. We propose AULLM++, a reasoning-oriented framework leveraging Large Language Models (LLMs), which injects visual features into textual prompts as actionable semantic premises to guide inference. It formulates AU prediction into three stages: evidence construction, structure modeling, and deduction-based prediction. Specifically, a Multi-Granularity Evidence-Enhanced Fusion Projector (MGE-EFP) fuses mid-level texture cues with high-level semantics, distilling them into a compact Content Token (CT). Furthermore, inspired by micro- and macro-expression AU correspondence, we encode AU relationships as a sparse structural prior and learn interaction strengths via a Relation-Aware AU Graph Neural Network (R-AUGNN), producing an Instruction Token (IT). We then fuse CT and IT into a structured textual prompt and introduce Counterfactual Consistency Regularization (CCR) to construct counterfactual samples, enhancing the model's generalization. Extensive experiments demonstrate AULLM++ achieves state-of-the-art performance on standard benchmarks and exhibits superior cross-domain generalization.
Zhishu Liu, Kaishen Yuan, Bo Zhao +2
Great Bay University, Dongguan, China · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Shenzhen International Graduate School, Tsinghua University, Shenzhen, China
Recently, cross-domain few-shot facial expression recognition (CF-FER) has received considerable attention. However, the performance of existing CF-FER methods is still unsatisfactory due to inferior transferable feature learning under large domain discrepancy and limited target samples. Fortunately, the action units (AUs), which indicate the movements of different facial muscles, provide consistent conceptual semantics for describing expressions within and across domains. Inspired by this, we propose a novel Action Unit-based Consistency-aware Hypergraph Network (AUCH-Net), which constructs consistency-aware hypergraphs on AUs, for CF-FER. Specifically, AUCH-Net presents a new AU feature learning (AFL) module and a new visual feature learning (VFL) module. The AFL module learns AU features under the guidance of a novel relation consistency loss and an AU regularization loss, while the VFL module learns visual features supervised by a relation consistency loss and a classification loss. By learning consistent AU features, AUCH-Net effectively models the connections between AUs and expression categories. As a result, we can bridge the gap between fine-grained facial variations and high-level expression categories, greatly facilitating the learning of transferable feature representations.Extensive experiments on both in-the-lab and in-the-wild datasets show that our method consistently outperforms several state-of-the-art methods. Our results clearly show that modeling the relationships among AUs holds significant potential for FER under cross-domain few-shot scenarios.
Xinhan Qiu, Yan Yan, Rui Zhu +2
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, School of Informatics, Xiamen University, Xiamen 361102, China · Fujian Key Laboratory of Pattern Recognition and Image Understanding, Xiamen University of Technology, Xiamen 361024, China · Bayes Business School, City St George’s, University of London, London, EC1Y 8TZ, UK +1