Organizations: the School of Computer Science and Engineering, and the Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University, Nanjing 210096, China. · the School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, 210094, China. · Key Laboratory of Child Development and Learning Science (Ministry of Education), School of Biological Sciences and Medical Engineering, Southeast University, Nanjing, China. · the School of Computer Science and Engineering, Nanyang Technological University, 50 Nanyang Avenue, Singapore, 639798.
Facial Action Unit (AU) detection seeks to recognize subtle facial muscle activations as defined by the Facial Action Coding System (FACS). A primary challenge w.r.t AU detection is the effective learning of discriminative and generalizable AU representations under conditions of limited annotated data. To address this, we propose a Hierarchical Vision-language Interaction for AU Understanding (HiVA) method, which leverages textual AU descriptions as semantic priors to guide and enhance AU detection. Specifically, HiVA employs a large language model to generate diverse and contextually rich AU descriptions to strengthen language-based representation learning. To capture both fine-grained and holistic vision-language associations, HiVA introduces an AU-aware dynamic graph module that facilitates the learning of AU-specific visual representations. These features are further integrated within a hierarchical cross-modal attention architecture comprising two complementary mechanisms: Disentangled Dual Cross-Attention (DDCA), which establishes fine-grained, AU-specific interactions between visual and textual features, and Contextual Dual Cross-Attention (CDCA), which models global inter-AU dependencies. This collaborative, cross-modal learning paradigm enables HiVA to leverage multi-grained vision-based AU features in conjunction with refined language-based AU details, culminating in robust and semantically enriched AU detection capabilities. Extensive experiments show that HiVA consistently surpasses state-of-the-art approaches. Besides, qualitative analyses reveal that HiVA produces semantically meaningful activation patterns, highlighting its efficacy in learning robust and interpretable cross-modal correspondences for comprehensive facial behavior analysis.
Figures & tables
Fig. 1: Overview of the proposed Hierarchical Vision-Language Interaction for Facial AU detection. (a) Illustrates the process of leveraging a large language model (e.g., GPT-4) to generate diverse and semantically rich AU descriptions, addressing the limitations of scarce textual knowledge. (b) depicts the core framework, which integrates visual features from input images and textual features from AU descriptions through hierarchical vision-language interaction, utilizing both local and global attention mechanisms for robust AU detection.
Fig. 2: Framework of the proposed Hierarchical Vision-Language Attention for AU Understanding (HiVA). HiVA consists of four main components: (a) AU-Oriented Visual Feature Encoding, which extracts AU-specific visual representations using a CNN and transformer backbone; (b) Language-Based AU Description Modeling, which encodes textual AU descriptions to capture both intra- and inter-AU semantics; (c) AU-Aware Dynamic Graph Construction, which builds AU-specific graphs based on visual similarity to model adaptive inter-AU relations; and (d) Hierarchical Vision-Language Interaction, which employs Disentangled Dual Cross-Attention (DDCA) and Contextual Dual Cross-Attention (CDCA) to interact visual and textual features at both local and global levels.
Fig. 3: Illustration of the Hierarchical Vision-Language Interaction module in HiVA. This component integrates two complementary attention mechanisms: Disentangled Dual Cross-Attention (DDCA) for fine-grained one-to-one interaction between AU-specific visual features and corresponding textual descriptions, and Contextual Dual Cross-Attention (CDCA) between global visual features and all AU descriptions. This bidirectional, cross-modal interaction enables the model to capture both localized semantic grounding and global inter-AU dependencies for robust AU detection.
Methods/AU
1
2
4
6
7
10
12
14
15
17
23
24
AVE
EAC-Net [ 7 ]
39.0
35.2
48.6
76.1
72.9
81.9
86.2
58.8
37.5
59.1
35.9
35.8
55.9
ATF [ 17 ]
39.2
35.2
45.9
71.6
71.9
79.0
83.7
65.5
33.8
60.0
37.3
41.8
55.4
IdenNet [ 18 ]
50.5
35.9
50.6
77.2
74.2
82.9
85.1
65.3
42.2
60.8
42.1
46.5
59.3
DSIN [ 9 ]
51.7
40.4
56.0
76.1
73.5
79.9
85.4
62.7
37.3
62.9
38.8
41.6
58.9
JAA-Net [ 10 ]
47.2
44.0
54.9
77.5
74.6
84.0
86.9
61.9
43.6
60.3
42.7
41.9
60.0
ARL [ 26 ]
45.8
39.8
55.1
75.7
77.2
82.3
86.6
58.8
47.6
62.1
47.4
55.4
61.1
TABLE I: F1 scores on BP4D dataset. Bold and underline denote the best and second best results, respectively.
Methods/AU
1
2
4
6
9
12
25
26
AVE
EAC-Net [ 7 ]
41.5
26.4
66.4
50.7
80.5
89.3
88.9
15.6
48.5
ATF [ 17 ]
45.2
39.7
47.1
48.6
32.0
55.0
86.4
39.2
49.2
IdenNet [ 18 ]
25.5
34.8
64.5
45.2
44.6
70.7
81.0
55.0
52.6
DSIN [ 9 ]
42.4
39.0
68.4
28.6
46.8
70.8
90.4
42.2
53.6
JAA-Net [ 10 ]
43.7
46.2
56.0
41.4
44.7
69.6
88.3
58.4
56.0
ARL [ 26 ]
43.9
42.1
63.6
41.8
40.0
76.2
95.2
66.8
58.7
TABLE II: F1 scores on DISFA dataset. Bold and underline denote the best and second best results, respectively.
Methods/AU
1
2
4
6
10
12
14
15
23
24
AVE
AlexNet [ 38 ]
44
46
2
73
72
82
5
19
43
42
42.8
EAC-Net [ 7 ]
15.5
56.6
0.1
81.0
76.1
84.0
0.1
38.5
57.8
51.2
46.1
ARL [ 26 ]
51.9
45.9
13.7
79.2
75.5
82.8
0.1
44.9
59.2
47.5
50.1
Ertugrul et al. [ 39 ]
43.7
44.9
19.8
74.6
76.5
79.8
50.0
33.9
16.8
12.9
45.3
AAR [ 35 ]
66.3
53.9
23.7
81.5
73.6
84.2
43.8
53.8
58.2
46.5
58.5
MAL [ 21 ]
52.4
57.0
54.1
74.5
78.0
84.9
43.1
47.7
54.4
51.9
59.8
TABLE III: F1 score on GFT dataset. Bold and underline denote the best and second best results, respectively.
Fig. 4: Comparison of per-AU F1 score improvements between the proposed HiVA and its vision-only baseline. HiVA consistently enhances AU detection performance, particularly for infrequently activated AUs. This trend highlights the effectiveness of language-based AU descriptions in improving feature robustness and modeling inter-AU dependencies via dual cross-modal attention mechanisms.
Methods/AU
AU1
AU2
AU4
AU6
AU7
AU10
AU12
AU14
AU15
AU17
AU23
AU24
AVE
HiVA (w/o AUG, w/o Ldiff )
50.9
47.5
62.1
78.2
78.8
83.6
87.9
66.0
52.6
61.5
48.0
53.0
64.2
HiVA (w/o AUG)
52.6
49.0
62.9
78.3
78.3
83.6
88.6
65.8
54.9
61.1
49.7
53.3
64.9
HiVA (w/o Ldiff )
53.2
48.1
63.0
78.2
79.2
83.7
88.5
67.1
54.9
61.5
50.8
55.2
65.4
HiVA (Ours)
54.1
49.7
63.3
79.3
79.8
84.5
88.8
68.5
57.0
62.6
53.1
56.8
66.5
VLU
48.1
44.3
61.3
77.9
78.9
83.3
88.4
65.1
51.5
61.6
45.6
51.6
63.1
HiVA (w/o DDCA)
51.4
48.8
63.0
78.5
79.2
83.4
88.4
65.8
54.8
60.4
50.2
53.4
64.8
TABLE IV: Ablation study on BP4D dataset. Bold and underline denote the best and second best results, respectively.
Methods/AU
AU1
AU2
AU4
AU6
AU9
AU12
AU25
AU26
AVE
HiVA (w/o AUG, w/o Ldiff )
47.4
49.7
73.7
51.0
54.8
73.5
95.5
67.8
64.2
HiVA (w/o AUG)
51.2
51.1
73.1
50.6
53.6
76.7
94.8
65.6
64.6
HiVA (w/o Ldiff )
45.9
46.6
78.5
53.9
59.2
72.7
95.1
66.0
64.6
HiVA (Ours)
60.6
58.4
75.4
51.0
61.2
74.8
93.9
63.8
67.4
Baseline
43.5
45.8
71.1
45.6
53.8
70.1
93.4
60.0
60.4
HiVA (w/o DDCA)
52.4
50.8
72.0
50.1
54.3
72.7
95.0
67.3
64.3
TABLE V: Ablation study on DISFA dataset. Bold and underline denote the best and second best results, respectively.
Methods/AU
AU1
AU2
AU4
AU6
AU10
AU12
AU14
AU15
AU23
AU24
AVE
HiVA (w/o AUG, w/o Ldiff )
50.8
57.1
44.8
80.2
75.9
77.3
39.3
45.3
55.0
50.6
57.6
HiVA (w/o AUG)
50.9
56.8
47.5
81.3
76.2
78.0
40.7
46.4
56.6
51.2
58.6
HiVA (w/o Ldiff )
51.0
58.7
54.7
80.4
77.0
80.1
38.0
48.4
58.9
47.7
59.5
HiVA (Ours)
56.5
59.6
52.4
82.6
77.7
80.2
40.8
47.7
59.3
49.9
60.7
Baseline
51.1
53.5
34.0
81.0
74.6
77.7
37.9
41.6
48.7
48.8
54.9
HiVA (w/o CDCA, w/o DDCA)
50.6
57.3
36.7
83.3
75.7
78.5
37.3
45.5
55.8
49.7
57.0
TABLE VI: Ablation study on GFT dataset. Bold and underline denote the best and second best results, respectively.
Fig. 5: Visualization of the cosine similarity matrices among textual AU embeddings at different training epochs for HiVA (w/o Ldiff ) (top) and HiVA (bottom). HiVA progressively learns more discriminative and diverse AU embeddings, while the variant without Ldiff shows persistently high similarity across different AUs, indicating limited semantic differentiation.
Fig. 6: Visualization of the learned cross-modality attention maps for several AUs (as examples) of different subjects on BP4D (a), DISFA (b), and GFT (c) datasets. The first three attention maps for each example are derived from DDCA. The final composite attention map is generated by CDCA.
Fig. 7: Visualization of the learned graph structures on the BP4D (a), DISFA (b), and GFT (c) datasets. Each node is connected to its K nearest neighbors, with nodes corresponding to activated AUs typically exhibiting a higher number of connections compared to those associated with inactivated AUs. HiVA-generated graphs tend to show more structured and meaningful inter-AU connections.
Facial Action Unit (AU) detection in in-the-wild environments remains a formidable challenge due to severe spatial-temporal heterogeneity, unconstrained poses, and complex audio-visual dependencies. While recent multimodal approaches have made progress, they often rely on capacity-limited encoders and shallow fusion mechanisms that fail to capture fine-grained semantic shifts and ultra-long temporal contexts. To bridge this gap, we propose a novel multimodal framework driven by Hierarchical Granularity Alignment and State Space Models.Specifically, we leverage powerful foundation models, namely DINOv2 and WavLM, to extract robust and high-fidelity visual and audio representations, effectively replacing traditional feature extractors. To handle extreme facial variations, our Hierarchical Granularity Alignment module dynamically aligns global facial semantics with fine-grained local active patches. Furthermore, we overcome the receptive field limitations of conventional temporal convolutional networks by introducing a Vision-Mamba architecture. This approach enables temporal modeling with O(N) linear complexity, effectively capturing ultra-long-range dynamics without performance degradation. A novel asymmetric cross-attention mechanism is also introduced to deeply synchronize paralinguistic audio cues with subtle visual movements.Extensive experiments on the challenging Aff-Wild2 dataset demonstrate that our approach significantly outperforms existing baselines, achieving state-of-the-art performance. Notably, this framework secured top rankings in the AU Detection track of the 10th Affective Behavior Analysis in-the-wild Competition.
Micro-expression Action Unit (AU) detection identifies localized AUs from subtle facial muscle activations, providing a foundation for decoding affective cues. Previous methods face three key limitations: (1) heavy reliance on low-density visual information, rendering discriminative evidence vulnerable to background noise; (2) coarse-grained feature processing that misaligns with the demand for fine-grained representations; and (3) neglect of inter-AU correlations, restricting the parsing of complex expression patterns. We propose AULLM++, a reasoning-oriented framework leveraging Large Language Models (LLMs), which injects visual features into textual prompts as actionable semantic premises to guide inference. It formulates AU prediction into three stages: evidence construction, structure modeling, and deduction-based prediction. Specifically, a Multi-Granularity Evidence-Enhanced Fusion Projector (MGE-EFP) fuses mid-level texture cues with high-level semantics, distilling them into a compact Content Token (CT). Furthermore, inspired by micro- and macro-expression AU correspondence, we encode AU relationships as a sparse structural prior and learn interaction strengths via a Relation-Aware AU Graph Neural Network (R-AUGNN), producing an Instruction Token (IT). We then fuse CT and IT into a structured textual prompt and introduce Counterfactual Consistency Regularization (CCR) to construct counterfactual samples, enhancing the model's generalization. Extensive experiments demonstrate AULLM++ achieves state-of-the-art performance on standard benchmarks and exhibits superior cross-domain generalization.
Zhishu Liu, Kaishen Yuan, Bo Zhao +2
Great Bay University, Dongguan, China · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Shenzhen International Graduate School, Tsinghua University, Shenzhen, China
Facial action unit (AU) detection remains challenging because it involves heterogeneous, AU-specific uncertainties arising at both the representation and decision stages. Recent methods have improved discriminative feature learning, but they often treat the AU representations as deterministic, overlooking uncertainty caused by visual noise, subject-dependent appearance variations, and ambiguous inter-AU relationships, all of which can substantially degrade robustness. Meanwhile, conventional point-estimation classifiers often provide poorly calibrated confidence, producing overconfident predictions, especially under the severe label imbalance typical of AU datasets. We propose UAU-Net, an Uncertainty-aware AU detection framework that explicitly models uncertainty at both stages. At the representation stage, we introduce CV-AFE, a conditional VAE (CVAE)-based AU feature extraction module that learns probabilistic AU representations by jointly estimating feature means and variances across multiple spatio-temporal scales; conditioning on AU labels further enables CV-AFE to capture uncertainty associated with inter-AU dependencies. At the decision stage, we design AB-ENN, an Asymmetric Beta Evidential Neural Network for multi-label AU detection, which parameterizes predictive uncertainty with Beta distributions and mitigates overconfidence via an asymmetric loss tailored to highly imbalanced binary labels. Extensive experiments on BP4D and DISFA show that UAU-Net achieves strong AU detection performance, and further analyses indicate that modeling uncertainty in both representation learning and evidential prediction improves robustness and reliability.
Yuze Li, Zhilei Liu
School of Artificial Intelligence Tianjin University Tianjin, China