Organizations: the School of Computer Science and Engineering, and the Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University, Nanjing 210096, China. · the School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, 210094, China. · Key Laboratory of Child Development and Learning Science (Ministry of Education), School of Biological Sciences and Medical Engineering, Southeast University, Nanjing, China. · the School of Computer Science and Engineering, Nanyang Technological University, 50 Nanyang Avenue, Singapore, 639798.
Facial Action Unit (AU) detection seeks to recognize subtle facial muscle activations as defined by the Facial Action Coding System (FACS). A primary challenge w.r.t AU detection is the effective learning of discriminative and generalizable AU representations under conditions of limited annotated data. To address this, we propose a Hierarchical Vision-language Interaction for AU Understanding (HiVA) method, which leverages textual AU descriptions as semantic priors to guide and enhance AU detection. Specifically, HiVA employs a large language model to generate diverse and contextually rich AU descriptions to strengthen language-based representation learning. To capture both fine-grained and holistic vision-language associations, HiVA introduces an AU-aware dynamic graph module that facilitates the learning of AU-specific visual representations. These features are further integrated within a hierarchical cross-modal attention architecture comprising two complementary mechanisms: Disentangled Dual Cross-Attention (DDCA), which establishes fine-grained, AU-specific interactions between visual and textual features, and Contextual Dual Cross-Attention (CDCA), which models global inter-AU dependencies. This collaborative, cross-modal learning paradigm enables HiVA to leverage multi-grained vision-based AU features in conjunction with refined language-based AU details, culminating in robust and semantically enriched AU detection capabilities. Extensive experiments show that HiVA consistently surpasses state-of-the-art approaches. Besides, qualitative analyses reveal that HiVA produces semantically meaningful activation patterns, highlighting its efficacy in learning robust and interpretable cross-modal correspondences for comprehensive facial behavior analysis.
Figures & tables
Fig. 1: Overview of the proposed Hierarchical Vision-Language Interaction for Facial AU detection. (a) Illustrates the process of leveraging a large language model (e.g., GPT-4) to generate diverse and semantically rich AU descriptions, addressing the limitations of scarce textual knowledge. (b) depicts the core framework, which integrates visual features from input images and textual features from AU descriptions through hierarchical vision-language interaction, utilizing both local and global attention mechanisms for robust AU detection.
Fig. 2: Framework of the proposed Hierarchical Vision-Language Attention for AU Understanding (HiVA). HiVA consists of four main components: (a) AU-Oriented Visual Feature Encoding, which extracts AU-specific visual representations using a CNN and transformer backbone; (b) Language-Based AU Description Modeling, which encodes textual AU descriptions to capture both intra- and inter-AU semantics; (c) AU-Aware Dynamic Graph Construction, which builds AU-specific graphs based on visual similarity to model adaptive inter-AU relations; and (d) Hierarchical Vision-Language Interaction, which employs Disentangled Dual Cross-Attention (DDCA) and Contextual Dual Cross-Attention (CDCA) to interact visual and textual features at both local and global levels.
Fig. 3: Illustration of the Hierarchical Vision-Language Interaction module in HiVA. This component integrates two complementary attention mechanisms: Disentangled Dual Cross-Attention (DDCA) for fine-grained one-to-one interaction between AU-specific visual features and corresponding textual descriptions, and Contextual Dual Cross-Attention (CDCA) between global visual features and all AU descriptions. This bidirectional, cross-modal interaction enables the model to capture both localized semantic grounding and global inter-AU dependencies for robust AU detection.
Methods/AU
1
2
4
6
7
10
12
14
15
17
23
24
AVE
EAC-Net [ 7 ]
39.0
35.2
48.6
76.1
72.9
81.9
86.2
58.8
37.5
59.1
35.9
35.8
55.9
ATF [ 17 ]
39.2
35.2
45.9
71.6
71.9
79.0
83.7
65.5
33.8
60.0
37.3
41.8
55.4
IdenNet [ 18 ]
50.5
35.9
50.6
77.2
74.2
82.9
85.1
65.3
42.2
60.8
42.1
46.5
59.3
DSIN [ 9 ]
51.7
40.4
56.0
76.1
73.5
79.9
85.4
62.7
37.3
62.9
38.8
41.6
58.9
JAA-Net [ 10 ]
47.2
44.0
54.9
77.5
74.6
84.0
86.9
61.9
43.6
60.3
42.7
41.9
60.0
ARL [ 26 ]
45.8
39.8
55.1
75.7
77.2
82.3
86.6
58.8
47.6
62.1
47.4
55.4
61.1
TABLE I: F1 scores on BP4D dataset. Bold and underline denote the best and second best results, respectively.
Methods/AU
1
2
4
6
9
12
25
26
AVE
EAC-Net [ 7 ]
41.5
26.4
66.4
50.7
80.5
89.3
88.9
15.6
48.5
ATF [ 17 ]
45.2
39.7
47.1
48.6
32.0
55.0
86.4
39.2
49.2
IdenNet [ 18 ]
25.5
34.8
64.5
45.2
44.6
70.7
81.0
55.0
52.6
DSIN [ 9 ]
42.4
39.0
68.4
28.6
46.8
70.8
90.4
42.2
53.6
JAA-Net [ 10 ]
43.7
46.2
56.0
41.4
44.7
69.6
88.3
58.4
56.0
ARL [ 26 ]
43.9
42.1
63.6
41.8
40.0
76.2
95.2
66.8
58.7
TABLE II: F1 scores on DISFA dataset. Bold and underline denote the best and second best results, respectively.
Methods/AU
1
2
4
6
10
12
14
15
23
24
AVE
AlexNet [ 38 ]
44
46
2
73
72
82
5
19
43
42
42.8
EAC-Net [ 7 ]
15.5
56.6
0.1
81.0
76.1
84.0
0.1
38.5
57.8
51.2
46.1
ARL [ 26 ]
51.9
45.9
13.7
79.2
75.5
82.8
0.1
44.9
59.2
47.5
50.1
Ertugrul et al. [ 39 ]
43.7
44.9
19.8
74.6
76.5
79.8
50.0
33.9
16.8
12.9
45.3
AAR [ 35 ]
66.3
53.9
23.7
81.5
73.6
84.2
43.8
53.8
58.2
46.5
58.5
MAL [ 21 ]
52.4
57.0
54.1
74.5
78.0
84.9
43.1
47.7
54.4
51.9
59.8
TABLE III: F1 score on GFT dataset. Bold and underline denote the best and second best results, respectively.
Fig. 4: Comparison of per-AU F1 score improvements between the proposed HiVA and its vision-only baseline. HiVA consistently enhances AU detection performance, particularly for infrequently activated AUs. This trend highlights the effectiveness of language-based AU descriptions in improving feature robustness and modeling inter-AU dependencies via dual cross-modal attention mechanisms.
Methods/AU
AU1
AU2
AU4
AU6
AU7
AU10
AU12
AU14
AU15
AU17
AU23
AU24
AVE
HiVA (w/o AUG, w/o Ldiff )
50.9
47.5
62.1
78.2
78.8
83.6
87.9
66.0
52.6
61.5
48.0
53.0
64.2
HiVA (w/o AUG)
52.6
49.0
62.9
78.3
78.3
83.6
88.6
65.8
54.9
61.1
49.7
53.3
64.9
HiVA (w/o Ldiff )
53.2
48.1
63.0
78.2
79.2
83.7
88.5
67.1
54.9
61.5
50.8
55.2
65.4
HiVA (Ours)
54.1
49.7
63.3
79.3
79.8
84.5
88.8
68.5
57.0
62.6
53.1
56.8
66.5
VLU
48.1
44.3
61.3
77.9
78.9
83.3
88.4
65.1
51.5
61.6
45.6
51.6
63.1
HiVA (w/o DDCA)
51.4
48.8
63.0
78.5
79.2
83.4
88.4
65.8
54.8
60.4
50.2
53.4
64.8
TABLE IV: Ablation study on BP4D dataset. Bold and underline denote the best and second best results, respectively.
Methods/AU
AU1
AU2
AU4
AU6
AU9
AU12
AU25
AU26
AVE
HiVA (w/o AUG, w/o Ldiff )
47.4
49.7
73.7
51.0
54.8
73.5
95.5
67.8
64.2
HiVA (w/o AUG)
51.2
51.1
73.1
50.6
53.6
76.7
94.8
65.6
64.6
HiVA (w/o Ldiff )
45.9
46.6
78.5
53.9
59.2
72.7
95.1
66.0
64.6
HiVA (Ours)
60.6
58.4
75.4
51.0
61.2
74.8
93.9
63.8
67.4
Baseline
43.5
45.8
71.1
45.6
53.8
70.1
93.4
60.0
60.4
HiVA (w/o DDCA)
52.4
50.8
72.0
50.1
54.3
72.7
95.0
67.3
64.3
TABLE V: Ablation study on DISFA dataset. Bold and underline denote the best and second best results, respectively.
Methods/AU
AU1
AU2
AU4
AU6
AU10
AU12
AU14
AU15
AU23
AU24
AVE
HiVA (w/o AUG, w/o Ldiff )
50.8
57.1
44.8
80.2
75.9
77.3
39.3
45.3
55.0
50.6
57.6
HiVA (w/o AUG)
50.9
56.8
47.5
81.3
76.2
78.0
40.7
46.4
56.6
51.2
58.6
HiVA (w/o Ldiff )
51.0
58.7
54.7
80.4
77.0
80.1
38.0
48.4
58.9
47.7
59.5
HiVA (Ours)
56.5
59.6
52.4
82.6
77.7
80.2
40.8
47.7
59.3
49.9
60.7
Baseline
51.1
53.5
34.0
81.0
74.6
77.7
37.9
41.6
48.7
48.8
54.9
HiVA (w/o CDCA, w/o DDCA)
50.6
57.3
36.7
83.3
75.7
78.5
37.3
45.5
55.8
49.7
57.0
TABLE VI: Ablation study on GFT dataset. Bold and underline denote the best and second best results, respectively.
Fig. 5: Visualization of the cosine similarity matrices among textual AU embeddings at different training epochs for HiVA (w/o Ldiff ) (top) and HiVA (bottom). HiVA progressively learns more discriminative and diverse AU embeddings, while the variant without Ldiff shows persistently high similarity across different AUs, indicating limited semantic differentiation.
Fig. 6: Visualization of the learned cross-modality attention maps for several AUs (as examples) of different subjects on BP4D (a), DISFA (b), and GFT (c) datasets. The first three attention maps for each example are derived from DDCA. The final composite attention map is generated by CDCA.
Fig. 7: Visualization of the learned graph structures on the BP4D (a), DISFA (b), and GFT (c) datasets. Each node is connected to its K nearest neighbors, with nodes corresponding to activated AUs typically exhibiting a higher number of connections compared to those associated with inactivated AUs. HiVA-generated graphs tend to show more structured and meaningful inter-AU connections.
Great Bay University, Dongguan, China · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Shenzhen International Graduate School, Tsinghua University, Shenzhen, China