This paper addresses the expression (EXPR) recognition challenge in the 10th Affective Behavior Analysis in-the-Wild (ABAW) workshop and competition, which requires frame-level classification of eight facial emotional expressions from unconstrained videos. This task is challenging due to inaccurate face localization, large pose and scale variations, motion blur, temporal instability, and other confounding factors across adjacent frames. We propose a two-stage dual-modal (audio-visual) model to address these difficulties. Stage I focuses on robust visual feature extraction with a pretrained DINOv2-based encoder. Specifically, DINOv2 ViT-L/14 is used as the backbone, a padding-aware augmentation (PadAug) strategy is employed for image padding and data preprocessing from raw videos, and a mixture-of-experts (MoE) training head is introduced to enhance classifier diversity. Stage II addresses modality fusion and temporal consistency. For the visual modality, faces are re-cropped from raw videos at multiple scales, and the extracted visual features are averaged to form a robust frame-level representation. Concurrently, frame-aligned Wav2Vec 2.0 audio features are derived from short audio windows to provide complementary acoustic cues. These dual-modal features are integrated via a lightweight gated fusion module, followed by inference-time temporal smoothing. Experiments on the ABAW dataset demonstrate the effectiveness of the proposed method. The two-stage model achieves a Macro-F1 score of 0.5368 on the official validation set and 0.5122 +/- 0.0277 under 5-fold cross-validation, outperforming the official baselines.
Figures & tables
Figure 1: Overview of the proposed model. For the video modality, three facial crops at different scales for each target frame are extracted from the raw video and encoded by an adapted DINOv2 backbone. For the audio modality, a Wav2Vec 2.0 encoder is used to extract window-based features aligned with the visual frame. The resulting multi-scale visual representation is fused with the audio features through a gated fusion module for frame-level facial expression recognition. During inference, lightweight temporal smoothing is further applied to improve prediction stability and temporal consistency.
Figure 2: The visual adaptation pipeline in Stage-I. The DINOv2-based encoder is adapted on AffectNet and RAF-DB. PadAug simulates boundary padding artifacts caused by large facial crops, while a training-only MoE head provides sample-dependent expert routing to enhance DINOv2 adaptation. After Stage I, the MoE head is discarded and only the adapted DINOv2 backbone is retained.
Figure 3: Illustration of the proposed padding-aware augmentation (PadAug). The top row shows original facial crops, while the bottom row presents augmented samples where artificial padding is applied to different image boundaries to simulate boundary artifacts caused by imperfect face cropping in raw videos.
Teams
Total Score
Ours
0.391
HSEmotion
0.386
USTC-IAT-United
0.360
IMLAB
0.320
baseline
0.225
Table 1: Official challenge server results on the EXPR Recognition Challenge.
Training Method
Official Face Crop + Linear
FaceCrop-0.9 + Linear
FaceCrop-1.2 + Linear
FaceCrop-1.5 + Linear
Multi-scale Mean (0.9/1.2/1.5) + Linear
Pretrained
0.3007
0.3199
0.3307
0.3192
0.3355
Fine-tune + MLP
0.3378
0.3697
0.3773
0.3738
0.3800
Fine-tune + MoE
0.3576
0.4040
0.4251
0.4223
0.4257
Fine-tune + MoE + PadAug
0.3525
0.3992
0.4205
0.4245
0.4344
Table 2: Ablation on face crop scale and visual encoder training strategy. Results are reported as F1 on the official validation set using a linear classifier on top of the extracted frozen visual features.
Alignment Strategy
F1
Nearest
0.2423
Window Mean ( 0.25 s)
0.2876
Window Mean ( 0.50 s)
0.2901
Window Mean ( 0.75 s)
0.2886
Table 3: Ablation on audio-frame alignment strategy. Results are reported as F1 on the official validation set.
Fusion
Params
Official Val F1
5-Fold Mean F1
Fold-1
Fold-2
Fold-3
Fold-4
Fold-5
A only
–
0.2901
–
–
–
–
–
–
V only
–
0.4344
–
–
–
–
–
–
Concat + Linear
16K
0.4717
0.4527
0.4527
0.4203
0.4629
0.4946
0.4331
Concat + MLP
1.05M
0.5079
0.4838
0.4670
0.4725
0.4983
0.5340
0.4474
Bilinear
1.32M
0.4794
0.4502
0.4312
0.4341
0.4670
0.4714
0.4472
Dynamic Weighting
2.10M
0.4712
0.4558
0.4466
0.4442
0.4651
0.4672
0.4557
Table 4: Comparison of different audio–visual fusion strategies. Results are reported as F1 on the official validation set and under 5-fold cross-validation.
Figure 4: Macro-averaged F1 score under different temporal smoothing strategies and window sizes.
Strategy
Official Val F1
5-Fold Mean F1 ± Std
Single-frame Baseline
0.5131
0.4842±0.0212
Mean Smoothing
0.5335
0.5114±0.0303
Median Filter
0.5368
0.5122±0.0277
Gaussian Weighting
0.5305
0.5098±0.0273
Hard Voting
0.5364
0.5095±0.0258
Table 5: Temporal smoothing results with a fixed window size of 101. Results are reported as F1 on the official validation set and under 5-fold cross-validation.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Module
Input setting
Output shape
Parameters
FLOPs
DINOv2 ViT-L/14 visual encoder
face crop, 224×224
1×1024
303,227,904
81.437G
Wav2Vec 2.0 large-lv60k audio encoder
20-s audio, 16 kHz
999×1024
315,470,496
413.011G
Lightweight gated fusion head
visual: 1×1024 ; audio: 1×1024
1×8
2,366,472
2.371M
Appendix
Table A1: Complexity statistics of the main modules used in our final system. FLOPs are reported under single-sample inference. For the visual encoder, the input is a face crop of size 224×224 . For the audio encoder, the input is a 20-second waveform sampled at 16 kHz.
Figure A1 : Detailed structure of the lightweight gated fusion head used in Stage II. The 1024-dimensional visual and audio features are first projected into a 512-dimensional space. A sigmoid gate is then predicted from the concatenated bimodal input to perform element-wise adaptive fusion, followed by a linear classifier for frame-level expression recognition.
Expression recognition in in-the-wild video data remains challenging due to substantial variations in facial appearance, background conditions, audio noise, and the inherently dynamic nature of human affect. Relying on a single modality, such as facial expressions or speech, is often insufficient for capturing these complex emotional cues. To address this limitation, we propose a multimodal emotion recognition framework for the Expression (EXPR) task in the 10th Affective Behavior Analysis in-the-wild (ABAW) Challenge. Our framework builds on large-scale pre-trained models for visual and audio representation learning and integrates them in a unified multimodal architecture. To better capture temporal patterns in facial expression sequences, we incorporate temporal visual modeling over video windows. We further introduce a bi-directional cross-attention fusion module that enables visual and audio features to interact in a symmetric manner, facilitating cross-modal contextualization and complementary emotion understanding. In addition, we employ a text-guided contrastive objective to encourage semantically meaningful visual representations through alignment with emotion-related text prompts. Experimental results on the ABAW 10th EXPR benchmark demonstrate the effectiveness of the proposed framework, achieving a Macro F1 score of 0.32 compared to the baseline score of 0.25, and highlight the benefit of combining temporal visual modeling, audio representation learning, and cross-modal fusion for robust emotion recognition in unconstrained real-world environments.
Emotion recognition in real-world environments is hindered by partial occlusions, missing modalities, and severe class imbalance. To address these issues, particularly for the Affective Behavior Analysis in-the-wild (ABAW) Expression challenge, we propose a multimodal framework that dynamically fuses visual and audio representations. Our approach uses a dual-branch Transformer architecture featuring a safe cross-attention mechanism and a modality dropout strategy. This design allows the network to rely on audio-based predictions when visual cues are absent. To mitigate the long-tail distribution of the Aff-Wild2 dataset, we apply focal loss optimization, combined with a sliding-window soft voting strategy to capture dynamic emotional transitions and reduce frame-level classification jitter. Experiments demonstrate that our framework effectively handles missing modalities and complex spatiotemporal dependencies, achieving an accuracy of 60.79% and an F1-score of 0.5029 on the Aff-Wild2 validation set.
Jun Yu, Naixiang Zheng, Guoyuan Wang +5
University of Science and Technology of China · Unisound AI Technology Co., Ltd.
Facial Action Unit (AU) detection in in-the-wild environments remains a formidable challenge due to severe spatial-temporal heterogeneity, unconstrained poses, and complex audio-visual dependencies. While recent multimodal approaches have made progress, they often rely on capacity-limited encoders and shallow fusion mechanisms that fail to capture fine-grained semantic shifts and ultra-long temporal contexts. To bridge this gap, we propose a novel multimodal framework driven by Hierarchical Granularity Alignment and State Space Models.Specifically, we leverage powerful foundation models, namely DINOv2 and WavLM, to extract robust and high-fidelity visual and audio representations, effectively replacing traditional feature extractors. To handle extreme facial variations, our Hierarchical Granularity Alignment module dynamically aligns global facial semantics with fine-grained local active patches. Furthermore, we overcome the receptive field limitations of conventional temporal convolutional networks by introducing a Vision-Mamba architecture. This approach enables temporal modeling with O(N) linear complexity, effectively capturing ultra-long-range dynamics without performance degradation. A novel asymmetric cross-attention mechanism is also introduced to deeply synchronize paralinguistic audio cues with subtle visual movements.Extensive experiments on the challenging Aff-Wild2 dataset demonstrate that our approach significantly outperforms existing baselines, achieving state-of-the-art performance. Notably, this framework secured top rankings in the AU Detection track of the 10th Affective Behavior Analysis in-the-wild Competition.