Emotion recognition in real-world environments is hindered by partial occlusions, missing modalities, and severe class imbalance. To address these issues, particularly for the Affective Behavior Analysis in-the-wild (ABAW) Expression challenge, we propose a multimodal framework that dynamically fuses visual and audio representations. Our approach uses a dual-branch Transformer architecture featuring a safe cross-attention mechanism and a modality dropout strategy. This design allows the network to rely on audio-based predictions when visual cues are absent. To mitigate the long-tail distribution of the Aff-Wild2 dataset, we apply focal loss optimization, combined with a sliding-window soft voting strategy to capture dynamic emotional transitions and reduce frame-level classification jitter. Experiments demonstrate that our framework effectively handles missing modalities and complex spatiotemporal dependencies, achieving an accuracy of 60.79% and an F1-score of 0.5029 on the Aff-Wild2 validation set.
Figures & tables
Figure 1: The proposed multimodal emotion recognition framework processes video and audio inputs through BEiT-Large and WavLM-Large, aligns them in a unified embedding space, and dynamically fuses the representations via a multimodal attention network for MLP classification.
Visual Backbone
Accuracy
F1-Score
ResNet50
0.4531
0.3345
EfficientNetV2-M
0.4921
0.3932
MAE-ViT-Base
0.4924
0.3686
BEiT-large
0.5421
0.4268
Table 1: Comparison of different visual backbones on the Aff-Wild2 validation set.
Weight
Accuracy
F1-Score
0.0
0.4265
0.3150
0.5
0.5656
0.4427
0.7
0.5698
0.4436
1.0
0.5542
0.4368
Table 2: Baseline performance with different modality fusion weights ( λ ).
p
d
l
Accuracy
F1-Score
0.0
256
2
0.5677
0.4628
0.0
512
2
0.5820
0.4739
0.0
256
3
0.5824
0.4764
0.0
512
3
0.5730
0.4626
0.10
256
3
0.6079
0.5029
0.10
256
4
0.5981
0.4814
Table 3: Ablation studies on modality dropout probability ( p ), dimension ( d ), and the number of self-attention layers ( l ).
St. Petersburg Federal Research Center of the Russian Academy of Sciences, St. Petersburg, Russia · St. Petersburg Federal Research Center of the Russian Academy of Sciences St. Petersburg, Russia · ITMO University, St. Petersburg, Russia +1