Team RAS in 10th ABAW Competition: Multimodal Valence and Arousal Estimation Approach
Authors: Elena Ryumina, Maxim Markitantov, Alexandr Axyonov, Dmitry Ryumin, Mikhail Dolgushin, Denis Dresvyanskiy, Alexey Karpov
Organizations: St. Petersburg Federal Research Center of the Russian Academy of Sciences, St. Petersburg, Russia · St. Petersburg Federal Research Center of the Russian Academy of Sciences St. Petersburg, Russia · ITMO University, St. Petersburg, Russia · ITMO University St. Petersburg, Russia
Continuous emotion recognition in terms of valence and arousal under in-the-wild (ITW) conditions remains a challenging problem due to large variations in appearance, head pose, illumination, occlusions, and subject-specific patterns of affective expression. We present a multimodal method for valence-arousal estimation ITW. Our method combines three complementary modalities: face, behavior, and audio. The face modality relies on GRADA-based frame-level embeddings and Transformer-based temporal regression. We use Qwen3-VL-4B-Instruct to extract behavior-relevant information from video segments, while Mamba is used to model temporal dynamics across segments. The audio modality relies on WavLM-Large with attention-statistics pooling and includes a cross-modal filtering stage to reduce the influence of unreliable or non-speech segments. To fuse modalities, we explore two fusion strategies: a Directed Cross-Modal Mixture-of-Experts Fusion Strategy that learns interactions between modalities with adaptive weighting, and a Reliability-Aware Audio-Visual Fusion Strategy that combines visual features at the frame-level while using audio as complementary context. The results are reported on the Aff-Wild2 dataset following the 10th Affective Behavior Analysis in-the-Wild (ABAW) challenge protocol. Experiments demonstrate that the proposed multimodal fusion strategy achieves a Concordance Correlation Coefficient (CCC) of 0.658 on the Aff-Wild2 development set.
Figures & tables
Figure 1: Pipeline of the proposed multimodal method for VA estimation.
Figure 2: Prompt used for behavior description extraction with Qwen3.
ID
Model
Aff-Wild2 (devel. set)
Aff-Wild2 (test set)
Valence
Arousal
Avg.
Avg.
1
Video-based model. GRADA + Transformer
0.5869
0.6508
0.6189
0.54
2
Visual-Based Behavior Description Model.
0.2499
0.5515
0.4007
–
Qwen3 + Mamba
3
Multimodal-Based Behavior Description Model.
0.4290
0.6480
0.5385
–
Qwen3 + Mamba
Table 1: Experimental results on Aff-Wild2 VA estimation.
Central University, Moscow, Russia · Sber AI Lab, Moscow, Russia · HSE University, Laboratory of Algorithms and Technologies for Network Analysis, Nizhny Novgorod, Russia