cs.CVMar 13, 2026

Team RAS in 10th ABAW Competition: Multimodal Valence and Arousal Estimation Approach

Authors: Elena Ryumina, Maxim Markitantov, Alexandr Axyonov, Dmitry Ryumin, Mikhail Dolgushin, Denis Dresvyanskiy, Alexey Karpov

Organizations: St. Petersburg Federal Research Center of the Russian Academy of Sciences, St. Petersburg, Russia · St. Petersburg Federal Research Center of the Russian Academy of Sciences St. Petersburg, Russia · ITMO University, St. Petersburg, Russia · ITMO University St. Petersburg, Russia

Abstract

Continuous emotion recognition in terms of valence and arousal under in-the-wild (ITW) conditions remains a challenging problem due to large variations in appearance, head pose, illumination, occlusions, and subject-specific patterns of affective expression. We present a multimodal method for valence-arousal estimation ITW. Our method combines three complementary modalities: face, behavior, and audio. The face modality relies on GRADA-based frame-level embeddings and Transformer-based temporal regression. We use Qwen3-VL-4B-Instruct to extract behavior-relevant information from video segments, while Mamba is used to model temporal dynamics across segments. The audio modality relies on WavLM-Large with attention-statistics pooling and includes a cross-modal filtering stage to reduce the influence of unreliable or non-speech segments. To fuse modalities, we explore two fusion strategies: a Directed Cross-Modal Mixture-of-Experts Fusion Strategy that learns interactions between modalities with adaptive weighting, and a Reliability-Aware Audio-Visual Fusion Strategy that combines visual features at the frame-level while using audio as complementary context. The results are reported on the Aff-Wild2 dataset following the 10th Affective Behavior Analysis in-the-Wild (ABAW) challenge protocol. Experiments demonstrate that the proposed multimodal fusion strategy achieves a Concordance Correlation Coefficient (CCC) of 0.658 on the Aff-Wild2 development set.

Figures & tables

Explore similar work

CardsList
  1. HSEmotion Team at the 11th ABAW Challenge: Multi-Task Learning and Ambivalence/Hesitancy Video Recognition

    Jul 14, 2026Aleksei Bakin, Andrey V. SavchenkoMulti-Task LearningMultimodal Emotion Recognition

  2. Team LEYA in 10th ABAW Competition: Multimodal Ambivalence/Hesitancy Recognition Approach

    Mar 13, 2026Elena Ryumina, Alexandr Axyonov, Dmitry Sysoev +4Ambivalence and Hesitancy RecognitionMultimodal Classification