Organizations: School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea · Graduate School of Green Growth and Sustainability, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea
Audio-visual emotion recognition (AVER) methods typically fuse utterance-level features, and even frame-level attention models seldom address the frame-rate mismatch across modalities. In this paper, we propose a Transformer-based framework focusing on the temporal alignment of multimodal features. Our design employs a multimodal self-attention encoder that simultaneously captures intra- and inter-modal dependencies within a shared feature space. To address heterogeneous sampling rates, we incorporate Temporally-aligned Rotary Position Embeddings (TaRoPE), which implicitly synchronize audio and video tokens. Furthermore, we introduce a Cross-Temporal Matching (CTM) loss that enforces consistency among temporally proximate pairs, guiding the encoder toward better alignment. Experiments on CREMA-D and RAVDESS datasets demonstrate consistent improvements over recent baselines, suggesting that explicitly addressing frame-rate mismatch helps preserve temporal cues and enhances cross-modal fusion.
Figures & tables
Figure 1: Audio and video features are first extracted by modality-specific encoders and projected into a shared embedding space. Temporally-aligned RoPE (TaRoPE) implicitly synchronizes heterogeneous audio–visual sequences within the unified Transformer encoder by adapting rotary positional embeddings to different temporal resolutions. The Cross-Temporal Matching (CTM) loss explicitly enforces temporal consistency by aligning audio–video feature pairs that are temporally proximal along a shared time axis. The final representation is pooled for emotion classification.
Method
Year
CREMA-D
RAVDESS
Ghaleb et al. [ 7 ]
2019
74.00
67.70
MATER [ 6 ]
2020
67.20
76.30
TA-AVN [ 16 ]
2021
84.00
78.70
Luna-Jiménez et al. [ 11 ]
2021
-
80.08
Luna-Jiménez et al. [ 12 ]
2021
-
86.70
V8 + A4 [ 14 ]
2022
-
86.00
Table 1: Comparison of audio-visual emotion recognition performance (%) on CREMA-D and RAVDESS datasets.
Figure 2: Illustration of differenta fusion strategies using attention mechanisms.
Fusion Module
# of parameters
Acc.
Concat.
-
85.71
ISA + ISA
12.61M
87.98
ICA + ICA
87.49
ISA + ICA
87.71
ICA + ISA
88.31
MSA + MSA (Ours)
6.83M
88.95
Table 2: Ablation study on different fusion strategies conducted on CREMA-D.
Positional Encoding
w/o Lctm
w/ Lctm
Sinusoidal
88.09
88.79
Learnable
87.44
88.79
RoPE
87.76
89.00
TaRoPE
88.95
89.49
Table 3: Ablation study on positional encoding strategies and CTM loss conducted on CREMA-D.
Multimodal emotion recognition has attracted growing interest due to its importance in human-computer interaction, remote education, and healthcare. This paper proposes a novel multimodal emotion recognition framework that integrates rich audio and visual feature extraction with an attention-based fusion strategy. For audio, we extract three complementary feature types: semantic embeddings from Wav2Vec2, MFCC features, and statistical acoustic descriptors such as pitch, energy, and rhythm. These are aligned and fused via a BiLSTM to capture temporal dependencies. For video, we propose a ResNet50-BiLSTM architecture that combines deep residual learning and sequential modeling to extract expressive spatiotemporal features from facial sequences. To enhance multimodal synergy, we introduce a feature-level fusion mechanism based on multi-head attention, allowing the model to adaptively weigh contributions across modalities. Experiments conducted on the MELD and IEMOCAP datasets demonstrate that our model significantly outperforms baselines in both accuracy and robustness. Furthermore, ablation studies show that the attention-based fusion strategy significantly improves performance in unbalanced data settings. Our findings suggest that the proposed framework effectively captures diverse emotional cues from speech and visual expressions, and offers a practical and generalizable approach for real-world multimodal emotion recognition tasks.
Xu Lin, Ke Wang, Hui Kang +1
College of Computer Science and Technology, Jilin University, Changchun 130012, China · Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun 130012, China
Expression recognition in in-the-wild video data remains challenging due to substantial variations in facial appearance, background conditions, audio noise, and the inherently dynamic nature of human affect. Relying on a single modality, such as facial expressions or speech, is often insufficient for capturing these complex emotional cues. To address this limitation, we propose a multimodal emotion recognition framework for the Expression (EXPR) task in the 10th Affective Behavior Analysis in-the-wild (ABAW) Challenge. Our framework builds on large-scale pre-trained models for visual and audio representation learning and integrates them in a unified multimodal architecture. To better capture temporal patterns in facial expression sequences, we incorporate temporal visual modeling over video windows. We further introduce a bi-directional cross-attention fusion module that enables visual and audio features to interact in a symmetric manner, facilitating cross-modal contextualization and complementary emotion understanding. In addition, we employ a text-guided contrastive objective to encourage semantically meaningful visual representations through alignment with emotion-related text prompts. Experimental results on the ABAW 10th EXPR benchmark demonstrate the effectiveness of the proposed framework, achieving a Macro F1 score of 0.32 compared to the baseline score of 0.25, and highlight the benefit of combining temporal visual modeling, audio representation learning, and cross-modal fusion for robust emotion recognition in unconstrained real-world environments.
While Multimodal Large Language Models (MLLMs) have advanced video understanding, achieving precise temporal and cross-modal alignment in audiovisual video captioning remains a formidable challenge. Most existing approaches suffer from modality detachment and temporal incoherence, failing to accurately bind auditory events to visual entities or capture complex causal dynamics. To address these deficiencies, we propose TCA-Captioner, a framework specifically engineered to enhance Temporal and Cross-Modal Alignment for audiovisual video captioning. We first introduce the Observer-Checker-Corrector (OCC) framework, an iterative refinement strategy that generates high-fidelity, meticulously grounded training data. Leveraging a curated high-density human interaction dataset, TCA-Captioner is optimized to model sophisticated audiovisual interactions. Furthermore, we present TCA-Bench, a diagnostic benchmark utilizing a Decoupled Evaluation Protocol to isolate and quantify model proficiency in audiovisual binding and temporal relational reasoning. Extensive experiments demonstrate that TCA-Captioner sets a new standard for temporally-coherent and synchronized audiovisual narratives.
Chen Zhao, Jiajun Ma, Qilong Huang +6
Meituan, China · Nanjing University, State Key Laboratory for Novel Software Technology, China