Multimodal Emotion Regression with Multi-Objective Optimization and VAD-Aware Audio Modeling for the 10th ABAW EMI Track

Jiawen Huang; Chenxi Huang; Zhuofan Wen; Hailiang Yao; Shun Chen; Longjiang Yang; Cong Yu; Fengyu Zhang

cs.AIMar 14, 2026

Multimodal Emotion Regression with Multi-Objective Optimization and VAD-Aware Audio Modeling for the 10th ABAW EMI Track

Authors: Jiawen Huang, Chenxi Huang, Zhuofan Wen, Hailiang Yao, Shun Chen, Longjiang Yang, Cong Yu, Fengyu Zhang, +2 more

Organizations: School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Tianjin Normal University

Abstract

We participated in the 10th ABAW Challenge, focusing on the Emotional Mimicry Intensity (EMI) Estimation track on the Hume-Vidmimic2 dataset. This task aims to predict six continuous emotion dimensions: Admiration, Amusement, Determination, Empathic Pain, Excitement, and Joy. Through systematic multimodal exploration of pretrained high-level features, we found that, under our pretrained feature setting, direct feature concatenation outperformed the more complex fusion strategies we tested. This empirical finding motivated us to design a systematic approach built upon three core principles: (i) preserving modality-specific attributes through feature-level concatenation; (ii) improving training stability and metric alignment via multi-objective optimization; and (iii) enriching acoustic representations with a VAD-inspired latent prior. Our final framework integrates concatenation-based multimodal fusion, a shared six-dimensional regression head, multi-objective optimization with MSE, Pearson-correlation, and auxiliary branch supervision, EMA for parameter stabilization, and a VAD-inspired latent prior for the acoustic branch. On the official validation set, the proposed scheme achieved our best mean Pearson Correlation Coefficient of 0.478567.

Figures & tables

Figure 1: Architecture of the proposed multimodal framework. The model employs three modality-specific branches with auxiliary supervision, VAD-Aware audio fusion, feature concatenation, and a shared six-dimensional regression head for emotional mimicry intensity estimation.

Figure 2: Overview of the VAD-aware audio representation learning module. After audio linear projection, the acoustic feature is split into a main audio stream and a VAD prediction branch. The VAD branch produces a 3D latent VAD representation, which is combined with the main audio stream to construct the audio embedding. This embedding is then used for multimodal fusion and auxiliary loss, while the latent VAD representation is regularized to improve stability and prevent extreme deviations.

Modality	Model	Features	$\bar{p}$	Admiration	Amusement	Determination	Empathic Pain	Excitement	Joy
Faces	Baseline ViT [ 8 ]		0.0900	-	-	-	-	-	-
Audio	wav2Vec2 [ 8 ]		0.2400	-	-	-	-	-	-
Audio+Video	ViT+wav2Vec2 [ 8 ]		0.2500	-	-	-	-	-	-
Audio+Video	ResNet18+AUs+Wav2Vec2.0 [ 21 ]		0.3288	-	-	-	-	-	-
Audio	Wav2Vec2.0+VAD [ 1 ]		0.3890	-	-	-	-	-	-
Audio+Video	EmoViT+HuBERT+ChatGLM3 [ 19 ]		0.5851	0.7155	0.6159	0.6303	0.3488	0.6174	0.5793

Table 1: Pearson’s correlation for EMI Estimation on the Hume-Vidmimic2’s validation set [ 20 ] .

Partition	Duration	Samples
Train	15:07:03	8072
Validation	9:12:02	4588
Test	9:04:05	4586
All	33:23:10	17246

Table 2: Hume-Vidmimic2 partition statistics.

	Visual	Audio	Text	$\bar{p}$
Baseline	Yes			0.09
Ours	Yes			0.092504
Baseline		Yes		0.24
Ours		Yes		0.355879
Ours			Yes	0.464771

Table 3: Comparison of average Pearson correlation coefficients on the validation set in the single-modality setting.

Method	Fusion	$\bar{p}$
Baseline	Average	0.467417
Baseline	Concat	0.472604
Baseline + Temporal Enhancement	Concat	0.441066
Baseline + Multi-Objective	Concat	0.476641
Baseline + Multi-Objective+ VAD	Concat	0.478567

Table 4: Ablation study of our method on the validation set.

Explore similar work

Two-Stage Multimodal Framework for Emotion Mimicry Intensity Prediction
May 21, 2026Dinithi Dissanayake, Shaveen Silva, Ovindu Atukorala +2Multimodal Emotion Recognition Multimodal Fusion
Augmented Human Lab, National University of Singapore, Singapore · University of Moratuwa, Sri Lanka
Anchoring Emotions in Text: Robust Multimodal Fusion for Mimicry Intensity Estimation
Mar 16, 2026Lingsi Zhu, Yuefeng Zou, Yunxiang Zhang +7Multimodal Fusion Multimodal Emotion Recognition
University of Science and Technology of China · Unisound AI Technology Co., Ltd. · Pingan Technology Co., Ltd.
Team RAS in 10th ABAW Competition: Multimodal Valence and Arousal Estimation Approach
Mar 13, 2026Elena Ryumina, Maxim Markitantov, Alexandr Axyonov +4Valence-Arousal Modeling Affective Computing
St. Petersburg Federal Research Center of the Russian Academy of Sciences, St. Petersburg, Russia · St. Petersburg Federal Research Center of the Russian Academy of Sciences St. Petersburg, Russia · ITMO University, St. Petersburg, Russia +1

May 21, 2026J/K move · Enter open · S save

Two-Stage Multimodal Framework for Emotion Mimicry Intensity Prediction

Dinithi Dissanayake, Shaveen Silva, Ovindu Atukorala, Prasanth Sasikumar+1

Augmented Human Lab, National University of Singapore, Singapore · University of Moratuwa, Sri Lanka

Multimodal Emotion Recognition Multimodal Fusion Multimodal Understanding

Open paper AskPDF arXiv

We present our submission to the Hume-ABAW10 Emotional Mimicry Intensity (EMI) Challenge, which aims to predict six continuous emotion intensity dimensions: Admiration, Amusement, Determination, Empathic Pain, Excitement, and Joy, from in-the-wild multimodal video clips. We propose a staged multimodal framework that combines textual, acoustic, and visual representations, with an optional motion branch. Our approach first trains modality-specific encoders independently and then fuses their learned representations through a lightweight regressor with modality dropout and controlled encoder adaptation. Across our submitted systems, the best validation performance is obtained by the text--audio--vision--motion fusion model under the expanded 4:1 split, achieving an average Pearson correlation of 0.4722. Although the motion branch yields only very slight gains, its behavior can be interesting to study. Our team was placed third in the EMI challenge, achieving an average Pearson correlation of 0.57 for the test set. Overall, we provide a practical and reproducible baseline for EMI prediction.

Abstract

Explore similar work

Two-Stage Multimodal Framework for Emotion Mimicry Intensity Prediction

Anchoring Emotions in Text: Robust Multimodal Fusion for Mimicry Intensity Estimation

Team RAS in 10th ABAW Competition: Multimodal Valence and Arousal Estimation Approach