Foundation Model Embeddings Meet Blended Emotions: A Multimodal Fusion Approach for the BLEMORE Challenge

Masoumeh Chapariniya; Aref Farhadipour; Sarah Ebling; Volker Dellwo; Teodora Vukovic

cs.CVMar 24, 2026

Foundation Model Embeddings Meet Blended Emotions: A Multimodal Fusion Approach for the BLEMORE Challenge

Authors: Masoumeh Chapariniya, Aref Farhadipour, Sarah Ebling, Volker Dellwo, Teodora Vukovic

Organizations: Department of Computational Linguistics, University of Zürich, Switzerland

Abstract

We present our system for the BLEMORE Challenge at FG 2026 on blended emotion recognition with relative salience prediction. Our approach combines six encoder families through late probability fusion: an S4D-ViTMoE face encoder adapted with soft-label KL training, frozen layer-selective Wav2Vec2 audio features, finetuned body-language encoders (TimeSformer, VideoMAE), and -- for the first time in emotion recognition -- Gemini Embedding 2.0, a large multimodal model whose video embeddings produce competitive presence accuracy (ACCP = 0.320) from only 2 seconds of input. Three key findings emerge from our experiments: selecting prosody-encoding layers (6--12) from frozen Wav2Vec2 outperforms end-to-end finetuning (Score 0.207 vs. 0.161), as the non-verbal nature of BLEMORE audio makes phonetic layers irrelevant; the post-processing salience threshold $β$ varies from 0.05 to 0.43 across folds, revealing that personalized expression styles are the primary bottleneck; and task-adapted encoders collectively receive 62% of ensemble weight over general-purpose baselines. Our 12-encoder system achieves Score = 0.279 (ACCP = 0.391, ACCS = 0.168) on the test set, placing 6th.

Figures & tables

Panel of Figure 1 — Fig. 1: Overview of the multimodal ensemble pipeline. A video clip is processed by five encoder families operating on different input representations: face-cropped frames (S4D-ViTMoE), body frames (TimeSformer, VideoMAE), audio waveforms (Wav2Vec2 layers 6–12), short video clips (Gemini Embedding 2.0), and pre-extracted features from challenge baselines. Each encoder produces a 6-dimensional probability vector over emotions, which are combined through optimized weighted averaging. The fused probabilities are converted to discrete predictions via threshold-based post-processing ( $\alpha$ for presence, $\beta$ for salience).

Our adapted/finetuned models $\ddagger$
Encoder	M	ACC ${}_{\text{P}}$	ACC ${}_{\text{S}}$	Score
S4D-ViTMoE (face)	V	$.340{\scriptstyle\pm.024}$	$.140{\scriptstyle\pm.018}$	$.240$
Gemini Embed. 2.0 (2s)	AV	$.320{\scriptstyle\pm.033}$	$.137{\scriptstyle\pm.009}$	$.223$
VideoMAE body (ft)	V	$.291{\scriptstyle\pm.016}$	$.144{\scriptstyle\pm.026}$	$.218$
Wav2Vec2 frozen+MLP-1024	A	$.294{\scriptstyle\pm.025}$	$.120{\scriptstyle\pm.018}$	$.207$
TimeSformer body (ft)	V	$.259{\scriptstyle\pm.031}$	$.131{\scriptstyle\pm.015}$	$.195$

TABLE I: Individual encoder results (5-fold CV). M = modality.

\dagger

Baselines from [ 2 ] .

\ddagger

Our finetuned models.

BLEMORE baselines (test) $\dagger$
Configuration	ACC ${}_{\text{P}}$	ACC ${}_{\text{S}}$	Score
VideoMAEv2 + HuBERT $\dagger$	.332	.114	.223
ImageBind + WavLM $\dagger$	.327	.114	.221
HiCMAE $\dagger$	.268	.180	.224
Our submissions (test)
S4D + Wav2Vec2	.327	.159	.243

TABLE II: Fusion results compared to BLEMORE baselines [ 2 ] . Val = 5-fold CV. Test = competition server.

	Encoder	9-enc	12-enc
Ours	S4D-ViTMoE (face)	.094	.117
	Wav2Vec2 (audio)	.170	.111
	Gemini Embed. 2.0	—	.192
	TimeSformer (body)	—	.090
	VideoMAE (body)	—	.110
Baseline	HiCMAE	.261	.103

TABLE III: Optimized encoder weights for two ensemble configurations. Weights are averaged across 5-fold cross-validation. Dashes indicate encoders not included in that configuration.

Explore similar work

Ordering Matters: Rank-Aware Selective Fusion for Blended Emotion Recognition
May 20, 2026Junghyun Lee, Hyunseo Kim, Hanna Jang +1Multimodal Emotion Recognition Multimodal Fusion
Department of Artificial Intelligence and Software, Ewha Womans University, Seoul, Republic of Korea
Two-Stage Multimodal Framework for Emotion Mimicry Intensity Prediction
May 21, 2026Dinithi Dissanayake, Shaveen Silva, Ovindu Atukorala +2Multimodal Emotion Recognition Multimodal Fusion
Augmented Human Lab, National University of Singapore, Singapore · University of Moratuwa, Sri Lanka
Enhancing Multimodal Emotion Recognition via Multi-Feature Encoding and Attention-Based Fusion
Sep 4, 2026Xu Lin, Ke Wang, Hui Kang +1Multimodal Emotion Recognition Multimodal Fusion
College of Computer Science and Technology, Jilin University, Changchun 130012, China · Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun 130012, China

May 20, 2026J/K move · Enter open · S save

Ordering Matters: Rank-Aware Selective Fusion for Blended Emotion Recognition

Junghyun Lee, Hyunseo Kim, Hanna Jang, Junhyug Noh

Department of Artificial Intelligence and Software, Ewha Womans University, Seoul, Republic of Korea

Multimodal Emotion Recognition Multimodal Fusion Feature Fusion Emotion Recognition

Open paper AskPDF arXiv

Blended emotion recognition is challenging because emotions are often expressed as mixtures of subtle and overlapping multimodal cues rather than a single dominant signal. We propose a rank-aware multi-encoder framework that selectively combines complementary representations from diverse pre-extracted video and audio encoders. Our method projects heterogeneous encoder features into a shared latent space, estimates sample-wise encoder importance through an attention-based gating module, and fuses only the top-n most informative encoders. To better model blended emotions, we decouple prediction into presence and salience heads and align them through probability-level fusion. We further incorporate feature-level unsupervised domain adaptation without pseudo-labeling to improve robustness under distribution shift. Experiments on the BlEmoRE challenge show that the proposed framework outperforms strong individual encoders and naïve multi-encoder fusion baselines. Our final system ranked 2nd in the competition, supporting the effectiveness of rank-aware selective fusion for fine-grained blended emotion recognition.

Abstract

Explore similar work

Ordering Matters: Rank-Aware Selective Fusion for Blended Emotion Recognition

Two-Stage Multimodal Framework for Emotion Mimicry Intensity Prediction

Enhancing Multimodal Emotion Recognition via Multi-Feature Encoding and Attention-Based Fusion