MUPA$^{2}$E: Multimodal Unified Perception with Asymmetric Attention for Emotion Assessment

Stefanos Gkikas; Eric Nichols; Christian Arzate Cruz; Randy Gomez

cs.AIAug 17, 2026

MUPA $^{2}$ E: Multimodal Unified Perception with Asymmetric Attention for Emotion Assessment

Authors: Stefanos Gkikas, Eric Nichols, Christian Arzate Cruz, Randy Gomez

Organizations: Honda Research Institute Japan, Wako City, Japan

Abstract

Automatic emotion assessment can benefit from combining neural and behavioral signals, but many multimodal approaches rely on separate, modality-specific feature-extraction pipelines before fusion. This paper presents MUPA\textsuperscript{2}E, a unified perception framework that processes facial video and electroencephalography (EEG) through a single shared asymmetric-attention backbone. Facial video is represented through axis-folded frame tokens, while EEG is processed either as a raw multichannel waveform or projected into the spatial domain for multimodal fusion. The framework is evaluated on the DMER dataset under a stratified subject-independent protocol, comparing unimodal video, unimodal EEG, and fused video--EEG configurations with per-channel and merged EEG projections. Using the original recordings, with shorter trials zero-padded to match the longest duration, merged fusion at stride~ $30$ achieves the highest validation performance and a test accuracy of $70.07\%$ . Further analysis revealed that recording duration is unevenly distributed across the affective classes, making the padding pattern a potential classification cue. Controlling for this factor by cropping all recordings to a common duration of $20$ seconds yielded a test accuracy of $62.71\%$ , providing a stricter duration-controlled assessment of the framework in which differences in recording length are removed as a potential classification cue. These findings demonstrate the feasibility of processing structurally different neural and visual signals within a compact unified architecture while highlighting the importance of controlling duration-related cues in affective datasets.

Figures & tables

Method/Parameter	Value
Video augmentations
AugMix	$p\in[0.20,0.40]$
TrivialAugment	$p\in[0.20,0.40]$
Center Crop	$p\in[0.20,0.40]$ , $200{\times}200$
Noise	$p\in[0.20,0.40]$ , $\sigma=100$
Masking-1	$p\in[0.20,0.80]$ , 3 blocks

Table 1: Augmentation, regularization, and training configuration.

Figure 1: Overview of MUPA 2 E. Facial video is axis-folded into a $224\times 224\times 3L$ tensor; the EEG waveform ( $C\times L$ ) is projected to a $224\times 224\times 1$ grid via bilinear interpolation ( $z=\alpha\times\beta$ ). The two representations are channel-concatenated, flattened into $50176$ Fourier-augmented spatial tokens, and segmented into $32$ groups of $1568$ tokens, processed by the asymmetric attention backbone.

Split	Difficulty Group
	Q1 – Hard	Q2 – Med-Hard	Q3 – Med-Easy	Q4 – Easy
Training (48)	8, 12, 20, 25, 36, 37, 43, 45, 56, 60, 66, 71	14, 30, 33, 35, 39, 50, 51, 53, 55, 65, 72, 73	5, 6, 9, 10, 15, 23, 32, 52, 54, 59, 70, 78	11, 24, 26, 34, 40, 42, 49, 61, 63, 64, 68, 76
Validation (12)	19, 48, 58	22, 41, 79	29, 47, 77	62, 67, 69
Testing (13)	28, 74, 75, 80	21, 38, 57	1, 2, 7	18, 44, 46

Table 2: Subject-level split by difficulty group. Subjects are ranked by combined z-score and assigned to four quartile-based groups (Q1 = hardest, Q4 = easiest).

Modality	Stride	Performance
		Accuracy	Precision	F1	Average
Video	10	68.75	69.05	68.36	68.72
Video	15	69.27	69.63	69.13	69.34
Video	20	68.75	69.05	68.63	68.81
Video	30	69.27	69.53	69.17	69.32
EEG	1	69.79	69.87	69.76	69.81

Table 3: Unimodal performance across stride settings. Average : arithmetic mean of Accuracy, Precision, and F1, used as the primary performance criterion throughout. Bold marks the highest Average ; underline marks the second-highest.

Fusion	Stride	Performance
		Accuracy	Precision	F1	Average
Per-channel	10	68.75	69.05	68.63	68.81
Per-channel	15	69.27	69.53	69.17	69.32
Per-channel	20	69.27	69.75	69.08	69.37
Per-channel	30	68.85	69.05	68.63	68.84
Merged	10	68.75	69.05	68.63	68.81

Table 4: Multimodal fusion performance using Videos and EEG across stride settings.

Figure 2: Average validation performance (arithmetic mean of Accuracy, Precision, and F1) as a function of video temporal stride for unimodal (a) and multimodal (b) configurations. Unimodal EEG at stride 1 is shown as a dashed reference in both panels.

Fusion	Stride	Computational Cost		Inference Cost		Accuracy
		Params (M)	GFLOPs	Latency (ms) GPU $\downarrow$	Samples/s GPU $\uparrow$
Merged	30	1.89	3.13	6.55	152.65	70.07

Table 5: Test accuracy, computational cost, and inference efficiency of the best validation-selected merged-fusion configuration.

Study	Modality	Validation Protocol	Samples	Accuracy
Ours	Video+EEG	Hold-out	Original	70.07
Ours	Video+EEG	Hold-out	Cropped	62.71
Ours	Video+EEG	LOSO	Cropped	76.37
[ 10 ]	EEG	Hold-out	Cropped	65.22

Table 6: Accuracy comparison across sample preprocessing and validation protocols.

Explore similar work

Aug 10, 2026

A Multi-Scale Temporal Framework with Dynamic Fusion for EEG-Based Emotion Recognition

Mixed emotions represent a clinically relevant but still underexplored target for automatic emotion recognition. EEG provides millisecond-level access to neural activity, yet most EEG pipelines analyze the signal through a single temporal window, thereby fixing the temporal structure available to the model. This study introduces a multi-scale temporal framework for EEG-based emotion recognition. The EEG waveform is decomposed into windows of one or several durations, processed by a shared attention-based encoder, and integrated through a dynamic fusion module that assigns sample-specific weights across temporal scales. The framework is evaluated under a subject-independent protocol in binary and three-class settings, with the three-class task including the mixed affective category. The best results are 65.22% for the two-class task and 45.43% for the three-class task. Both are obtained with three-scale dynamic-fusion configurations and remain substantially above the full-signal baseline. The best-performing temporal scales differ between the two tasks. Dynamic fusion outperforms concatenation in the highest-scoring two-class configuration and slightly exceeds it in the highest-scoring three-class configuration, although these multi-scale settings require substantially more computation than the full-signal baseline.

Stefanos Gkikas, Yang Guo, Guangliang Li +3

Honda Research Institute Japan Wako City, Japan · Faculty of Information Science and Engineering Ocean University of China Qingdao, China · BioSIS (Biosensing & Intelligent Systems) Lab Centre for Intelligent Computing and Systems University of Canberra Canberra, Australia +1

Multi-Scale Feature Fusion EEG Emotion Recognition Emotion Recognition

Ask PDF

Sep 25, 2026

Differential Attention Unlocks Complementary EEG and Speech Fusion for Emotion Recognition

Multimodal emotion recognition (MER) increasingly pairs EEG with speech, treating internal neural signals and external vocal expression as informative views of affect. In practice, naive fusion underperforms the stronger single modality, because EEG artifacts inject noise that corrupts the shared representation. We introduce EmoSpeechBrain, a multimodal framework built on the insight that noise suppression is a precondition for effective fusion. Its EEG encoder uses differential attention, taking the difference between two attention maps to cancel shared noise and isolate discriminative neural activity. An attention-based gating adapter aligns both modalities in a shared space and weights each one's contribution to the prediction. On two datasets - PME4 and EAV, EmoSpeechBrain improves MER accuracy by up to 12.9% over other state-of-the-art (SOTA) EEG encoders, and surpasses unimodal speech and EEG baselines by up to 13.1% and 23.1%. These results show that once EEG noise is suppressed, fusion delivers gains that naive combination cannot.

Philip H. Lee, Shreeram Suresh Chandra, John H. L. Hansen

Independent Researcher, USA · Center for Language and Speech Processing (CLSP), Johns Hopkins University, Baltimore, MD, USA · Center for Robust Speech Systems (CRSS), University of Texas at Dallas, Richardson, TX, USA

Multimodal Emotion Recognition EEG Emotion Recognition Multimodal Fusion Electroencephalography

Ask PDF

Jul 19, 2026

EmoEUS: Uncertainty Supervision for Multimodal Emotion Recognition in Conversation

Multimodal emotion recognition in conversation (MERC) can leverage multimodal and contextual cues to boost recognition performance. However, existing fusion approaches in MERC often ignore modality-specific uncertainty across utterances caused by conflicting cues, varying noise, and missing modality-specific signals. We propose EmoEUS, an explicit uncertainty supervision framework for MERC. EmoEUS performs uncertainty-aware multimodal fusion by dynamically weighting modalities using learned variance estimates. We also introduce an explicitly supervised loss that aligns each utterance's predicted variance with the distance between the utterance's distributional representation and its emotion- and modality-specific cluster center. Experiments on IEMOCAP and MELD show that EmoEUS consistently outperforms state-of-the-art methods.

Zilong Huang, Kong Aik Lee, Junjie Li +2

Dept. of Electrical and Electronic Engineering, The Hong Kong Polytechnic University · Speech, Language, and Cognition Laboratory, The University of Hong Kong

Emotion Recognition in Conversations Multimodal Emotion Recognition Uncertainty-Aware Multimodal Fusion Multimodal Fusion

Ask PDF