Differential Attention Unlocks Complementary EEG and Speech Fusion for Emotion Recognition
Authors: Philip H. Lee, Shreeram Suresh Chandra, John H. L. Hansen
Organizations: Independent Researcher, USA · Center for Language and Speech Processing (CLSP), Johns Hopkins University, Baltimore, MD, USA · Center for Robust Speech Systems (CRSS), University of Texas at Dallas, Richardson, TX, USA
Multimodal emotion recognition (MER) increasingly pairs EEG with speech, treating internal neural signals and external vocal expression as informative views of affect. In practice, naive fusion underperforms the stronger single modality, because EEG artifacts inject noise that corrupts the shared representation. We introduce EmoSpeechBrain, a multimodal framework built on the insight that noise suppression is a precondition for effective fusion. Its EEG encoder uses differential attention, taking the difference between two attention maps to cancel shared noise and isolate discriminative neural activity. An attention-based gating adapter aligns both modalities in a shared space and weights each one's contribution to the prediction. On two datasets - PME4 and EAV, EmoSpeechBrain improves MER accuracy by up to 12.9% over other state-of-the-art (SOTA) EEG encoders, and surpasses unimodal speech and EEG baselines by up to 13.1% and 23.1%. These results show that once EEG noise is suppressed, fusion delivers gains that naive combination cannot.
Figures & tables
Figure 1: Overall architecture of EmoSpeechBrain. (a) The EEG Encoder applies differential attention to isolate discriminative neural activity, while a frozen WavLM-Large encoder extracts speech features. The two modalities interact through parallel cross-modal attention and are fused by the Gating Alignment Module (GAM) before concatenation and classification. (b) The GAM concatenates EEG and speech features, refines them with self-attention and a sigmoid-gated bottleneck, then splits the result into aligned EEG and speech streams, Z^eeg and Z^speech .
Figure 2: Proposed EEG encoder architectural design for EmoSpeechBrain.
Method
PME4
EAV
Balanced Acc. ↑
Weighted F1 ↑
Balanced Acc. ↑
Weighted F1 ↑
EEGNet [ 28 ]
0.503±0.035
0.492±0.046
0.551±0.111
0.556±0.101
BENDR [ 29 ]
0.543±0.065
0.522±0.059
0.646±0.015
0.656±0.017
BIOT [ 30 ]
0.522±0.071
0.536±0.069
0.635±0.023
0.639±0.019
EEGPT [ 31 ]
0.557±0.058
0.548±0.061
0.713±0.072
0.718±0.077
LaBraM [ 32 ]
0.587±0.562
0.594±0.052
0.730±0.082
0.733±0.081
Table 1: Comparison of EmoSpeechBrain against baseline and state-of-the-art EEG encoders on the PME4 and EAV datasets. We use ∗ to indicate statistically significant improvement over all baselines (two-tailed paired t -test, p<0.05 ).
Method
PME4
EAV
Balanced Acc. ↑
Weighted F1 ↑
Balanced Acc. ↑
Weighted F1 ↑
Ablation on Each Modality
Speech Only
0.603±0.022
0.594±0.024
0.642±0.012
0.621±0.013
EEG Only
0.503±0.030
0.502±0.031
0.562±0.032
0.565±0.032
Ablation on Different Components
w/o Local Feature Extraction
0.699±0.021
0.693±0.022
0.808±0.083
0.807±0.063
Table 2: Ablation study table measuring the accuracy of different model variants to evaluate the contribution of each component. We use ∗ to indicate statistically significant improvement over all baselines (two-tailed paired t -test, p<0.05 ).
Figure 3: Ablation study showing balanced accuracy of differential attention and vanilla attention on PME4 and EAV when each EEG frequency band is removed. None denotes the EEG has no frequency bands removed.
Mixed emotions represent a clinically relevant but still underexplored target for automatic emotion recognition. EEG provides millisecond-level access to neural activity, yet most EEG pipelines analyze the signal through a single temporal window, thereby fixing the temporal structure available to the model. This study introduces a multi-scale temporal framework for EEG-based emotion recognition. The EEG waveform is decomposed into windows of one or several durations, processed by a shared attention-based encoder, and integrated through a dynamic fusion module that assigns sample-specific weights across temporal scales. The framework is evaluated under a subject-independent protocol in binary and three-class settings, with the three-class task including the mixed affective category. The best results are 65.22% for the two-class task and 45.43% for the three-class task. Both are obtained with three-scale dynamic-fusion configurations and remain substantially above the full-signal baseline. The best-performing temporal scales differ between the two tasks. Dynamic fusion outperforms concatenation in the highest-scoring two-class configuration and slightly exceeds it in the highest-scoring three-class configuration, although these multi-scale settings require substantially more computation than the full-signal baseline.
Stefanos Gkikas, Yang Guo, Guangliang Li +3
Honda Research Institute Japan Wako City, Japan · Faculty of Information Science and Engineering Ocean University of China Qingdao, China · BioSIS (Biosensing & Intelligent Systems) Lab Centre for Intelligent Computing and Systems University of Canberra Canberra, Australia +1
Automatic emotion assessment can benefit from combining neural and behavioral signals, but many multimodal approaches rely on separate, modality-specific feature-extraction pipelines before fusion. This paper presents MUPA\textsuperscript{2}E, a unified perception framework that processes facial video and electroencephalography (EEG) through a single shared asymmetric-attention backbone. Facial video is represented through axis-folded frame tokens, while EEG is processed either as a raw multichannel waveform or projected into the spatial domain for multimodal fusion. The framework is evaluated on the DMER dataset under a stratified subject-independent protocol, comparing unimodal video, unimodal EEG, and fused video--EEG configurations with per-channel and merged EEG projections. Using the original recordings, with shorter trials zero-padded to match the longest duration, merged fusion at stride~30 achieves the highest validation performance and a test accuracy of 70.07%. Further analysis revealed that recording duration is unevenly distributed across the affective classes, making the padding pattern a potential classification cue. Controlling for this factor by cropping all recordings to a common duration of 20 seconds yielded a test accuracy of 62.71%, providing a stricter duration-controlled assessment of the framework in which differences in recording length are removed as a potential classification cue. These findings demonstrate the feasibility of processing structurally different neural and visual signals within a compact unified architecture while highlighting the importance of controlling duration-related cues in affective datasets.
Stefanos Gkikas, Eric Nichols, Christian Arzate Cruz +1
Multimodal emotion recognition in conversation (MERC) can leverage multimodal and contextual cues to boost recognition performance. However, existing fusion approaches in MERC often ignore modality-specific uncertainty across utterances caused by conflicting cues, varying noise, and missing modality-specific signals. We propose EmoEUS, an explicit uncertainty supervision framework for MERC. EmoEUS performs uncertainty-aware multimodal fusion by dynamically weighting modalities using learned variance estimates. We also introduce an explicitly supervised loss that aligns each utterance's predicted variance with the distance between the utterance's distributional representation and its emotion- and modality-specific cluster center. Experiments on IEMOCAP and MELD show that EmoEUS consistently outperforms state-of-the-art methods.
Zilong Huang, Kong Aik Lee, Junjie Li +2
Dept. of Electrical and Electronic Engineering, The Hong Kong Polytechnic University · Speech, Language, and Cognition Laboratory, The University of Hong Kong