Differential Attention Unlocks Complementary EEG and Speech Fusion for Emotion Recognition
Authors: Philip H. Lee, Shreeram Suresh Chandra, John H. L. Hansen
Organizations: Independent Researcher, USA · Center for Language and Speech Processing (CLSP), Johns Hopkins University, Baltimore, MD, USA · Center for Robust Speech Systems (CRSS), University of Texas at Dallas, Richardson, TX, USA
Multimodal emotion recognition (MER) increasingly pairs EEG with speech, treating internal neural signals and external vocal expression as informative views of affect. In practice, naive fusion underperforms the stronger single modality, because EEG artifacts inject noise that corrupts the shared representation. We introduce EmoSpeechBrain, a multimodal framework built on the insight that noise suppression is a precondition for effective fusion. Its EEG encoder uses differential attention, taking the difference between two attention maps to cancel shared noise and isolate discriminative neural activity. An attention-based gating adapter aligns both modalities in a shared space and weights each one's contribution to the prediction. On two datasets - PME4 and EAV, EmoSpeechBrain improves MER accuracy by up to 12.9% over other state-of-the-art (SOTA) EEG encoders, and surpasses unimodal speech and EEG baselines by up to 13.1% and 23.1%. These results show that once EEG noise is suppressed, fusion delivers gains that naive combination cannot.
Figures & tables
Figure 1: Overall architecture of EmoSpeechBrain. (a) The EEG Encoder applies differential attention to isolate discriminative neural activity, while a frozen WavLM-Large encoder extracts speech features. The two modalities interact through parallel cross-modal attention and are fused by the Gating Alignment Module (GAM) before concatenation and classification. (b) The GAM concatenates EEG and speech features, refines them with self-attention and a sigmoid-gated bottleneck, then splits the result into aligned EEG and speech streams, Z^eeg and Z^speech .
Figure 2: Proposed EEG encoder architectural design for EmoSpeechBrain.
Method
PME4
EAV
Balanced Acc. ↑
Weighted F1 ↑
Balanced Acc. ↑
Weighted F1 ↑
EEGNet [ 28 ]
0.503±0.035
0.492±0.046
0.551±0.111
0.556±0.101
BENDR [ 29 ]
0.543±0.065
0.522±0.059
0.646±0.015
0.656±0.017
BIOT [ 30 ]
0.522±0.071
0.536±0.069
0.635±0.023
0.639±0.019
EEGPT [ 31 ]
0.557±0.058
0.548±0.061
0.713±0.072
0.718±0.077
LaBraM [ 32 ]
0.587±0.562
0.594±0.052
0.730±0.082
0.733±0.081
Table 1: Comparison of EmoSpeechBrain against baseline and state-of-the-art EEG encoders on the PME4 and EAV datasets. We use ∗ to indicate statistically significant improvement over all baselines (two-tailed paired t -test, p<0.05 ).
Method
PME4
EAV
Balanced Acc. ↑
Weighted F1 ↑
Balanced Acc. ↑
Weighted F1 ↑
Ablation on Each Modality
Speech Only
0.603±0.022
0.594±0.024
0.642±0.012
0.621±0.013
EEG Only
0.503±0.030
0.502±0.031
0.562±0.032
0.565±0.032
Ablation on Different Components
w/o Local Feature Extraction
0.699±0.021
0.693±0.022
0.808±0.083
0.807±0.063
Table 2: Ablation study table measuring the accuracy of different model variants to evaluate the contribution of each component. We use ∗ to indicate statistically significant improvement over all baselines (two-tailed paired t -test, p<0.05 ).
Figure 3: Ablation study showing balanced accuracy of differential attention and vanilla attention on PME4 and EAV when each EEG frequency band is removed. None denotes the EEG has no frequency bands removed.
Honda Research Institute Japan Wako City, Japan · Faculty of Information Science and Engineering Ocean University of China Qingdao, China · BioSIS (Biosensing & Intelligent Systems) Lab Centre for Intelligent Computing and Systems University of Canberra Canberra, Australia +1
Dept. of Electrical and Electronic Engineering, The Hong Kong Polytechnic University · Speech, Language, and Cognition Laboratory, The University of Hong Kong