cs.SDFeb 21, 2026

Multi-Channel Speech Enhancement for Cocktail Party Speech Emotion Recognition

Authors: Youjun Chen, Guinan Li, Mengzhe Geng, Xurong Xie, Shujie Hu, Huimeng Wang, Haoning Xu, Chengxi Deng, +4 more

Organizations: The Chinese University of Hong Kong, Hong Kong SAR, China · National Research Council Canada, Canada · Institute of Software, Chinese Academy of Sciences, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

This paper highlights the critical importance of multi-channel speech enhancement (MCSE) for speech emotion recognition (ER) in cocktail party scenarios. A multi-channel speech dereverberation and separation front-end integrating DNN-WPE and mask-based MVDR is used to extract the target speaker's speech from the mixture speech, before being fed into the downstream ER back-end using HuBERT- and ViT-based speech and visual features. Experiments on mixture speech constructed using the IEMOCAP and MSP-FACE datasets suggest the MCSE output consistently outperforms domain fine-tuned single-channel speech representations produced by: a) Conformer-based metric GANs; and b) WavLM SSL features with optional SE-ER dual task fine-tuning. Statistically significant increases in weighted, unweighted accuracy and F1 measures by up to 9.5%, 8.5% and 9.1% absolute (17.1%, 14.7% and 16.0% relative) are obtained over the above single-channel baselines. The generalization of IEMOCAP trained MCSE front-ends are also shown when being zero-shot applied to out-of-domain MSP-FACE data.

Figures & tables

Figure 1: Illustration of multi-channel speech enhancement (MCSE) and emotion recognition (ER) system. The MCSE front-end integrates DNN-WPE based speech dereverberation and mask-based MVDR speech separation components. Three different emotion recognition decoders are designed for audio-only and audio-visual ER systems, including: (a) audio-only ER system (MCSE-ER (Audio-only)) , (b) early-fusion audio-visual ER system (MCSE-ER (AV-Early)) using cross-modal attention-based fusion block and (c) late-fusion audio-visual ER system (MCSE-ER (AV-Late)) using the weights of audio ( waw_{a} ) and visual ( wvw_{v} ) to calculate the final classified probability.
IDSystemExperimental SetupSE Front-endER Back-end
 SE Front-endER Back-endTrained & Evaluated on IEMOCAPFine-tuned & Evaluated on IEMOCAPFine-tuned & Evaluated on MSP-FACE (Via zero-shot SE)
 
 
 #Chn.Audio Enc.Visual Enc.AV FusionSRMR ↑\uparrowPESQ ↑\uparrowSTOI ↑\uparrowWA% ↑\uparrowUA% ↑\uparrowF1% ↑\uparrowWA% ↑\uparrowUA% ↑\uparrowF1% ↑\uparrow
The Raw First Channel of Mixture Speech3.161.160.48/
1WavLM [ 31 ] + ER fine-tuningSingleWavLM///54.355.655.138.432.131.9
Table 1: Performance comparison between multi-channel speech enhancement (MCSE) front-end and single-channel speech enhancement baselines for audio-only (Sys. 5 vs. Sys.1-4) and audio-visual (Sys. 10 vs. Sys.6-9) emotion recognition (ER) on multi-channel IEMOCAP and MSP-FACE datasets. When these systems are evaluated on MSP-FACE, the IEMOCAP trained SE front-end is directly used to produce enhanced speech for ER back-end fine-tuning, marked as “zero-shot SE” in the last column of this table. “ ∗\ast ”, “ †\dagger ”, “ ‡\ddagger ” and “ ∘\circ ” represent statistically significant (Paired Single-tailed T-test [ 34 ] , pp =0.05) accuracy improvements over Sys.2, Sys.4, Sys.7 and Sys.9, respectively.
IDSystemSE Front-endER Back-end
SRMRPESQSTOIWA%UA%F1%
1MCSE-ER(Audio-only)6.692.820.7665.266.265.9
2w/o dervb.5.522.560.7063.263.964.0 ∗
3w/o sep.5.831.730.6656.657.256.8 ∗
4w/o dervb. & sep.3.161.160.4852.554.253.2 ∗
5MCSE-ER(AV-Late)6.692.820.7675.978.177.5
Table 2: Ablation studies on the importance of MCSE front-end components and audio-visual fusion approaches for ER on IEMOCAP data. “ ∗\ast ”, “ †\dagger ” and ‡\ddagger represent a statistically significant (Paired Single-tailed T-test, pp =0.05) accuracy decrease over Sys.1, Sys.5 and Sys.9 with both speech dereverberation (dervb.) and separation (sep.) components in the MCSE front-ends, respectively.

Explore similar work

CardsList

  1. From Speech to Interaction: Analyzing Multimodal Systems in Cocktail-Party Scenarios

    Aug 9, 2026Thai-Binh Nguyen, Zhaolin Li, Jan Niehues +1Target Speaker ExtractionAutomatic Speech Recognition

    Karlsruhe Institute of Technology Karlsruhe, Germany · Carnegie Mellon University Pittsburgh, USA

  2. SISER: Speaker-Invariant Speech Emotion Recognition with Entropy-Based Adversarial Training

    Aug 31, 2026Eunseo Choi, Hyunku Kang, Chanwoo KimSpeech Emotion RecognitionInvariant Representation Learning

    Korea Univiersity, South Korea

  3. SEER: Source-Conditioned Emotion Enhancement via Retrieval for Cochlear-Implant Speech

    Oct 8, 2026Hsing-Hang Chou, Yun-Shao Lin, Ching-Chin Sung +1Speech ProcessingSpeech Enhancement

    Department of Electrical Engineering, National Tsing Hua University, Taiwan · Research Center for Information Technology Innovation, Academia Sinica, Taiwan