MOUSSE: Multilingual Overlapped Speech Separation with Predictive Uncertainty Estimation
Authors: Su Yeon Shim, Yejin Jeon, David Ifeoluwa Adelani, Young-Joo Suh
Organizations: KT Corporation · Mila - Quebec AI Institute · McGill University · Canada CIFAR AI Chair · Graduate School of Artificial Intelligence, POSTECH · Department of Computer Science and Engineering, POSTECH
Recent generative speech separation methods have shown promising performance, yet mixtures of overlapping speech in different languages remain underexplored. To this end, we first curate MuSSeD, an English-Korean benchmark comprising two- to four- speaker mixtures across varying overlap ratios. We further propose MOUSSE, a flow matching framework that incorporates boundary-aware training and frame-wise predictive uncertainty through a permutation-invariant Gaussian negative log-likelihood. The predicted scale is further used to adapt classifier-free guidance during sampling. MOUSSE achieves 13.18 dB SI-SDRi and 0.175 ESTOIi, outperforming discriminative and flow matching-based separation baselines, with a 1.58 dB SI-SDRi improvement over the strongest model.
Figures & tables
Figure 1: Boundary-aware weight maps for two- to four-speaker mixtures, which illustrate frames near activity boundaries.
Figure 2: Overview of MOUSSE with boundary-aware training and adaptive uncertainty-guided sampling.
Model
SDRi
SI-SDRi
ESTOIi
Conv-TASNET [ 8 ]
9.60
9.72
0.032
ReSepformer [ 3 ]
5.51
5.56
-0.003
SR-CorrNet [ 12 ]
11.59
11.60
0.092
SpeechFlow [ 7 ]
5.04
3.65
-0.072
MOUSSE (Ours)
13.20
13.18
0.175
Table 1: Overall separation performance on MuSSeD.
Configuration
SDRi
SI-SDRi
ESTOIi
Full model
13.20
13.18
0.175
– Boundary-aware weighting
12.11
12.10
0.164
– NLL loss ( λ=0.166 )
11.95
11.93
0.142
Fixed guidance ( λ=0.166 )
13.00
12.98
0.172
PIT MSE training
10.53
10.52
0.138
Table 2: Ablation study of MOUSSE components.
2 spk
3 spk
4 spk
SI-SDRi
Lang.
Eng.
0.166
0.168
0.172
14.79
Kor.
0.193
0.193
0.181
10.81
Mixed
0.168
0.171
0.170
13.89
Overlap Ratio
0%
0.149
0.137
0.131
13.77
25%
0.166
0.152
0.142
13.95
50%
0.181
0.175
0.162
13.21
Table 3: Mean conditional predicted scale σcond and SI-SDRi across language compositions and overlap ratios.
Figure 3: Predictive uncertainty and adaptive guidance analysis.
Speech separation (SS) has witnessed substantial progress in recent years with the advent of neural network-based methods, achieving notable improvements on signal-level evaluation metrics. Existing speech separation systems are mostly optimized with signal-level objectives, and often yield outputs that are acceptable in waveform metrics but suboptimal for intelligibility-sensitive downstream tasks. To address this gap, we propose SLM-SS, a token-based generative speech separation framework that models multi-talker mixtures with speech language models. We first discretize clean speech into multi-codebook token sequences with neural codecs and serialize multi-speaker targets via Serialized Output Training (SOT). An encoder-decoder autoregressive speech language model predicts the zero-order sequence to establish speaker ordering and coarse linguistic content, and a non-autoregressive module then refines higher-order residual codebooks for efficient reconstruction. We further introduce a per-talker non-autoregressive strategy that simplifies high-order modeling and improves the stability. On Libri2Mix, SLM-SS with per-talker NAR achieves 6.92% WER, approaching codec-only reconstruction. The WavTokenizer variant eliminates NAR refinement and achieves the highest mean subjective quality score among evaluated separation systems. Its UTMOS and DNSMOS advantages over discriminative baselines persist on unseen noisy and Chinese mixtures without fine-tuning.
Chenda Li, Tianhua Li, Yiming Chen +4
Auditory Cognition and Computational Acoustics Lab, Shanghai Jiao Tong University, Shanghai, 200240, P. R. China
Target speaker extraction (TSE) isolates a desired speaker from a multi-speaker mixture using cues that identify the target. Prior work has explored acoustic and textual cues, such as enrollment audio and different types of speaker descriptors, but typically relies on a single cue family. Yet no single cue is both broadly available and reliably discriminative. Moreover, most TSE systems adopt mask-prediction models, which are prone to audible artifacts and generalize poorly across acoustic conditions. To address these limitations, we propose PromptSep-Speaker, a generative TSE model based on latent diffusion that unifies four interchangeable cues: speaker temporal order, transcript, enrollment audio, and speaker caption. The generative formulation delivers higher separation quality and multi-modal prompting improves applicability across scenarios. To fully exploit each cue, we further introduce a frozen WavLM representation with an auxiliary alignment loss that supervises query-mixture cross-attention without extra labeling. Across four evaluation sets and a listening study, PromptSep-Speaker achieves state-of-the-art performance over TSE and universal source separation baselines, improving separation accuracy and quality across cue modalities.
Yutong Wen, Ke Chen, Prem Seetharaman +5
Adobe Research, USA · Massachusetts Institute of Technology, USA
While discriminative models for multi-channel speech separation excel in reference-based metrics, they often exhibit suboptimal human listening quality. To address this, we propose a novel MeanFlow-based one-step generative corrector (MeCo). MeCo learns a conditional average velocity field to map discriminative estimates directly onto the clean speech manifold in a single step. To maximize one-step generation performance, we introduce Data-Space Optimization (DSO). DSO integrates an xr-loss, which penalizes prediction errors on longer displacement intervals to serve as a generative objective for human listening quality, with an Endpoint SI-SDR loss that directly optimizes terminal signal fidelity. Experiments demonstrate that MeCo achieves state-of-the-art (SOTA) performance with minimal computational overhead, simultaneously achieving superior signal fidelity and human listening quality in both in-domain and out-of-domain scenarios.
Dohwan Kim, Jung-Woo Choi
School of Electrical Engineering, KAIST, Daejeon, Republic of Korea