MOUSSE: Multilingual Overlapped Speech Separation with Predictive Uncertainty Estimation
Organizations: KT Corporation · Mila - Quebec AI Institute · McGill University · Canada CIFAR AI Chair ·
Abstract
Recent generative speech separation methods have shown promising performance, yet mixtures of overlapping speech in different languages remain underexplored. To this end, we first curate MuSSeD, an English-Korean benchmark comprising two- to four- speaker mixtures across varying overlap ratios. We further propose MOUSSE, a flow matching framework that incorporates boundary-aware training and frame-wise predictive uncertainty through a permutation-invariant Gaussian negative log-likelihood. The predicted scale is further used to adapt classifier-free guidance during sampling. MOUSSE achieves 13.18 dB SI-SDRi and 0.175 ESTOIi, outperforming discriminative and flow matching-based separation baselines, with a 1.58 dB SI-SDRi improvement over the strongest model.
Figures & tables
| Model | SDRi | SI-SDRi | ESTOIi |
| Conv-TASNET [ 8 ] | 9.60 | 9.72 | 0.032 |
| ReSepformer [ 3 ] | 5.51 | 5.56 | -0.003 |
| SR-CorrNet [ 12 ] | 11.59 | 11.60 | 0.092 |
| SpeechFlow [ 7 ] | 5.04 | 3.65 | -0.072 |
| MOUSSE (Ours) | 13.20 | 13.18 | 0.175 |
| Configuration | SDRi | SI-SDRi | ESTOIi |
| Full model | 13.20 | 13.18 | 0.175 |
| – Boundary-aware weighting | 12.11 | 12.10 | 0.164 |
| – NLL loss ( ) | 11.95 | 11.93 | 0.142 |
| Fixed guidance ( ) | 13.00 | 12.98 | 0.172 |
| PIT MSE training | 10.53 | 10.52 | 0.138 |
| 2 spk | 3 spk | 4 spk | SI-SDRi | ||
| Lang. | Eng. | 0.166 | 0.168 | 0.172 | 14.79 |
| Kor. | 0.193 | 0.193 | 0.181 | 10.81 | |
| Mixed | 0.168 | 0.171 | 0.170 | 13.89 | |
| Overlap Ratio | 0% | 0.149 | 0.137 | 0.131 | 13.77 |
| 25% | 0.166 | 0.152 | 0.142 | 13.95 | |
| 50% | 0.181 | 0.175 | 0.162 | 13.21 |
Speech Separation
Speech separation isolates individual voices from overlapping speech, supporting clearer communication, transcription, and speaker-aware audio processing.
See all 40 papers