Speech Separation

Follow this topic Ask about this topic

Speech separation studies how to recover individual speaker streams from mixtures, including recordings with overlap, noise, reverberation, or moving speakers. Common approaches use time- or frequency-domain neural networks, spatial cues from microphone arrays, and target-speaker conditioning; newer generative methods use diffusion, flow matching, or autoregressive audio models. Systems are increasingly evaluated on real conversations and by downstream tasks such as speech recognition and diarization, not only signal-quality scores. Key challenges include generalizing beyond simulated mixtures, keeping speaker identities aligned across long recordings, and balancing quality with streaming latency and computation.

Latest papers

36 papers7 papers in the last four weeks, up 75% on the four weeks before. 0.0% of all new papers.

SLM-SS: Token-Based Speech Separation with Speech Language Models
Oct 20, 2026Chenda Li, Tianhua Li, Yiming Chen +4Speech Separation Speech Language Models
Auditory Cognition and Computational Acoustics Lab, Shanghai Jiao Tong University, Shanghai, 200240, P. R. China
Beyond Two Tracks: Speaker-Anchored Full-Duplex Dialogue Recovery
Oct 19, 2026Yuxuan Chen, Yang Xiao, Jack Sun +3Full-Duplex Spoken Dialogue Systems Speaker Diarization
The Chinese University of Hong Kong, Shenzhen · The University of Melbourne · Lulula
Soloist: A Generative Model for Blind Voice Separation in the Wild
Oct 14, 2026Chen Yang, ChuFan Yu, Botian Jiang +5Audio Source Separation Speech Separation
Fudan University · Shanghai Innovation Institute
Frontend-Assisted Large Audio-Language Models for SLT2026 SmartGlasses Challenge
Oct 9, 2026Yushi Wang, Zeyan Song, Xiaobin Rong +3Speaker-Attributed ASR Automatic Speech Recognition
Key Laboratory of Modern Acoustics, Nanjing University, Nanjing, China · NJU-Horizon Intelligent Audio Lab, Horizon Robotics, Beijing, China · Qwen Business Unit of Alibaba, Hangzhou, China
VM-ARRAYDPS: Virtual Microphone Augmented Diffusion Posterior Sampling for Unsupervised Blind Speech Separation
Oct 7, 2026Jingqi Sun, Haozhan Tang, Shulin He +1Speech Separation Microphone Array Processing
Department of Computer Science and Engineering Southern University of Science and Technology, Shenzhen, China
SEAL: Mixture-Closed Additive Reconstruction and Refinement-Aware Expert Routing for Efficient Speech Separation
Oct 5, 2026Shao-Chun Hu, Zi-Xiang Lin, Jeih-Weih Hung +1Speech Separation Sparse Mixture-of-Experts
Department of Electrical Engineering, National Chi Nan University · Graduate Institute of AI Interdisciplinary Applied Technology, National Taiwan Normal University
SepRQ : Self-Supervised Speech Mixture Representation Learning via Mask-Free, Multi-Scale Source Separation
Oct 3, 2026Séverin Baroudi, Hervé Bredin, Ricard MarxerSelf-Supervised Speech Representation Learning Speech Separation
Univ Toulon, Aix Marseille Univ, CNRS, LIS, Toulon, France · CNRS, ILLS, Montr´eal, Canada · pyannoteAI, Toulouse, France
RadarVox: Radar-Audio Multimodal Cocktail-Party Speech Separation with Speaker-Aware Cross-Modal Matching
Sep 20, 2026Yanlin Xu, Yiwei Ru, Mupei Li +3Speech Separation
New Laboratory of Pattern Recognition (NLPR), State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences (CASIA), Beijing 100190, China · School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS), Beijing 100049, China · Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China +2
Neural Multichannel Distant Speaker Diarization with Heavy-tailed Source Separation Model
Sep 10, 2026Sicheng Mao, Baihan Li, Mathieu Fontaine +2Audio Source Separation Speaker Diarization
LTCI, Telecom Paris, Institut Polytechnique Paris, France · SJTU Paris Elite Institute of Technology, Shanghai Jiao Tong University, China · LIUM, Le Mans University, France
EConv-TasNet: Efficient Conv-TasNet for Effective Speech Separation
Sep 10, 2026Pei-Chun Chang, Chuan-Yi LiuEfficient Neural Network Inference Temporal Convolutional Networks
Novatek Microelectronics Corporation Hsinchu, Taiwan
Geometric Ceilings on Time-Frequency Masking for Single-Channel Separation
Sep 3, 2026Maxime BaeldeMusic Information Retrieval Audio Source Separation
Independent researcher, Lille, France
Dynamic Clustering for Cross-Segment Permutation Alignment in Long Speech Separation
Aug 10, 2026Yuzhu Wang, Archontis Politis, Konstantinos Drossos +1Audio Source Separation Speech Separation
Signal Processing Research Centre, Tampere University, Tampere, Finland · Nokia Technologies, Espoo, Finland
DAVE: A Decoupled Audio-Visual Enhancement Framework for Real-World Speech Separation
Aug 10, 2026Wei Zhou, Wanyi Ning, Yinshang Guo +3Speech Enhancement Noise Robustness
1Yijiahe AI · 2Tianjin University · 3Nanjing University
From Speech to Interaction: Analyzing Multimodal Systems in Cocktail-Party Scenarios
Aug 9, 2026Thai-Binh Nguyen, Zhaolin Li, Jan Niehues +1Target Speaker Extraction Automatic Speech Recognition
Karlsruhe Institute of Technology Karlsruhe, Germany · Carnegie Mellon University Pittsburgh, USA
SAGE: Switch-Aware EEG-Guided Soft Gating for Target Speaker Extraction with In-Trial Switching
Aug 3, 2026Xuefei Wang, Ximin Chen, Yuting Ding +2Target Speaker Extraction Audio Source Separation
Department of Electronic and Electrical Engineering, Southern University of Science and Technology, Shenzhen, China · School of Biomedical Engineering, Capital Medical University, Beijing, China
TF-MossFormer: Integrating Convolution Gated Local-Global Attentions for Enhanced Time-Frequency Domain Monaural Speech Separation
Jul 23, 2026Shengkui Zhao, Zexu Pan, Haoxu Wang +3Hybrid CNN-Transformer Architectures Frequency-Domain Feature Learning
Token Foundry, Alibaba Group, Singapore
SLT 2026 REAL-TSE Challenge: Real-world Target Speaker Extraction from Conversational Recordings
Jul 16, 2026Shuai Wang, Zihan Qian, Ke Zhang +9Target Speaker Extraction Speech Processing
Nanjing University, China · Chinese University of Hong Kong (Shenzhen), China · Brno University of Technology, Czechia +3
Teaching Speech Enhancement Models to Sing: Domain Adaptation from Speech Enhancement to Singing Voice Separation
Jul 13, 2026Paul A. Bereuter, Mark D. Plumbley, Alois SontacchiFine-Tuning Domain Adaptation
Institute of Electronic Music and Acoustics, University of Music and Performing Arts, Graz, Austria · Department of Informatics, King’s College London, London, United Kingdom
The SonicAGI System for the REAL-TSE Challenge
Jul 13, 2026Kai Li, Wendi Sang, Jintao Cheng +1Target Speaker Extraction Speech Processing
Department of Computer Science and Technology, Institute for AI, BNRist, Tsinghua University, Beijing, China · IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing, China · Chinese Institute for Brain Research (CIBR), Beijing, China
PS4: Proxy-Supervised Joint Training for Real Target Speaker Extraction
Jul 9, 2026Wanyi Ning, Wei Zhou, Yingpeng Li +3Target Speaker Extraction Speech Processing
Yijiahe AI, Nanjing, China · Tianjin University, Tianjin, China · Nanjing University, Nanjing, China
Flow Matching-Based Speech Source Separation with Best-of-N Biometric Sampling
Jul 7, 2026Anastasia Zorkina, Alexandr Anikin, Nikita Khmelev +5Flow Matching Speech Processing
ITMO University, Speech Processing Group, Russia · Speech Technology Center Ltd., R&D department, Russia.
TF-MoE: Time-Frequency Mixture-of-Experts for Efficient Speech Separation
Jun 28, 2026Qinzhe Hu, Chenda Li, Wangyou Zhang +3Sparse Mixture-of-Experts Speech Processing
Auditory Cognition and Computational Acoustics Lab Shanghai Jiao Tong University, Shanghai, China · Microsoft Research Asia, China
Position-Aware Target Speaker Extraction for Long-Form Multi-Party Conversations: A Diarization-Free Framework for ASR
Jun 28, 2026Yichi Wang, Junzhe Chen, Wangjin Zhou +1Target Speaker Extraction Speech Processing
Graduate School of Informatics, Kyoto University, Kyoto, Japan
MeCo: One-Step MeanFlow-based Corrector for Multi-Channel Speech Separation
Jun 8, 2026Dohwan Kim, Jung-Woo ChoiFlow Matching MeanFlow
School of Electrical Engineering, KAIST, Daejeon, Republic of Korea
Feasibility of Time-Domain DNN-Based Speech Enhancement on Embedded FPGA for Hearing Aid
Jun 2, 2026Feyisayo Olalere, Umut Altin, Kiki van der Heijden +1Efficient Neural Network Inference Speech Processing
Radboud University, Donders Institute for Brain, Cognition, and Behaviour, The Netherlands · Mortimer B. Zuckerman Mind, Brain, Behavior Institute, Columbia University, USA.
Echo: A Joint-Embedding Predictive Architecture for Speaker Diarization and Speech Recognition in a Shared Latent Space
Jun 1, 2026Louis MouchonAudio Representation Learning Joint-Embedding Predictive Architecture
Independent Research
Cross-Talk Speech Reduction, by Separation, for Separation
May 19, 2026Zhong-Qiu Wang, Samuele CornellSpeech Processing Speech Separation
Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen 518055, China · Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA
Predictive-Generative Drift Decomposition for Speech Enhancement and Separation
May 7, 2026Julius Richter, Yoshiki Masuyama, Christoph Boeddeker +3Stochastic Interpolants Speech Processing
MERL Cambridge, MA, USA
DialogueSidon: Recovering Full-Duplex Dialogue Tracks from In-the-Wild Dialogue Audio
Apr 10, 2026Wataru Nakata, Yuki Saito, Kazuki Yamauchi +2Audio Diffusion Models Speech Processing
The University of Tokyo, Tokyo, Japan · National Institute of Advanced Industrial Science and Technology (AIST), Tokyo, Japan
Asymmetric Encoder-Decoder Based on Time-Frequency Correlation for Speech Separation
Mar 31, 2026Ui-Hyeop Shin, Hyung-Min ParkSpeech Separation Speaker Disentanglement
The authors are with the Department of Electronic Engineering, Sogang University, Seoul 04107, Republic of Korea

Oct 20, 2026J/K move · Enter open · S save

SLM-SS: Token-Based Speech Separation with Speech Language Models

Chenda Li, Tianhua Li, Yiming Chen, Wei Wang+3

Auditory Cognition and Computational Acoustics Lab, Shanghai Jiao Tong University, Shanghai, 200240, P. R. China

Speech Separation Speech Language Models

Open paper AskPDF arXiv

Speech separation (SS) has witnessed substantial progress in recent years with the advent of neural network-based methods, achieving notable improvements on signal-level evaluation metrics. Existing speech separation systems are mostly optimized with signal-level objectives, and often yield outputs that are acceptable in waveform metrics but suboptimal for intelligibility-sensitive downstream tasks. To address this gap, we propose SLM-SS, a token-based generative speech separation framework that models multi-talker mixtures with speech language models. We first discretize clean speech into multi-codebook token sequences with neural codecs and serialize multi-speaker targets via Serialized Output Training (SOT). An encoder-decoder autoregressive speech language model predicts the zero-order sequence to establish speaker ordering and coarse linguistic content, and a non-autoregressive module then refines higher-order residual codebooks for efficient reconstruction. We further introduce a per-talker non-autoregressive strategy that simplifies high-order modeling and improves the stability. On Libri2Mix, SLM-SS with per-talker NAR achieves

6.92\%

WER, approaching codec-only reconstruction. The WavTokenizer variant eliminates NAR refinement and achieves the highest mean subjective quality score among evaluated separation systems. Its UTMOS and DNSMOS advantages over discriminative baselines persist on unseen noisy and Chinese mixtures without fine-tuning.