Neural Multichannel Distant Speaker Diarization and Source Separation with Beta Speaker Activity Prior

Sicheng Mao; Mathieu Fontaine; Anthony Larcher; Roland Badeau

cs.SDAug 21, 2026

Neural Multichannel Distant Speaker Diarization and Source Separation with Beta Speaker Activity Prior

Authors: Sicheng Mao, Mathieu Fontaine, Anthony Larcher, Roland Badeau

Organizations: LTCI, Telecom Paris, Institut Polytechnique de Paris, France · LIUM, Universite du Mans, France

Abstract

Distant speaker diarization remains challenging due to adverse acoustic conditions, varying numbers of speakers and overlapping speech. While data-driven approaches have shown strong performance, model-driven methods offer a compelling alternative by leveraging spatial information from multichannel recordings. This paper is motivated to propose a Bayesian diarization model for a model-driven method called neural FCASA to enhance its robustness. Specifically, we propose a beta prior over speaker activity and hence a variational lower bound objective that can be seen as a regularized continuous speaker activity score in place of the original cross-entropy loss to train the diarization model. Our experiments show significant improvements in terms of Diarization Error Rate by at least 3% (16% relatively) and Jaccard Error Rate by at least 4% (20% relatively) on the AMI dataset compared to the baseline.

Figures & tables

Figure 1: The overview of our model, training and inference

Figure 2: Beta Distribution with different hyperparameters

Figure 3: Beta distribution with different PERT hyperparameters

$m,\lambda$	Setup	Miss ( $\downarrow$ )	FA ( $\downarrow$ )	Conf. ( $\downarrow$ )	DER ( $\downarrow$ )	JER ( $\downarrow$ )
(baseline)	Forgiving	5.59	8.16	0.73	14.48	13.50
	Fair	8.61	6.33	0.79	15.73	23.16
	Full	10.77	6.81	1.16	18.73	27.65
	Overlap	19.40	3.80	0.90	24.11	26.53
0.3, 4	Forgiving	5.37	4.31	0.43	10.11	9.73
	Fair	8.00	3.39	0.44	11.84	16.85

Table 1: Diarization metrics of the trained models on AMI evaluation set. All metrics are reported in percentage with the best score in bold and the second-best score u nderlined. We also list the single channel diarization baseline Pyannote 3.1 [ 7 ] as a reference. Note that the evaluation of our method is performed chunk-wisely (10 seconds), which is not directly comparable to the Pyannote baselines.

Speaker Diarization

Speaker diarization identifies who spoke when in audio, including speaker changes, overlapping speech, and speaker identity across conversations.

See all 60 papers

Neural Multichannel Distant Speaker Diarization with Heavy-tailed Source Separation Model
Sep 10, 2026Sicheng Mao, Baihan Li, Mathieu Fontaine +2Audio Source Separation Speaker Diarization
LTCI, Telecom Paris, Institut Polytechnique Paris, France · SJTU Paris Elite Institute of Technology, Shanghai Jiao Tong University, China · LIUM, Le Mans University, France
S2SND-v2: Unified Online and Offline Speaker Diarization with Speaker-Adapted w2v-BERT 2.0
Oct 26, 2026Jincheng Wang, Li Li, Weixin Zhu +4Speaker Diarization Speaker Embedding
School of Artificial Intelligence, Wuhan University, Wuhan, China · School of Artificial Intelligence, the Chinese University of Hong Kong, Shenzhen, China · Tencent Ethereal Audio Lab, Tencent, Shenzhen, China
DiarizationLM: Speaker Diarization Post-Processing with Large Language Models
Jan 7, 2024Quan Wang, Yiling Huang, Guanlong Zhao +3Speaker Diarization
Google LLC, USA

Sep 10, 2026J/K move · Enter open · S save

Neural Multichannel Distant Speaker Diarization with Heavy-tailed Source Separation Model

Sicheng Mao, Baihan Li, Mathieu Fontaine, Anthony Larcher+1

LTCI, Telecom Paris, Institut Polytechnique Paris, France · SJTU Paris Elite Institute of Technology, Shanghai Jiao Tong University, China · LIUM, Le Mans University, France

Audio Source Separation Speaker Diarization Speech Separation

Open paper AskPDF arXiv

Distant speaker diarization remains challenging due to difficult acoustic environments, varying numbers of speakers and overlapping speech. Model-driven methods are proposed to exploit the speech source features in multi-channel recordings that help diarization. This paper generalizes a neural model that jointly learns to perform blind source separation and diarization over speech mixtures (neural FCASA) with heavy-tailed models. The popular Gaussian distribution has been applied for variance modeling in the original source separation model, which we replace with two families of heavy-tailed models (Leptokurtic Generalized Gaussian distribution and Student's t distribution) to better capture the heavy-tailedness in speech signals. Thanks to the Gaussian scale mixture model, we are able to unify the proposed method and the original one under the same form of learning objective. Our experiments show consistent large improvements in Diarization Error Rate (DER) and Jaccard Error Rate (JER) compared to the baseline on various corpora.

Abstract

Speaker Diarization

Explore similar work

Neural Multichannel Distant Speaker Diarization with Heavy-tailed Source Separation Model

S2SND-v2: Unified Online and Offline Speaker Diarization with Speaker-Adapted w2v-BERT 2.0

DiarizationLM: Speaker Diarization Post-Processing with Large Language Models