Distant speaker diarization remains challenging due to adverse acoustic conditions, varying numbers of speakers and overlapping speech. While data-driven approaches have shown strong performance, model-driven methods offer a compelling alternative by leveraging spatial information from multichannel recordings. This paper is motivated to propose a Bayesian diarization model for a model-driven method called neural FCASA to enhance its robustness. Specifically, we propose a beta prior over speaker activity and hence a variational lower bound objective that can be seen as a regularized continuous speaker activity score in place of the original cross-entropy loss to train the diarization model. Our experiments show significant improvements in terms of Diarization Error Rate by at least 3% (16% relatively) and Jaccard Error Rate by at least 4% (20% relatively) on the AMI dataset compared to the baseline.
Figures & tables
Figure 1: The overview of our model, training and inference
Figure 2: Beta Distribution with different hyperparameters
Figure 3: Beta distribution with different PERT hyperparameters
m,λ
Setup
Miss ( ↓ )
FA ( ↓ )
Conf. ( ↓ )
DER ( ↓ )
JER ( ↓ )
(baseline)
Forgiving
5.59
8.16
0.73
14.48
13.50
Fair
8.61
6.33
0.79
15.73
23.16
Full
10.77
6.81
1.16
18.73
27.65
Overlap
19.40
3.80
0.90
24.11
26.53
0.3, 4
Forgiving
5.37
4.31
0.43
10.11
9.73
Fair
8.00
3.39
0.44
11.84
16.85
Table 1: Diarization metrics of the trained models on AMI evaluation set. All metrics are reported in percentage with the best score in bold and the second-best score u nderlined. We also list the single channel diarization baseline Pyannote 3.1 [ 7 ] as a reference. Note that the evaluation of our method is performed chunk-wisely (10 seconds), which is not directly comparable to the Pyannote baselines.
Distant speaker diarization remains challenging due to difficult acoustic environments, varying numbers of speakers and overlapping speech. Model-driven methods are proposed to exploit the speech source features in multi-channel recordings that help diarization. This paper generalizes a neural model that jointly learns to perform blind source separation and diarization over speech mixtures (neural FCASA) with heavy-tailed models. The popular Gaussian distribution has been applied for variance modeling in the original source separation model, which we replace with two families of heavy-tailed models (Leptokurtic Generalized Gaussian distribution and Student's t distribution) to better capture the heavy-tailedness in speech signals. Thanks to the Gaussian scale mixture model, we are able to unify the proposed method and the original one under the same form of learning objective. Our experiments show consistent large improvements in Diarization Error Rate (DER) and Jaccard Error Rate (JER) compared to the baseline on various corpora.
Sicheng Mao, Baihan Li, Mathieu Fontaine +2
LTCI, Telecom Paris, Institut Polytechnique Paris, France · SJTU Paris Elite Institute of Technology, Shanghai Jiao Tong University, China · LIUM, Le Mans University, France
This paper presents S2SND-v2, a unified online and offline speaker diarization framework built on S2SND that improves acoustic feature extraction and the use of the resulting representations. For acoustic feature extraction, we introduce a speaker-adapted w2v-BERT 2.0 front end and compare full and pruned construction routes. To better use these features, overlap-aware activity-query masking improves speaker embedding extraction under overlap, while late-speaker training aligns unknown-speaker training with online inference conditions. Combining these improvements, the pruned system achieves competitive average diarization error rates of 14.52% online and 12.03% offline across eight test sets.
Jincheng Wang, Li Li, Weixin Zhu +4
School of Artificial Intelligence, Wuhan University, Wuhan, China · School of Artificial Intelligence, the Chinese University of Hong Kong, Shenzhen, China · Tencent Ethereal Audio Lab, Tencent, Shenzhen, China
In this paper, we introduce DiarizationLM, a framework to leverage large language models (LLM) to post-process the outputs from a speaker diarization system. Various goals can be achieved with the proposed framework, such as improving the readability of the diarized transcript, or reducing the word diarization error rate (WDER). In this framework, the outputs of the automatic speech recognition (ASR) and speaker diarization systems are represented as a compact textual format, which is included in the prompt to an optionally finetuned LLM. The outputs of the LLM can be used as the refined diarization results with the desired enhancement. As a post-processing step, this framework can be easily applied to any off-the-shelf ASR and speaker diarization systems without retraining existing components. Our experiments show that a finetuned PaLM 2-S model can reduce the WDER by rel. 55.5% on the Fisher telephone conversation dataset, and rel. 44.9% on the Callhome English dataset.