Large Audio-Language Models (LALMs) have demonstrated remarkable performance in end-to-end speaker diarization and recognition. However, their speaker discriminability remains limited due to the scarcity of large-scale conversational data and the absence of explicit speaker representation optimization. To address this, we propose GLSC-SDR, a paradigm that jointly trains speaker classification with diarization and recognition. We further introduce a Global-Local Speaker Classification strategy, which uses clustered speakers as global labels and re-encoded intra-cluster speakers as local labels. This hierarchical design enhances fine-grained speaker discrimination while preserving semantic transcription accuracy. Experiments on AliMeeting, AISHELL-4, and AMI-SDM demonstrate that GLSC-SDR achieves competitive or superior performance compared to simulation-based and multi-encoder approaches, without relying on large-scale real conversational data.
Figures & tables
Figure 1: Joint training Global-Local Speaker Classification and Speech Diarization and Recognition paradigm.
Model
Alimeeting
AISHELL-4
AMI-SDM
WER↓
cpWER↓
△ cp↓
SCA↑
WER↓
cpWER↓
△ cp↓
SCA↑
WER↓
cpWER↓
△ cp↓
SCA↑
Related work
TagSpeech [ 7 ]
25.42
33.84
8.42
81.63
-
-
-
-
31.62
42.55
10.93
70.01
VibeVoice-ASR [ 18 ]
27.4
29.33
1.93
-
21.4
24.99
3.59
-
24.65
28.82
4.17
-
Gemini-2.5-pro*
27.43
41.64
14.2
-
22.42
31.59
9.17
-
22.35
34.78
12.43
-
Gemini-3-pro*
26.75
32.84
6.09
-
22.75
27.43
4.68
-
22.09
26.91
4.82
-
Table 1: Main results in Alimeeting, AISHELL-4 and AMI-SDM. (* indicates the results are coverted from [ 18 ] )
Figure 2: GLSC data construction pipeline.
Classification Method
WER↓
cpWER↓
△ cp↓
SCA↑
GSC-SDR
22.61
30.81
8.2
78.56
SC-SDR
22.83
30.8
8.06
78.56
GLSC-SDR
22.71
29.93
7.22
79.74
Table 2: The impact of different speaker classification methods on SDR effectiveness shown on Alimeeting.
exp_group
cluster num
WER ↓
cpWER ↓
△ cp ↓
SCA ↑
baseline
-
23.33
32.18
8.85
77.84
GLSC_Kmeans
50
23.05
31.53
8.48
72.2
GLSC_Kmeans
200
22.58
30.32
7.74
78.58
GLSC_Kmeans
430
22.7
31.28
8.23
78.02
GLSC_HDBSCAN
430
22.71
29.93
7.22
79.74
Table 3: The impact of cluster number on GLSC and cluster algorithm, results shown on Alimeeting.
Multi-talker speech recognition is often addressed by combining automatic speech recognition (ASR) and speaker diarization in a pipeline system. Recently, LLM-based approaches have shown promise by jointly modeling semantic and speaker information, but they typically require large-scale multi-talker corpora that are costly to annotate. In this paper, we investigate how to efficiently train an LLM-based system with limited real-recorded data while maintaining high accuracy in speaker attribution. We propose several strategies: (1) a dual-encoder architecture to extract semantic and speaker features, (2) a feature interleaving format to merge these features as the inputs to the LLM, (3) a length-aware speaker ID loss to enhance diarization capability, and (4) an adaptive threshold strategy for ASR loss computation to mitigate hallucinations caused by speech overlaps. These strategies balance training between ASR and diarization tasks. Our system outperforms open-source baseline approaches, achieving relative improvements of 18% on the AliMeeting corpus and 24% on the Aishell4 corpus.
We present TagSpeech, a unified LLM-based framework that utilizes Temporal Anchor Grounding for joint multi-speaker ASR and diarization. The framework is built on two key designs: (1) decoupled semantic and speaker streams fine-tuned via Serialized Output Training (SOT) to learn turn-taking dynamics; and (2) an interleaved time anchor mechanism that not only supports fine-grained timestamp prediction but also acts as a synchronization signal between semantic understanding and speaker tracking. Compared to previous works that primarily focus on speaker-attributed ASR or implicit diarization, TagSpeech addresses the challenge of fine-grained speaker-content alignment and explicitly models "who spoke what and when" in an end-to-end manner. Experiments on AMI and AliMeeting benchmarks demonstrate that our method achieves consistent improvements in Diarization Error Rate (DER) over strong end-to-end baselines, including Qwen-Omni and Gemini, particularly in handling complex speech overlaps. Moreover, TagSpeech employs a parameter-efficient training paradigm in which the LLM backbone is frozen and only lightweight projectors are trained, resulting in strong performance with low computational cost.
Mingyue Huo, Yiwen Shao, Yuheng Zhang
University of Illinois Urbana-Champaign · Johns Hopkins University
In this paper, we introduce DiarizationLM, a framework to leverage large language models (LLM) to post-process the outputs from a speaker diarization system. Various goals can be achieved with the proposed framework, such as improving the readability of the diarized transcript, or reducing the word diarization error rate (WDER). In this framework, the outputs of the automatic speech recognition (ASR) and speaker diarization systems are represented as a compact textual format, which is included in the prompt to an optionally finetuned LLM. The outputs of the LLM can be used as the refined diarization results with the desired enhancement. As a post-processing step, this framework can be easily applied to any off-the-shelf ASR and speaker diarization systems without retraining existing components. Our experiments show that a finetuned PaLM 2-S model can reduce the WDER by rel. 55.5% on the Fisher telephone conversation dataset, and rel. 44.9% on the Callhome English dataset.