Joint Learning Global-Local Speaker Classification to Enhance End-to-End Speaker Diarization and Recognition

Yuhang Dai; Haopeng Lin; Jiale Qian; Ruiqi Yan; Hao Meng; Hanke Xie; Hanlin Wen; Shunshun Yin

cs.SDMar 26, 2026

Joint Learning Global-Local Speaker Classification to Enhance End-to-End Speaker Diarization and Recognition

Authors: Yuhang Dai, Haopeng Lin, Jiale Qian, Ruiqi Yan, Hao Meng, Hanke Xie, Hanlin Wen, Shunshun Yin,

Organizations: Soul AI Lab, China · X-LANCE Lab, Shanghai Jiao Tong University, China

Abstract

Large Audio-Language Models (LALMs) have demonstrated remarkable performance in end-to-end speaker diarization and recognition. However, their speaker discriminability remains limited due to the scarcity of large-scale conversational data and the absence of explicit speaker representation optimization. To address this, we propose GLSC-SDR, a paradigm that jointly trains speaker classification with diarization and recognition. We further introduce a Global-Local Speaker Classification strategy, which uses clustered speakers as global labels and re-encoded intra-cluster speakers as local labels. This hierarchical design enhances fine-grained speaker discrimination while preserving semantic transcription accuracy. Experiments on AliMeeting, AISHELL-4, and AMI-SDM demonstrate that GLSC-SDR achieves competitive or superior performance compared to simulation-based and multi-encoder approaches, without relying on large-scale real conversational data.

Figures & tables

Figure 1: Joint training Global-Local Speaker Classification and Speech Diarization and Recognition paradigm.

Model	Alimeeting				AISHELL-4				AMI-SDM
Model	WER↓	cpWER↓	$\triangle$ cp↓	SCA↑	WER↓	cpWER↓	$\triangle$ cp↓	SCA↑	WER↓	cpWER↓	$\triangle$ cp↓	SCA↑
Related work
TagSpeech [ 7 ]	25.42	33.84	8.42	81.63	-	-	-	-	31.62	42.55	10.93	70.01
VibeVoice-ASR [ 18 ]	27.4	29.33	1.93	-	21.4	24.99	3.59	-	24.65	28.82	4.17	-
Gemini-2.5-pro*	27.43	41.64	14.2	-	22.42	31.59	9.17	-	22.35	34.78	12.43	-
Gemini-3-pro*	26.75	32.84	6.09	-	22.75	27.43	4.68	-	22.09	26.91	4.82	-

Table 1: Main results in Alimeeting, AISHELL-4 and AMI-SDM. (* indicates the results are coverted from [ 18 ] )

Figure 2: GLSC data construction pipeline.

Classification Method	WER↓	cpWER↓	$\triangle$ cp↓	SCA↑
GSC-SDR	22.61	30.81	8.2	78.56
SC-SDR	22.83	30.8	8.06	78.56
GLSC-SDR	22.71	29.93	7.22	79.74

Table 2: The impact of different speaker classification methods on SDR effectiveness shown on Alimeeting.

exp_group	cluster num	WER $\downarrow$	cpWER $\downarrow$	$\triangle$ cp $\downarrow$	SCA $\uparrow$
baseline	-	23.33	32.18	8.85	77.84
GLSC_Kmeans	50	23.05	31.53	8.48	72.2
GLSC_Kmeans	200	22.58	30.32	7.74	78.58
GLSC_Kmeans	430	22.7	31.28	8.23	78.02
GLSC_HDBSCAN	430	22.71	29.93	7.22	79.74

Table 3: The impact of cluster number on GLSC and cluster algorithm, results shown on Alimeeting.

Speaker Diarization

Speaker diarization identifies who spoke when in audio, including speaker changes, overlapping speech, and speaker identity across conversations.

See all 60 papers

Balancing ASR and diarization in end-to-end LLMs for multi-talker speech recognition
Jun 11, 2026Naijun Zheng, Yuke Lin, Sanli Tian +4Speaker-Attributed ASR Multi-Speaker ASR
Huawei Technologies, China
TagSpeech: End-to-End Multi-Speaker ASR and Diarization with Fine-Grained Temporal Grounding
Jan 11, 2026Mingyue Huo, Yiwen Shao, Yuheng ZhangMulti-Speaker ASR Audio Understanding
University of Illinois Urbana-Champaign · Johns Hopkins University
DiarizationLM: Speaker Diarization Post-Processing with Large Language Models
Jan 7, 2024Quan Wang, Yiling Huang, Guanlong Zhao +3Speaker Diarization
Google LLC, USA

Jun 11, 2026J/K move · Enter open · S save

Balancing ASR and diarization in end-to-end LLMs for multi-talker speech recognition

Naijun Zheng, Yuke Lin, Sanli Tian, Mengtian Li+3

Huawei Technologies, China

Speaker-Attributed ASR Multi-Speaker ASR Automatic Speech Recognition Speech Language Models Speaker Diarization

Open paper AskPDF arXiv

Multi-talker speech recognition is often addressed by combining automatic speech recognition (ASR) and speaker diarization in a pipeline system. Recently, LLM-based approaches have shown promise by jointly modeling semantic and speaker information, but they typically require large-scale multi-talker corpora that are costly to annotate. In this paper, we investigate how to efficiently train an LLM-based system with limited real-recorded data while maintaining high accuracy in speaker attribution. We propose several strategies: (1) a dual-encoder architecture to extract semantic and speaker features, (2) a feature interleaving format to merge these features as the inputs to the LLM, (3) a length-aware speaker ID loss to enhance diarization capability, and (4) an adaptive threshold strategy for ASR loss computation to mitigate hallucinations caused by speech overlaps. These strategies balance training between ASR and diarization tasks. Our system outperforms open-source baseline approaches, achieving relative improvements of 18% on the AliMeeting corpus and 24% on the Aishell4 corpus.

Abstract

Speaker Diarization

Explore similar work

Balancing ASR and diarization in end-to-end LLMs for multi-talker speech recognition

TagSpeech: End-to-End Multi-Speaker ASR and Diarization with Fine-Grained Temporal Grounding

DiarizationLM: Speaker Diarization Post-Processing with Large Language Models