TS-SP: Learning Speaker-Preserving Representations in Audio Large Language Models

Junjie Li; Zheng Liang; Zhe Li; Tianchi Liu; Kong Aik Lee

cs.SDOct 4, 2026

TS-SP: Learning Speaker-Preserving Representations in Audio Large Language Models

Authors: Junjie Li, Zheng Liang, Zhe Li, Tianchi Liu, Kong Aik Lee

Organizations: Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong SAR, China · Speech, Language, and Cognition Laboratory, The University of Hong Kong, Hong Kong SAR, China · National University of Singapore

Abstract

Audio large language models (ALLMs) can understand speech content, yet their ability to use speaker identity for verification remains limited. We propose TS-SP (Two-Stage Speaker Preservation), a parameter-efficient framework for learning speaker-preserving representations and making them accessible to an ALLM's language-model component. We instantiate and evaluate TS-SP on Qwen2.5-Omni-7B. First, we adapt the audio encoder with speaker identity supervision. We then freeze the adapted encoder and train the language model to compare speakers. Both stages use low-rank adaptation (LoRA), keeping the pretrained base weights fixed. On Vox1-O, TS-SP reduces the equal error rate (EER) from 7.01% for the Paired Loss Adaptation Baseline to 4.37%. EER remains within 4.31--4.79% under unseen prompts. Cross-domain evaluation on CN-Celeb yields a similar EER to the baseline, but lower accuracy at the native decision threshold. These findings support two-stage adaptation for improving speaker verification on the evaluated backbone.

Figures & tables

Figure 1: TS-SP: Stage 1 learns audio LoRA with intermediate speaker supervision; Stage 2 freezes the adapted audio pathway and trains LM LoRA for speaker comparison. All pretrained base weights remain fixed.

IN-DOMAIN EVALUATION
System	Adaptation	EER (%)	minDCF	Target ACC (%)	Nontarget ACC (%)	Overall ACC (%)
Vox1-O
Kimi-Audio-7B-Instruct [ 6 ]	Zero-shot	62.26	1.000	43.44	32.94	38.19
Qwen2-Audio-7B-Instruct [ 2 ]	Zero-shot	33.39	1.000	100.00	0.00	50.00
Qwen2.5-Omni-7B [ 23 ]	Zero-shot	32.03	1.000	60.99	72.55	66.77
Paired Loss Adaptation Baseline [ 16 ]	Fine-tuned	7.01	0.915	97.99	87.77	92.88

Table 1: SV results on Vox1-O/E/H and CN-Celeb. Only the Paired Loss Adaptation Baseline and TS-SP undergo task-specific fine-tuning (VoxCeleb2); other systems are evaluated zero-shot. CN-Celeb uses 348,430 trials (1,776 target, 346,654 nontarget), without domain adaptation. Lower EER/minDCF and higher ACC are better; ACC uses

s=0

. Bold marks the best value per metric and trial list.

ID	Usage	Prompt supplied with the audio pair	Positive	Negative	EER (%)	minDCF	Target ACC (%)	Nontarget ACC (%)
P0	Train/test	Are these two audio segments spoken by the same speaker? Answer only Yes or No.	Yes	No	4.37	0.611	97.49	93.82
P1	Test only	Do these two audio segments belong to the same speaker? Respond only with Yes or No.	Yes	No	4.33	0.611	97.55	93.66
P2	Test only	The two audio segments were produced by the same speaker. Is this statement true or false? Answer only True or False.	True	False	4.39	0.671	97.15	94.33
P3	Test only	Compare the speakers in the two audio segments. Output 1 if they are the same speaker and 2 otherwise.	1	2	4.79	0.824	98.69	90.44
P4	Test only	Are the speakers in the two audio segments the same or different? Answer only Same or Different.	Same	Different	4.31	0.631	96.30	95.16
P5	Test only	这两段音频是否由同一个说话人说出？只回答“是”或“不是”。	是	不是	4.50	0.643	99.88	79.88

Table 2: Prompt-robustness results of TS-SP on the cleaned Vox1-O trial list. P0 is the only prompt used in Stage 2 training, while P1–P5 are unseen test-time prompts. Positive and negative verbalizers correspond to same-speaker and different-speaker trials, respectively. Target and nontarget accuracies use the native threshold

s_{v}=0

Speaker Verification

Speaker verification determines whether two speech recordings come from the same person, enabling voice authentication and identity analysis.

See all 61 papers

Speaker Verification with Speech-Aware LLMs: Evaluation and Augmentation
Mar 11, 2026Thomas Thebaud, Yuzhe Wang, Laureano Moro-Velazquez +2Speaker Embedding Speaker Verification
Electrical and Computer Engineering Department, Johns Hopkins University, Baltimore, MD, USA · Human Language Technology Center of Excellence, Johns Hopkins University, Baltimore, MD, USA
Large Audio Language Models for Spoofing-Aware Speaker Verification
Jul 16, 2026Sofya Savelyeva, Mariia Perunova, Evgeny Kushnir +3Audio-Language Model Evaluation Audio Deepfake Detection
Applied AI Institute · MIRAI · AXXX, HSE +3
SpeakerLLM: A Speaker-Specialized Audio-LLM for Speaker Understanding and Verification Reasoning
May 14, 2026KiHyun Nam, Jungwoo Heo, Siu Bae +2Audio Representation Learning Speaker Verification
Korea Advanced Institute of Science and Technology (KAIST) · University of Seoul

Mar 11, 2026J/K move · Enter open · S save

Speaker Verification with Speech-Aware LLMs: Evaluation and Augmentation

Thomas Thebaud, Yuzhe Wang, Laureano Moro-Velazquez, Jesus Villalba-Lopez+1

Electrical and Computer Engineering Department, Johns Hopkins University, Baltimore, MD, USA · Human Language Technology Center of Excellence, Johns Hopkins University, Baltimore, MD, USA

Speaker Embedding Speaker Verification Speech Language Models Audio Representation Learning

Open paper AskPDF arXiv

Speech-aware large language models (LLMs) can accept speech inputs, yet their training objectives largely emphasize linguistic content or specific fields such as emotions or the speaker's gender, leaving it unclear whether they encode speaker identity. First, we propose a model-agnostic scoring protocol that produces continuous verification scores for both API-only and open-weight models, using confidence scores or log-likelihood ratios from the Yes/No token probabilities. Using this protocol, we benchmark recent speech-aware LLMs and observe weak speaker discrimination (EERs above 20% on VoxCeleb1). Second, we introduce a lightweight augmentation that equips an LLM with ASV capability by injecting frozen ECAPA-TDNN speaker embeddings through a learned projection and training only LoRA adapters. On TinyLLaMA-1.1B, the resulting ECAPA-LLM achieves 1.03% EER on VoxCeleb1-E, approaching a dedicated speaker verification system while preserving a natural-language interface.

Abstract

Speaker Verification

Explore similar work

Speaker Verification with Speech-Aware LLMs: Evaluation and Augmentation

Large Audio Language Models for Spoofing-Aware Speaker Verification

SpeakerLLM: A Speaker-Specialized Audio-LLM for Speaker Understanding and Verification Reasoning