Speaker Verification with Speech-Aware LLMs: Evaluation and Augmentation

Thomas Thebaud; Yuzhe Wang; Laureano Moro-Velazquez; Jesus Villalba-Lopez; Najim Dehak

cs.SDMar 11, 2026

Speaker Verification with Speech-Aware LLMs: Evaluation and Augmentation

Authors: Thomas Thebaud, Yuzhe Wang, Laureano Moro-Velazquez, Jesus Villalba-Lopez, Najim Dehak

Organizations: Electrical and Computer Engineering Department, Johns Hopkins University, Baltimore, MD, USA · Human Language Technology Center of Excellence, Johns Hopkins University, Baltimore, MD, USA

Abstract

Speech-aware large language models (LLMs) can accept speech inputs, yet their training objectives largely emphasize linguistic content or specific fields such as emotions or the speaker's gender, leaving it unclear whether they encode speaker identity. First, we propose a model-agnostic scoring protocol that produces continuous verification scores for both API-only and open-weight models, using confidence scores or log-likelihood ratios from the Yes/No token probabilities. Using this protocol, we benchmark recent speech-aware LLMs and observe weak speaker discrimination (EERs above 20% on VoxCeleb1). Second, we introduce a lightweight augmentation that equips an LLM with ASV capability by injecting frozen ECAPA-TDNN speaker embeddings through a learned projection and training only LoRA adapters. On TinyLLaMA-1.1B, the resulting ECAPA-LLM achieves 1.03% EER on VoxCeleb1-E, approaching a dedicated speaker verification system while preserving a natural-language interface.

Figures & tables

Figure 1: Schematic of the pipeline to train and test a speaker-aware LLM.

Model	EER $\%\downarrow$			Failure	Gender	Gender	Accent	Accent
	Vox1-O	Vox1-E	Vox1-H	Rate $\%\downarrow$	Accuracy $\%\uparrow$	Predicted $\%\uparrow$	Accuracy $\%\uparrow$	Predicted $\%\uparrow$
Qwen-2.5-7B [ 6 ]	37.01	34.83	45.43	0.68	97.98	99.82	75.45	76.76
Kimi-audio-7B [ 35 ]	43.58	43.02	43.93	16.12	-	0	-	0
Gemini3-flash [ 26 ]	45.13	45.64	44.26	0.38	92.16	99.75	84.99	85.72
Gemini2.5-flash-lite [ 26 ]	36.15	37.09	47.55	16.56	91.40	99.91	83.32	83.90
GPT4.0-audio [ 3 ]	22.62	21.88	38.91	0.05	97.32	99.42	82.65	82.90

Table 1: Speaker-awareness evaluation of a set of Speech-aware LLMs using confidence scoring. EER (%) is shown for each split of Voxceleb1-test, as well as the confidence score failure rate, accuracy and failure rate for gender and accent prediction. Accuracies are left empty for Kimi as those fields were never predicted. Note that in case of failure to provide a score for a trial, the trial is ignored in the computation of the EER.

Model	Vox1-O	Vox1-E	Vox1-H
ECAPA-TDNN [ 13 ]	0.89	0.45	0.96
SA-Ministral3	14.76	15.88	21.04
SA-TinyLLaMA	1.87	1.03	2.20
SA-TinyLLaMA F	5.48	4.21	6.60
SA-TinyLLaMA XS	3.57	2.21	3.44
SA-TinyLLaMA ${}_{XS}^{F}$	27.01	27.82	28.55

Table 2: EER (%) results on the splits of VoxCeleb1 for our proposed speaker-augmented LLMs, evaluated using log-likelihood based scoring. The ECAPA-TDNN system is shown as a comparison, evaluated using cosine scoring.

Speaker Embedding

See all 9 papers

SpeakerLLM: A Speaker-Specialized Audio-LLM for Speaker Understanding and Verification Reasoning
May 14, 2026KiHyun Nam, Jungwoo Heo, Siu Bae +2Audio Representation Learning Speaker Verification
Korea Advanced Institute of Science and Technology (KAIST) · University of Seoul
Large Audio Language Models for Spoofing-Aware Speaker Verification
Jul 16, 2026Sofya Savelyeva, Mariia Perunova, Evgeny Kushnir +3Audio-Language Model Evaluation Audio Deepfake Detection
Applied AI Institute · MIRAI · AXXX, HSE +3
TS-SP: Learning Speaker-Preserving Representations in Audio Large Language Models
Oct 4, 2026Junjie Li, Zheng Liang, Zhe Li +2Speaker Verification Audio-Language Models
Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong SAR, China · Speech, Language, and Cognition Laboratory, The University of Hong Kong, Hong Kong SAR, China · National University of Singapore

May 14, 2026J/K move · Enter open · S save

SpeakerLLM: A Speaker-Specialized Audio-LLM for Speaker Understanding and Verification Reasoning

KiHyun Nam, Jungwoo Heo, Siu Bae, Ha-Jin Yu+1

Korea Advanced Institute of Science and Technology (KAIST) · University of Seoul

Audio Representation Learning Speaker Verification Speech Processing Audio Understanding Audio-Language Models

Open paper AskPDF arXiv

As audio-first agents become increasingly common in physical AI, conversational robots, and screenless wearables, audio large language models (audio-LLMs) must integrate speaker-specific understanding to support user authorization, personalization, and context-aware interaction. This requires modeling who is speaking, how the voice sounds, and how recording conditions affect speaker cues. Conventional speaker verification systems provide strong scalar scores but little linguistic evidence, while current audio-LLMs and speaker-aware language models have limited ability to organize speaker information beyond binary labels or descriptive profiles. We present SpeakerLLM, a speaker-specialized audio-LLM framework that unifies single-utterance speaker profiling, recording-condition understanding, utterance-pair speaker comparison, and evidence-organized verification reasoning within a natural-language interface. We construct verification-reasoning targets and a decision-composition policy that separate profile-level evidence from the final same-or-different decision and organize recording condition, profile evidence, and the decision into a structured trace. At its core, SpeakerLLM uses a hierarchical speaker tokenizer designed to capture multiple granularities of speaker evidence. Utterance-level speaker embeddings summarize identity and profile-level cues, whereas frame-level speaker features preserve fine-grained acoustic descriptors. Experiments show that SpeakerLLM-Base improves speaker-profile and recording-condition understanding over general audio-LLMs, while SpeakerLLM-VR preserves strong generated-verdict accuracy and produces decision traces grounded in the supervised verification reasoning schema. We will release the metadata-enriched supervision dataset and target-construction code for reproducibility.

Abstract

Speaker Embedding

Explore similar work

SpeakerLLM: A Speaker-Specialized Audio-LLM for Speaker Understanding and Verification Reasoning

Large Audio Language Models for Spoofing-Aware Speaker Verification

TS-SP: Learning Speaker-Preserving Representations in Audio Large Language Models