CoLMbo-SV: A Grounded Language Model for Explainable Speaker Verification

Massa Baali; Sarthak Bisht; Ziyue Qiu; Joseph Konan; Rita Singh; Bhiksha Raj

cs.CLSep 27, 2026

CoLMbo-SV: A Grounded Language Model for Explainable Speaker Verification

Authors: Massa Baali, Sarthak Bisht, Ziyue Qiu, Joseph Konan, Rita Singh, Bhiksha Raj

Organizations: Carnegie Mellon University, USA

Abstract

Speaker verification systems achieve high accuracy but provide little account of the acoustic evidence behind their judgments. Making these systems inspectable requires exposing interpretable evidence while retaining the richer information on which their decisions depend. We present CoLMbo-SV, a speaker language model that combines strong speaker discrimination with structured, acoustically grounded comparison reports. By connecting a pretrained speaker encoder to a language model and supplying explicit acoustic measurements, CoLMbo-SV makes voice comparisons inspectable without restricting verification to the evidence verbalized in its reports. We additionally introduce VoxReason, paired recordings with measured acoustic properties and comparison reports filtered through numerical and qualitative checks, providing supervision for this combined capability. We also develop an evaluation framework that separates what acoustic information a speaker representation encodes, what influences the verification score, and what the generated report discusses. On VoxCeleb1-O, CoLMbo-SV achieves 0.99% EER, reducing verification error by approximately 80% relative to the strongest audio-language baseline fine-tuned on VoxReason, while attaining a numerical-grounding score of 0.82. Our analysis further demonstrates that acoustic correctness and decision relevance are distinct properties of an explanation, exposing a gap that numerical-grounding metrics miss. Together, these contributions substantially advance audio-language speaker verification, bring its accuracy toward that of dedicated speaker encoders while adding checkable acoustic reporting, and establish an empirical framework for connecting natural-language explanations to the decisions they explain.

Figures & tables

Figure 1: CoLMbo-SV. A frozen speaker encoder embeds each recording; a mapper turns the embeddings and the prompt into a soft-token prefix for a LoRA-adapted language model; a raw-evidence bypass carries the uncompressed embeddings to a verification head and to a comparison token in the prefix; the extracted measurements enter as text. The language model writes the report and the verdict scored for EER; the head is a second, continuous decision path.

Figure 2: VoxReason construction: measurements are extracted from each recording, an LLM writes a comparison report from the two measurement records and the label, and a deterministic numerical validator and an LLM judge filter the reports.

Dedicated speaker encoders (cosine)
System	Report	Vox1-O	SITW
ReDimNet-B6	no	0.17	0.28
W2V-BERT 2.0 (our encoder)	no	0.34	0.87
ECAPA-TDNN	no	0.90	1.45
Audio-language models
Qwen2-Audio-7B, zero-shot	yes	47.01	47.67

Table 1: Verification EER (%) on VoxCeleb1-O and SITW, full scale; audio LMs fine-tuned on VoxReason unless marked zero-shot.

System	Strict $\uparrow$	Answ. $\uparrow$	Unk. $\downarrow$	Score $\uparrow$	Perf. $\uparrow$	$\geq$ 50% $\uparrow$	Cov. $\uparrow$	Rep. $\downarrow$
	Verdict (%)			Numerical grounding
Qwen2-Audio, zero-shot	3.83	46.94	91.83	0.068	17.5	56.7	25.7	0.035
Qwen2-Audio, fine-tuned	89.33	90.39	1.17	0.671	16.2	80.8	77.6	0.015
CoLMbo-SV (ours)	94.00	94.16	0.17	0.820	0.2	100.0	94.7	0.074
Qwen2-Audio, FT, measurements in prompt ∗	24.00	84.71	71.67	0.734	19.3	99.8	67.8	0.028

Table 2: Report quality on VoxCeleb1-O,

N{=}600

identical pairs, no numerical values in any baseline prompt.

	Variant	EER $\downarrow$	Ground. $\uparrow$	$\geq$ 50% $\uparrow$	Unk. $\downarrow$	Rep. $\downarrow$
A	stage 1: verifier, no evidence block in targets	1.06	0.639	89.0	0.00	0.081
B	stage 2: report objective, verifier frozen	0.78	0.631	88.2	0.17	0.077
C	+ evidence-block targets, no measurements given	1.37	0.658	90.3	0.00	0.070
D	+ measurements as 8 soft tokens	0.76	0.686	95.0	0.67	0.070
E	+ measurements as text (CoLMbo-SV, stage 3)	0.99	0.820	100.0	0.17	0.074

Table 3: Ablation. B is fine-tuned from A; C, D and E are each fine-tuned from B with the change named; E is CoLMbo-SV. Verdict EER on VoxCeleb1-O; report metrics on the 600 pairs of Table 2 .

System	feats	DG	Cov	DG@3	shuf.	lift
CoLMbo-SV	6.8	0.835	0.466	0.523	0.496	0.027
+ prompt names this pair’s top-3	6.8	0.834	0.464	0.522	0.494	0.028
+ prompt names another pair’s top-3	6.8	0.835	0.465	0.517	0.492	0.024
Qwen2-Audio, fine-tuned	4.8	0.830	0.320	0.443	0.351	0.092
Qwen2-Audio, FT, measurements in prompt	6.9	0.936	0.487	0.534	0.512	0.023
Qwen2-Audio, zero-shot	1.0	0.917	0.071	0.181	0.172	0.010

Table 4: Decision grounding of the reports of Table 2 against the language model’s decision-relevant set (

|C^{*}|{=}13

of 16 report features; random selection scores DG

{}=0.812

). DG: share of discussed features in

C^{*}

; Cov: share of

C^{*}

covered; DG@3: share of the pair’s own top-3 discussed; shuf.: same against a random other pair; lift: their difference.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Split	Source	Pairs	Same	Different	Speakers
Train	TIMIT	20,000	10,000	10,000	462
Train	VoxCeleb2	20,000	10,000	10,000	700
Test	TIMIT	1,000	500	500	168
Test	VoxCeleb2	1,000	–	–	118

Appendix

Table 5: VoxReason composition. Train and test speakers are disjoint within each source. Every pair carries one validated report; the 761,151 training rows are the reports and their per-section and per-feature decompositions used as instruction targets.

	Qwen3-Omni	CoLMbo-SV
Strict verdict (%) $\uparrow$	24.31	77.10
Answered verdict (%) $\uparrow$	48.54	77.10
Unknown rate (%) $\downarrow$	49.92	0.00
Numerical grounding $\uparrow$	0.23	0.70
Perfectly grounded (%) $\uparrow$	9.12	47.47
$\geq 50\%$ grounded (%) $\uparrow$	23.20	84.42

Appendix

Table 6: In-domain comparison against zero-shot Qwen3-Omni-30B-A3B-Thinking on the TIMIT pairs test split (

N{=}1917

matched pairs). Repetition here is the fraction of reports with any repeated 3-gram. Best in bold.

SNR bucket	$n$	Mean SNR (dB)	Verdict (%)	Grounding	Perfect (%)
$\leq 15$ dB †	4	13.06	25.00	0.44	0.00
$15$ – $25$ dB	142	21.75	71.83	0.71	48.23
$25$ – $35$ dB	1672	30.96	73.15	0.73	48.85
$>35$ dB	1182	37.83	79.02	0.72	44.59

Appendix

Table 7: Verdict accuracy and numerical grounding on TIMIT pairs stratified by minimum SNR (

N{=}3000

). Best in bold. † Sample size is small; read as suggestive.

Figure 3: Absolute prefix shift $|h_{\text{probe}}-h_{\text{neutral}}|$ induced by six feature-specific prompts relative to the neutral pairwise prompt, over the 120 most variable prefix dimensions (columns) and the prefix positions (rows), averaged over 100 test pairs.

Probe	$\Delta_{F}$	Cos. dist.	$M$ (sensitivity)
Breathiness	0.146	0.010	0.140
F0	0.180	0.015	0.098
Jitter	0.189	0.017	0.116
HNR	0.197	0.018	0.122
Formants	0.228	0.025	0.079
VOT	0.228	0.024	0.059

Appendix

Table 8: Mapper prefix shift from the neutral baseline under feature-specific prompts (100 pairs), alongside each feature’s decision-sensitivity magnitude

M

for the language-model verdict (best of its constituent measurements).

Rep. penalty	No-repeat $n$ -gram	Grounding $\uparrow$	Rep. rate $\downarrow$	Unk. (%) $\downarrow$
1.00	–	0.740	0.277	6.0
1.15	–	0.728	0.127	8.0
1.15	6	0.696	0.070	1.5
1.15	4	0.678	0.036	0.5

Appendix

Table 9: Decoding controls (variant D of Table 3 , first 200 evaluation pairs).

Concept	Probe	Enc.	$M$ (LM)	$p$	LM	Head
					Decision-relevant
roughness (VQ)	0.875	✓	0.142	$<$ 0.005	✓	–
breathiness	0.878	✓	0.140	$<$ 0.005	✓	–
shimmer	0.786	✓	0.125	$<$ 0.005	✓	–
voicing ratio	0.778	✓	0.124	$<$ 0.005	✓	–
HNR	0.824	✓	0.122	$<$ 0.005	✓	–

Appendix

Table 10: Which acoustic concepts the frozen speaker embedding encodes, and which of them the verdict is sensitive to. Probe is speaker-disjoint held-out balanced accuracy (gate

\geq 0.70

);

M

and

p

are the magnitude share and its empirical

p

-value against 200 random directions for the language-model verdict of CoLMbo-SV. The last two columns mark concepts that are both encoded and significant after BH correction, for the language-model verdict and for the verification head. a significant but not encoded, so the direction is not interpretable. Rows sorted by

M

Prompt	Generated response
Overall assessment. “Give an overall assessment of whether these two recordings come from the same speaker.”	## 1. Recording Conditions. “Utterance 1 exhibits a higher signal-to-noise ratio (SNR) of 35.49 dB compared to utterance 2’s 31.21 dB. … While utterance 1 benefits from a cleaner recording environment, the differences in SNR and noise floor are not so substantial as to preclude a reliable acoustic comparison.” ## 2. Prosodic & Voice Quality. “Mean F0: 113.24 vs 116.56 Hz; jitter higher in utterance 1 (1.99 vs 1.69); HNR 10.50 vs 11.90 dB.” ## 3. Spectral. “Utterance 1 F1=800 Hz, F2=1800 Hz, F3=2830 Hz; Utterance 2 F1=750, F2=1750, F3=2750.”
Segmental analysis. “Analyze the segmental acoustic features F1, F2, F3 means, voiced and voiceless VOT and finish with a Conclusion line on speaker identity.”	“Utterance 2 mean F1 827.08 Hz vs utterance 1’s 756.44 Hz; utterance 1 F2=1859 Hz, F3=2919 Hz vs utterance 2’s F2=1759, F3=2811. Voiceless VOT 80 ms (utt 2) vs 70 ms (utt 1); voiced VOT $-15$ vs $-18$ ms.” “Most discriminating evidence: F1 (z=1.16, p=0.245); F3 (z=0.78, p=0.437); F2 (z=0.80, p=0.424). Combining 5 features via Fisher’s method ( $\chi^{2}$ =7.98, df=10, p=0.635) yields same speaker, high confidence.” Conclusion: Same speaker, high confidence.
Single-feature query (HNR). “Examine HNR differences and explain how vocal clarity contributes to speaker comparison.”	“Utterance 1 has higher HNR of 12.95 dB compared to utterance 2’s 10.54 dB, indicating cleaner vocal tone.” “Typical within-speaker variation for HNR is 1.50 (utterance 1 12.95 dB, utterance 2 10.54 dB; z=1.64, p=0.100), indicating different speaker, medium confidence.” Conclusion: Different speakers, medium confidence.
Single-feature query (breathiness). “Analyze breathiness in both utterances and explain whether it supports or weakens speaker differentiation.”	“Breathiness index slightly higher in utterance 2 (0.55) vs utterance 1 (0.51).” “Most discriminating evidence: breathiness index (z=0.52, p=0.605); voice_quality.breathiness (z=0.52, p=0.605); voice_quality.whisper (z=0.12, p=0.903). Combining 3 features via Fisher’s method ( $\chi^{2}$ =2.33, df=6, p=0.886) yields same speaker, high confidence.” Conclusion: Same speaker, high confidence.

Appendix

Table 11: Representative CoLMbo-SV outputs across prompt types, from the TIMIT-trained checkpoint of Appendix C . Numerical citations match the measurements (grounding

\geq 0.9

in each case). The closing statistical-strength sentence and confidence label are part of the VoxReason report format (Appendix B ) and are not evaluated.

Speaker Verification

Speaker verification determines whether two speech recordings come from the same person, enabling voice authentication and identity analysis.

See all 61 papers

SpeakerLLM: A Speaker-Specialized Audio-LLM for Speaker Understanding and Verification Reasoning
May 14, 2026KiHyun Nam, Jungwoo Heo, Siu Bae +2Audio Representation Learning Speaker Verification
Korea Advanced Institute of Science and Technology (KAIST) · University of Seoul
SpeakerCard-1M: An Evidence-Grounded Corpus for In-the-Wild Speaker Verification
Jun 2, 2026Junyi Peng, Oldřich Plchot, Xiao Song +9Audio-Language Model Evaluation Speaker Verification
Brno University of Technology, Czechia · Peking University, China · Xiaomi, China +3
Speaker Verification with Speech-Aware LLMs: Evaluation and Augmentation
Mar 11, 2026Thomas Thebaud, Yuzhe Wang, Laureano Moro-Velazquez +2Speaker Embedding Speaker Verification
Electrical and Computer Engineering Department, Johns Hopkins University, Baltimore, MD, USA · Human Language Technology Center of Excellence, Johns Hopkins University, Baltimore, MD, USA

May 14, 2026J/K move · Enter open · S save

SpeakerLLM: A Speaker-Specialized Audio-LLM for Speaker Understanding and Verification Reasoning

KiHyun Nam, Jungwoo Heo, Siu Bae, Ha-Jin Yu+1

Korea Advanced Institute of Science and Technology (KAIST) · University of Seoul

Audio Representation Learning Speaker Verification Speech Processing Audio Understanding Audio-Language Models

Open paper AskPDF arXiv

As audio-first agents become increasingly common in physical AI, conversational robots, and screenless wearables, audio large language models (audio-LLMs) must integrate speaker-specific understanding to support user authorization, personalization, and context-aware interaction. This requires modeling who is speaking, how the voice sounds, and how recording conditions affect speaker cues. Conventional speaker verification systems provide strong scalar scores but little linguistic evidence, while current audio-LLMs and speaker-aware language models have limited ability to organize speaker information beyond binary labels or descriptive profiles. We present SpeakerLLM, a speaker-specialized audio-LLM framework that unifies single-utterance speaker profiling, recording-condition understanding, utterance-pair speaker comparison, and evidence-organized verification reasoning within a natural-language interface. We construct verification-reasoning targets and a decision-composition policy that separate profile-level evidence from the final same-or-different decision and organize recording condition, profile evidence, and the decision into a structured trace. At its core, SpeakerLLM uses a hierarchical speaker tokenizer designed to capture multiple granularities of speaker evidence. Utterance-level speaker embeddings summarize identity and profile-level cues, whereas frame-level speaker features preserve fine-grained acoustic descriptors. Experiments show that SpeakerLLM-Base improves speaker-profile and recording-condition understanding over general audio-LLMs, while SpeakerLLM-VR preserves strong generated-verdict accuracy and produces decision traces grounded in the supervised verification reasoning schema. We will release the metadata-enriched supervision dataset and target-construction code for reproducibility.

Abstract

Speaker Verification

Explore similar work

SpeakerLLM: A Speaker-Specialized Audio-LLM for Speaker Understanding and Verification Reasoning

SpeakerCard-1M: An Evidence-Grounded Corpus for In-the-Wild Speaker Verification

Speaker Verification with Speech-Aware LLMs: Evaluation and Augmentation