Speaker verification systems achieve high accuracy but provide little account of the acoustic evidence behind their judgments. Making these systems inspectable requires exposing interpretable evidence while retaining the richer information on which their decisions depend. We present CoLMbo-SV, a speaker language model that combines strong speaker discrimination with structured, acoustically grounded comparison reports. By connecting a pretrained speaker encoder to a language model and supplying explicit acoustic measurements, CoLMbo-SV makes voice comparisons inspectable without restricting verification to the evidence verbalized in its reports. We additionally introduce VoxReason, paired recordings with measured acoustic properties and comparison reports filtered through numerical and qualitative checks, providing supervision for this combined capability. We also develop an evaluation framework that separates what acoustic information a speaker representation encodes, what influences the verification score, and what the generated report discusses. On VoxCeleb1-O, CoLMbo-SV achieves 0.99% EER, reducing verification error by approximately 80% relative to the strongest audio-language baseline fine-tuned on VoxReason, while attaining a numerical-grounding score of 0.82. Our analysis further demonstrates that acoustic correctness and decision relevance are distinct properties of an explanation, exposing a gap that numerical-grounding metrics miss. Together, these contributions substantially advance audio-language speaker verification, bring its accuracy toward that of dedicated speaker encoders while adding checkable acoustic reporting, and establish an empirical framework for connecting natural-language explanations to the decisions they explain.
Figures & tables
Figure 1: CoLMbo-SV. A frozen speaker encoder embeds each recording; a mapper turns the embeddings and the prompt into a soft-token prefix for a LoRA-adapted language model; a raw-evidence bypass carries the uncompressed embeddings to a verification head and to a comparison token in the prefix; the extracted measurements enter as text. The language model writes the report and the verdict scored for EER; the head is a second, continuous decision path.
Figure 2: VoxReason construction: measurements are extracted from each recording, an LLM writes a comparison report from the two measurement records and the label, and a deterministic numerical validator and an LLM judge filter the reports.
System
Report
Vox1-O
SITW
Dedicated speaker encoders (cosine)
ReDimNet-B6
no
0.17
0.28
W2V-BERT 2.0 (our encoder)
no
0.34
0.87
ECAPA-TDNN
no
0.90
1.45
Audio-language models
Qwen2-Audio-7B, zero-shot
yes
47.01
47.67
Table 1: Verification EER (%) on VoxCeleb1-O and SITW, full scale; audio LMs fine-tuned on VoxReason unless marked zero-shot.
Verdict (%)
Numerical grounding
System
Strict ↑
Answ. ↑
Unk. ↓
Score ↑
Perf. ↑
≥ 50% ↑
Cov. ↑
Rep. ↓
Qwen2-Audio, zero-shot
3.83
46.94
91.83
0.068
17.5
56.7
25.7
0.035
Qwen2-Audio, fine-tuned
89.33
90.39
1.17
0.671
16.2
80.8
77.6
0.015
CoLMbo-SV (ours)
94.00
94.16
0.17
0.820
0.2
100.0
94.7
0.074
Qwen2-Audio, FT, measurements in prompt ∗
24.00
84.71
71.67
0.734
19.3
99.8
67.8
0.028
Table 2: Report quality on VoxCeleb1-O, N=600 identical pairs, no numerical values in any baseline prompt.
Variant
EER ↓
Ground. ↑
≥ 50% ↑
Unk. ↓
Rep. ↓
A
stage 1: verifier, no evidence block in targets
1.06
0.639
89.0
0.00
0.081
B
stage 2: report objective, verifier frozen
0.78
0.631
88.2
0.17
0.077
C
+ evidence-block targets, no measurements given
1.37
0.658
90.3
0.00
0.070
D
+ measurements as 8 soft tokens
0.76
0.686
95.0
0.67
0.070
E
+ measurements as text (CoLMbo-SV, stage 3)
0.99
0.820
100.0
0.17
0.074
Table 3: Ablation. B is fine-tuned from A; C, D and E are each fine-tuned from B with the change named; E is CoLMbo-SV. Verdict EER on VoxCeleb1-O; report metrics on the 600 pairs of Table 2 .
System
feats
DG
Cov
DG@3
shuf.
lift
CoLMbo-SV
6.8
0.835
0.466
0.523
0.496
0.027
+ prompt names this pair’s top-3
6.8
0.834
0.464
0.522
0.494
0.028
+ prompt names another pair’s top-3
6.8
0.835
0.465
0.517
0.492
0.024
Qwen2-Audio, fine-tuned
4.8
0.830
0.320
0.443
0.351
0.092
Qwen2-Audio, FT, measurements in prompt
6.9
0.936
0.487
0.534
0.512
0.023
Qwen2-Audio, zero-shot
1.0
0.917
0.071
0.181
0.172
0.010
Table 4: Decision grounding of the reports of Table 2 against the language model’s decision-relevant set ( ∣C∗∣=13 of 16 report features; random selection scores DG =0.812 ). DG: share of discussed features in C∗ ; Cov: share of C∗ covered; DG@3: share of the pair’s own top-3 discussed; shuf.: same against a random other pair; lift: their difference.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Split
Source
Pairs
Same
Different
Speakers
Train
TIMIT
20,000
10,000
10,000
462
Train
VoxCeleb2
20,000
10,000
10,000
700
Test
TIMIT
1,000
500
500
168
Test
VoxCeleb2
1,000
–
–
118
Appendix
Table 5: VoxReason composition. Train and test speakers are disjoint within each source. Every pair carries one validated report; the 761,151 training rows are the reports and their per-section and per-feature decompositions used as instruction targets.
Qwen3-Omni
CoLMbo-SV
Strict verdict (%) ↑
24.31
77.10
Answered verdict (%) ↑
48.54
77.10
Unknown rate (%) ↓
49.92
0.00
Numerical grounding ↑
0.23
0.70
Perfectly grounded (%) ↑
9.12
47.47
≥50% grounded (%) ↑
23.20
84.42
Appendix
Table 6: In-domain comparison against zero-shot Qwen3-Omni-30B-A3B-Thinking on the TIMIT pairs test split ( N=1917 matched pairs). Repetition here is the fraction of reports with any repeated 3-gram. Best in bold.
SNR bucket
n
Mean SNR (dB)
Verdict (%)
Grounding
Perfect (%)
≤15 dB †
4
13.06
25.00
0.44
0.00
15 – 25 dB
142
21.75
71.83
0.71
48.23
25 – 35 dB
1672
30.96
73.15
0.73
48.85
>35 dB
1182
37.83
79.02
0.72
44.59
Appendix
Table 7: Verdict accuracy and numerical grounding on TIMIT pairs stratified by minimum SNR ( N=3000 ). Best in bold. † Sample size is small; read as suggestive.
Figure 3: Absolute prefix shift ∣hprobe−hneutral∣ induced by six feature-specific prompts relative to the neutral pairwise prompt, over the 120 most variable prefix dimensions (columns) and the prefix positions (rows), averaged over 100 test pairs.
Probe
ΔF
Cos. dist.
M (sensitivity)
Breathiness
0.146
0.010
0.140
F0
0.180
0.015
0.098
Jitter
0.189
0.017
0.116
HNR
0.197
0.018
0.122
Formants
0.228
0.025
0.079
VOT
0.228
0.024
0.059
Appendix
Table 8: Mapper prefix shift from the neutral baseline under feature-specific prompts (100 pairs), alongside each feature’s decision-sensitivity magnitude M for the language-model verdict (best of its constituent measurements).
Rep. penalty
No-repeat n -gram
Grounding ↑
Rep. rate ↓
Unk. (%) ↓
1.00
–
0.740
0.277
6.0
1.15
–
0.728
0.127
8.0
1.15
6
0.696
0.070
1.5
1.15
4
0.678
0.036
0.5
Appendix
Table 9: Decoding controls (variant D of Table 3 , first 200 evaluation pairs).
Decision-relevant
Concept
Probe
Enc.
M (LM)
p
LM
Head
roughness (VQ)
0.875
✓
0.142
< 0.005
✓
–
breathiness
0.878
✓
0.140
< 0.005
✓
–
shimmer
0.786
✓
0.125
< 0.005
✓
–
voicing ratio
0.778
✓
0.124
< 0.005
✓
–
HNR
0.824
✓
0.122
< 0.005
✓
–
Appendix
Table 10: Which acoustic concepts the frozen speaker embedding encodes, and which of them the verdict is sensitive to. Probe is speaker-disjoint held-out balanced accuracy (gate ≥0.70 ); M and p are the magnitude share and its empirical p -value against 200 random directions for the language-model verdict of CoLMbo-SV. The last two columns mark concepts that are both encoded and significant after BH correction, for the language-model verdict and for the verification head. a significant but not encoded, so the direction is not interpretable. Rows sorted by M .
Prompt
Generated response
Overall assessment. “Give an overall assessment of whether these two recordings come from the same speaker.”
## 1. Recording Conditions. “Utterance 1 exhibits a higher signal-to-noise ratio (SNR) of 35.49 dB compared to utterance 2’s 31.21 dB. … While utterance 1 benefits from a cleaner recording environment, the differences in SNR and noise floor are not so substantial as to preclude a reliable acoustic comparison.” ## 2. Prosodic & Voice Quality. “Mean F0: 113.24 vs 116.56 Hz; jitter higher in utterance 1 (1.99 vs 1.69); HNR 10.50 vs 11.90 dB.” ## 3. Spectral. “Utterance 1 F1=800 Hz, F2=1800 Hz, F3=2830 Hz; Utterance 2 F1=750, F2=1750, F3=2750.”
Segmental analysis. “Analyze the segmental acoustic features F1, F2, F3 means, voiced and voiceless VOT and finish with a Conclusion line on speaker identity.”
“Utterance 2 mean F1 827.08 Hz vs utterance 1’s 756.44 Hz; utterance 1 F2=1859 Hz, F3=2919 Hz vs utterance 2’s F2=1759, F3=2811. Voiceless VOT 80 ms (utt 2) vs 70 ms (utt 1); voiced VOT −15 vs −18 ms.” “Most discriminating evidence: F1 (z=1.16, p=0.245); F3 (z=0.78, p=0.437); F2 (z=0.80, p=0.424). Combining 5 features via Fisher’s method ( χ2 =7.98, df=10, p=0.635) yields same speaker, high confidence.” Conclusion: Same speaker, high confidence.
Single-feature query (HNR). “Examine HNR differences and explain how vocal clarity contributes to speaker comparison.”
“Utterance 1 has higher HNR of 12.95 dB compared to utterance 2’s 10.54 dB, indicating cleaner vocal tone.” “Typical within-speaker variation for HNR is 1.50 (utterance 1 12.95 dB, utterance 2 10.54 dB; z=1.64, p=0.100), indicating different speaker, medium confidence.” Conclusion: Different speakers, medium confidence.
Single-feature query (breathiness). “Analyze breathiness in both utterances and explain whether it supports or weakens speaker differentiation.”
“Breathiness index slightly higher in utterance 2 (0.55) vs utterance 1 (0.51).” “Most discriminating evidence: breathiness index (z=0.52, p=0.605); voice_quality.breathiness (z=0.52, p=0.605); voice_quality.whisper (z=0.12, p=0.903). Combining 3 features via Fisher’s method ( χ2 =2.33, df=6, p=0.886) yields same speaker, high confidence.” Conclusion: Same speaker, high confidence.
Appendix
Table 11: Representative CoLMbo-SV outputs across prompt types, from the TIMIT-trained checkpoint of Appendix C . Numerical citations match the measurements (grounding ≥0.9 in each case). The closing statistical-strength sentence and confidence label are part of the VoxReason report format (Appendix B ) and are not evaluated.
As audio-first agents become increasingly common in physical AI, conversational robots, and screenless wearables, audio large language models (audio-LLMs) must integrate speaker-specific understanding to support user authorization, personalization, and context-aware interaction. This requires modeling who is speaking, how the voice sounds, and how recording conditions affect speaker cues. Conventional speaker verification systems provide strong scalar scores but little linguistic evidence, while current audio-LLMs and speaker-aware language models have limited ability to organize speaker information beyond binary labels or descriptive profiles. We present SpeakerLLM, a speaker-specialized audio-LLM framework that unifies single-utterance speaker profiling, recording-condition understanding, utterance-pair speaker comparison, and evidence-organized verification reasoning within a natural-language interface. We construct verification-reasoning targets and a decision-composition policy that separate profile-level evidence from the final same-or-different decision and organize recording condition, profile evidence, and the decision into a structured trace. At its core, SpeakerLLM uses a hierarchical speaker tokenizer designed to capture multiple granularities of speaker evidence. Utterance-level speaker embeddings summarize identity and profile-level cues, whereas frame-level speaker features preserve fine-grained acoustic descriptors. Experiments show that SpeakerLLM-Base improves speaker-profile and recording-condition understanding over general audio-LLMs, while SpeakerLLM-VR preserves strong generated-verdict accuracy and produces decision traces grounded in the supervised verification reasoning schema. We will release the metadata-enriched supervision dataset and target-construction code for reproducibility.
KiHyun Nam, Jungwoo Heo, Siu Bae +2
Korea Advanced Institute of Science and Technology (KAIST) · University of Seoul
Modern speaker verification (SV) systems rely on speaker embeddings that are effective but difficult to interpret or query in natural language. Most existing speech-text corpora target controllable synthesis or utterance-level captioning, offering limited speaker-level supervision for in-the-wild speaker recognition. This paper introduces SpeakerCard-1M, a bilingual speaker resource for evidence-grounded SV, derived from VoxCeleb1/2 and CN-Celeb1/2, where the ``-1M'' suffix refers to the 1.78M utterance-level captions contained in the release. We adopt a tool-first, LLM-last approach in which ten acoustic probes produce field-level evidence, the evidence is aggregated into speaker profiles under a schema that separates relatively stable traits from utterance-level states, and bilingual Speaker Cards are rendered by a constrained LLM that sees only the structured fields. The release includes 56.7k Speaker Card records over 10.2k speakers, 1.78M utterance-level captions, and speaker-ID-disjoint hard-negative triplets. We further define two SV-oriented cross-modal protocols, bidirectional Speaker-Text Retrieval (T2S-R / S2T-R) and Attribute-Conditioned Verification (AC-Verify), and compare a dual-encoder baseline against recent audio language models under a zero-shot forced-choice setting. Joint audio-text training costs only 0.31% absolute EER on VoxCeleb1-O relative to the audio-only baseline. Under a style-symmetric LLM-generated counterfactual protocol, eight recent audio language models (7B-30B+ parameters, both open- and closed-source) score 49-77% on pitch-level AC-Verify in a 2-way forced-choice setting, compared with 88.66% for our dual encoder.
Junyi Peng, Oldřich Plchot, Xiao Song +9
Brno University of Technology, Czechia · Peking University, China · Xiaomi, China +3
Speech-aware large language models (LLMs) can accept speech inputs, yet their training objectives largely emphasize linguistic content or specific fields such as emotions or the speaker's gender, leaving it unclear whether they encode speaker identity. First, we propose a model-agnostic scoring protocol that produces continuous verification scores for both API-only and open-weight models, using confidence scores or log-likelihood ratios from the Yes/No token probabilities. Using this protocol, we benchmark recent speech-aware LLMs and observe weak speaker discrimination (EERs above 20% on VoxCeleb1). Second, we introduce a lightweight augmentation that equips an LLM with ASV capability by injecting frozen ECAPA-TDNN speaker embeddings through a learned projection and training only LoRA adapters. On TinyLLaMA-1.1B, the resulting ECAPA-LLM achieves 1.03% EER on VoxCeleb1-E, approaching a dedicated speaker verification system while preserving a natural-language interface.
Thomas Thebaud, Yuzhe Wang, Laureano Moro-Velazquez +2
Electrical and Computer Engineering Department, Johns Hopkins University, Baltimore, MD, USA · Human Language Technology Center of Excellence, Johns Hopkins University, Baltimore, MD, USA