Rethinking Automated Voice Similarity by Shifting from EER to Embedding Geometry

Szu-Chi Chen; Jia-Kai Dong; Yi-Cheng Lin; Sung-Feng Huang; Hung-yi Lee

eess.ASSep 27, 2026

Rethinking Automated Voice Similarity by Shifting from EER to Embedding Geometry

Authors: Szu-Chi Chen, Jia-Kai Dong, Yi-Cheng Lin, Sung-Feng Huang, Hung-yi Lee

Organizations: National Taiwan University, Taipei, Taiwan · NVIDIA, Taiwan · Artificial Intelligence Center of Research Excellence (NTU AI-CoRE), NTU, Taiwan

Abstract

Speaker verification (SV) models are commonly assumed to better capture nuances among speaker characteristics as verification accuracy improves, leading to their widespread use as automated proxies for human voice similarity in speech generation tasks. However, by establishing a human perceptual alignment metric and conducting systematic analysis, we demonstrate that perceptual alignment is governed far more by how a model is trained (its learning objective) than by how well it performs (EER). Notably, standard margin-based classification losses (e.g., AAM-Softmax) yield substantially lower perceptual alignment than prototypical metric losses, while EER itself fails to track human judgment, directly challenging the community's implicit assumption. We trace this divergence to embedding geometry, where a model's effective dimensionality ( $d_{\mathrm{eff}}$ ) tracks perceptual alignment with a $-0.95$ rank correlation, revealing that the dimensional spread favored by classification losses fundamentally clashes with the low-dimensional nature of human voice perception. Imposing a dimensionality bottleneck compresses $d_{\mathrm{eff}}$ and raises perceptual alignment ( $ρ_{\mathrm{align}}$ ) from 0.08 to 0.74, establishing a principled geometric criterion for evaluating voice similarity.

Figures & tables

Figure 1: Overview of our study. Top: measurement of perceptual alignment; bottom: controlled experiments, effective dimensionality analysis, and embedding dimension bottlenecks.

Metric: prototypical
	ECAPA-TDNN			Fast ResNet-34			ReDimNet-B2			WavLM-p1			WavLM-p2
loss	EER $\downarrow$	$\rho_{\mathrm{align}}$ $\uparrow$	$d_{\mathrm{eff}}$	EER $\downarrow$	$\rho_{\mathrm{align}}$ $\uparrow$	$d_{\mathrm{eff}}$	EER $\downarrow$	$\rho_{\mathrm{align}}$ $\uparrow$	$d_{\mathrm{eff}}$	EER $\downarrow$	$\rho_{\mathrm{align}}$ $\uparrow$	$d_{\mathrm{eff}}$	EER $\downarrow$	$\rho_{\mathrm{align}}$ $\uparrow$	$d_{\mathrm{eff}}$
Prototypical	1.92 $\pm$ .04	.395 $\pm$ .010	27.1	2.39 $\pm$ .13	.491 $\pm$ .034	18.3	1.46 $\pm$ .05	.423 $\pm$ .001	27.9	1.75 $\pm$ .09	.462 $\pm$ .029	21.2	1.37 $\pm$ .04	.428 $\pm$ .011	25.2
Angular Proto.	1.47 $\pm$ .04	.406 $\pm$ .007	33.0	2.35 $\pm$ .12	.499 $\pm$ .012	21.2	1.15 $\pm$ .01	.425 $\pm$ .011	31.0	1.42 $\pm$ .06	.455 $\pm$ .024	26.1	1.09 $\pm$ .10	.401 $\pm$ .003	31.2
Classification
NSL	2.10 $\pm$ .03	.207 $\pm$ .009	59.1	2.82 $\pm$ .14	.249 $\pm$ .016	45.9	1.82 $\pm$ .11	.171 $\pm$ .006	67.4	1.83 $\pm$ .10	.298 $\pm$ .012	44.2	2.31 $\pm$ .09	.215 $\pm$ .015	54.9

Table 1: The controlled experimental matrix: EER (%, VoxCeleb1-O), perceptual alignment

\rho_{\mathrm{align}}

, and effective dimensionality

d_{\mathrm{eff}}

(Eq. 2 ); mean over 3 seeds (

\pm

sd for EER and

\rho_{\mathrm{align}}

;

d_{\mathrm{eff}}

seed sd

\leq 4.5

). Blue bold and red mark the highest and lowest

\rho_{\mathrm{align}}

per model condition; bold marks the best EER.

Within model conditions
Population	$n$	Spearman $(\mathrm{EER},\rho_{\mathrm{align}})$	Spearman $(d_{\mathrm{eff}},\rho_{\mathrm{align}})$
ECAPA-TDNN	7	$+0.11$ $(p=.84)$	$-0.96$ $(p=.003)$
Fast ResNet-34	7	$+0.43$ $(p=.35)$	$-0.93$ $(p=.007)$
ReDimNet-B2	7	$-0.68$ $(p=.11)$	$-0.71$ $(p=.088)$
WavLM-p1	7	$+0.61$ $(p=.17)$	$-1.00$ $(p<.001)$
WavLM-p2	7	$-0.36$ $(p=.44)$	$-1.00$ $(p<.001)$

Table 2: Spearman rank correlation of

\rho_{\mathrm{align}}

with EER (%, VoxCeleb1-O) and with

d_{\mathrm{eff}}

, each training condition one observation (three seeds averaged). Parentheses: two-sided permutation-test

p

-values, exact for

n{=}7

and Monte Carlo (

2{\times}10^{6}

permutations) for

n{=}35

Figure 2: Verification vs. perceptual alignment on ECAPA (circles; 3-seed mean $\pm$ sd) and the four public checkpoints (stars), with EER scored on the VoxSim pairs.

Figure 3: Effective dimensionality ( $d_{\mathrm{eff}}$ ) tracks human perceptual alignment ( $\rho_{\mathrm{align}}$ ). ECAPA, 7 losses $\times$ 3 seeds (21 model points): Spearman $(d_{\mathrm{eff}},\rho_{\mathrm{align}})=-0.95$ .

	AAM-Softmax			AM-Softmax
$n_{\mathrm{out}}$	EER% $\downarrow$	$\rho_{\mathrm{align}}$ $\uparrow$	$d_{\mathrm{eff}}$	EER% $\downarrow$	$\rho_{\mathrm{align}}$ $\uparrow$	$d_{\mathrm{eff}}$
512	1.43 $\pm$ .03	0.107 $\pm$ .007	75.8	1.56 $\pm$ .02	0.079 $\pm$ .014	101.6
192 ∗	1.47 $\pm$ .12	0.111 $\pm$ .013	74.5	1.62 $\pm$ .05	0.080 $\pm$ .009	98.3
32	2.31 $\pm$ .25	0.300 $\pm$ .006	22.7	2.74 $\pm$ .84	0.252 $\pm$ .010	23.9
10	13.99 $\pm$ 7.24	0.685 $\pm$ .111	3.1	6.63 $\pm$ .41	0.501 $\pm$ .029	5.4
3	37.32 $\pm$ 12.93 †	0.287 $\pm$ .401	1.2	20.48 $\pm$ .35	0.738 $\pm$ .018	1.5

Table 3: Embedding dimension bottleneck on ECAPA-TDNN (3-seed mean

\pm

sd; EER on VoxCeleb1-O;

d_{\mathrm{eff}}

seed sd

\leq 2.7

\ast

: native width. † : two of three seeds collapse to

d_{\mathrm{eff}}\approx 1.1

and EER

>40

\rho_{\mathrm{align}}

( blue ) and EER ( red ) are shaded as a heat map, darker meaning larger.

Explore similar work

Jun 4, 2026cs.SD

Do speech foundation models perceive speaker similarity as humans do?

This study presents a comparative analysis between the speaker embeddings of speech foundation models and human subjective perception of speaker similarity. Human listeners have the ability to judge speaker similarity on a continuous scale discerning how similar two voices are. In contrast, speech foundation models embed speaker characteristics into numerical representation. However, a question remains: does the numerical distance between speaker embeddings in these models truly align with the similarity perceived by humans? To address this, we conduct a comprehensive investigation using more than 40 models to compare model-derived distances with human-perceived similarity scores. Furthermore, we identify which factors in model configuration contribute most to a speaker embedding that mirrors human perception. Our findings provide insights for the development of more perceptually grounded speech foundation models.

Minoru Kishi, Hayato Yagi, Shinnosuke Takamichi +1

Keio University, Japan · The University of Tokyo, Japan

Ask PDF

Sep 23, 2026cs.SD

When Entanglement Lower-Bounds Disparity: Auditing and Repairing Demographic Fairness in Audio Understanding Models

Speech technology penalizes some voices: recognition errs nearly twice as often for Black speakers, and accuracy declines for second-language accents and older speakers. We introduce TRIAD, an audit grid crossing 120 texts, 24 rendered demographic voice profiles (gender, age band, accent), and ten expressive styles via controllable text-to-speech, isolating perceived demographic attributes from content and affect. For ten open-weights encoders we define axis-fidelity functionals, principal-angle leakage between axis subspaces, and group-conditional gaps; a proposition proves that average probe disparity grows with the same aggregate voice-semantic leakage

Λ

we measure, and a corollary shows that peak leakage forces worst-case disparity inside an active region. The measured mean-square probe disparity tracks

Λ

(Pearson r = 0.93), and a black-box protocol exposes the same signature in two closed-source models. ORCA, an adapter combining axis-specific contrastive heads, an orthogonality penalty, and group-balanced sampling, cuts leakage 72% and roughly halves the gaps.

Kian Shamsaie, Iman Modarressi

People Make Things

Speaker Disparities Algorithmic Fairness Model Auditing Entanglement

Ask PDF

Jul 24, 2026cs.CL

Simple Language Normalization Wins: Cross-Lingual Speaker Verification for the TidyVoice 2026 Challenge

Cross-lingual mismatch remains a key source of overall degradation in modern speaker verification. The TidyVoice2026 Challenge targets this setting with text-independent verification, comprising 3,666 training and 808 development speakers in 40 languages and 2,200 evaluation speakers in 38 unseen languages, without language labels at test time. Starting from the official SimAM-ResNet34 baseline pretrained on VoxBlink2 and VoxCeleb2 and fine-tuned on TidyVoice, we revisit Nuisance Attribute Projection (NAP) as a simple language-normalization step in the embedding space. We estimate a compact language subspace from cross-language same-speaker differences and project embeddings onto its orthogonal complement before cosine scoring with Adaptive Symmetric score normalization. This reduces development EER from 2.97% with cosine and 2.70% with AS-Norm to 2.18% and yields a Codabench evaluation score of 8.40, showing that simple back-end language normalization can rival more complex systems.

Nina Hosseini-Kivanani

University of Luxembourg & Radio T´el´evisioun L¨etzebuerg (RTL), Luxembourg

Automatic Speaker Verification Cross-Lingual Consistency Projection

Ask PDF