Rethinking Automated Voice Similarity by Shifting from EER to Embedding Geometry

Szu-Chi Chen; Jia-Kai Dong; Yi-Cheng Lin; Sung-Feng Huang; Hung-yi Lee

eess.ASSep 27, 2026

Rethinking Automated Voice Similarity by Shifting from EER to Embedding Geometry

Authors: Szu-Chi Chen, Jia-Kai Dong, Yi-Cheng Lin, Sung-Feng Huang, Hung-yi Lee

Organizations: National Taiwan University, Taipei, Taiwan · NVIDIA, Taiwan · Artificial Intelligence Center of Research Excellence (NTU AI-CoRE), NTU, Taiwan

Abstract

Speaker verification (SV) models are commonly assumed to better capture nuances among speaker characteristics as verification accuracy improves, leading to their widespread use as automated proxies for human voice similarity in speech generation tasks. However, by establishing a human perceptual alignment metric and conducting systematic analysis, we demonstrate that perceptual alignment is governed far more by how a model is trained (its learning objective) than by how well it performs (EER). Notably, standard margin-based classification losses (e.g., AAM-Softmax) yield substantially lower perceptual alignment than prototypical metric losses, while EER itself fails to track human judgment, directly challenging the community's implicit assumption. We trace this divergence to embedding geometry, where a model's effective dimensionality ( $d_{\mathrm{eff}}$ ) tracks perceptual alignment with a $-0.95$ rank correlation, revealing that the dimensional spread favored by classification losses fundamentally clashes with the low-dimensional nature of human voice perception. Imposing a dimensionality bottleneck compresses $d_{\mathrm{eff}}$ and raises perceptual alignment ( $ρ_{\mathrm{align}}$ ) from 0.08 to 0.74, establishing a principled geometric criterion for evaluating voice similarity.

Figures & tables

Figure 1: Overview of our study. Top: measurement of perceptual alignment; bottom: controlled experiments, effective dimensionality analysis, and embedding dimension bottlenecks.

Metric: prototypical
	ECAPA-TDNN			Fast ResNet-34			ReDimNet-B2			WavLM-p1			WavLM-p2
loss	EER $\downarrow$	$\rho_{\mathrm{align}}$ $\uparrow$	$d_{\mathrm{eff}}$	EER $\downarrow$	$\rho_{\mathrm{align}}$ $\uparrow$	$d_{\mathrm{eff}}$	EER $\downarrow$	$\rho_{\mathrm{align}}$ $\uparrow$	$d_{\mathrm{eff}}$	EER $\downarrow$	$\rho_{\mathrm{align}}$ $\uparrow$	$d_{\mathrm{eff}}$	EER $\downarrow$	$\rho_{\mathrm{align}}$ $\uparrow$	$d_{\mathrm{eff}}$
Prototypical	1.92 $\pm$ .04	.395 $\pm$ .010	27.1	2.39 $\pm$ .13	.491 $\pm$ .034	18.3	1.46 $\pm$ .05	.423 $\pm$ .001	27.9	1.75 $\pm$ .09	.462 $\pm$ .029	21.2	1.37 $\pm$ .04	.428 $\pm$ .011	25.2
Angular Proto.	1.47 $\pm$ .04	.406 $\pm$ .007	33.0	2.35 $\pm$ .12	.499 $\pm$ .012	21.2	1.15 $\pm$ .01	.425 $\pm$ .011	31.0	1.42 $\pm$ .06	.455 $\pm$ .024	26.1	1.09 $\pm$ .10	.401 $\pm$ .003	31.2
Classification
NSL	2.10 $\pm$ .03	.207 $\pm$ .009	59.1	2.82 $\pm$ .14	.249 $\pm$ .016	45.9	1.82 $\pm$ .11	.171 $\pm$ .006	67.4	1.83 $\pm$ .10	.298 $\pm$ .012	44.2	2.31 $\pm$ .09	.215 $\pm$ .015	54.9

Table 1: The controlled experimental matrix: EER (%, VoxCeleb1-O), perceptual alignment

\rho_{\mathrm{align}}

, and effective dimensionality

d_{\mathrm{eff}}

(Eq. 2 ); mean over 3 seeds (

\pm

sd for EER and

\rho_{\mathrm{align}}

;

d_{\mathrm{eff}}

seed sd

\leq 4.5

). Blue bold and red mark the highest and lowest

\rho_{\mathrm{align}}

per model condition; bold marks the best EER.

Within model conditions
Population	$n$	Spearman $(\mathrm{EER},\rho_{\mathrm{align}})$	Spearman $(d_{\mathrm{eff}},\rho_{\mathrm{align}})$
ECAPA-TDNN	7	$+0.11$ $(p=.84)$	$-0.96$ $(p=.003)$
Fast ResNet-34	7	$+0.43$ $(p=.35)$	$-0.93$ $(p=.007)$
ReDimNet-B2	7	$-0.68$ $(p=.11)$	$-0.71$ $(p=.088)$
WavLM-p1	7	$+0.61$ $(p=.17)$	$-1.00$ $(p<.001)$
WavLM-p2	7	$-0.36$ $(p=.44)$	$-1.00$ $(p<.001)$

Table 2: Spearman rank correlation of

\rho_{\mathrm{align}}

with EER (%, VoxCeleb1-O) and with

d_{\mathrm{eff}}

, each training condition one observation (three seeds averaged). Parentheses: two-sided permutation-test

p

-values, exact for

n{=}7

and Monte Carlo (

2{\times}10^{6}

permutations) for

n{=}35

Figure 2: Verification vs. perceptual alignment on ECAPA (circles; 3-seed mean $\pm$ sd) and the four public checkpoints (stars), with EER scored on the VoxSim pairs.

Figure 3: Effective dimensionality ( $d_{\mathrm{eff}}$ ) tracks human perceptual alignment ( $\rho_{\mathrm{align}}$ ). ECAPA, 7 losses $\times$ 3 seeds (21 model points): Spearman $(d_{\mathrm{eff}},\rho_{\mathrm{align}})=-0.95$ .

	AAM-Softmax			AM-Softmax
$n_{\mathrm{out}}$	EER% $\downarrow$	$\rho_{\mathrm{align}}$ $\uparrow$	$d_{\mathrm{eff}}$	EER% $\downarrow$	$\rho_{\mathrm{align}}$ $\uparrow$	$d_{\mathrm{eff}}$
512	1.43 $\pm$ .03	0.107 $\pm$ .007	75.8	1.56 $\pm$ .02	0.079 $\pm$ .014	101.6
192 ∗	1.47 $\pm$ .12	0.111 $\pm$ .013	74.5	1.62 $\pm$ .05	0.080 $\pm$ .009	98.3
32	2.31 $\pm$ .25	0.300 $\pm$ .006	22.7	2.74 $\pm$ .84	0.252 $\pm$ .010	23.9
10	13.99 $\pm$ 7.24	0.685 $\pm$ .111	3.1	6.63 $\pm$ .41	0.501 $\pm$ .029	5.4
3	37.32 $\pm$ 12.93 †	0.287 $\pm$ .401	1.2	20.48 $\pm$ .35	0.738 $\pm$ .018	1.5

Table 3: Embedding dimension bottleneck on ECAPA-TDNN (3-seed mean

\pm

sd; EER on VoxCeleb1-O;

d_{\mathrm{eff}}

seed sd

\leq 2.7

\ast

: native width. † : two of three seeds collapse to

d_{\mathrm{eff}}\approx 1.1

and EER

>40

\rho_{\mathrm{align}}

( blue ) and EER ( red ) are shaded as a heat map, darker meaning larger.

Explore similar work

Do speech foundation models perceive speaker similarity as humans do?
Jun 4, 2026Minoru Kishi, Hayato Yagi, Shinnosuke Takamichi +1Speaker Similarity Speaker
Keio University, Japan · The University of Tokyo, Japan
When Entanglement Lower-Bounds Disparity: Auditing and Repairing Demographic Fairness in Audio Understanding Models
Sep 23, 2026Kian Shamsaie, Iman ModarressiSpeaker Disparities
People Make Things
Simple Language Normalization Wins: Cross-Lingual Speaker Verification for the TidyVoice 2026 Challenge
Jul 24, 2026Nina Hosseini-KivananiAutomatic Speaker Verification Cross-Lingual Consistency
University of Luxembourg & Radio T´el´evisioun L¨etzebuerg (RTL), Luxembourg

Jun 4, 2026, cs.SDJ/K move · Enter open · S save

Do speech foundation models perceive speaker similarity as humans do?

Minoru Kishi, Hayato Yagi, Shinnosuke Takamichi, Yuki Saito

Keio University, Japan · The University of Tokyo, Japan

Open paper AskPDF arXiv

This study presents a comparative analysis between the speaker embeddings of speech foundation models and human subjective perception of speaker similarity. Human listeners have the ability to judge speaker similarity on a continuous scale discerning how similar two voices are. In contrast, speech foundation models embed speaker characteristics into numerical representation. However, a question remains: does the numerical distance between speaker embeddings in these models truly align with the similarity perceived by humans? To address this, we conduct a comprehensive investigation using more than 40 models to compare model-derived distances with human-perceived similarity scores. Furthermore, we identify which factors in model configuration contribute most to a speaker embedding that mirrors human perception. Our findings provide insights for the development of more perceptually grounded speech foundation models.

Abstract

Explore similar work

Do speech foundation models perceive speaker similarity as humans do?

When Entanglement Lower-Bounds Disparity: Auditing and Repairing Demographic Fairness in Audio Understanding Models

Simple Language Normalization Wins: Cross-Lingual Speaker Verification for the TidyVoice 2026 Challenge