PhiNet: Speaker Verification with Phonetic Interpretability

Yi Ma; Shuai Wang; Tianchi Liu; Haizhou Li

eess.ASApr 2, 2026

PhiNet: Speaker Verification with Phonetic Interpretability

Authors: Yi Ma, Shuai Wang, Tianchi Liu, Haizhou Li

Organizations: Department of Electrical and Computer Engineering, National University of Singapore, 119077, Singapore · School of Intelligence Science and Technology, Nanjing University, Suzhou, 215163, China · Shenzhen Loop Area Institute, Shenzhen, China · LIGHTSPEED, 068897, Singapore ·

Abstract

Despite remarkable progress, automatic speaker verification (ASV) systems typically lack the transparency required for high-accountability applications. Motivated by how human experts perform forensic speaker comparison (FSC), we propose a speaker verification network with phonetic interpretability, PhiNet, designed to enhance both local and global interpretability by leveraging phonetic evidence in decision-making. For users, PhiNet provides detailed phonetic-level comparisons that enable manual inspection of speaker-specific features and facilitate a more critical evaluation of verification outcomes. For developers, it offers explicit reasoning behind verification decisions, simplifying error tracing and informing hyperparameter selection. In our experiments, we demonstrate PhiNet's interpretability with practical examples, including its application in analyzing the impact of different hyperparameters. We conduct both qualitative and quantitative evaluations of the proposed interpretability methods and assess speaker verification performance across multiple benchmark datasets, including VoxCeleb, SITW, and LibriSpeech. Results show that PhiNet achieves performance comparable to traditional black-box ASV models while offering meaningful, interpretable explanations for its decisions, bridging the gap between ASV and forensic analysis.

Figures & tables

Fig. 1: Block diagram of the proposed framework for speaker verification with phonetic interpretability.

Fig. 2: Block diagram of the phonetic trait extractor.

Fig. 3: Illustration of two conditions of adjusting phonetic scores based on phoneme distinctiveness. The top figure shows the adjustment when the phonetic score is closer to

a

, while the bottom figure depicts the adjustment when the score is closer to

b

. Here,

a

and

b

represent the scores for completely different speakers and the same speaker, respectively.

Fig. 4: Visualization of the decision making process for a non-target trial (top) and a target trial (bottom). Phonetic boundary are marked by dotted lines in enrollment and test utterances. “Er. Seg” and “Te. Seg” show the segments within each phoneme in enrollment and test utterance. The results shown are obtained using the model trained under System (10) in Table II .

Fig. 5: Phonetic weight distribution across different phonemes. The results shown are obtained using the model trained under System (10) in Table II .

Dur	Whole Utt.		5s		2s
Dur	EER	minDCF	EER	minDCF	EER	minDCF
1s	8.411	0.556	8.818	0.594	12.547	0.811
2s	5.946	0.447	6.204	0.501	10.973	0.750
3s	5.673	0.439	6.053	0.479	12.047	0.790
4s	6.364	0.440	6.622	0.516	15.216	0.851
5s	6.563	0.457	7.029	0.514	19.248	0.857

TABLE I: Speaker verification performance between models trained on samples of different durations. The systems are trained under System (4), (6), (10), (12) and (14) in Table II , and evaluated on Vox1-O with the whole test sample (“Whole Utt”), the first 5 seconds (“5s”), and the first 2 seconds (“2s”). “Mix” indicates that the training duration in each mini-batch is randomly selected from 1 to 5 seconds

Fig. 6: The phonetic weight for models trained with various duration. The configuration of these models are same with System (4), (6), (10), (12) and (14) in Table II respectively.

Fig. 7: Results of leave- $i$ th-phoneme-out experiments on SITW-eval, Vox1-O and LibriSpeech. Leave- $i$ th-phoneme-out experiments conducted in input spectrogram are shown as “spec-sitw-eval”, “spec-vox1-O” and “spec-libriSpeech”. Correspondingly, “trait” means the phonetic trait of each phoneme is left out and ‘baseline’ shows the EER of the network without leaving anything out.

Panel of Figure 9 — Fig. 8: Similarity heatmaps between the individual phonetic traits and the centric of each phonetic traits for two speakers in the test set. The results are obtained using the model trained under System (10) in Table II . The phoneme order in the heatmaps (top-to-bottom and left-to-right) follows the same order as in Fig 5 . Brighter points indicate higher cosine similarity.

Model	Idx	Dur	$\alpha$	$\beta$	Vox1-O
Model	Idx	Dur	$\alpha$	$\beta$	EER( $\%$ )	minDCF	Fidelity $\downarrow$
ECAPA	(1)	2s	-	-	3.403	0.272	-
-TDNN	(2)	3s	-	-	3.541	0.279	-
PhiNet	(3)	1s	0	0	8.366	0.549	0.048
	(4)	1s	0.001	0.0015	8.411	0.556	0.053
	(5)	2s	0	0	6.244	0.442	0.068

TABLE II: Performance in a comparative study between hyper-parameters used in network training. The fidelity score quantifies the alignment between phonetic trait-based and spectrogram-based leave-one-out setting. Lower values indicate better faithfulness. All the networks are trained on VoxCeleb1. “Dur” represents the duration of seconds of training sample. “

\alpha

” and “

\beta

” represent the weights assigned to different components in the loss function. The bolded values indicate the optimal configurations.

Dur	$\alpha$	$\beta$	Vox1-O
Dur	$\alpha$	$\beta$	EER( $\%$ )	minDCF	Fidelity $\downarrow$
3s	0.001	0.0015	5.673	0.439	0.081
	0.0012	0.0017	5.689	0.435	0.084
	0.0015	0.002	6.244	0.459	0.076
5s	0.001	0.0015	6.563	0.457	0.113
	0.0012	0.0017	6.465	0.462	0.129

TABLE III: Impact of different the weight of

\alpha

and

\beta

on model performance during training.

$c$	Vox1-O
$c$	EER( $\%$ )	minDCF	Fidelity $\downarrow$
1	5.808	0.448	0.108
2	5.673	0.439	0.081
3	5.585	0.417	0.095
4	5.955	0.448	0.057

TABLE IV: Performance evaluation of models trained with different dimension of mapping vector.

c

is the dimension of the mapping vector in eq. 2 .

Loss	$K$	Aug	Vox1-O
Loss	$K$	Aug	EER( $\%$ )	minDCF	Fidelity $\downarrow$
$\mathcal{L}_{veri}$	32	✓	6.100	0.445	0.083
	32	×	6.571	0.483	0.100
	64	✓	5.840	0.430	0.096
	64	×	6.747	0.486	0.147
	128	✓	5.920	0.436	0.110

TABLE V: Impact of data augmentation on model performance during training.

Function	Vox1-O
Function	EER( $\%$ )	minDCF	Fidelity $\downarrow$
Min-Max	5.673	0.439	0.081
Sigmoid	5.981	0.444	0.092
Min-Shift	5.705	0.433	0.097
ReLU	5.967	0.458	0.096

TABLE VI: Performance evaluation of models trained with different function to generate phonetic weights.

System	Dur	Params	Training	Vox1-O		Vox1-E		Vox1-H		SITW-eval		LibriSpeech
System	Dur	(Million)	Data	EER	Fidelity	EER	Fidelity	EER	Fidelity	EER	Fidelity	EER	Fidelity
ECAPA-TDNN (Pre-trained)	2s	6.19	Vox2	1.223	-	1.387	-	2.567	-	2.352	-	0.696	-
ECAPA-TDNN (Reproduced)	2s	5.40	Vox1	3.403	-	-	-	-	-	4.893	-	2.061	-
	3s		Vox1	3.541	-	-	-	-	-	4.547	-	2.280	-
	2s		Vox2	2.648	-	2.794	-	5.474	-	3.318	-	1.685	-
PhiNet	3s	4.81	Vox1	5.673	0.081	-	-	-	-	6.944	0.079	3.518	0.060

TABLE VII: Performance of networks evaluated on the Vox1-O/E/H, SITW and LibriSpeech. The system is trained on training set of VoxCeleb1 (‘Vox1’) and training set of VoxCeleb2 (“Vox2”). “Pre-trained” refers to the pre-trained ECAPA-TDNN baseline released from WeSpeaker, while “Reproduced” denotes the ECAPA-TDNN trained with same strategy as our model.

Speaker Verification

Speaker verification determines whether two speech recordings come from the same person, enabling voice authentication and identity analysis.

See all 62 papers

LISE : Listenable Interpretable Speaker Embeddings
Jun 19, 2026Xiaoliang Wu, Chongxin Gan, Ke Liu +2Audio Representation Learning Speaker Verification
University of Southampton, United Kingdom · The Hong Kong Polytechnic University, Hong Kong SAR, China · University of Edinburgh, United Kingdom
Integrated Spoofing-Robust Automatic Speaker Verification via a Three-Class Formulation and LLR
Mar 14, 2026Kai Tan, Lin Zhang, Ruiteng Zhang +6Speaker Verification ASVspoof
HLTCOE & 2 CLSP, Johns Hopkins University, USA · Tianjin University, China
CoLMbo-SV: A Grounded Language Model for Explainable Speaker Verification
Sep 27, 2026Massa Baali, Sarthak Bisht, Ziyue Qiu +3Speaker Verification Faithfulness of Language Model Explanations
Carnegie Mellon University, USA

Jun 19, 2026J/K move · Enter open · S save

LISE : Listenable Interpretable Speaker Embeddings

Xiaoliang Wu, Chongxin Gan, Ke Liu, Peter Bell+1

University of Southampton, United Kingdom · The Hong Kong Polytechnic University, Hong Kong SAR, China · University of Edinburgh, United Kingdom

Audio Representation Learning Speaker Verification Neural Network Interpretability Speaker Embedding

Open paper AskPDF arXiv

Deep neural network-based automatic speaker verification (ASV) systems achieve impressive performance but their embedding representations remain opaque, lacking a structured and perceptually verifiable explanation of the vocal characteristics they encode. Existing approaches either require annotation of speaker attributes or introduce alternative representations whose interpretability is unvalidated with listeners. We propose Listenable Interpretable Speaker Embeddings (LISE), a label-free framework that decomposes pretrained speaker embeddings into a small set of components. This decomposition yields a structured representation that supports the analysis of what information has been encoded by speaker embeddings. LISE preserves ASV performance with negligible EER degradation on x-vector and ECAPA-TDNN. Crucially, the interpretability of these components for human listeners is demonstrated through listening experiments, where participants distinguished speakers with 83.9% accuracy.

Abstract

Speaker Verification

Explore similar work

LISE : Listenable Interpretable Speaker Embeddings

Integrated Spoofing-Robust Automatic Speaker Verification via a Three-Class Formulation and LLR

CoLMbo-SV: A Grounded Language Model for Explainable Speaker Verification