Where Does Timbre Hide? Localizing Speaker Information in Speech Signals

Yiquan Zhou; Yixuan Wang; Xiaozhe Qi; Xin Gao; Shihao Li

cs.SDOct 21, 2026

Where Does Timbre Hide? Localizing Speaker Information in Speech Signals

Authors: Yiquan Zhou, Yixuan Wang, Xiaozhe Qi, Xin Gao, Shihao Li

Organizations: School of Software Engineering, Xi’an Jiaotong University, Xi’an, China, and also with the SYKI-SPEECH Team, Xi’an, China · Division of Music and Audio, Union Wheatland Culture and Media Ltd., Chengdu, China

Abstract

Timbre similarity is a key evaluation target for generative speech systems, including voice conversion, speaker-cloning text-to-speech, and zero-shot speech models. When the generated voice does not match the target speaker, the problem often reflects imperfect separation of speaker-related timbre from content, making it important to identify where timbre information resides and which representations still carry it. Using speaker identity as a measurable proxy, we probe frequency bands, phase/magnitude structure, WORLD source/filter decomposition, voiced/unvoiced regions, self-supervised learning (SSL) layers, EnCodec/HuBERT tokens, and a 45-listener test across Mandarin, English, and Japanese speech/singing data. Speaker information is distributed: 1--8 kHz magnitude and vocal-tract spectral-envelope structure dominate perception, above-8 kHz bands remain machine-discriminative, shallow SSL layers expose compact speaker information, and token leakage depends on tokenizer/layer choice. These findings indicate where generation systems should condition on or transform speaker information, and where content, codec, and token pipelines should audit residual speaker leakage.

Figures & tables

Experiment group	Data scale	What it checks
AISHELL-1	20 spk.	lowpass/highpass, SSL, tokens
AISHELL-3	98 spk.	high-sample-rate bands, SSL/token validation, listening
VCTK	100 spk.	48 kHz high-band and unvoiced validation
ESD/CREMA-D	20/91 spk.	emotion robustness
NUS/JVS	12/100 spk.	speech–singing robustness
Self-rec.	4 spk.	controlled band-swap and voiced/unvoiced pilot

TABLE I: Data used in the main checks.

Fig. 1: Speaker-information localization protocol. The study combines signal edits, model-based and model-free measurements, representation probes, and human similarity ratings, so speaker information measured by models can be kept separate from perceived speaker-related timbre information.

Dataset/Band	1–4	4–8	8–12	High band
VCTK acc.	92.1	89.3	88.1	50.1 (20–24)
VCTK shuf.	1.2	1.5	1.1	0.8
VCTK dur./eng.	4.5	5.8	5.4	5.4
AISHELL-3 acc.	82.5	72.4	55.0	42.8 (16–20)
AISHELL-3 shuf.	0.9	1.0	1.0	0.9
AISHELL-3 dur./eng.	6.4	5.6	5.3	6.0

TABLE II: Band MLP check. Values are speaker-classification accuracy (%); shuffled labels stay near chance and duration/energy baselines are much lower.

Fig. 2: Bandwidth comparison. SV lowpass curves saturate near 4–8 kHz, while mel probes show weaker but measurable above-8 kHz information.

Condition	Mean $\pm$ SD	95% CI
Different full	2.29 $\pm$ 0.53	[2.13, 2.45]
Same 12–20 kHz	1.07 $\pm$ 0.28	[0.99, 1.16]
Same $>$ 8 kHz	1.24 $\pm$ 0.29	[1.15, 1.32]
Same $<$ 4 kHz	3.81 $\pm$ 0.58	[3.64, 3.99]
Same 1–4 kHz	2.92 $\pm$ 0.69	[2.71, 3.13]
Same $<$ 8 kHz	4.45 $\pm$ 0.35	[4.35, 4.56]

TABLE III: Human similarity ratings (45 listeners, 1–5 scale).

Fig. 3: Listening test. Low/mid-frequency same-speaker pairs receive high similarity ratings; high-frequency-only pairs receive low ratings.

Backend	Condition	Score	Ret.
WavLM	magnitude kept	disc. 0.195	81.6%
WavLM	phase kept	disc. -0.199	–
WavLM	filter kept	sim. 0.906	92.4%
WavLM	F0 kept	sim. 0.804	82.0%
ECAPA	magnitude kept	disc. 0.284	77.1%
ECAPA	phase kept	disc. 0.003	0.7%

TABLE IV: Phase/magnitude and source/filter summary across two SV backends. Ret. denotes retention relative to the full or original condition.

Fig. 4: Second-backend validation. WavLM and ECAPA both favor magnitude/filter information over phase/F0-dominant information.

Model	Peak	Final	32d	16d	8d
WavLM	L1 99.8	L12 94.8	99.8	98.3	85.3
HuBERT	L5 99.5	L12 95.3	99.0	93.8	69.2
Wav2Vec2	L1 99.3	L12 95.0	99.0	95.8	82.2
Whisper	L3 97.5	L6 92.0	96.5	90.2	74.5

TABLE V: SSL representation results on AISHELL-1. PCA columns use each model’s peak layer.

HuBERT k-means tokens
Layer	$k$ =50	$k$ =100	$k$ =200	$k$ =500	$k$ =1000
L1	60.2	68.5	78.0	89.0	94.0
L3	46.5	65.7	70.7	84.0	92.7
L6	43.0	57.8	75.3	87.3	96.2
L9	37.7	52.0	64.7	75.2	88.5
L12	40.7	57.2	68.3	74.2	84.2

TABLE VI: Token and codec results. HuBERT and EnCodec values are speaker-classification accuracies (%).

Condition	Quantitative detail
ESD emotion	Same-speaker cross-emotion cosine 0.899 vs. different-speaker same-emotion 0.644; gap +0.255.
CREMA-D	Leave-one-emotion-out mean 89.2%; held-out anger 75.4%, neutral 95.1%.
NUS-48E	Sing–read same-speaker similarity 0.909; cross-modal discriminability 0.246.
JVS/JVS-MuSiC	Speech-trained classifier reaches 23.7% on singing; centroid gap 0.272.
VCTK V/UV	Unvoiced-only 70.9 $\pm$ 1.2%; voiced-vs.-unvoiced cosine gap 0.463.

TABLE VII: Robustness details across style, modality, and phonation changes.

Speech Processing

Speech processing develops computational methods for recognizing, synthesizing, analyzing, and transforming spoken audio.

See all 412 papers

A Comprehensive Study of Content Representations for Speech Synthesis
Sep 25, 2026Diego Torres, Axel Roebel, Nicolas ObinAudio Representation Learning Discrete Speech Representation Learning
STMS Lab IRCAM, CNRS, Sorbonne Universit´e Paris, France
Speech Generation Speaker Poisoning: Capability Erasure in Zero-Shot Text-to-Speech
Mar 8, 2026Thanathai Lertpetchpun, Thanapat Trachu, Sai Praneeth Karimireddy +1TTS Synthesis Zero-Shot TTS
Signal Analysis and Interpretation Laboratory, University of Southern California · Thomas Lord Department of Computer Science, University of Southern California
Universal Speech Content Factorization
Mar 9, 2026Henry Li Xinyuan, Zexin Cai, Lin Zhang +5Voice Conversion Speaker Disentanglement
Human Language Technology Center of Excellence (COE), Johns Hopkins University, USA · Center for Language and Speech Processing, Johns Hopkins University, USA

Sep 25, 2026J/K move · Enter open · S save

A Comprehensive Study of Content Representations for Speech Synthesis

Diego Torres, Axel Roebel, Nicolas Obin

STMS Lab IRCAM, CNRS, Sorbonne Universit´e Paris, France

Audio Representation Learning Discrete Speech Representation Learning Speaker Disentanglement Speech Generation Evaluation

Open paper AskPDF arXiv

Speech content representations are central to voice conversion, speech-to-speech translation, and multimodal language models, yet they are rarely compared under a common generative framework that directly measures what each representation contains. We address this by training a generative model conditioned solely on each representation and evaluating the generated audio along the content, speaker identity, and prosody axes. Across SSL features, supervised tokens, posteriorgrams, and neural audio codecs, we find two distinct regimes: representations that nearly reconstruct the original audio, and representations that effectively disentangle speaker identity. These results show that disentanglement depends not on supervision alone, but on the interaction between the training objective and the representation's information capacity: supervised representations only disentangle speaker identity when their capacity is sufficiently constrained.

Abstract

Speech Processing

Explore similar work

A Comprehensive Study of Content Representations for Speech Synthesis

Speech Generation Speaker Poisoning: Capability Erasure in Zero-Shot Text-to-Speech

Universal Speech Content Factorization