Universal Speech Content Factorization

Henry Li Xinyuan; Zexin Cai; Lin Zhang; Leibny Paola García-Perera; Berrak Sisman; Sanjeev Khudanpur; Nicholas Andrews; Matthew Wiesner

eess.ASMar 9, 2026

Universal Speech Content Factorization

Authors: Henry Li Xinyuan, Zexin Cai, Lin Zhang, Leibny Paola García-Perera, Berrak Sisman, Sanjeev Khudanpur, Nicholas Andrews, Matthew Wiesner

Organizations: Human Language Technology Center of Excellence (COE), Johns Hopkins University, USA · Center for Language and Speech Processing, Johns Hopkins University, USA

Abstract

We propose Universal Speech Content Factorization (USCF), a simple and invertible linear method for extracting a low-rank speech representation in which speaker timbre is suppressed while phonetic content is preserved. USCF extends Speech Content Factorization, a closed-set voice conversion (VC) method, to an open-set setting by learning a universal speech-to-content mapping via least-squares optimization and deriving speaker-specific transformations from only a few seconds of target speech. We show through embedding analysis that USCF effectively removes speaker-dependent variation. As a zero-shot VC system, USCF achieves competitive intelligibility, naturalness, and speaker similarity compared to methods that require substantially more target-speaker data or additional neural training. Finally, we demonstrate that as a training-efficient timbre-disentangled speech feature, USCF features can serve as the acoustic representation for training timbre-prompted text-to-speech models. Speech samples and code are publicly available.

Figures & tables

Figure 1: Full pipeline for voice conversion using USCF.

Figure 2: Decomposing speech into a content-factorized form through SCF. $\mathbf{X}_{i}$ are content-aligned WavLM features for different speakers. Content alignment for $\mathbf{X}_{i}$ is performed through kNN matching.

Figure 3: Left: formulation for $\mathbf{W}_{1}$ , one of our proposed universal speech-to-content mappings. Right: Derivation of speaker transformation matrix $\mathbf{S}_{4}$ for unseen speaker $4$

Method	WER (%) $\downarrow$	UTMOS $\uparrow$	Spk Sim $\uparrow$
USCF $\mathbf{W}_{1}$	2.70	2.805	0.524
USCF $\mathbf{W}_{2}$	4.04	2.519	0.557
USCF $\mathbf{W}_{3}$	2.31	2.826	0.420
kNN-VC	3.16	2.855	0.666
LinearVC	2.69	2.765	0.621
SCF	2.18	2.886	0.603

Table 1: Comparisons of USCF VC with baseline systems according to objective metrics. Color intensity reflects metric directionality.

Method	MOS (%) $\uparrow$	SMOS (%) $\uparrow$
original	3.80 $\pm$ 0.17	4.29 $\pm$ 0.15
USCF $\mathbf{W}_{1}$	3.42 $\pm$ 0.17	3.00 $\pm$ 0.23
USCF $\mathbf{W}_{1}$ 100s	3.66 $\pm$ 0.17	2.94 $\pm$ 0.29
kNN-VC	3.53 $\pm$ 0.18	3.29 $\pm$ 0.25
LinearVC	3.65 $\pm$ 0.16	3.17 $\pm$ 0.21
SCF	3.63 $\pm$ 0.18	3.08 $\pm$ 0.25

Table 2: Comparison of subjective metrics between USCF and baselines. Mean scores with

95\%

intervals.

Model	Rank	Spk EER $\uparrow$	Phoneme EER $\downarrow$
USCF	75	36.40%	11.43%
USCF	1024	35.33%	11.43%
WavLM	1024	21.77%	11.56%
ContentVec	792	27.98%	8.82%
WavLM - kNN	1024	35.14%	12.70%

Table 3: Speaker ID using same-phoneme embeddings.

Rank	WER (%) $\downarrow$	UTMOS $\uparrow$	Spk Sim $\uparrow$
100	2.77	2.81	0.504
75	2.70	2.805	0.524
50	2.69	2.738	0.529
30	2.96	2.607	0.513
20	3.98	2.388	0.489

Table 4: VC performance of USCF across different ranks.

Num frames	WER (%) $\downarrow$	UTMOS $\uparrow$	Spk Sim $\uparrow$
10000	2.28	2.935	0.564
5000	2.42	2.923	0.564
2000	2.47	2.915	0.564
1000	2.51	2.904	0.546
500	2.70	2.805	0.524
200	4.94	2.431	0.42

Table 5: VC performance of USCF when output output speaker transformation matrix is derived with different number of frames of target speaker speech.

Target Features	ASR WER $\downarrow$	Epochs	UTMOS-v2 $\uparrow$
USCF features	11.44%	25	2.881
mel	27.93%	39	2.741
mel (normalized)	11.92%	33	2.732

Table 6: TTS models trained using different target features. Fbank (normalized) is where each utterance is first normalized to the same voice using kNN-VC [ 3 ] , then converted into Fbank features.

Voice Conversion

Voice conversion changes a speaker's vocal identity or delivery while preserving the spoken content.

See all 35 papers

A Comprehensive Study of Content Representations for Speech Synthesis
Sep 25, 2026Diego Torres, Axel Roebel, Nicolas ObinAudio Representation Learning Discrete Speech Representation Learning
STMS Lab IRCAM, CNRS, Sorbonne Universit´e Paris, France
Zero-VC: Zero-Lookahead Streaming Voice Conversion via Speaker Anonymization
Jun 18, 2026Yudong Li, Zihao Fang, Junwen Qiu +4Speech Processing Speaker Anonymization
The Chinese University of Hong Kong, Shenzhen · Shenzhen Transsion Holdings Co., Ltd. · Shenzhen Loop Area Institute +1
Where Does Timbre Hide? Localizing Speaker Information in Speech Signals
Oct 21, 2026Yiquan Zhou, Yixuan Wang, Xiaozhe Qi +2Speech Processing
School of Software Engineering, Xi’an Jiaotong University, Xi’an, China, and also with the SYKI-SPEECH Team, Xi’an, China · Division of Music and Audio, Union Wheatland Culture and Media Ltd., Chengdu, China

Sep 25, 2026J/K move · Enter open · S save

A Comprehensive Study of Content Representations for Speech Synthesis

Diego Torres, Axel Roebel, Nicolas Obin

STMS Lab IRCAM, CNRS, Sorbonne Universit´e Paris, France

Audio Representation Learning Discrete Speech Representation Learning Speaker Disentanglement Speech Generation Evaluation

Open paper AskPDF arXiv

Speech content representations are central to voice conversion, speech-to-speech translation, and multimodal language models, yet they are rarely compared under a common generative framework that directly measures what each representation contains. We address this by training a generative model conditioned solely on each representation and evaluating the generated audio along the content, speaker identity, and prosody axes. Across SSL features, supervised tokens, posteriorgrams, and neural audio codecs, we find two distinct regimes: representations that nearly reconstruct the original audio, and representations that effectively disentangle speaker identity. These results show that disentanglement depends not on supervision alone, but on the interaction between the training objective and the representation's information capacity: supervised representations only disentangle speaker identity when their capacity is sufficiently constrained.

Abstract

Voice Conversion

Explore similar work

A Comprehensive Study of Content Representations for Speech Synthesis

Zero-VC: Zero-Lookahead Streaming Voice Conversion via Speaker Anonymization

Where Does Timbre Hide? Localizing Speaker Information in Speech Signals