A Comprehensive Study of Content Representations for Speech Synthesis

Diego Torres; Axel Roebel; Nicolas Obin

eess.ASSep 25, 2026

A Comprehensive Study of Content Representations for Speech Synthesis

Authors: Diego Torres, Axel Roebel, Nicolas Obin

Organizations: STMS Lab IRCAM, CNRS, Sorbonne Universit´e Paris, France

Abstract

Speech content representations are central to voice conversion, speech-to-speech translation, and multimodal language models, yet they are rarely compared under a common generative framework that directly measures what each representation contains. We address this by training a generative model conditioned solely on each representation and evaluating the generated audio along the content, speaker identity, and prosody axes. Across SSL features, supervised tokens, posteriorgrams, and neural audio codecs, we find two distinct regimes: representations that nearly reconstruct the original audio, and representations that effectively disentangle speaker identity. These results show that disentanglement depends not on supervision alone, but on the interaction between the training objective and the representation's information capacity: supervised representations only disentangle speaker identity when their capacity is sufficiently constrained.

Figures & tables

Figure 1: Diagram of the evaluation methodology.

		Content	Identity			Prosody
	$K$	dWER	EER	Gender	Sim.	Reg.	Range	Corr.
Real audio	–	–	0.5	96.9	–	–	–	–
Vocos	–	0.2	0.6	97.8	0.89	6	8	0.98
HuBERT	$\infty$	0.5	1.3	98.7	0.61	48	30	0.94
	1024	1.7	32.4	91.7	0.16	181	81	0.48
	256	2.3	36.1	78.4	0.14	206	90	0.41

Table 1: Evaluation results for the considered speech content representations. dWER, EER and gender accuracy are measured in percentage. Register and range errors are measured in cents. dWER is computed corpus-wise, prosody metrics are shown as the median, while gender and speaker similarity are the mean.

K

indicates the number of discrete codes, and is set to

\infty

whenever a representation is continuous.

Figure 2: Scatter plot of EER and pitch correlation versus dWER. Numbers next to the markers indicate the codebook size, and their absence indicates that the features are continuous. Bars indicate the 95% confidence intervals of: the mean of EER, the median of pitch correlation, and the corpus-wise dWER.

	$K$	dWER (%)	EER (%)	Gender (%)	Pitch Corr.
VQ-ASR 8-d	256	$\uparrow$ 3.2	46.8	55.9	$\downarrow$ 0.29
	$\infty$	$\uparrow$ 2.4	$\uparrow$ 39.7	$\downarrow$ 71.9	$\downarrow$ 0.35

Table 2: Evaluation of VQ-ASR when the codebook dimension is decreased to 8. Both the quantized version with 256 codes and the continuous version are considered.

		dWER (%)		EER (%)		UTMOS
Representation	bit/s	$w{=}1$	$w{=}2$	$w{=}1$	$w{=}2$	$w{=}1$	$w{=}2$
Real audio	–	–		0.5		4.14
Vocos	–	0.2		0.6		3.77
HuBERT	1.2 M	0.5	0.5	1.3	1.3	3.77	3.77
ContentVec	1.2 M	1.0	$\downarrow$ 0.6	19.6	$\downarrow$ 16.3	3.54	$\uparrow$ 3.86
Soft units	410 k	0.5	0.5	3.2	$\downarrow$ 2.2	3.74	$\uparrow$ 3.82

Table 3: Comparison of metrics when guidance scale is changed (

w=1.0

w=2.0

	$K$	dWER (%)	EER (%)	Gender (%)	Pitch Corr.
$k$ -means	256	2.3	34.8	85.0	$\uparrow$ 0.43
VQ-ASR	256	$\downarrow$ 2.5	47.7	50.5	$\uparrow$ 0.37

Table 4: Evaluation with a decoder with three times as many parameters.

Audio Representation Learning

Audio representation learning builds reusable features that capture speech, music, and sound for recognition, retrieval, generation, and analysis.

See all 211 papers

SPEAR-Gen: Generation-Aware Pre-training for Unified Speech Representations
Sep 28, 2026Xiaoyu Yang, Arthur Hinsvark, Antonios Alexos +3Representation Learning Self-Supervised Speech Representation Learning
Department of Engineering, University of Cambridge, Cambridge, UK · Meta Superintelligence Labs, Menlo Park, USA
Where Does Timbre Hide? Localizing Speaker Information in Speech Signals
Oct 21, 2026Yiquan Zhou, Yixuan Wang, Xiaozhe Qi +2Speech Processing
School of Software Engineering, Xi’an Jiaotong University, Xi’an, China, and also with the SYKI-SPEECH Team, Xi’an, China · Division of Music and Audio, Union Wheatland Culture and Media Ltd., Chengdu, China
Universal Speech Content Factorization
Mar 9, 2026Henry Li Xinyuan, Zexin Cai, Lin Zhang +5Voice Conversion Speaker Disentanglement
Human Language Technology Center of Excellence (COE), Johns Hopkins University, USA · Center for Language and Speech Processing, Johns Hopkins University, USA

Sep 28, 2026J/K move · Enter open · S save

SPEAR-Gen: Generation-Aware Pre-training for Unified Speech Representations

Xiaoyu Yang, Arthur Hinsvark, Antonios Alexos, Osama Hanna+2

Department of Engineering, University of Cambridge, Cambridge, UK · Meta Superintelligence Labs, Menlo Park, USA

Representation Learning Self-Supervised Speech Representation Learning Discrete Speech Representation Learning

Open paper AskPDF arXiv

Speech understanding and generation place different demands on speech representations, and existing models are typically optimised towards one capability or the other. To reduce this gap, we introduce SPEAR-Gen, a speech representation model that learns a single representation for both capabilities. Task-aligned feature aggregation consolidates complementary linguistic and paralinguistic information across a frozen encoder into discrete targets for masked prediction, while a coarse-to-fine objective combines log-Mel reconstruction with residual flow matching to preserve spectral structure and fine-grained acoustic variation. Experiments on SUPERB and speech resynthesis show that SPEAR-Gen maintains strong understanding performance while substantially improving resynthesis quality and speaker preservation. These results demonstrate that a single speech representation can effectively support both understanding and generation.

Abstract

Audio Representation Learning

Explore similar work

SPEAR-Gen: Generation-Aware Pre-training for Unified Speech Representations

Where Does Timbre Hide? Localizing Speaker Information in Speech Signals

Universal Speech Content Factorization