Energy-Based Models For Speech Synthesis

Wanli Sun; Zehai Tu; Anton Ragni

cs.SDOct 19, 2023

Energy-Based Models For Speech Synthesis

Authors: Wanli Sun, Zehai Tu, Anton Ragni

Organizations: Department of Computer Science, University of Sheffield, Sheffield, UK

Abstract

Recently there has been a lot of interest in non-autoregressive (non-AR) models for speech synthesis, such as FastSpeech 2 and diffusion models. Unlike AR models, these models do not have autoregressive dependencies among outputs which makes inference efficient. This paper expands the range of available non-AR models with another member called energy-based models (EBMs). The paper describes how noise contrastive estimation, which relies on the comparison between positive and negative samples, can be used to train EBMs. It proposes a number of strategies for generating effective negative samples, including using high-performing AR models. It also describes how sampling from EBMs can be performed using Langevin Markov Chain Monte-Carlo (MCMC). The use of Langevin MCMC enables to draw connections between EBMs and currently popular diffusion models. Experiments on LJSpeech dataset show that the proposed approach offers improvements over Tacotron 2.

Figures & tables

Figure 1: Architecture of EBMs examined in this work (inspired by Transformer TTS)

Model	MCD $\downarrow$	FFE $\downarrow$	$\log f_{\mathrm{o}}\downarrow$	Parameters
Tacotron 2	4.218	47.31%	0.292	28.19M
EBM (1) (small)	4.163	47.06%	0.289	2.30M
EBM (1) (large)	4.178	47.05%	0.291	7.64M

Table 1: Comparison between Tacotron 2 and two EBMs utilising Tacotron 2 hypotheses as negative samples

Condition		MCD $\downarrow$	FFE $\downarrow$	$\log f_{\mathrm{o}}\downarrow$
TM:	5%	4.149	46.89%	0.290
	10%	4.166	46.98%	0.284
	15%	4.161	47.30%	0.285
FM:	5%	4.166	46.88%	0.292
	10%	4.138	47.27%	0.286
	15%	4.097	47.35%	0.284

Table 2: Negative sampling methods (95% confidence intervals)

Step	MCD $\downarrow$	FFE $\downarrow$	$\log f_{\mathrm{o}}\downarrow$
0	4.218	47.31%	0.292
1	4.217	47.31%	0.292
100	3.943	46.16%	0.282
300	$\bm{3.937}$	$\bm{45.85\%}$	$\bm{0.276}$

Table 3: Simplified Langevin MCMC (95% confidence intervals)

RM	TM	FM	TW	MCD $\downarrow$	FFE $\downarrow$	$\log f_{\mathrm{o}}\downarrow$
30%	5%	5%	1.2	MCD $\downarrow$	FFE $\downarrow$	$\log f_{\mathrm{o}}\downarrow$
$\checkmark$				4.013	46.57%	0.280
$\checkmark$	$\checkmark$			3.958	46.31%	0.276
$\checkmark$		$\checkmark$		3.997	45.63%	0.278
$\checkmark$			$\checkmark$	3.927	45.98%	0.267
$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\bm{3.882}$	$\bm{45.36\%}$	$\bm{0.258}$

Table 4: Combination of negative sampling methods (95% confidence intervals)

Method	MOS
Ground Truth	4.53 $\pm$ 0.05
Ground Truth (log-Mel + WaveGlow)	4.39 $\pm$ 0.07
Tacotron 2 (log-Mel + WaveGlow)	3.77 $\pm$ 0.11
EBM (log-Mel + WaveGlow)	3.84 $\pm$ 0.13

Table 5: Subjective evaluation

MOS	1	2	3	4	5
Tacotron 2	0	3	121	344	15
EBM	0	1	94	370	18

Table 6: MOS score counts

TTS Synthesis

Text-to-speech synthesis converts written language into intelligible, natural-sounding speech with control over voice, prosody, style, and timing.

See all 275 papers

Score-Based Training for Energy-Based TTS Models
May 19, 2025Wanli Sun, Anton RagniEnergy-Based Models Score Matching
School of Computer Science, University of Sheffield, Sheffield, UK
Equilibrium Training of Energy-Based Models with Parallel Trajectory Tempering
Jul 29, 2026Nicolas Béreux, Aurélien Decelle, Cyril Furtlehner +1Markov Chain Monte Carlo Energy-Based Models
Université Paris-Saclay, CNRS, INRIA, LISN, 91190 Gif-sur-Yvette, France · Escuela Técnica Superior de Ingenieros Industriales, Universidad Politécnica de Madrid, 28006 Madrid, Spain · Departamento de Física Teórica & IPARCOS, Universidad Complutense de Madrid, 28040 Madrid, Spain.
Fast Text-to-Audio Generation with One-Step Sampling via Energy-Scoring and Auxiliary Contextual Representation Distillation
May 1, 2026Kuan-Po Huang, Bo-Ru Lu, Byeonggeun Kim +8Text-to-Audio Generation One-Step Generative Modeling
National Taiwan University · Amazon AGI

May 19, 2025J/K move · Enter open · S save

Score-Based Training for Energy-Based TTS Models

Wanli Sun, Anton Ragni

School of Computer Science, University of Sheffield, Sheffield, UK

Energy-Based Models Score Matching Score-Based Generative Modeling TTS Synthesis

Open paper AskPDF arXiv

Noise contrastive estimation (NCE) is a popular method for training energy-based models (EBM) with intractable normalisation terms. The key idea of NCE is to learn by comparing unnormalised log-likelihoods of the reference and noisy samples, thus avoiding explicitly computing normalisation terms. However, NCE critically relies on the quality of noisy samples. Recently, sliced score matching (SSM) has been popularised by closely related diffusion models (DM). Unlike NCE, SSM learns a gradient of log-likelihood, or score, by learning distribution of its projections on randomly chosen directions. However, both NCE and SSM disregard the form of log-likelihood function, which is problematic given that EBMs and DMs make use of first-order optimisation during inference. This paper proposes a new criterion that learns scores more suitable for first-order schemes. Experiments contrasts these approaches for training EBMs.

Abstract

TTS Synthesis

Explore similar work

Score-Based Training for Energy-Based TTS Models

Equilibrium Training of Energy-Based Models with Parallel Trajectory Tempering

Fast Text-to-Audio Generation with One-Step Sampling via Energy-Scoring and Auxiliary Contextual Representation Distillation