Score-Based Training for Energy-Based TTS Models

Wanli Sun; Anton Ragni

cs.SDMay 19, 2025

Score-Based Training for Energy-Based TTS Models

Authors: Wanli Sun, Anton Ragni

Organizations: School of Computer Science, University of Sheffield, Sheffield, UK

Abstract

Noise contrastive estimation (NCE) is a popular method for training energy-based models (EBM) with intractable normalisation terms. The key idea of NCE is to learn by comparing unnormalised log-likelihoods of the reference and noisy samples, thus avoiding explicitly computing normalisation terms. However, NCE critically relies on the quality of noisy samples. Recently, sliced score matching (SSM) has been popularised by closely related diffusion models (DM). Unlike NCE, SSM learns a gradient of log-likelihood, or score, by learning distribution of its projections on randomly chosen directions. However, both NCE and SSM disregard the form of log-likelihood function, which is problematic given that EBMs and DMs make use of first-order optimisation during inference. This paper proposes a new criterion that learns scores more suitable for first-order schemes. Experiments contrasts these approaches for training EBMs.

Figures & tables

Figure 1: Two ways of computing scores for EBMs: analytic (top) and predictive (bottom)

Figure 2: Different score functions: from left to right, the hypotheses list from the worst to the best.

SSM	Delta	MCD $\downarrow$	$\log f{\mathrm{0}}\downarrow$	S-BERT $\uparrow$	UTMOSv2 $\uparrow$
✓	✗	$\bm{5.312}$	$\bm{0.296}$	$\bm{0.905}$	3.710
✗	✓	5.399	0.299	0.900	2.959
✓	✓	$\bm{5.312}$	$\bm{0.296}$	$\bm{0.905}$	$\bm{3.718}$
Tacotron 2		5.770	0.300	0.882	3.409

Table 1: Objective evaluation of score-based training, 100 inference steps, full validation (1800) set

Table 2: Objective evaluation of inference for score-based EBMs, partial validation (600) set

Loss	Step	MCD $\downarrow$	$\log f_{\mathrm{o}}\downarrow$	S-BERT $\uparrow$	UTMOSv2 $\uparrow$
$\text{NCE}^{*}$	100	5.762	$\bm{0.295}$	0.875	2.572
SSM	1	5.294	0.299	0.904	$\bm{3.717}$
Delta	1	$\bm{5.292}$	0.299	0.904	3.703

Table 3: Objective evaluation of NCE (full validation set) and score-based (partial validation set) trained EBMs

#	Method	MOS
1	Ground Truth	4.29 $\pm$ 0.04
2	Ground Truth (log Mel + HiFi-GAN)	4.15 $\pm$ 0.06
3	Tacotron 2 (log Mel + HiFi-GAN)	3.74 $\pm$ 0.08
4	SSM (1 steps) (log Mel + HiFi-GAN)	3.90 $\pm$ 0.06
5	Delta (1 steps) (log Mel + HiFi-GAN)	3.95 $\pm$ 0.05
6	SSM (10 steps) (log Mel + HiFi-GAN)	3.82 $\pm$ 0.09

Table 4: Subjective evaluation

Panel of Figure 3 — Figure 3: Detailed breakdown of MOS score counts

Energy-Based Models

Energy-based models assign scalar energies to configurations, defining probabilities and guiding generation, inference, prediction, and associative memory.

See all 91 papers

Energy-Based Models For Speech Synthesis
Oct 19, 2023Wanli Sun, Zehai Tu, Anton RagniTTS Synthesis Energy-Based Models
Department of Computer Science, University of Sheffield, Sheffield, UK
Learning Energy-Based Models from Stochastic Interpolants using Spatiotemporal Differences
May 26, 2026Hanlin Yu, RuiKang OuYang, Partha Kaushik +3Contrastive Learning Stochastic Interpolants
University of Helsinki · University of Cambridge · Carnegie Mellon University +1
Equilibrium Training of Energy-Based Models with Parallel Trajectory Tempering
Jul 29, 2026Nicolas Béreux, Aurélien Decelle, Cyril Furtlehner +1Markov Chain Monte Carlo Energy-Based Models
Université Paris-Saclay, CNRS, INRIA, LISN, 91190 Gif-sur-Yvette, France · Escuela Técnica Superior de Ingenieros Industriales, Universidad Politécnica de Madrid, 28006 Madrid, Spain · Departamento de Física Teórica & IPARCOS, Universidad Complutense de Madrid, 28040 Madrid, Spain.

Oct 19, 2023J/K move · Enter open · S save

Energy-Based Models For Speech Synthesis

Wanli Sun, Zehai Tu, Anton Ragni

Department of Computer Science, University of Sheffield, Sheffield, UK

TTS Synthesis Energy-Based Models

Open paper AskPDF arXiv

Recently there has been a lot of interest in non-autoregressive (non-AR) models for speech synthesis, such as FastSpeech 2 and diffusion models. Unlike AR models, these models do not have autoregressive dependencies among outputs which makes inference efficient. This paper expands the range of available non-AR models with another member called energy-based models (EBMs). The paper describes how noise contrastive estimation, which relies on the comparison between positive and negative samples, can be used to train EBMs. It proposes a number of strategies for generating effective negative samples, including using high-performing AR models. It also describes how sampling from EBMs can be performed using Langevin Markov Chain Monte-Carlo (MCMC). The use of Langevin MCMC enables to draw connections between EBMs and currently popular diffusion models. Experiments on LJSpeech dataset show that the proposed approach offers improvements over Tacotron 2.

Abstract

Energy-Based Models

Explore similar work

Energy-Based Models For Speech Synthesis

Learning Energy-Based Models from Stochastic Interpolants using Spatiotemporal Differences

Equilibrium Training of Energy-Based Models with Parallel Trajectory Tempering