Bayesian Speech Synthesizers Can Learn from Multiple Teachers

Ziyang Zhang; Yifan Gao; Xuenan Xu; Baoxiang Li; Wen Wu; Chao Zhang

cs.SDOct 28, 2025

Bayesian Speech Synthesizers Can Learn from Multiple Teachers

Authors: Ziyang Zhang, Yifan Gao, Xuenan Xu, Baoxiang Li, Wen Wu, Chao Zhang

Organizations: Tsinghua University · Shanghai Artificial Intelligence Laboratory.

Abstract

Text-to-speech (TTS) is a "one-to-many" problem: one text can be spoken in many valid ways, so its realizations form a distribution whose spread depends on the input. Most TTS models do not model this spread explicitly: they leave it implicit in the sampling process or fix it in advance. To model this distribution explicitly, we propose BELLE (Bayesian evidential learning with language modelling), which places a Bayesian prior over the mean and variance of each mel-spectrogram frame and predicts this prior from the input, while keeping the architecture and single-pass sampling of common continuous-valued autoregressive TTS models such as MELLE. Estimating this variance requires several recordings of each text, whereas standard TTS corpora provide one. We therefore construct two parallel-speech corpora: ParallelVox from real human recordings and LS+6TTS, which adds speech from six pre-trained TTS teachers to LibriSpeech (LS). Trained on these corpora, BELLE outperforms MELLE under the same training conditions, is competitive with larger TTS systems on the objective metrics of LibriSpeech test-clean, and supports streaming generation. Analyses of repeated human readings show that real speech has the properties the Bayesian prior assumes and that the variance predicted by BELLE follows the variation across these readings. Parallel speech improves BELLE, and synthetic parallel speech performs comparably to real parallel speech at the same volume and is far easier to scale.

Figures & tables

Figure 1: Overview of BELLE. Left: the mel-based AR TTS architecture (Sec. 3.2 ). Right: the Sampling Module, which predicts a Bayesian prior over the mean and variance of each frame and samples the frame from it (Sec. 4 ).

Figure 2: Interleaved text and mel-spectrogram chunks in BELLE-stream.

System	Training data	Continuation		Cross-Sentence		Diversity	Subjective
System	Training data	WER-H $\downarrow$	SIM-o $\uparrow$	WER-H $\downarrow$	SIM-o $\uparrow$	Cos. $\uparrow$	MOS $\uparrow$	SMOS $\uparrow$
Ground Truth	-	2.15	0.668	2.15	0.672	46.7	4.20	3.81
VALL-E	ParallelVox	7.46	0.340	16.3	0.452	17.4	-	-
MELLE	ParallelVox	3.06	0.406	5.60	0.538	19.9	-	-
BELLE	ParallelVox	3.00	0.449	5.73	0.570	21.5	-	-
BELLE	LS+6TTS-881h	2.57	0.510	4.05	0.634	20.3	-	-

Table 1: Controlled comparison under a shared training and evaluation pipeline. WER-H and SIM-o are measured on LibriSpeech test-clean, and Cos. (

\times 10^{-4}

) on the ParallelVox single-speaker repeat test set. MOS and SMOS were collected only for Ground Truth and the models trained on LS+6TTS.

System	WER-H $\downarrow$	SIM-o $\uparrow$
SMLLE	6.37	0.489
IST-LM	4.53	0.653
BELLE-stream	4.01	0.557

Table 3: Cross-Sentence results of streaming systems: SMLLE ( Sun et al., 2025 ) , IST-LM ( Yang et al., 2024 ) , and BELLE-stream trained on LS+6TTS.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Source	Recordings	Hours	Speakers	License
LibriHeavy ( Kang et al., 2024 )	312,959	1,017.7	3,392	CC BY 4.0
Parallel Audiobook Corpus	84,791	134.4	53	Public Domain
VCTK 0.92 ( Yamagishi et al., 2019 )	34,981	20.7	109	CC BY 4.0
CMU ARCTIC ( Kominek and Black, 2004 ) + L2-Arctic ( Zhao et al., 2018 )	18,625	15.5	42	Mixed
Common Voice 17.0 ( Ardila et al., 2020 ) , parallel subset	284	0.2	255	CC-0
Total	451,640	1,188.5	3,851

Appendix

Table 4: Source corpora of ParallelVox after VAD trimming and before WER filtering.

WER $\leq$	MOS $\geq$	Texts	Recordings	Hours
10%	—	123,640	344,303	903.0
10%	3.8	60,012	175,356	400.1
10%	4.0	38,618	114,741	244.4
5%	—	92,734	262,337	638.5
3%	—	73,683	214,831	474.9

Appendix

Table 5: Size of ParallelVox under different WER and MOS thresholds. The first row is the setting used in this paper.

Human	Each teacher	Continuation		Cross-Sentence
Human	Each teacher	WER-H $\downarrow$	SIM-o $\uparrow$	WER-H $\downarrow$	SIM-o $\uparrow$
0.16	0.14	2.64	0.415	5.17	0.453
0.22	0.13	2.57	0.409	4.73	0.449
0.28	0.12	2.78	0.406	5.47	0.444
0.34	0.11	2.76	0.405	5.47	0.446

Appendix

Table 6: Weights of the human recording and of each teacher realization in Eq. ( 13 ), on LS-100+6TTS . The second row is the setting used in this paper.

Panel of Figure 3 — Figure 3: Instructions shown to the raters in the subjective evaluation.

Group source	Duration	Median F0	F0 s.d.	Energy	Energy s.d.
Ground Truth (human readings)	0.54	0.70	0.61	0.21	0.65
BELLE ParallelVox	0.07	0.37	0.25	0.34	0.33
BELLE LS+6TTS	0.19	0.38	0.25	0.29	0.29

Appendix

Table 7: Spearman’s

\rho

between the within-group variation of prosodic quantities and the within-group WavLM distance.

Metric	ParallelVox	LS	LS+6TTS
Continuation WER-H	$-0.07$ ( $p=0.52$ )	$-0.39$ ( $p<0.001$ )	$-0.46$ ( $p<0.001$ )
Continuation SIM-o	$+0.044$ ( $p<0.001$ )	$+0.005$ ( $p=0.004$ )	$+0.031$ ( $p<0.001$ )
Cross-Sentence WER-H	$+0.13$ ( $p=0.56$ )	$-3.75$ ( $p<0.001$ )	$-0.84$ ( $p<0.001$ )
Cross-Sentence SIM-o	$+0.032$ ( $p<0.001$ )	$+0.015$ ( $p<0.001$ )	$+0.036$ ( $p<0.001$ )
Diversity (Cos.)	$+1.67$ ( $p=0.007$ )	$+6.90$ ( $p<0.001$ )	$+2.60$ ( $p<0.001$ )
MOS	–	–	$+0.19$ ( $p=0.021$ )

Appendix

Table 8: BELLE

-

MELLE differences and

p

-values, from a paired bootstrap for the objective metrics and a paired Wilcoxon test for MOS and SMOS. MOS and SMOS were collected only on LS+6TTS.

Sampling	Flux loss	Continuation				Cross-Sentence
Sampling	Flux loss	WER-C	WER-H	SIM-r	SIM-o	WER-C	WER-H	SIM-r	SIM-o
✗	✓	4.30	5.00	0.382	0.344	12.04	13.07	0.392	0.351
✓	✗	2.50	3.25	0.388	0.347	5.61	6.37	0.417	0.372
✓	✓	2.04	2.66	0.398	0.362	4.12	4.84	0.433	0.395

Appendix

Table 9: Ablation of the Sampling Module and the flux loss on BELLE LS-100+2TTS . ✓ indicates enabled, and ✗ indicates disabled.

Sampling Module and loss	Continuation WER-H $\downarrow$	Cross-Sentence WER-H $\downarrow$
None	5.00	13.07
Gaussian, KL toward unit variance (MELLE)	3.24	6.83
Gaussian, NLL with free variance	3.88	7.31
NIG, evidential loss (BELLE)	2.66	4.84

Appendix

Table 10: Sampling Modules and their losses on LS-100+2TTS .

Explore similar work

Paradee: Distilling Kokoro-82M into an 8M-Parameter Single-Voice Text-to-Speech Model
Oct 5, 2026Sahil MahendrakarTTS Synthesis
Speaking Longer Than Learned: Training-Free Long-Form Generation for Full-Stream Text-to-Speech
Oct 14, 2026Jeseung Lim, Jiseong Woo, Heeseung KimTTS Synthesis Streaming TTS Synthesis
Department of Physics, University of Seoul, Seoul, Republic of Korea · Department of AI, University of Seoul, Seoul, Republic of Korea
BanglaKontho: Closing the Long-Form Gap in Bangla Text-to-Speech
Sep 24, 2026Mizbaul Haque MarufTTS Synthesis Low-Resource TTS Synthesis
Vivasoft Limited, Dhaka, Bangladesh

Oct 5, 2026J/K move · Enter open · S save

Paradee: Distilling Kokoro-82M into an 8M-Parameter Single-Voice Text-to-Speech Model

Sahil Mahendrakar

TTS Synthesis

Open paper AskPDF arXiv

We distill Kokoro-82M, a widely used open text-to-speech model with 54 voices, into Paradee, an 8.07M-parameter model that speaks one of them. Paradee keeps Kokoro's architecture with much narrower layers, and each of its two halves is trained separately against the frozen teacher. It has 10x fewer parameters and needs 15x less compute. We first synthesize a corpus with the teacher and keep its durations, pitch, energy and phoneme features. We then train a small text side to predict these values, and a small decoder to turn the teacher's saved values into the teacher's audio, first with spectral losses and then adversarially. Finally, we connect the two halves and quantize the weights to int8. It needs no alignment learning and no joint training, and it runs on one laptop. Stored in int8, Paradee is 8.5 MB, runs 25x faster than real time on one CPU thread, and scores 4.41 on UTMOS against the teacher's 4.52. The student initially kept a slight buzz, which we trace to the phase of voiced speech between 2 and 8 kHz. A phase-locking filter applied after synthesis removes most of it, with no training and no extra parameters. Code, model files and audio samples are at https://github.com/sahilmahendrakar/paradee

Abstract

Explore similar work

Paradee: Distilling Kokoro-82M into an 8M-Parameter Single-Voice Text-to-Speech Model

Speaking Longer Than Learned: Training-Free Long-Form Generation for Full-Stream Text-to-Speech

BanglaKontho: Closing the Long-Form Gap in Bangla Text-to-Speech