We investigate the use of zero-shot text-to-speech (ZS-TTS) as a data augmentation source for low-resource personalized speech synthesis. While synthetic augmentation can provide linguistically rich and phonetically diverse speech, naively mixing large amounts of synthetic speech with limited real recordings often leads to speaker similarity degradation during fine-tuning. To address this issue, we propose ZeSTA, a simple domain-conditioned training framework that distinguishes real and synthetic speech via a lightweight domain embedding, combined with real-data oversampling to stabilize adaptation under extremely limited target data, without modifying the base architecture. Experiments on LibriTTS and an in-house dataset with two ZS-TTS sources demonstrate that our approach improves speaker similarity over naive synthetic augmentation while preserving intelligibility and perceptual quality. Audio samples are available on our web page.
Figures & tables
Figure 1: ZeSTA framework: training pipeline (a) and inference pipeline (b).
Corpus
#Spk (M/F)
Train
Valid
Test
LibriTTS
8 (4/4)
[198,286]
[20,29]
[20,29]
YoBind
6 (3/3)
[133,375]
[13,37]
[13,37]
Table 1: Summary statistics of the LibriTTS and YoBind datasets. Both datasets are gender-balanced. Ranges denote the minimum and maximum number of utterances per speaker.
Corpus
Fish-Speech
CosyVoice 2
LibriTTS
0.763
0.794
YoBind
0.756
0.788
Table 2: SECS scores of synthetic speech used as Synth 90% .
Source ZS-TTS
Training Data
DC
OS
LibriTTS
YoBind
SECS ↑
CER ↓
WER ↓
SECS ↑
CER ↓
WER ↓
Real-Only Training
Real 10%
–
–
0.818
5.932
12.520
0.813
5.063
11.035
Real 100%
–
–
0.832
6.539
13.645
0.840
4.743
10.424
Fish-Speech [ 26 ]
Real 10% + Synth 90%
–
–
0.765
4.738
10.348
0.764
4.081
9.358
Real 10% + Synth 90%
–
✓
0.770
4.802
10.470
0.770
4.085
9.542
Real 10% + Synth 90%
✓
–
0.807
4.962
10.879
0.792
4.471
9.965
Table 3: Objective evaluation results on LibriTTS and YoBind datasets. (+Extra Synth) indicates extra synthetic augmentation. Arrows indicate whether higher ( ↑ ) or lower ( ↓ ) values are better.
ZS-TTS
Training Data
DC
OS
LibriTTS
YoBind
MOS (Naturalness)
Ground Truth (Recordings)
–
–
4.31 ± 0.28
4.31 ± 0.30
Real-Only
Real 10%
–
–
3.58 ± 0.42
2.85 ± 0.47
Real 100%
–
–
3.67 ± 0.34
3.50 ± 0.40
FS
Real 10% + Synth 90%
–
–
3.86 ± 0.37
3.50 ± 0.35
Real 10% + Synth 90%
✓
✓
3.92 ± 0.27
3.58 ± 0.38
Table 4: Subjective evaluation results on LibriTTS and YoBind datasets. MOS scores (95% CI) measure perceived naturalness. ABX values indicate preference (%) for the proposed system over the baseline ( Real 10% + Synth 90% without DC or OS). All ABX results are statistically significant under a two-sided binomial test ( p<0.05 ).
Figure 2: t-SNE visualization of latent representations for a target speaker under different synthetic augmentation settings.
Emb. Size
SECS ↑
CER ↓
WER ↓
16
0.807
5.118
11.229
64
0.807
4.962
10.879
256
0.800
5.036
11.033
Table 5: Effect of domain embedding size on speaker similarity and intelligibility evaluated on LibriTTS. Synthetic data are generated using Fish-Speech. All models are trained with Real 10% + Synth 90% data using DC without OS.
Training Data
SECS ↑
CER ↓
WER ↓
Real 10% (baseline)
0.818
5.932
12.520
Speaker-matched
0.807
4.962
10.879
Speaker-mismatched
0.792
5.817
12.354
Table 6: Ablation on speaker-matched vs. speaker-mismatched synthetic data. Synthetic augmentation uses the same setup ( Real 10% + Synth 90% ) with DC and without OS.