Instruction-guided text-to-speech (ITTS) lets users specify voice properties in natural language. Applications from conversational AI character voices to synthetic training data for downstream speech models depend on the system producing many distinct voices when the same broad category is requested repeatedly. No prior work audits whether ITTS outputs are in fact diverse, either compared to real speech or across attribute values. We evaluate seven open-source ITTS systems using speaker embeddings whose representational capacity we independently validate. Every system produces speaker distributions markedly narrower than real speech. Furthermore, female prompts produce less diverse voice pools than male prompts across most systems. In contrast, accent and emotion prompts barely alter speaker diversity in most systems, but reveal a strong controllability bias toward North American English and neutral speech. These results show that diversity and controllability capture complementary weaknesses, neither of which alone is sufficient to evaluate ITTS. Our code and data are available at https://github.com/dlion168/ITTS-Speaker-Diversity.
Figures & tables
Figure 1: Overview. (a) A broad description should produce many distinct voices. (b) We keep only encoders whose diversity scores track speaker count (RQ1). (c) ITTS voices are less diverse than real speech (RQ2), and diversity and controllability vary across voice attributes (RQ3).
Figure 2: Encoder calibration with MPCD and Vendi. Mean MPCD and Vendi over 100 random subsets at each ground-truth speaker count k , for all 11 candidate encoders. Solid lines denote the eight encoders meeting calibration criteria and retained for evaluation. Dashed lines indicate the three excluded candidates.
Figure 3: Overall speaker diversity under the neutral prompt. ITTS systems (blue) are compared against the Real reference (orange). Error bars are batch-to-batch standard deviations across the three generation batches.
Figure 4: Speaker diversity under different sampling parameters. Vendi for the three systems that expose decoding controls, under (a) sampling temperature T for MOSS-VoiceGenerator and (b) classifier-free guidance scale γ for VoxInstruct and VoxCPM2. Both panels share the same y-axis. Each point is the mean over the eight validated encoders and three seeds. The dashed horizontal line marks the real-speech baseline.
Figure 5: Controllability versus diversity across gender and age conditions. Each color denotes an ITTS model, while marker shapes identify specific gender and age values. The dashed vertical line marks chance-level recall of 0.25 for age classification. Error bars show 95% confidence intervals, which are smaller than the markers.
Figure 6: Controllability versus diversity across accent and emotion conditions. Dashed vertical lines mark chance-level recall baselines for each attribute. Error bars show 95% confidence intervals, which are smaller than the markers.
Age
Gender
MOSS-VG
Parler-large
Parler-mini
CapSpeech
Qwen3-TTS
VoxInstruct
VoxCPM2
Child
Female
27.8
36.0
32.0
14.2
6.5
13.3
31.5
Male
43.3
35.8
36.1
17.1
25.2
21.2
38.4
Young adult
Female
36.3
36.3
32.1
12.4
9.5
12.6
28.0
Male
46.5
36.6
35.9
13.3
19.9
17.8
19.9
Middle-aged
Female
40.7
35.8
32.4
19.2
24.6
12.9
23.5
Male
47.3
35.0
35.1
21.8
27.3
18.5
21.1
Table 1: Speaker diversity under joint gender and age conditions. Vendi scores for joint gender and age prompts, averaged across eight encoders. Rows are organized by age group to facilitate female and male comparisons within each bracket. Bold numbers highlight the lowest diversity score in each column.
Single attribute
Intersectional
System
Gender
Age
Gender
Age
MOSS-VG
female
child
female
child
CapSpeech
female
young adult
female
young adult
Qwen3-TTS
female
child
female
child
VoxInstruct
female
child
female
young adult †
VoxCPM2
male
middle-aged
male
young adult
Table 2: Gender and age group with the lowest Vendi in each system. The three row groups are systems where the single-attribute and gender × age results agree, systems where they differ, and systems without a clear lowest age group. Bold marks where the gender × age result differs from the single-attribute result. † Overlaps with the second lowest in confidence interval.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Encoder
MPCD
VS
chamfer
MMD
TIT
0.97
1.00
0.96
0.99
SBRN
0.96
1.00
0.93
0.99
SBEC
0.94
1.00
0.95
0.99
ER2
0.79
0.99
0.95
0.99
WS221
0.96
1.00
0.95
0.99
CAM
0.74
0.99
0.95
0.98
Appendix
Table 3: Spearman rank calibration ρ across all four metrics for each evaluated encoder. Higher values indicate stronger monotonic agreement between metric scores and ground-truth speaker counts k . The top partition contains the eight validated encoders adopted for our main analysis. The lower partition contains XV, WLM, and USAT. Chamfer and MMD achieve high correlations of at least 0.91 across all architectures, offering limited sensitivity for encoder selection.
Figure 7: Encoder validation with Chamfer distance and MMD. Mean Chamfer distance and MMD between 100 random subsets and the full reference pool at each ground-truth speaker count k , for all 11 candidate encoders. Solid lines mark the eight encoders retained for evaluation, and dashed lines mark the three excluded encoders.
Attribute
Classifier
Classes
Gender
Vox-Profile WavLM sex head
2
Age
Vox-Profile WavLM age regressor, binned at 20/40/60
Table 4: Classifiers for measuring controllability. The classifier and number of classes used for each attribute.
Region
Source(s)
Train clips (speakers)
Hours
North America
CV17
40,000 (4,350)
58.6
British English
CV17
20,000 (1,410)
29.0
South Asia
CV17 + L2-ARCTIC Hindi
16,000 (1,221)
24.8
Oceania
CV17
14,000 (548)
20.6
South African
AfriSpeech-200 ZA + CV17
8,495 (313)
15.4
Germanic
CV17 + VoxPopuli en_de / en_nl
1,799 (109)
3.5
Appendix
Table 5: Accent classifier training data. CV17 denotes CommonVoice-17. VoxPopuli en_de / en_nl adds 75 Germanic speakers (999 clips) from the European Parliament channel. AfriSpeech-200 ZA adds 119 South-African speakers (2,793 clips). L2-ARCTIC Hindi adds 76 clips (2 speakers) to South Asia.
Region
WavLM
Whisper
Ensemble
North America
0.868
0.880
0.881
British English
0.802
0.771
0.825
South Asia
0.905
0.920
0.918
Germanic
0.842
0.906
0.887
Oceania
0.840
0.793
0.873
South African
0.832
0.869
0.871
Appendix
Table 6: Per-class F1 on the speaker-independent held-out test split. The ensemble (production model used for controllability measurement in § 3.5 ) achieves the best macro-F1; on per-class F1 Whisper edges the ensemble on South Asia and Germanic by small margins.
Figure 8: Emotion confusion matrices for the seven ITTS systems. Rows give the prompted emotion and columns the emotion predicted by the classifier, with each row normalized over its 3,000 clips. Outlined diagonal cells equal the per-class recall plotted in Figure 6 (b).
#
Male (gender)
Child (age)
1
Speak in a male voice.
Speak in a child’s voice.
2
Speak with the voice of a man.
Speak with the voice of a young child.
3
Speak so you sound masculine.
Speak so you sound like a young child.
4
Speak as a male speaker.
Speak as a child around eight years old.
5
Speak with a voice that has masculine qualities.
Speak with a voice that has childlike qualities.
Appendix
Table 7: Single-attribute prompt templates , illustrated for the male-gender and child-age values. Gender, age, accent, and emotion each use the same five frames with the attribute keyword swapped.
#
Word order
Template (male × child)
1
age-first
Speak as a young boy.
gender-first
Speak as a boy who is young.
2
age-first
Use the voice of a child who is male.
gender-first
Use the voice of a male who is a child.
3
age-first
Sound like a child who is male.
gender-first
Sound like a male who is a child.
Appendix
Table 8: Prompt templates combining gender and age attributes , illustrated for the male child category. All gender and age combinations use ten templates derived from five sentence frames tested under both age-first and gender-first word orders.
System
Release
Params
License
MOSS-VoiceGenerator
OpenMOSS-Team/MOSS-VoiceGenerator
1.7B
Apache-2.0
Parler-TTS Mini v1
parler-tts/parler-tts-mini-v1
0.9B
Apache-2.0
Parler-TTS Large v1
parler-tts/parler-tts-large-v1
2.3B
Apache-2.0
Qwen3-TTS Voice Design
Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign
1.7B
Apache-2.0
VoxInstruct
thuhcsi/VoxInstruct
1.37B
MIT
CapSpeech-NAR
OpenSound/CapSpeech-models
0.9B
CC BY-NC 4.0
Appendix
Table 9: ITTS systems in the panel. Parameter counts exclude separately-loaded vocoder and codec modules. CapSpeech-NAR is the only non-commercial release; all generations and derived statistics for CapSpeech-NAR inherit the CC BY-NC 4.0 restriction.
Encoder (code)
Toolkit / Release
License
TitaNet (TIT)
nvidia/speakerverification_en_titanet_large
CC-BY-4.0
SpeechBrain-ECAPA (SBEC)
speechbrain/spkrec-ecapa-voxceleb
Apache-2.0
SpeechBrain-ResNet (SBRN)
speechbrain/spkrec-resnet-voxceleb
Apache-2.0
WeSpeaker-ResNet221 (WS221)
WeSpeaker voxceleb_resnet221_LM
CC-BY-4.0
WeSpeaker-SimAM-ResNet34 (WSVB)
WeSpeaker voxblink2_samresnet34
CC-BY-NC-SA-4.0
ERes2Net (ER2)
iic/speech_eres2net_sv_zh-cn_16k-common
Apache-2.0
Appendix
Table 10: Speaker encoders used in the validation study (§ 3.2 ).
Classifier
Release
License
Gender (binary sex head)
tiantiaf/wavlm-large-age-sex
Apache-2.0
Age (regressor)
tiantiaf/wavlm-large-age-sex
Apache-2.0
Emotion
emotion2vec/emotion2vec_plus_large
Apache-2.0 †
Accent (in-house)
Trained for this paper
Research-only (ours; see below)
Frozen backbones used in the in-house accent classifier
WavLM-Large
microsoft/wavlm-large
MIT
Appendix
Table 11: Controllability classifiers. † : the emotion2vec_plus_large model card lists Apache-2.0; the companion _seed and _base releases use a FunASR-linked “model license” instead, so re-users should re-verify from the Hugging Face page.
Dataset
Use in this paper
License
Emilia
Real-speech reference sampling (§ 4 , § 5 )
CC BY-NC 4.0
LibriSpeech
Encoder validation
CC BY 4.0
CommonVoice-17
Accent classifier training (all 6 regions)
CC0 1.0
VoxPopuli en_de / en_nl
Accent classifier training (Germanic class)
CC0
AfriSpeech-200 ZA
Accent classifier training (South African class)
CC BY-NC-SA 4.0
L2-ARCTIC Hindi
Accent classifier training (South Asia augmentation)
CC BY-NC 4.0
Appendix
Table 12: Datasets referenced in the paper and their licenses. Emilia, AfriSpeech-200, and L2-ARCTIC are non-commercial; derived artifacts that embed these corpora inherit their restrictions.