cs.CLNov 14, 2026

Beyond Controllability: Speaker Diversity in Instruction-Guided Text-to-Speech

Authors: Yi-Cheng Lin, Jia-Kai Dong, Kuan-Yu Chen, Szu-Chi Chen, Jian-Jiun Ding, Sung-Feng Huang, Yusuke Hirota, Hung-yi Lee

Organizations: National Taiwan University · NVIDIA · Artificial Intelligence Center of Research Excellence, National Taiwan University

Abstract

Instruction-guided text-to-speech (ITTS) lets users specify voice properties in natural language. Applications from conversational AI character voices to synthetic training data for downstream speech models depend on the system producing many distinct voices when the same broad category is requested repeatedly. No prior work audits whether ITTS outputs are in fact diverse, either compared to real speech or across attribute values. We evaluate seven open-source ITTS systems using speaker embeddings whose representational capacity we independently validate. Every system produces speaker distributions markedly narrower than real speech. Furthermore, female prompts produce less diverse voice pools than male prompts across most systems. In contrast, accent and emotion prompts barely alter speaker diversity in most systems, but reveal a strong controllability bias toward North American English and neutral speech. These results show that diversity and controllability capture complementary weaknesses, neither of which alone is sufficient to evaluate ITTS. Our code and data are available at https://github.com/dlion168/ITTS-Speaker-Diversity.

Figures & tables

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. InstCharVoice: Grounding Natural-Language Instructions for Character-Level Control in Text-to-Speech

    Sep 28, 2026Sihang Nie, Xueru Li, Xiaofen Xing +4TTS SynthesisLanguage Model-Based Control

  2. The Binding Effect: Analyzing How Multi-Dimensional Cues Form Gender Bias in Instruction TTS

    Mar 21, 2026Kuan-Yu Chen, Yi-Cheng Lin, Po-Chung Hsieh +5Text-to-SpeechSocial Bias in Language Models

  3. VoTSpeech: Decoupling Voice Design from Speech Generation in Instruction-Following TTS

    Oct 10, 2026Wenbing Yang, Qihang Lu, Zihan Sun +4TTS SynthesisControllable Speech Generation