Voice design from natural language aims to generate speaker timbres directly from free-form textual descriptions, allowing users to create voices tailored to specific roles, personalities, and emotions. Such controllable voice creation benefits a wide range of downstream applications-including storytelling, game dubbing, role-play agents, and conversational assistants, making it a significant task for modern Text-to-Speech models. However, existing models are largely trained on carefully recorded studio data, which produces speech that is clean and well-articulated, yet lacks the lived-in qualities of real human voices. To address these limitations, we present MOSS-VoiceGenerator, an open-source instruction-driven voice generation model that creates new timbres directly from natural language prompts. Motivated by the hypothesis that exposure to real-world acoustic variation produces more perceptually natural voices, we train on large-scale expressive speech data sourced from cinematic content. Subjective preference studies demonstrate its superiority in overall performance, instruction-following, and naturalness compared to other voice design models.
Figures & tables
Figure 1: Illustration of the MOSS-VoiceGenerator inference. The voice description and text are concatenated and fed into a causal language model with delay-pattern generation; the output audio tokens are decoded by MOSS-Audio-Tokenizer.
Figure 2: Data collection pipeline for MOSS-VoiceGenerator. Phase 1 annotates cinematic audio via speaker diarization, denoising and quality filtering, single-speaker filtering, and ASR transcription, followed by speech captioning and timbre instruction generation. Phase 2 augments the corpus by training a speech-text embedding model for retrieval from internal TTS data and a fine-tuned caption model for scalable annotation.
Figure 3: A snapshot of the training corpus profiled along three perceptual dimensions based on the caption results. The distributions reveal broad, naturalistic coverage of everyday speaking styles.
Model
InstructTTSEval-EN
InstructTTSEval-ZH
APS
DSD
RP
APS
DSD
RP
Gemini-TTS-Pro
87.6
86.0
67.2
89.0
90.1
75.5
GPT-4o-mini-TTS
76.4
74.3
54.8
54.9
52.3
46.0
VoxInstruct
54.9
57.0
39.3
47.5
52.3
42.6
VoiceSculptor-VD
-
-
-
75.7
64.7
61.5
MIMO-Audio-7B-Instruct
80.6
77.6
59.5
75.7
74.3
61.5
Table 1: Instruction-following accuracy (%) on InstructTTSEval.
Figure 4: Pairwise preference results (Win / Tie / Lose) of MOSS-VoiceGenerator against three baselines across three evaluation dimensions. Each bar reports the percentage of comparisons won, tied, or lost. MOSS-VoiceGenerator consistently wins on all three dimensions against all baselines.
Instruction-following text-to-speech (TTS) typically leaves voice design implicit, as conventional speaker embeddings encode reference audio rather than text-driven targets. We propose VoTSpeech, which decouples voice design from speech generation via an explicit intermediate voice latent. Built on a shared language model backbone, VoTSpeech first predicts a continuous voice latent from text instructions using a flow-based module. This latent serves as a shared condition for both the language model and the Acoustic DiT, guiding autoregressive sequence modeling and acoustic latent generation, respectively, before an audio VAE decoder converts the acoustic latents into waveforms. To capture speaker identity and complementary acoustic nuances, the voice latent is jointly supervised by speaker embeddings and temporally pooled audio VAE features. With task-specific adaptation, VoTSpeech performs strongly on InstructTTSEval-ZH, demonstrating competitive capabilities against strong open-source models while maintaining high naturalness. These results confirm that explicit voice latent design and its conditioning interface effectively empower text-driven voice design for controllable speech synthesis. Code, model, and audio samples are available at https://github.com/ywbn/VoTSpeech .
Wenbing Yang, Qihang Lu, Zihan Sun +4
Beijing University of Posts and Telecommunications, Beijing, China · Hello Group, Beijing, China
Voice design from natural language descriptions is emerging as a new task in text-to-speech multimodal generation, aiming to synthesize speech with target timbre and speaking style without relying on reference audio. However, existing methods mainly focus on single-utterance generation, leaving conversational voice design largely unexplored. In this work, we extend voice design to dialogue, enabling better target speaker modeling and turn-level expressive control in natural conversational settings. We propose CapTalk, a unified caption-conditioned text-audio autoregressive framework for both single-utterance and dialogue voice design. CapTalk uses utterance-level captions for single-utterance voice design and speaker-level captions for dialogue speaker modeling, and further introduces a CoT control sequence in dialogue to explicitly plan turn-level dynamic attributes. To resolve the conflict between stable timbre preservation and context-adaptive expression, we propose a hierarchical variational conditioning module with an utterance-level speaker encoder to better balance stable timbre preservation and context-adaptive expression. This enables timbre reuse while keeping expression adaptive to the current utterance and, in dialogue, the surrounding context. We also build a comprehensive evaluation protocol for both single-utterance and dialogue settings. Experiments show that CapTalk achieves state-of-the-art performance on a single-utterance voice design benchmark and delivers better expression controllability and contextual appropriateness in multi-turn dialogue. Audio samples are available at: https://anonymous.4open.science/api/repo/Captalk-D601/file/index.html.
Xiaosu Su, Zihan Sun, Peilei Jia +1
University of Chinese Academy of Sciences Beijing, China · Hello Group Inc. Beijing, China
Recent breakthroughs in generative models have made text-to-voice generation (TTV) possible, enabling the synthesis of speech directly from textual voice descriptions. However, existing systems face two key challenges. First, they struggle to generate a diverse range of voices, spanning real-world human speakers and fictional characters. Second, they lack robust and flexible voice editing capabilities, such as voice cloning and the ability to modify attributes like emotion and tone. In this paper, we propose VoiceDesigner, a unified framework for text-to-voice generation and editing that supports diverse and controllable voice design. To tackle the above challenges, we propose solutions from two perspectives. First, we develop a hybrid data pipeline that leverages digital signal processing techniques and speech generation models to construct a diverse voice dataset covering both real-world and fictional voices. Second, we introduce a diffusion transformer with architectural improvements to better handle complex conditioning and enhance multi-task performance, enabling unified voice generation and editing. Through subjective and objective evaluations, VoiceDesigner achieves superior prompt alignment with both voice descriptions and editing instructions, while maintaining competitive perceptual quality and voice usability compared to state-of-the-art TTV models.