VoiceDesigner: Text-to-Voice Generation and Editing via Unified Diffusion Modeling and Data Augmentation

Jiarui Hai; Karan Thakkar; Ke Chen; Yunyun Wang; Jiaqi Su; Rithesh Kumar; Mounya Elhilali; Zeyu Jin

eess.ASAug 12, 2026

VoiceDesigner: Text-to-Voice Generation and Editing via Unified Diffusion Modeling and Data Augmentation

Authors: Jiarui Hai, Karan Thakkar, Ke Chen, Yunyun Wang, Jiaqi Su, Rithesh Kumar, Mounya Elhilali, Zeyu Jin

Organizations: Johns Hopkins University · Adobe Research

Abstract

Recent breakthroughs in generative models have made text-to-voice generation (TTV) possible, enabling the synthesis of speech directly from textual voice descriptions. However, existing systems face two key challenges. First, they struggle to generate a diverse range of voices, spanning real-world human speakers and fictional characters. Second, they lack robust and flexible voice editing capabilities, such as voice cloning and the ability to modify attributes like emotion and tone. In this paper, we propose VoiceDesigner, a unified framework for text-to-voice generation and editing that supports diverse and controllable voice design. To tackle the above challenges, we propose solutions from two perspectives. First, we develop a hybrid data pipeline that leverages digital signal processing techniques and speech generation models to construct a diverse voice dataset covering both real-world and fictional voices. Second, we introduce a diffusion transformer with architectural improvements to better handle complex conditioning and enhance multi-task performance, enabling unified voice generation and editing. Through subjective and objective evaluations, VoiceDesigner achieves superior prompt alignment with both voice descriptions and editing instructions, while maintaining competitive perceptual quality and voice usability compared to state-of-the-art TTV models.

Figures & tables

Model	Voice Generation Scope			Editing Capability
Model	Speaker Traits	Style Traits	Character Design	Voice Cloning	Voice Editing
PromptTTS [ 14 , 5 ]	✓	▲	✗	✗	✗
ParlerTTS [ 15 ]	✓	▲	✗	✗	✗
VoxInstruct [ 16 ]	✓	✓	✗	✗	✗
CapSpeech [ 7 ]	✓	✓	✗	✗	✗
Qwen3TTS [ 17 ]	✓	✓	✓	▲	✗

TABLE I: TTV capabilities across models ( ▲ : limited support).

Fig. 1: Generation modes of VoiceDesigner.

Fig. 2: An overview of framework design of VoiceDesigner.

Fig. 3: An overview of data simulation pipeline.

Training Stage	Batch Size	Training Steps	Learning Rate
Pretraining	$512\times 20$ s	400k	$1.5\times 10^{-4}$
Task Adaptation	$384\times 30$ s	100k	$8\times 10^{-5}$
Quality Refinement	$128\times 30$ s	10k	$1\times 10^{-5}$

TABLE II: Training configurations for different training stages.

Model	Objective Evaluation		Subjective: TTV-Traits			Subjective: TTV-Character
Model	WER (%) $\downarrow$	Style-ACC (%) $\uparrow$	MOS-C $\uparrow$	MOS-U $\uparrow$	MOS-Q $\uparrow$	MOS-C $\uparrow$	MOS-U $\uparrow$	MOS-Q $\uparrow$
ElevenLabs-TTV API ∗ [ 26 ]	1.68	0.58	$\mathbf{4.00}\pm 0.06$	$\mathbf{4.10}\pm 0.05$	$3.97\pm 0.05$	$\mathbf{4.26}\pm 0.05$	$\mathbf{4.24}\pm 0.05$	$\mathbf{4.23}\pm 0.05$
CapSpeech-NAR [ 7 ]	2.12	0.57	$3.26\pm 0.07$	$3.34\pm 0.06$	$3.96\pm 0.05$	–	–	–
Qwen3TTS-VoiceDesign [ 17 ]	1.39	0.50	$3.10\pm 0.08$	$3.57\pm 0.06$	$\mathbf{4.20}\pm 0.05$	$3.59\pm 0.07$	$\underline{4.03}\pm 0.05$	$\underline{4.19}\pm 0.05$
VoiceDesigner	1.22	0.66	$3.93\pm 0.06$	$3.57\pm 0.06$	$3.96\pm 0.05$	$3.71\pm 0.06$	$3.72\pm 0.06$	$3.60\pm 0.06$
VoiceDesigner (+Enhance)	–	–	$\underline{3.94}\pm 0.06$	$\underline{3.64}\pm 0.06$	$\underline{4.17}\pm 0.05$	$\underline{3.76}\pm 0.06$	$3.93\pm 0.05$	$3.92\pm 0.05$

TABLE III: Voice generation performance across objective evaluations and two subjective listening tests. ∗ indicates the commercial closed-source systems accessed via API. All MOS results are reported with 95% confidence intervals.

Model	WER (%) $\downarrow$	SIM-o $\uparrow$
Ground Truth	2.06	0.734
SeedTTS ${}^{*}_{\text{DiT}}$ [ 64 ]	1.73	0.790
E2-TTS [ 1 , 3 ]	2.19	0.710
F5-TTS [ 1 ]	1.83	0.670
CosyVoice-3-1.5B ∗ [ 10 ]	2.21	0.720
CosyVoice-3-0.5B [ 10 ]	2.02	0.718

TABLE IV: Voice cloning performance on the Seed-TTS test-en benchmark, where ∗ denotes closed-source models; results are from official technical reports.

Model	MOS-T $\uparrow$	MOS-S $\uparrow$
CosyVoice-3-0.5B [ 10 ]	$2.48\pm 0.08$	$2.63\pm 0.08$
IndexTTS-2 [ 2 ]	$\underline{3.85}\pm 0.07$	$\underline{3.94}\pm 0.06$
VoiceDesigner	$\mathbf{3.93}\pm 0.07$	$\mathbf{4.00}\pm 0.06$

TABLE V: Subjective evaluation for challenging voice cloning.

Model	MOS-E $\uparrow$	SIM-t $\uparrow$
Ground Truth	$4.147\pm 0.056$	–
IndexTTS-2 [ 2 ]	$\underline{3.494}\pm 0.072$	0.856
Step-Audio-EditX [ 13 ]	$3.333\pm 0.064$	0.734
VoiceDesigner	$\textbf{4.129}\pm 0.054$	0.884

TABLE VI: Evaluation results for voice editing performance.

Fig. 4: Ablation study on the model architecture.

Fig. 5: Ablation study on generative data augmentation.

TTS Synthesis

Text-to-speech synthesis converts written language into intelligible, natural-sounding speech with control over voice, prosody, style, and timing.

See all 267 papers

MOSS-VoiceGenerator: Create Realistic Voices with Natural Language Descriptions
Mar 30, 2026Kexin Huang, Liwei Fan, Botian Jiang +11Controllable Speech Generation TTS Synthesis
CookVoice: Unified Framework for Style Controllable Multi-Modal Human Voice Generation
Aug 12, 2026Haowei Lou, Hye-Young Paik, Dai Jia +2Singing Voice Synthesis TTS Synthesis
UNSW Sydney · Dolby Laboratories
VoTSpeech: Decoupling Voice Design from Speech Generation in Instruction-Following TTS
Oct 10, 2026Wenbing Yang, Qihang Lu, Zihan Sun +4TTS Synthesis Controllable Speech Generation
Beijing University of Posts and Telecommunications, Beijing, China · Hello Group, Beijing, China

Mar 30, 2026J/K move · Enter open · S save

MOSS-VoiceGenerator: Create Realistic Voices with Natural Language Descriptions

Kexin Huang, Liwei Fan, Botian Jiang, Yaozhou Jiang+10

Controllable Speech Generation TTS Synthesis Emotional Speech Synthesis

Open paper AskPDF arXiv

Voice design from natural language aims to generate speaker timbres directly from free-form textual descriptions, allowing users to create voices tailored to specific roles, personalities, and emotions. Such controllable voice creation benefits a wide range of downstream applications-including storytelling, game dubbing, role-play agents, and conversational assistants, making it a significant task for modern Text-to-Speech models. However, existing models are largely trained on carefully recorded studio data, which produces speech that is clean and well-articulated, yet lacks the lived-in qualities of real human voices. To address these limitations, we present MOSS-VoiceGenerator, an open-source instruction-driven voice generation model that creates new timbres directly from natural language prompts. Motivated by the hypothesis that exposure to real-world acoustic variation produces more perceptually natural voices, we train on large-scale expressive speech data sourced from cinematic content. Subjective preference studies demonstrate its superiority in overall performance, instruction-following, and naturalness compared to other voice design models.

Abstract

TTS Synthesis

Explore similar work

MOSS-VoiceGenerator: Create Realistic Voices with Natural Language Descriptions

CookVoice: Unified Framework for Style Controllable Multi-Modal Human Voice Generation

VoTSpeech: Decoupling Voice Design from Speech Generation in Instruction-Following TTS