Instruction-following text-to-speech (TTS) typically leaves voice design implicit, as conventional speaker embeddings encode reference audio rather than text-driven targets. We propose VoTSpeech, which decouples voice design from speech generation via an explicit intermediate voice latent. Built on a shared language model backbone, VoTSpeech first predicts a continuous voice latent from text instructions using a flow-based module. This latent serves as a shared condition for both the language model and the Acoustic DiT, guiding autoregressive sequence modeling and acoustic latent generation, respectively, before an audio VAE decoder converts the acoustic latents into waveforms. To capture speaker identity and complementary acoustic nuances, the voice latent is jointly supervised by speaker embeddings and temporally pooled audio VAE features. With task-specific adaptation, VoTSpeech performs strongly on InstructTTSEval-ZH, demonstrating competitive capabilities against strong open-source models while maintaining high naturalness. These results confirm that explicit voice latent design and its conditioning interface effectively empower text-driven voice design for controllable speech synthesis. Code, model, and audio samples are available at https://github.com/ywbn/VoTSpeech .
Figures & tables
Figure 1: Overview of VoTSpeech. Voice DiT predicts a continuous voice latent from an instruction-derived LLM state. The resulting latent conditions both the LLM and Acoustic DiT through dual pathways and is trained against a speech-derived target combining speaker embeddings and pooled audio VAE features.
Objective Evaluation
Subjective Evaluation
Model
APS
DSD
RP
CER ↓
NMOS
EMOS
AMOS
Ming-Omni-TTS-0.5B
84.9
72.2
53.9
2.48
3.71±0.27
3.71±0.24
3.75±0.25
Qwen3-TTS-12Hz-1.7B-VD
87.1
76.0
55.2
2.99
4.08±0.21
4.10±0.19
4.06±0.20
MOSS-VoiceGenerator
73.1
70.2
58.9
5.03
3.86±0.22
3.72±0.18
3.70±0.19
VoxCPM2
84.7
71.8
56.8
2.58
3.80±0.25
3.82±0.18
3.82±0.17
dots.tts-soar (fine-tuning)
85.3
75.0
64.3
2.71
3.87±0.23
4.12±0.18
3.96±0.19
Table 1: Comparison with existing systems and direct fine-tuning. Bold and underline mark the best and second-best means.
Variant
APS
DSD
RP
VoTSpeech (Full)
84.8
78.5
66.9
w/o LLM-side Voice Conditioning
83.8
76.7
63.9
w/o VAE Target Features
85.1
77.7
63.6
Table 2: Ablation of dual-path voice conditioning and target design. Bold marks the best result in each column.
Instruction-based text-to-speech (ITTS) systems enable natural-language control of expressive speech generation, but often offer limited transparency and fine-grained control over individual text units. Character-level controllable TTS systems provide explicit acoustic control, yet typically rely on user-specified acoustic attributes. To bridge this gap, we propose InstCharVoice, a unified framework that grounds natural-language instructions in character-level acoustic control. We first construct grounded instruction annotations on the WordVoice-5A-zh corpus using Qwen3-Omni. With this supervision, we train an autoregressive model to identify instruction-relevant characters and predict their acoustic attributes before generating the corresponding speech tokens. Keyword prediction and grounding-aware loss weighting help the model focus on instruction-relevant characters and attributes. Experiments show improved instruction following and keyword-level acoustic control over representative ITTS systems, with competitive speech naturalness and explicit character-level controllability. Audio samples are available at https://xxh333.github.io/instcharvoice-demo/.
Sihang Nie, Xueru Li, Xiaofen Xing +4
South China University of Technology · Huya Inc. · The Hongkong Polytechnic University
Voice design from natural language aims to generate speaker timbres directly from free-form textual descriptions, allowing users to create voices tailored to specific roles, personalities, and emotions. Such controllable voice creation benefits a wide range of downstream applications-including storytelling, game dubbing, role-play agents, and conversational assistants, making it a significant task for modern Text-to-Speech models. However, existing models are largely trained on carefully recorded studio data, which produces speech that is clean and well-articulated, yet lacks the lived-in qualities of real human voices. To address these limitations, we present MOSS-VoiceGenerator, an open-source instruction-driven voice generation model that creates new timbres directly from natural language prompts. Motivated by the hypothesis that exposure to real-world acoustic variation produces more perceptually natural voices, we train on large-scale expressive speech data sourced from cinematic content. Subjective preference studies demonstrate its superiority in overall performance, instruction-following, and naturalness compared to other voice design models.
Recent full-stream text-to-speech (TTS) models enable low-latency personalized synthesis as input text arrives. However, sustaining generation beyond the lengths seen during training without additional training remains largely unexplored. We systematically analyze VoXtream2, which extends generation by periodically discarding past context while retaining the acoustic prompt. We find that this strategy does not prevent degradation: pronunciation accuracy deteriorates near the end of the training range due to the range of positions in which the model operates rather than the loss of past context, while speaker similarity gradually declines throughout generation. Based on these findings, we propose two training-free modifications that reduce the attention window and strengthen attention to the acoustic prompt. Experiments show that these modifications preserve pronunciation accuracy and speaker similarity throughout long-form generation while retaining the naturalness of the original model. Ablations further show complementary effects of the two modifications.
Jeseung Lim, Jiseong Woo, Heeseung Kim
Department of Physics, University of Seoul, Seoul, Republic of Korea · Department of AI, University of Seoul, Seoul, Republic of Korea