Voice design from natural language descriptions is emerging as a new task in text-to-speech multimodal generation, aiming to synthesize speech with target timbre and speaking style without relying on reference audio. However, existing methods mainly focus on single-utterance generation, leaving conversational voice design largely unexplored. In this work, we extend voice design to dialogue, enabling better target speaker modeling and turn-level expressive control in natural conversational settings. We propose CapTalk, a unified caption-conditioned text-audio autoregressive framework for both single-utterance and dialogue voice design. CapTalk uses utterance-level captions for single-utterance voice design and speaker-level captions for dialogue speaker modeling, and further introduces a CoT control sequence in dialogue to explicitly plan turn-level dynamic attributes. To resolve the conflict between stable timbre preservation and context-adaptive expression, we propose a hierarchical variational conditioning module with an utterance-level speaker encoder to better balance stable timbre preservation and context-adaptive expression. This enables timbre reuse while keeping expression adaptive to the current utterance and, in dialogue, the surrounding context. We also build a comprehensive evaluation protocol for both single-utterance and dialogue settings. Experiments show that CapTalk achieves state-of-the-art performance on a single-utterance voice design benchmark and delivers better expression controllability and contextual appropriateness in multi-turn dialogue. Audio samples are available at: https://anonymous.4open.science/api/repo/Captalk-D601/file/index.html.
Figures & tables
Figure 1. Overview of CapTalk. (a) Hierarchical variational timbre conditioning. The bottom part shows the unified caption-conditioned text-audio autoregressive framework for both single-utterance and dialogue voice design. Overview of CapTalk with hierarchical variational timbre conditioning and a unified caption-conditioned text-audio autoregressive generation framework.
Figure 2. Overview of data construction and evaluation. (A) Qwen3-Omni-based caption annotation for public and internal speech data. (B) Single-utterance and dialogue data construction with dialogue CoT extraction. (C) Evaluation for single-utterance and dialogue voice design. data_processing
Model
APS
DSD
RP
AVG
Qwen3TTS-12Hz-1.7B-VD
87.10
76.00
55.20
72.77
Ming-omni-tts-0.5B
84.90
72.20
53.90
70.33
VoiceSculptor
73.77
65.40
47.60
62.26
Fish Speech S2 Pro
29.61
50.80
42.60
41.00
CapTalk-1.5B (Ours)
84.10
75.40
61.70
73.73
Table 1. Results on the InstructTTSEval-ZH Benchmark.
Model
Overall
Identity
Timbre
Express.
Role
Stabil.
MOS
Ming-omni-tts-0.5B
3.95
4.22
4.03
4.06
3.88
4.27
3.91
Qwen3TTS-12Hz-1.7B-VD
4.20
3.78
4.15
4.12
3.85
4.35
3.82
VoiceSculptor
3.00
3.15
2.89
2.74
2.81
3.25
2.87
Fish Speech S2 Pro
2.07
2.19
2.02
1.82
1.86
2.29
2.11
CapTalk (Ours)
4.24
3.98
3.59
4.10
4.17
4.38
4.20
Table 2. Human evaluation results for single-utterance voice design. All metrics are rated on a 1–5 scale. MOS denotes human-rated naturalness. Best results are in bold .
Evaluation Category
Metric
Emotion
Tone
Pitch
Energy
Speed
CoT Prediction
Accuracy
0.7850
0.7675
0.8375
0.8250
0.9125
CoT Controllability
Success Rate
0.7675
0.7675
0.8400
0.8550
0.8675
Table 3. Automatic evaluation results for dialogue voice design.
Model Setting
Gemini
Human Pairwise Preference
Dialogue Eval. (w/ CoT)
72%
65.5%
Dialogue Eval. (w/o CoT)
28%
34.5%
Table 4. Context coherence comparison in dialogue evaluation.
Data Scale
Type
APS
DSD
RP
AVG
300h
Acted
82.78
71.41
49.10
67.76
1500h
Conversat.
70.20
57.80
41.20
56.40
3000h
Conversat.
74.00
62.40
46.80
61.07
5000h
Conversat.
79.10
68.30
55.10
67.50
5000h+300h
Conversat.+Acted
84.10
75.40
61.70
73.73
Table 5. Scaling law results for single-utterance voice design on InstructTTSEval-ZH.
Data Scale
Evaluation Category
Emotion
Tone
Pitch
Energy
Speed
17,258 sess., 3373 h
CoT Prediction
0.7325
0.6950
0.8050
0.7875
0.8825
24,198 sess., 4482.35 h
CoT Prediction
0.7550
0.7225
0.8250
0.8025
0.9000
32,171 sess., 6270.68 h
CoT Prediction
0.7850
0.7675
0.8375
0.8250
0.9125
17,258 sess., 3373 h
CoT Controllability
0.7025
0.6900
0.7800
0.8125
0.8275
24,198 sess., 4482.35 h
CoT Controllability
0.7375
0.7300
0.8125
0.8350
0.8475
32,171 sess., 6270.68 h
CoT Controllability
0.7675
0.7675
0.8400
0.8550
0.8675
Table 6. Scaling law results for dialogue voice design under CoT prediction and CoT controllability evaluation.
Model
SIM ↑
Context Coherence ↑
MOS ↑
Fish S2 Pro
0.806
3.61
3.80
CapTalk
0.808
4.18
4.12
Table 7. Cross-system dialogue comparison between CapTalk and Fish Speech S2 Pro. SIM measures cross-turn target-speaker timbre consistency in dialogue; Context Coherence and MOS are human ratings (1–5).
Type
APS
DSD
RP
AVG
Baseline (w/o caption_loss)
79.10
68.30
55.10
67.50
w/ caption_loss
77.20
69.10
54.40
66.90
Table 8. Ablation study on explicit caption_loss supervision in the single-utterance setting.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Evaluation Category
Purpose
Method
Reported Results
CoT Prediction Evaluation
Evaluate whether the predicted CoT for the current utterance is reasonable
Gemini-assisted evaluation and human evaluation are used to judge whether the predicted CoT is consistent with the dialogue context
Accuracy
CoT Controllability Evaluation
Evaluate the effectiveness of CoT-based control
Controlled generation is performed by fixing the text, context, and random seed while changing only one CoT attribute at a time
Success Rate
Overall Dialogue Evaluation
Evaluate whether introducing CoT brings benefits to dialogue generation
Gemini-assisted evaluation and human pairwise evaluation are used to compare dialogue generation with and without CoT
Context Coherence Preference
Appendix
Table 9. Summary of the dialogue evaluation protocol.
Setting
Cross-utterance SIM ↑
Fixed espk (timbre reuse)
0.92
Resampled espk (per-utterance)
0.42
Appendix
Table 10. Timbre reuse analysis. SIM is the average pairwise cosine similarity between WavLM speaker embeddings extracted from generated utterances of the same designed voice.
Style
Single-utterance caption
Dialogue caption
APS
Gender: Female. Age: Teenager. Pitch: Female pitch range, relatively high at the beginning, naturally falling in the middle, and slightly rising at the end of the sentence. Speaking rate: The overall speaking rate tends to be relatively fast, with a compact rhythm and frequent local speed changes. Volume: The overall loudness is moderate, but some segments within the utterance are relatively louder, with a moderate dynamic range. Clarity: Clear articulation, standard pronunciation, and complete final consonants. Fluency: Fluent delivery, with no obvious pauses or filler words. Accent: Standard Mandarin. Timbre texture: Bright timbre with a slight childlike quality, soft and warm. Personality: Lively, cheerful, good at guiding others, and approachable.
Gender: Male. Age: Middle-aged. Pitch: The pitch center tends to be moderately low, with strong F0 stability and solid chest resonance. Speaking rate: The overall speaking rate tends to be medium-slow, with a steady rhythm, though it becomes slightly faster in some narrative segments. Volume: The overall loudness is moderate, with a relatively narrow dynamic range and little loudness variation across segments. Clarity: Clear articulation, accurate placement, and complete final consonants. Fluency: Fluent and coherent speech, with occasional minor discourse fillers. Accent: A noticeable northern accent, with relatively hard pronunciation and flatter intonation. Timbre texture: Slightly rough voice quality with mild graininess, with resonance concentrated in the oral and chest cavities. Personality: Easygoing and natural, with a willingness to narrate and a tendency to share reflections from everyday life.
DSD
This is a young adult male voice with a somewhat dark and slightly husky timbre, as if he has just woken up or is in a low mood. He speaks at an unhurried pace, giving the impression of being immersed in his own memories.
This voice sounds like that of a middle-aged northern man with some life experience. His timbre is steady and slightly husky, and he speaks at an unhurried pace with an easygoing manner. He sounds like someone who enjoys chatting with others and remains curious about new things.
RP
An experienced team leader is reminding the group to prepare mentally for the high-intensity challenge ahead. The tone is solemn and authoritative, emphasizing the importance of focus and psychological readiness.
An experienced elder woman is patiently analyzing a complicated interpersonal relationship. Her speech carries both life experience and practical reminders, and her tone is gentle yet firm, as if she were guiding a younger person through confusion.
Appendix
Table 11. Illustrative comparison between single-utterance and dialogue captions.
Voice design from natural language aims to generate speaker timbres directly from free-form textual descriptions, allowing users to create voices tailored to specific roles, personalities, and emotions. Such controllable voice creation benefits a wide range of downstream applications-including storytelling, game dubbing, role-play agents, and conversational assistants, making it a significant task for modern Text-to-Speech models. However, existing models are largely trained on carefully recorded studio data, which produces speech that is clean and well-articulated, yet lacks the lived-in qualities of real human voices. To address these limitations, we present MOSS-VoiceGenerator, an open-source instruction-driven voice generation model that creates new timbres directly from natural language prompts. Motivated by the hypothesis that exposure to real-world acoustic variation produces more perceptually natural voices, we train on large-scale expressive speech data sourced from cinematic content. Subjective preference studies demonstrate its superiority in overall performance, instruction-following, and naturalness compared to other voice design models.
Instruction-following text-to-speech (TTS) typically leaves voice design implicit, as conventional speaker embeddings encode reference audio rather than text-driven targets. We propose VoTSpeech, which decouples voice design from speech generation via an explicit intermediate voice latent. Built on a shared language model backbone, VoTSpeech first predicts a continuous voice latent from text instructions using a flow-based module. This latent serves as a shared condition for both the language model and the Acoustic DiT, guiding autoregressive sequence modeling and acoustic latent generation, respectively, before an audio VAE decoder converts the acoustic latents into waveforms. To capture speaker identity and complementary acoustic nuances, the voice latent is jointly supervised by speaker embeddings and temporally pooled audio VAE features. With task-specific adaptation, VoTSpeech performs strongly on InstructTTSEval-ZH, demonstrating competitive capabilities against strong open-source models while maintaining high naturalness. These results confirm that explicit voice latent design and its conditioning interface effectively empower text-driven voice design for controllable speech synthesis. Code, model, and audio samples are available at https://github.com/ywbn/VoTSpeech .
Wenbing Yang, Qihang Lu, Zihan Sun +4
Beijing University of Posts and Telecommunications, Beijing, China · Hello Group, Beijing, China
Dynamic speaking style adaptation in multi-turn multimodal interaction remains a major challenge for text-to-speech (TTS) systems. Existing context-aware TTS (CTTS) methods typically map dialogue context to speech in an end-to-end manner. Such implicit modeling makes contextual style decisions difficult to supervise, while the entanglement of style, timbre, and content often leads to weak instruction-following and severe timbre drift across turns. To overcome these limitations, we propose Interactive TTS, a dynamic, style-adaptive framework for contextually appropriate and speaker-consistent speech generation. Interactive TTS decouples the process by explicitly modeling contextual style decisions as executable instructions. To bridge the gap between style decisions and speech generation, we introduce Iterative Rejection Sampling Fine-Tuning (Iterative RSFT) and Context-Aware Direct Preference Optimization (CADPO), which significantly enhance instruction-following and align the generated speech with conversational contexts. Extensive experiments demonstrate that Interactive TTS outperforms state-of-the-art models on VStyle and SpeechParaling-Bench. Demo is available at https://wjtian-wonderful.github.io/InteractiveTTS/