CapTalk: Unified Voice Design for Single-Utterance and Dialogue Speech Generation

Xiaosu Su; Zihan Sun; Peilei Jia; Jun Gao

cs.SDApr 9, 2026

CapTalk: Unified Voice Design for Single-Utterance and Dialogue Speech Generation

Authors: Xiaosu Su, Zihan Sun, Peilei Jia, Jun Gao

Organizations: University of Chinese Academy of Sciences Beijing, China · Hello Group Inc. Beijing, China

Abstract

Voice design from natural language descriptions is emerging as a new task in text-to-speech multimodal generation, aiming to synthesize speech with target timbre and speaking style without relying on reference audio. However, existing methods mainly focus on single-utterance generation, leaving conversational voice design largely unexplored. In this work, we extend voice design to dialogue, enabling better target speaker modeling and turn-level expressive control in natural conversational settings. We propose CapTalk, a unified caption-conditioned text-audio autoregressive framework for both single-utterance and dialogue voice design. CapTalk uses utterance-level captions for single-utterance voice design and speaker-level captions for dialogue speaker modeling, and further introduces a CoT control sequence in dialogue to explicitly plan turn-level dynamic attributes. To resolve the conflict between stable timbre preservation and context-adaptive expression, we propose a hierarchical variational conditioning module with an utterance-level speaker encoder to better balance stable timbre preservation and context-adaptive expression. This enables timbre reuse while keeping expression adaptive to the current utterance and, in dialogue, the surrounding context. We also build a comprehensive evaluation protocol for both single-utterance and dialogue settings. Experiments show that CapTalk achieves state-of-the-art performance on a single-utterance voice design benchmark and delivers better expression controllability and contextual appropriateness in multi-turn dialogue. Audio samples are available at: https://anonymous.4open.science/api/repo/Captalk-D601/file/index.html.

Figures & tables

Figure 1. Overview of CapTalk. (a) Hierarchical variational timbre conditioning. The bottom part shows the unified caption-conditioned text-audio autoregressive framework for both single-utterance and dialogue voice design. Overview of CapTalk with hierarchical variational timbre conditioning and a unified caption-conditioned text-audio autoregressive generation framework.

Figure 2. Overview of data construction and evaluation. (A) Qwen3-Omni-based caption annotation for public and internal speech data. (B) Single-utterance and dialogue data construction with dialogue CoT extraction. (C) Evaluation for single-utterance and dialogue voice design. data_processing

Model	APS	DSD	RP	AVG
Qwen3TTS-12Hz-1.7B-VD	87.10	76.00	55.20	72.77
Ming-omni-tts-0.5B	84.90	72.20	53.90	70.33
VoiceSculptor	73.77	65.40	47.60	62.26
Fish Speech S2 Pro	29.61	50.80	42.60	41.00
CapTalk-1.5B (Ours)	84.10	75.40	61.70	73.73

Table 1. Results on the InstructTTSEval-ZH Benchmark.

Model	Overall	Identity	Timbre	Express.	Role	Stabil.	MOS
Ming-omni-tts-0.5B	3.95	4.22	4.03	4.06	3.88	4.27	3.91
Qwen3TTS-12Hz-1.7B-VD	4.20	3.78	4.15	4.12	3.85	4.35	3.82
VoiceSculptor	3.00	3.15	2.89	2.74	2.81	3.25	2.87
Fish Speech S2 Pro	2.07	2.19	2.02	1.82	1.86	2.29	2.11
CapTalk (Ours)	4.24	3.98	3.59	4.10	4.17	4.38	4.20

Table 2. Human evaluation results for single-utterance voice design. All metrics are rated on a 1–5 scale. MOS denotes human-rated naturalness. Best results are in bold .

Evaluation Category	Metric	Emotion	Tone	Pitch	Energy	Speed
CoT Prediction	Accuracy	0.7850	0.7675	0.8375	0.8250	0.9125
CoT Controllability	Success Rate	0.7675	0.7675	0.8400	0.8550	0.8675

Table 3. Automatic evaluation results for dialogue voice design.

Model Setting	Gemini	Human Pairwise Preference
Dialogue Eval. (w/ CoT)	72%	65.5%
Dialogue Eval. (w/o CoT)	28%	34.5%

Table 4. Context coherence comparison in dialogue evaluation.

Data Scale	Type	APS	DSD	RP	AVG
300h	Acted	82.78	71.41	49.10	67.76
1500h	Conversat.	70.20	57.80	41.20	56.40
3000h	Conversat.	74.00	62.40	46.80	61.07
5000h	Conversat.	79.10	68.30	55.10	67.50
5000h+300h	Conversat.+Acted	84.10	75.40	61.70	73.73

Table 5. Scaling law results for single-utterance voice design on InstructTTSEval-ZH.

Data Scale	Evaluation Category	Emotion	Tone	Pitch	Energy	Speed
17,258 sess., 3373 h	CoT Prediction	0.7325	0.6950	0.8050	0.7875	0.8825
24,198 sess., 4482.35 h	CoT Prediction	0.7550	0.7225	0.8250	0.8025	0.9000
32,171 sess., 6270.68 h	CoT Prediction	0.7850	0.7675	0.8375	0.8250	0.9125
17,258 sess., 3373 h	CoT Controllability	0.7025	0.6900	0.7800	0.8125	0.8275
24,198 sess., 4482.35 h	CoT Controllability	0.7375	0.7300	0.8125	0.8350	0.8475
32,171 sess., 6270.68 h	CoT Controllability	0.7675	0.7675	0.8400	0.8550	0.8675

Table 6. Scaling law results for dialogue voice design under CoT prediction and CoT controllability evaluation.

Model	SIM $\uparrow$	Context Coherence $\uparrow$	MOS $\uparrow$
Fish S2 Pro	0.806	3.61	3.80
CapTalk	0.808	4.18	4.12

Table 7. Cross-system dialogue comparison between CapTalk and Fish Speech S2 Pro. SIM measures cross-turn target-speaker timbre consistency in dialogue; Context Coherence and MOS are human ratings (1–5).

Type	APS	DSD	RP	AVG
Baseline (w/o caption_loss)	79.10	68.30	55.10	67.50
w/ caption_loss	77.20	69.10	54.40	66.90

Table 8. Ablation study on explicit caption_loss supervision in the single-utterance setting.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Evaluation Category	Purpose	Method	Reported Results
CoT Prediction Evaluation	Evaluate whether the predicted CoT for the current utterance is reasonable	Gemini-assisted evaluation and human evaluation are used to judge whether the predicted CoT is consistent with the dialogue context	Accuracy
CoT Controllability Evaluation	Evaluate the effectiveness of CoT-based control	Controlled generation is performed by fixing the text, context, and random seed while changing only one CoT attribute at a time	Success Rate
Overall Dialogue Evaluation	Evaluate whether introducing CoT brings benefits to dialogue generation	Gemini-assisted evaluation and human pairwise evaluation are used to compare dialogue generation with and without CoT	Context Coherence Preference

Appendix

Table 9. Summary of the dialogue evaluation protocol.

Setting	Cross-utterance SIM $\uparrow$
Fixed $e_{\text{spk}}$ (timbre reuse)	0.92
Resampled $e_{\text{spk}}$ (per-utterance)	0.42

Appendix

Table 10. Timbre reuse analysis. SIM is the average pairwise cosine similarity between WavLM speaker embeddings extracted from generated utterances of the same designed voice.

Style	Single-utterance caption	Dialogue caption
APS	Gender: Female. Age: Teenager. Pitch: Female pitch range, relatively high at the beginning, naturally falling in the middle, and slightly rising at the end of the sentence. Speaking rate: The overall speaking rate tends to be relatively fast, with a compact rhythm and frequent local speed changes. Volume: The overall loudness is moderate, but some segments within the utterance are relatively louder, with a moderate dynamic range. Clarity: Clear articulation, standard pronunciation, and complete final consonants. Fluency: Fluent delivery, with no obvious pauses or filler words. Accent: Standard Mandarin. Timbre texture: Bright timbre with a slight childlike quality, soft and warm. Personality: Lively, cheerful, good at guiding others, and approachable.	Gender: Male. Age: Middle-aged. Pitch: The pitch center tends to be moderately low, with strong F0 stability and solid chest resonance. Speaking rate: The overall speaking rate tends to be medium-slow, with a steady rhythm, though it becomes slightly faster in some narrative segments. Volume: The overall loudness is moderate, with a relatively narrow dynamic range and little loudness variation across segments. Clarity: Clear articulation, accurate placement, and complete final consonants. Fluency: Fluent and coherent speech, with occasional minor discourse fillers. Accent: A noticeable northern accent, with relatively hard pronunciation and flatter intonation. Timbre texture: Slightly rough voice quality with mild graininess, with resonance concentrated in the oral and chest cavities. Personality: Easygoing and natural, with a willingness to narrate and a tendency to share reflections from everyday life.
DSD	This is a young adult male voice with a somewhat dark and slightly husky timbre, as if he has just woken up or is in a low mood. He speaks at an unhurried pace, giving the impression of being immersed in his own memories.	This voice sounds like that of a middle-aged northern man with some life experience. His timbre is steady and slightly husky, and he speaks at an unhurried pace with an easygoing manner. He sounds like someone who enjoys chatting with others and remains curious about new things.
RP	An experienced team leader is reminding the group to prepare mentally for the high-intensity challenge ahead. The tone is solemn and authoritative, emphasizing the importance of focus and psychological readiness.	An experienced elder woman is patiently analyzing a complicated interpersonal relationship. Her speech carries both life experience and practical reminders, and her tone is gentle yet firm, as if she were guiding a younger person through confusion.

Appendix

Table 11. Illustrative comparison between single-utterance and dialogue captions.

TTS Synthesis

Text-to-speech synthesis converts written language into intelligible, natural-sounding speech with control over voice, prosody, style, and timing.

See all 267 papers

MOSS-VoiceGenerator: Create Realistic Voices with Natural Language Descriptions
Mar 30, 2026Kexin Huang, Liwei Fan, Botian Jiang +11Controllable Speech Generation TTS Synthesis
VoTSpeech: Decoupling Voice Design from Speech Generation in Instruction-Following TTS
Oct 10, 2026Wenbing Yang, Qihang Lu, Zihan Sun +4TTS Synthesis Controllable Speech Generation
Beijing University of Posts and Telecommunications, Beijing, China · Hello Group, Beijing, China
Interactive TTS: Dynamic Speaking Style Adaptation for Expressive Speech Synthesis
Sep 22, 2026Wenjie Tian, Kangxiang Xia, Jingbin Hu +7TTS Synthesis Direct Preference Optimization
Alibaba ATH Token Foundry

Mar 30, 2026J/K move · Enter open · S save

MOSS-VoiceGenerator: Create Realistic Voices with Natural Language Descriptions

Kexin Huang, Liwei Fan, Botian Jiang, Yaozhou Jiang+10

Controllable Speech Generation TTS Synthesis Emotional Speech Synthesis

Open paper AskPDF arXiv

Voice design from natural language aims to generate speaker timbres directly from free-form textual descriptions, allowing users to create voices tailored to specific roles, personalities, and emotions. Such controllable voice creation benefits a wide range of downstream applications-including storytelling, game dubbing, role-play agents, and conversational assistants, making it a significant task for modern Text-to-Speech models. However, existing models are largely trained on carefully recorded studio data, which produces speech that is clean and well-articulated, yet lacks the lived-in qualities of real human voices. To address these limitations, we present MOSS-VoiceGenerator, an open-source instruction-driven voice generation model that creates new timbres directly from natural language prompts. Motivated by the hypothesis that exposure to real-world acoustic variation produces more perceptually natural voices, we train on large-scale expressive speech data sourced from cinematic content. Subjective preference studies demonstrate its superiority in overall performance, instruction-following, and naturalness compared to other voice design models.

Abstract

TTS Synthesis

Explore similar work

MOSS-VoiceGenerator: Create Realistic Voices with Natural Language Descriptions

VoTSpeech: Decoupling Voice Design from Speech Generation in Instruction-Following TTS

Interactive TTS: Dynamic Speaking Style Adaptation for Expressive Speech Synthesis