Organizations: The Hong Kong University of Science and Technology, Hong Kong SAR, China · JIUTIAN Research, China Mobile, Beijing, China · The State Key Laboratory of Multimedia Information Processing, Peking University, Beijing, China · Nanjing University · China Mobile (Hong Kong) Innovation Research Institute, Hong Kong SAR, China
Recently, text-to-speech systems have made significant progress in speech expressiveness and controllability. However, the speaking style of generated speech typically relies on clear user-specified instructions. In natural conversations, speaking style should be naturally inferred from the preceding conversational context. Therefore, we propose COT-TTS, a context-aware, reasoning-based text-to-speech task. Given historical conversation audio, target text, and a reference speech, the system should comprehend the conversational context, infer an explicit intermediate reasoning, and finally synthesize the target speech with the specified timbre. To support this task, we constructed a large-scale bilingual conversational speech dataset comprising 9 million training samples, including a high-quality subset of 1 million samples. We further constructed a source-disjoint benchmark with 800 human-verified samples and established strong task-specific baselines. Additionally, we developed end-to-end autoregressive models with parameter sizes of 0.6B and 1.7B, generating emotion-labeled transcripts, editable speech style inferences, and speech tokens. Experimental results show that the proposed model achieves performance comparable to large-scale baseline systems with significantly fewer parameters. At the same time, the model performs well in terms of duration consistency and emotional consistency, and can generate appropriate emotional, stress, and rhythmic variations based on the conversational context. To facilitate future research, we will publicly release the data construction pipeline, dataset, trained models, and related resources. The demo page and additional resources are available at https://luckybian.github.io/COT-TTS
Figures & tables
Fig. 1: Overview of the COT-TTS data construction framework. The figure illustrates data collection and preprocessing, multi-dimensional annotation, and the resulting structured sample format for context-aware reasoning TTS.
Fig. 2: Distributions of the normalized audio quality score, naturalness score, target-audio effective-speech ratio, and emotional expression intensity over 50K randomly sampled examples.
Fig. 3: Overview of the COT-TTS data filtering and enhancement pipeline, including deduplication, reference selection, quality screening, ASR retranscription, LLM-based filtering, and human evaluation.
Subset
Language
Hours
Sample Ratio
Samples
Stage-2
English
∼ 21K
51.73%
∼ 4.64M
Chinese
∼ 29K
48.27%
∼ 4.33M
Total
∼ 50K
100%
∼ 9.0M
Stage-3
English
∼ 2.5K
56%
∼ 0.56M
Chinese
∼ 3.0K
44%
∼ 0.44M
Total
∼ 5.5K
100%
∼ 1.0M
TABLE I: Statistics of the filtered COT-TTS data subsets.
Fig. 4: Overview of the end-to-end CoT-guided autoregressive model. Historical dialogue audio, target text, and reference speech are encoded into a unified token sequence. The model sequentially generates an emotion-tagged historical transcript, multi-dimensional CoT reasoning, and target speech tokens. The intermediate CoT can be inspected and edited before speech synthesis. For waveform reconstruction, the predicted semantic tokens are combined with either the reference or predicted global tokens.
Fig. 5: Overview of the three-stage training strategy for our end-to-end autoregressive models for COT-TTS. The figure summarizes the training data, model inputs, and outputs at each stage.
Stage
Data
Ratio
Zh:En
Epochs
1
ASR (70K h), TTS (70K h)
1:1
1:1
1—1
2
COT-TTS (9M) + Auxiliary tasks
7:3
48:52
2—3
3
COT-TTS (1M) + TTS (1M)
1:1
44:56
1—1
TABLE II: Training configurations of the three stages. Epochs are listed for the 0.6B/1.7B models.
Fig. 6: Overview of the three cascaded baseline architectures. (a) The three-stage ASR/diarization–LLM–TTS pipeline first converts historical dialogue audio into text, then infers the speaking manner with an LLM, and finally synthesizes the target speech using a controllable TTS model. (b) The two-stage AudioLLM–TTS pipeline directly analyzes the historical audio with an AudioLLM and uses the inferred speaking manner to control downstream TTS. (c) The two-stage AudioLLM–VC pipeline directly generates an intermediate target speech with an AudioLLM and subsequently applies voice conversion to condition the output on the reference speaker.
System
Model Size
Speech Quality and Acoustic Consistency
Context-Aware Reasoning Quality
Human Evaluation
UTMOSv2 ↑
DNSMOSPro ↑
WER ↓
Emo. Cons. ↑
Dur. Err. ↓
F0 Corr. ↑
Hist. Und. ↑
CoT Logic ↑
Eff. Info. Richness ↑
LLM Composite ↑
Human MOS ↑
three-dia-a3b-fish
36.2B
3.109
3.287
0.012
0.910
5.866
0.227
4.075
4.587
3.574
2.915
3.050
three-dia-a3b-voxcpm
34.2B
2.714
3.240
1.307
0.952
12.495
0.117
4.053
4.601
4.202
3.601
3.450
three-qwen_asr-a3b-fish
36.2B
3.102
3.275
0.092
0.917
5.267
0.217
3.816
4.759
2.941
2.258
3.500
three-qwen_asr-a3b-voxcpm
34.2B
2.663
3.223
0.165
0.944
5.852
0.133
3.820
4.753
4.141
3.510
3.300
two-a3b-fish
39.0B
3.138
3.269
0.090
0.920
5.112
0.213
3.689
4.836
2.644
1.969
3.050
TABLE III: Main results on the English COT-TTS benchmark. “ ↑ ” indicates that higher is better, and “ ↓ ” indicates that lower is better.
System
Model Size
Speech Quality and Acoustic Consistency
Context-Aware Reasoning Quality
Human Evaluation
UTMOSv2 ↑
DNSMOSPro ↑
CER ↓
Emo. Cons. ↑
Dur. Err. ↓
F0 Corr. ↑
Hist. Und. ↑
CoT Logic ↑
Eff. Info. Richness ↑
LLM Composite ↑
Human MOS ↑
three-dia-a3b-fish
36.2B
2.872
3.303
0.010
0.917
5.746
0.205
3.958
4.585
3.096
2.402
3.150
three-dia-a3b-voxcpm
34.2B
2.391
3.203
0.023
0.946
5.599
0.125
3.934
4.608
4.064
3.415
3.950
three-qwen_asr-a3b-fish
36.2B
2.874
3.279
0.055
0.921
5.286
0.188
3.708
4.789
2.932
2.234
3.700
three-qwen_asr-a3b-voxcpm
34.2B
2.372
3.155
0.056
0.943
5.248
0.140
3.747
4.798
3.942
3.282
3.200
two-a3b-fish
39.0B
2.880
3.283
0.052
0.921
5.144
0.218
3.531
4.812
2.545
1.847
3.150
TABLE IV: Main results on the Chinese COT-TTS benchmark. “ ↑ ” indicates that higher is better, and “ ↓ ” indicates that lower is better.
Training Setting
UTMOSv2
DNSMOSPro
WER/CER
Emo. Cons.
Dur. Err.
F0 Corr.
Human MOS
↑
↑
↓
↑
↓
↑
↑
Each cell is reported as EN / ZH. For the ASR error column, this corresponds to English WER / Chinese CER.
Stage 1 only
2.539 / 2.492
2.734 / 3.064
0.549 / 0.467
0.907 / 0.928
6.323 / 6.163
0.086 / 0.105
2.150 / 2.250
Stage 1 + Stage 2
2.619 / 2.544
2.900 / 3.222
0.089 / 0.229
0.948 / 0.949
5.770 / 5.888
0.160 / 0.158
3.150 / 3.050
Stage 1 + direct COT-TTS refinement
2.609 / 2.388
2.807 / 3.121
0.196 / 0.272
0.938 / 0.933
6.607 / 6.392
0.157 / 0.135
3.050 / 2.950
Three-stage w/o auxiliary tasks
2.630 / 2.542
2.928 / 3.161
0.163 / 0.285
0.946 / 0.945
7.174 / 7.332
0.138 / 0.120
3.650 / 3.750
TABLE V: Ablation results for the three-stage training strategy using the 0.6B model under non-edit inference. Each cell reports English / Chinese (EN / ZH) results. “ ↑ ” indicates that higher is better, and “ ↓ ” indicates that lower is better.
Control
Default
Edited Parameters
Duration (s)
5.54
4.5
6.5
7.5
8.5
Rhythm (s)
4.23/5.54
3.2/4.1
3.7/4.8
4.9/6.4
5.6/7.5
Intensity
0.77
0.57
0.67
0.87
0.97
TABLE VI: Editable-CoT settings for controlled speech synthesis. Only the specified CoT fields are modified.
Recent advances in text-to-speech (TTS) have greatly improved speech naturalness, speaker similarity, and controllability. However, most existing controllable TTS systems still rely on explicit user-provided style prompts, making it difficult to automatically determine how a sentence should be spoken in long and complex conversational scenarios. This proposal introduces the ISCSLP 2026 CoT-TTS Challenge, which aims to evaluate whether a system can infer the intended speaking manner from contextual information and generate speech consistent with both the reasoning output and the surrounding scene. The challenge contains two tracks: text-context-aware CoT-TTS and audio-context-aware CoT-TTS. We construct a large-scale bilingual training set from speech-rich media and provide carefully filtered evaluation data for leaderboard comparison. Each system is required to output both a chain-of-thought reasoning analysis and the generated speech waveform. The official evaluation combines objective metrics, multimodal LLM-based evaluation, and human subjective assessment. To facilitate reproducibility, we provide inference code together with a fine-tuning recipe for a 0.6B Qwen3-based model trained via a three-stage strategy. This challenge is expected to support research on context understanding, chain-of-thought reasoning, and expressive speech generation for applications such as film dubbing, audiobook production, virtual characters, and spoken dialogue agents. Further information about the associated challenge is available at:https://iscslp2026-cot-tts.github.io/challenge-website/
Wei Xue, Junlan Feng, Shilei Zhang +9
The Hong Kong University of Science and Technology · China Mobile · Jiutian Artificial Intelligence Technology (Beijing) Co., Ltd., China Mobile +3
Dynamic speaking style adaptation in multi-turn multimodal interaction remains a major challenge for text-to-speech (TTS) systems. Existing context-aware TTS (CTTS) methods typically map dialogue context to speech in an end-to-end manner. Such implicit modeling makes contextual style decisions difficult to supervise, while the entanglement of style, timbre, and content often leads to weak instruction-following and severe timbre drift across turns. To overcome these limitations, we propose Interactive TTS, a dynamic, style-adaptive framework for contextually appropriate and speaker-consistent speech generation. Interactive TTS decouples the process by explicitly modeling contextual style decisions as executable instructions. To bridge the gap between style decisions and speech generation, we introduce Iterative Rejection Sampling Fine-Tuning (Iterative RSFT) and Context-Aware Direct Preference Optimization (CADPO), which significantly enhance instruction-following and align the generated speech with conversational contexts. Extensive experiments demonstrate that Interactive TTS outperforms state-of-the-art models on VStyle and SpeechParaling-Bench. Demo is available at https://wjtian-wonderful.github.io/InteractiveTTS/
Recent zero-shot Text-to-Speech (TTS) systems can clone unseen speakers, yet controllable expressiveness remains limited by a gap between semantic intent and acoustic realization. Text descriptions offer a natural control interface but provide weak acoustic grounding, whereas speech style prompts provide concrete prosody but entangle style with the prompt speaker's timbre. To bridge this gap, we propose E3-System, a unified intent-to-speech framework for expressive zero-shot TTS that couples three requirements: Expressive style modeling, Explicit timbre-style control, and Easy-to-use natural-language interaction. Given a target speaker and either a natural-language style description or a speech style prompt, E3-System generates expressive speech while preserving the target timbre. The key idea is to close the full control loop: building a large expressive style space, grounding user intent into usable acoustic style references, and synthesizing speech with disentangled timbre and style. To realize this loop, EmoData provides a thousand-hour multilingual emotional speech dataset, Style-RAG grounds text descriptions into high-quality acoustic style prompts from these data, and Dual-TTS transfers the grounded style to arbitrary target speakers through VC-augmented timbre-style disentanglement. Experiments show that E3-System achieves strong speech quality, speaker preservation, and explicit expressive controllability.
Yiqiao Huang, Xiaolong Lin, Zhenpeng Zhan +1
Harvard University, Cambridge, MA, USA · Baidu Inc., Shenzhen, China · The Chinese University of Hong Kong, Shenzhen, China