Organizations: The Hong Kong University of Science and Technology, Hong Kong SAR, China · JIUTIAN Research, China Mobile, Beijing, China · The State Key Laboratory of Multimedia Information Processing, Peking University, Beijing, China · Nanjing University · China Mobile (Hong Kong) Innovation Research Institute, Hong Kong SAR, China
Recently, text-to-speech systems have made significant progress in speech expressiveness and controllability. However, the speaking style of generated speech typically relies on clear user-specified instructions. In natural conversations, speaking style should be naturally inferred from the preceding conversational context. Therefore, we propose COT-TTS, a context-aware, reasoning-based text-to-speech task. Given historical conversation audio, target text, and a reference speech, the system should comprehend the conversational context, infer an explicit intermediate reasoning, and finally synthesize the target speech with the specified timbre. To support this task, we constructed a large-scale bilingual conversational speech dataset comprising 9 million training samples, including a high-quality subset of 1 million samples. We further constructed a source-disjoint benchmark with 800 human-verified samples and established strong task-specific baselines. Additionally, we developed end-to-end autoregressive models with parameter sizes of 0.6B and 1.7B, generating emotion-labeled transcripts, editable speech style inferences, and speech tokens. Experimental results show that the proposed model achieves performance comparable to large-scale baseline systems with significantly fewer parameters. At the same time, the model performs well in terms of duration consistency and emotional consistency, and can generate appropriate emotional, stress, and rhythmic variations based on the conversational context. To facilitate future research, we will publicly release the data construction pipeline, dataset, trained models, and related resources. The demo page and additional resources are available at https://luckybian.github.io/COT-TTS
Figures & tables
Fig. 1: Overview of the COT-TTS data construction framework. The figure illustrates data collection and preprocessing, multi-dimensional annotation, and the resulting structured sample format for context-aware reasoning TTS.
Fig. 2: Distributions of the normalized audio quality score, naturalness score, target-audio effective-speech ratio, and emotional expression intensity over 50K randomly sampled examples.
Fig. 3: Overview of the COT-TTS data filtering and enhancement pipeline, including deduplication, reference selection, quality screening, ASR retranscription, LLM-based filtering, and human evaluation.
Subset
Language
Hours
Sample Ratio
Samples
Stage-2
English
∼ 21K
51.73%
∼ 4.64M
Chinese
∼ 29K
48.27%
∼ 4.33M
Total
∼ 50K
100%
∼ 9.0M
Stage-3
English
∼ 2.5K
56%
∼ 0.56M
Chinese
∼ 3.0K
44%
∼ 0.44M
Total
∼ 5.5K
100%
∼ 1.0M
TABLE I: Statistics of the filtered COT-TTS data subsets.
Fig. 4: Overview of the end-to-end CoT-guided autoregressive model. Historical dialogue audio, target text, and reference speech are encoded into a unified token sequence. The model sequentially generates an emotion-tagged historical transcript, multi-dimensional CoT reasoning, and target speech tokens. The intermediate CoT can be inspected and edited before speech synthesis. For waveform reconstruction, the predicted semantic tokens are combined with either the reference or predicted global tokens.
Fig. 5: Overview of the three-stage training strategy for our end-to-end autoregressive models for COT-TTS. The figure summarizes the training data, model inputs, and outputs at each stage.
Stage
Data
Ratio
Zh:En
Epochs
1
ASR (70K h), TTS (70K h)
1:1
1:1
1—1
2
COT-TTS (9M) + Auxiliary tasks
7:3
48:52
2—3
3
COT-TTS (1M) + TTS (1M)
1:1
44:56
1—1
TABLE II: Training configurations of the three stages. Epochs are listed for the 0.6B/1.7B models.
Fig. 6: Overview of the three cascaded baseline architectures. (a) The three-stage ASR/diarization–LLM–TTS pipeline first converts historical dialogue audio into text, then infers the speaking manner with an LLM, and finally synthesizes the target speech using a controllable TTS model. (b) The two-stage AudioLLM–TTS pipeline directly analyzes the historical audio with an AudioLLM and uses the inferred speaking manner to control downstream TTS. (c) The two-stage AudioLLM–VC pipeline directly generates an intermediate target speech with an AudioLLM and subsequently applies voice conversion to condition the output on the reference speaker.
System
Model Size
Speech Quality and Acoustic Consistency
Context-Aware Reasoning Quality
Human Evaluation
UTMOSv2 ↑
DNSMOSPro ↑
WER ↓
Emo. Cons. ↑
Dur. Err. ↓
F0 Corr. ↑
Hist. Und. ↑
CoT Logic ↑
Eff. Info. Richness ↑
LLM Composite ↑
Human MOS ↑
three-dia-a3b-fish
36.2B
3.109
3.287
0.012
0.910
5.866
0.227
4.075
4.587
3.574
2.915
3.050
three-dia-a3b-voxcpm
34.2B
2.714
3.240
1.307
0.952
12.495
0.117
4.053
4.601
4.202
3.601
3.450
three-qwen_asr-a3b-fish
36.2B
3.102
3.275
0.092
0.917
5.267
0.217
3.816
4.759
2.941
2.258
3.500
three-qwen_asr-a3b-voxcpm
34.2B
2.663
3.223
0.165
0.944
5.852
0.133
3.820
4.753
4.141
3.510
3.300
two-a3b-fish
39.0B
3.138
3.269
0.090
0.920
5.112
0.213
3.689
4.836
2.644
1.969
3.050
TABLE III: Main results on the English COT-TTS benchmark. “ ↑ ” indicates that higher is better, and “ ↓ ” indicates that lower is better.
System
Model Size
Speech Quality and Acoustic Consistency
Context-Aware Reasoning Quality
Human Evaluation
UTMOSv2 ↑
DNSMOSPro ↑
CER ↓
Emo. Cons. ↑
Dur. Err. ↓
F0 Corr. ↑
Hist. Und. ↑
CoT Logic ↑
Eff. Info. Richness ↑
LLM Composite ↑
Human MOS ↑
three-dia-a3b-fish
36.2B
2.872
3.303
0.010
0.917
5.746
0.205
3.958
4.585
3.096
2.402
3.150
three-dia-a3b-voxcpm
34.2B
2.391
3.203
0.023
0.946
5.599
0.125
3.934
4.608
4.064
3.415
3.950
three-qwen_asr-a3b-fish
36.2B
2.874
3.279
0.055
0.921
5.286
0.188
3.708
4.789
2.932
2.234
3.700
three-qwen_asr-a3b-voxcpm
34.2B
2.372
3.155
0.056
0.943
5.248
0.140
3.747
4.798
3.942
3.282
3.200
two-a3b-fish
39.0B
2.880
3.283
0.052
0.921
5.144
0.218
3.531
4.812
2.545
1.847
3.150
TABLE IV: Main results on the Chinese COT-TTS benchmark. “ ↑ ” indicates that higher is better, and “ ↓ ” indicates that lower is better.
Training Setting
UTMOSv2
DNSMOSPro
WER/CER
Emo. Cons.
Dur. Err.
F0 Corr.
Human MOS
↑
↑
↓
↑
↓
↑
↑
Each cell is reported as EN / ZH. For the ASR error column, this corresponds to English WER / Chinese CER.
Stage 1 only
2.539 / 2.492
2.734 / 3.064
0.549 / 0.467
0.907 / 0.928
6.323 / 6.163
0.086 / 0.105
2.150 / 2.250
Stage 1 + Stage 2
2.619 / 2.544
2.900 / 3.222
0.089 / 0.229
0.948 / 0.949
5.770 / 5.888
0.160 / 0.158
3.150 / 3.050
Stage 1 + direct COT-TTS refinement
2.609 / 2.388
2.807 / 3.121
0.196 / 0.272
0.938 / 0.933
6.607 / 6.392
0.157 / 0.135
3.050 / 2.950
Three-stage w/o auxiliary tasks
2.630 / 2.542
2.928 / 3.161
0.163 / 0.285
0.946 / 0.945
7.174 / 7.332
0.138 / 0.120
3.650 / 3.750
TABLE V: Ablation results for the three-stage training strategy using the 0.6B model under non-edit inference. Each cell reports English / Chinese (EN / ZH) results. “ ↑ ” indicates that higher is better, and “ ↓ ” indicates that lower is better.
Control
Default
Edited Parameters
Duration (s)
5.54
4.5
6.5
7.5
8.5
Rhythm (s)
4.23/5.54
3.2/4.1
3.7/4.8
4.9/6.4
5.6/7.5
Intensity
0.77
0.57
0.67
0.87
0.97
TABLE VI: Editable-CoT settings for controlled speech synthesis. Only the specified CoT fields are modified.