Current Text-to-Speech (TTS) systems typically use separate models for speech-prompted and text-prompted timbre control. While unifying both control signals into a single model is desirable, the challenge of cross-modal alignment often results in overly complex architectures and training objective. To address this challenge, we propose CAST-TTS, a simple yet effective framework for unified timbre control. Features are extracted from speech prompts and text prompts using pre-trained encoders. The multi-stage training strategy efficiently aligns the speech and projected text representations within a shared embedding space. A single cross-attention mechanism then allows the model to use either of these representations to control the timbre. Extensive experiments validate that the unified cross-attention mechanism is critical for achieving high-quality synthesis. CAST-TTS achieves performance comparable to specialized single-input models while operating within a unified architecture. The demo page can be accessed at https://HiRookie9.github.io/CAST-TTS-Page.
Figures & tables
Figure 1: An overview of CAST-TTS. The timbre encoder converts speech or text prompts into timbre embeddings, which condition the synthesis model through a cross-attention mechanism.
Objective Evaluations
Subjective Evaluations
Prompt
Model
WER(%)↓
SPK-Sim↑
Style-ACC(%)↑
UTMOS↑
N-MOS↑
Sim-MOS↑
Speech
F5-TTS-v1
2.31
75.4
-
3.87
4.13
4.17
MaskGCT
3.54
74.5
-
3.90
3.73
3.93
ZipVoice-L
1.77
66.7
-
4.26
3.90
4.01
CAST-TTS
2.05
78.4
-
3.91
3.86
4.09
Text
CapSpeech-NAR
5.11
-
88.93
4.06
4.08
4.05
Table 1: Objective and subjective evaluation results. The arrows (↓↑) indicate whether lower or higher scores are better.
Speech Prompt
Text Prompt
Model
Feature
WER(%)↓
SPK-Sim↑
UTMOS↑
WER(%)↓
Style-ACC(%)↑
UTMOS↑
CAST-SA
Melspec
3.74
35.8
3.97
4.17
81.25
3.78
ECAPA-TDNN
5.48
43.0
4.00
6.56
90.00
3.91
CAST-SACA
Melspec
2.67
35.5
3.95
4.52
90.10
3.89
ECAPA-TDNN
3.18
41.2
4.06
4.93
90.26
3.90
CAST-CA
ECAPA-TDNN
3.13
69.5
3.82
4.47
89.01
3.91
Table 2: Ablation study on model architectures. We evaluate the impact of different methods under both speech and text prompted conditions.
Recent Text-To-Speech (TTS) systems have achieved strong naturalness and zero-shot voice cloning performance, but fine-grained control of expressive speech at the word or phoneme level remains challenging. We propose CtrlSpeech, a controllable, expressive TTS framework with coarse-to-fine control. Built on the DiTAR architecture, CtrlSpeech combines global speaker conditioning with phone-aligned pitch, loudness, and duration signals, enabling localized prosodic control while preserving the target speaker's timbre. This design allows users to adjust expressive attributes at a fine temporal granularity, making speech refinement more flexible and controllable. Experimental results show that CtrlSpeech achieves competitive zero-shot TTS performance and improves controllability over expressive attributes, demonstrating its effectiveness for flexible and practical expressive speech synthesis.
While prompt-based text-to-speech (TTS) models enable natural language-driven speaking style control, they often provide limited fine-grained control and apply a single global style across an utterance. This restricts practical use cases that require continuous style attribute interpolation across utterances and time-varying style transitions within a single utterance. In this paper, we propose novel techniques to achieve both capabilities in existing prompt-based TTS models. For inter-utterance style interpolation, we compute direction vectors between contrastive style prompts in the embedding space and perform simple interpolation, enabling smooth transitions between style characteristics. For intra-utterance style transition, we first identify a strong attention bias toward early tokens in autoregressive TTS decoders, causing the initial audio realization to dominate subsequent generation. To mitigate this effect, we introduce KV-cache swapping and sliding-window attention masking. Experiments demonstrate that our proposed inter-utterance interpolation achieves a 99-100% success rate in gender conversion, up to 36 Hz pitch variation, and up to 1.6 syllables-per-second speed change. Our intra-utterance transition maintains a speaker similarity of 0.81-0.91 and achieves perceptual smoothness scores of 3.48-4.48.
Jaehoon Kang, Yejin Lee, Yoonji Park +1
Department of Artificial Intelligence, Sungkyunkwan University, Korea · Department of Computer Science and Engineering, Sungkyunkwan University, Korea
Controllable text-to-speech (TTS) has become a key research focus. However, methods based on either reference speech or text descriptions lack flexibility and precise control, and recent joint approaches remain loosely coupled, with speech modeling timbre and text controlling global style. We propose FineCombo-TTS, a unified framework for speech synthesis grounded in reference speech and guided by text descriptions, enabling flexible and precise control over acoustic attributes. Instead of explicit attribute disentanglement, we learn a unified acoustic representation and introduce a Conditional Flow Matching (CFM)-based Speech Variance Predictor to model fine-grained reference-to-target transformations guided by text descriptions. To support relative attribute control, we construct FineEdit, a structured paired dataset that explicitly encodes source-to-target attribute variations. Experiments demonstrate that our approach achieves flexible, precise, and expressive controllable TTS.
Shuoyi Zhou, Yixuan Zhou, Peiji Yang +4
Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · Tencent, Shenzhen, China · Inner Mongolia University Hohhot, China