Independent control of voice and emotional style characteristics of speech is a long-standing problem in text-to-speech (TTS) systems. We investigate how the conditioning location of voice and style prompts across TTS decoder layers affects the trade-off between preservation and expressiveness for voice identity and emotional style. We introduce a layer-dependent gated fusion mechanism and systematically study which layers should be influenced by which prompt type. Our findings show an asymmetric relationship. Voice benefits from early conditioning for identity preservation, while style benefits from early conditioning for expressiveness. Our findings provide practical guidance that helps neural network architects make an appropriate trade-off for the architecture of TTS systems according to their use cases.
Figures & tables
LRT χ2
Variance share
Metric
Model
M × PEM
Unit
Audio
Res.
Structured
Style spread
411.3
98.7
.38
.53
.09
Style drift
216.5
35.1
.72
.24
.04
Voice spread
104.8
104.8
.37
—
.63
Voice drift
366.9
311.6
.34
.25
.41
Templates
Style spread
63.8
102.3
.50
.42
.08
Table 1: Mixed-effects analysis. Likelihood-ratio tests for conditioning strategy (Model, df=3 ) and its interaction with the proxy embedding model (Model × PEM; df=3 for style, df=6 for voice), with variance shares of the random effects. All χ2 significant at p<.001 .
Style
Voice
Group 1
Group 2
Spread
Drift
Spread
Drift
Structured
Equal
Concat
−.153
−.246
−.003
−.017
Equal
Style
−.071
−.114
−.009
−.008
Equal
Voice
−.254
−.217
−.007
−.015
Concat
Style
−.224
−.131
−.006
−.009
Concat
Voice
−.101
−.029
−.004
−.033
Table 2: Pairwise model contrasts from the mixed-effects model (Holm-corrected, FWER =0.05 ), — = insignificant.
Model
EC ∗
TN ∗
WL ∗
D. ↓
S. ↑
D. ↓
S. ↑
D. ↓
S. ↑
Concat
.250 ±.025
.353 ±.006
.525 ±.039
.624 ±.020
.146 ±.016
.264 ±.010
Style
.250 ±.019
.352 ±.006
.505 ±.028
.613 ±.020
.139 ±.018
.257 ±.007
Voice
.227 ±.023
.349 ±.005
.470 ±.046
.621 ±.016
.125 ±.022
.260 ±.005
Equal
.242 ±.022
.365 ±.007
.491 ±.038
.618 ±.020
.136 ±.016
.267 ±.007
∗ EC=ECAPA-TDNN, TN=TitaNet, WL=WavLM
Table 3: Voice Embeddings Pairwise Distance mean ± std. Drift (D.) and Spread (S.)
Model
Audeering
Odyssey
Drift ↓
Spread ↑
Drift ↓
Spread ↑
Concat
2.897 ±.192
3.300 ±.205
2.815 ±.185
3.320 ±.182
Style
3.033 ±.213
3.536 ±.233
2.941 ±.198
3.532 ±.230
Voice
2.941 ±.184
3.239 ±.242
2.828 ±.164
3.179 ±.218
Equal
3.133 ±.176
3.461 ±.239
3.070 ±.148
3.466 ±.225
Table 4: Style Embeddings Pairwise Distance mean ± std
Figure 1: Style spread across conditioning strategies for structured and template prompting.
While prompt-based text-to-speech (TTS) models enable natural language-driven speaking style control, they often provide limited fine-grained control and apply a single global style across an utterance. This restricts practical use cases that require continuous style attribute interpolation across utterances and time-varying style transitions within a single utterance. In this paper, we propose novel techniques to achieve both capabilities in existing prompt-based TTS models. For inter-utterance style interpolation, we compute direction vectors between contrastive style prompts in the embedding space and perform simple interpolation, enabling smooth transitions between style characteristics. For intra-utterance style transition, we first identify a strong attention bias toward early tokens in autoregressive TTS decoders, causing the initial audio realization to dominate subsequent generation. To mitigate this effect, we introduce KV-cache swapping and sliding-window attention masking. Experiments demonstrate that our proposed inter-utterance interpolation achieves a 99-100% success rate in gender conversion, up to 36 Hz pitch variation, and up to 1.6 syllables-per-second speed change. Our intra-utterance transition maintains a speaker similarity of 0.81-0.91 and achieves perceptual smoothness scores of 3.48-4.48.
Jaehoon Kang, Yejin Lee, Yoonji Park +1
Department of Artificial Intelligence, Sungkyunkwan University, Korea · Department of Computer Science and Engineering, Sungkyunkwan University, Korea
Instruction-following text-to-speech (TTS) typically leaves voice design implicit, as conventional speaker embeddings encode reference audio rather than text-driven targets. We propose VoTSpeech, which decouples voice design from speech generation via an explicit intermediate voice latent. Built on a shared language model backbone, VoTSpeech first predicts a continuous voice latent from text instructions using a flow-based module. This latent serves as a shared condition for both the language model and the Acoustic DiT, guiding autoregressive sequence modeling and acoustic latent generation, respectively, before an audio VAE decoder converts the acoustic latents into waveforms. To capture speaker identity and complementary acoustic nuances, the voice latent is jointly supervised by speaker embeddings and temporally pooled audio VAE features. With task-specific adaptation, VoTSpeech performs strongly on InstructTTSEval-ZH, demonstrating competitive capabilities against strong open-source models while maintaining high naturalness. These results confirm that explicit voice latent design and its conditioning interface effectively empower text-driven voice design for controllable speech synthesis. Code, model, and audio samples are available at https://github.com/ywbn/VoTSpeech .
Wenbing Yang, Qihang Lu, Zihan Sun +4
Beijing University of Posts and Telecommunications, Beijing, China · Hello Group, Beijing, China
Style-captioned text-to-speech systems use natural language to control voice characteristics, but how individual words influence acoustic output remains unclear. Understanding this is critical for diagnosing failure modes and improving controllability in expressive TTS. We propose cross-attention attribution for speech diffusion models, adapting the DAAM framework to the speech domain for the first time, and apply it to CapSpeech-TTS. Our method extracts per-token heatmaps across 25 layers and 24 ODE steps. We analyze 3,600 (style caption, text transcript) combinations comprising 120 style captions conditioning the generation of 30 text transcripts each, revealing how caption tokens shape waveforms. Results show: (1) style tokens have lower temporal variance than content/function tokens, confirming global conditioning; (2) style attention correlates with F0 and energy; (3) style conditioning peaks in early steps and deep layers; (4) attention entropy reaches its minimum at layer 17, co-occurring with the style importance peak, indicating maximal network selectivity at the most style-critical stage. This is the first study of how natural language influences cross-attention in speech diffusion models