Unlocking Fine-Grained and Within-Utterance Speaking Style Control in Prompt-Based Text-to-Speech Models
Authors: Jaehoon Kang, Yejin Lee, Yoonji Park, Kyuhong Shim
Organizations: Department of Artificial Intelligence, Sungkyunkwan University, Korea · Department of Computer Science and Engineering, Sungkyunkwan University, Korea
While prompt-based text-to-speech (TTS) models enable natural language-driven speaking style control, they often provide limited fine-grained control and apply a single global style across an utterance. This restricts practical use cases that require continuous style attribute interpolation across utterances and time-varying style transitions within a single utterance. In this paper, we propose novel techniques to achieve both capabilities in existing prompt-based TTS models. For inter-utterance style interpolation, we compute direction vectors between contrastive style prompts in the embedding space and perform simple interpolation, enabling smooth transitions between style characteristics. For intra-utterance style transition, we first identify a strong attention bias toward early tokens in autoregressive TTS decoders, causing the initial audio realization to dominate subsequent generation. To mitigate this effect, we introduce KV-cache swapping and sliding-window attention masking. Experiments demonstrate that our proposed inter-utterance interpolation achieves a 99-100% success rate in gender conversion, up to 36 Hz pitch variation, and up to 1.6 syllables-per-second speed change. Our intra-utterance transition maintains a speaker similarity of 0.81-0.91 and achieves perceptual smoothness scores of 3.48-4.48.
Figures & tables
Figure 1: Overview of our training-free style control methods. (A) Inter-utterance style interpolation provides continuous control between source and target styles by adjusting the strength factor α . (B) Intra-utterance style transition unlocks style transitions within a single utterance during generation.
Figure 2: Inter-utterance Style Interpolation. (a) Given a source-target style prompt pair, we extract their embeddings e(s) and e(t) from the text encoder and compute the direction vector d=e(t)−e(s) . By varying the interpolation strength α , we generate new style embeddings to continuously control the speaking style attributes. (b) UMAP visualization of the embedding space shows that interpolated embeddings (colored points with α∈{0.4,0.8,1.2,1.6} ) form a smooth trajectory between source (male) and target (female) style representations.
Figure 3: Inter-utterance style interpolation results. (a) Gender conversion success rate, (b) average F0 change for pitch control, (c) average speaking rate change for speed control. All attributes show monotonic changes with α in the positive range, demonstrating continuous controllability.
Attribute
Direction
Success
Δ Metric
SIM
Gender
Female → Male
99.0%
–
–
Male → Female
100.0%
–
–
Pitch
High → Low
96.3%
− 36.1 Hz
0.76
Low → High
93.0%
+ 35.8 Hz
0.78
Speed
Quick → Slow
94.2%
− 1.4 SPS
0.84
Slow → Quick
95.7%
+ 1.6 SPS
0.84
Table 1: Objective evaluation of inter-utterance style interpolation.
Attr.
Direction
Style Change ( α )
MOS
0.5
1.0
2.0
Gender
Female → Male
0.09
0.12
1.74
4.25
Male → Female
0.18
0.62
2.00
4.40
Pitch
High → Low
0.95
1.09
1.71
4.26
Low → High
0.68
1.18
1.62
4.29
Speed
Quick → Slow
0.26
0.75
1.62
3.99
Table 2: Subjective evaluation of inter-utterance style interpolation.
Figure 4: Cross-attention weights between style text tokens (rows) and generated audio tokens (columns) across multiple decoder layers. Attention weights are actively updated during early generation, then remain nearly constant throughout subsequent generation.
Figure 5: Intra-utterance style transition. To convert speaking style mid-generation, we combine multiple techniques to avoid style self-referencing: (a) prepare an initial KV-cache derived from the target style prompt, and (b) swap the original KV-cache and style embeddings, together with (c) sliding-window attention masking.
Attribute
Direction
Window Size
Δ Metric
SIM
Trans. (%)
Smoothness
Pitch
High → Low
256
− 12.4 Hz
0.81
96.2
4.20
384
− 8.97 Hz
0.85
73.1
3.79
512
− 10.9 Hz
0.87
80.0
4.20
Full
− 11.5 Hz
0.90
55.6
4.00
Low → High
256
+ 27.4 Hz
0.84
96.2
3.48
384
+ 19.6 Hz
0.87
73.1
3.79
Table 3: Intra-utterance style transition results. Δ Metric indicates the average pitch (Hz) or speed (SPS) difference between the first and last 3-second segments. SIM indicates speaker similarity between the two segments. Trans. indicates the percentage of samples where listeners perceived a style transition. Smoothness rates the naturalness of style transition on a 1–5 scale. Results are reported using a KV-cache buffer size of k=48 .
Attribute
Direction
Style diff
SIM
Pitch
High → Low
− 4.3 Hz
0.92
Low → High
− 2.40 Hz
0.92
Speed
Quick → Slow
− 0.23 SPS
0.93
Slow → Quick
− 0.27 SPS
0.91
Table 4: Style transition results when only replacing the style embedding without KV-cache swap.
Figure 6: Ablation study on window size and KV-cache region size ( n=ntext+k ) for High → Low pitch conversion.
Figure 7: Inter-utterance style interpolation results when shifting all tokens. Results are comparable to attribute-only interpolation (Figure 3), demonstrating that manipulating only attribute tokens is sufficient for effective style control.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Evaluation interface for inter-utterance style interpolation. Participants compared source and converted audio, rating both the style conversion effectiveness (Q1) and audio quality (Q2).
Figure 9: Evaluation interface for intra-utterance style transition. Participants evaluated whether they detected the intended style transition (Q1) and rated the naturalness of the transition (Q2).
Attribute
Direction
Success (%)
Δ Metric
SIM
Gender
Female → Male
99.3
–
–
Male → Female
100
–
–
Pitch
High → Low
91.3
−32.4 Hz
0.79
Low → High
93.5
+35.4 Hz
0.79
Speed
Fast → Slow
95.5
−1.5 SPS
0.85
Slow → Fast
95.75
+1.7 SPS
0.85
Appendix
Table 5: Full vector interpolation results at α=2.0 (attribute tokens) and β=1.0 (non-attribute tokens).
Figure 10: Attention weight variance across audio generation positions for different decoder layers. During early generation, variance fluctuates as the model actively updates attention weights to establish style characteristics. Afterward, variance becomes stable, indicating fixed attention patterns.
Attribute
Prompt
Gender
“A male voice speaks moderate at a medium pitch with monotone modulation and a clean quality.”
“A female voice speaks moderate at a medium pitch with monotone modulation and a clean quality.”
Pitch
“A male voice speaks normally at a high pitch and a clean quality.”
“A male voice speaks normally at a low pitch and a clean quality.”
Speed
“A male voice speaks quickly at a normal pitch and a clean quality.”
“A male voice speaks slowly at a normal pitch and a clean quality.”
Appendix
Table 6: Style prompts used in experiments. The attribute keywords (underlined) are the tokens where direction vectors are computed and applied.
Dynamic speaking style adaptation in multi-turn multimodal interaction remains a major challenge for text-to-speech (TTS) systems. Existing context-aware TTS (CTTS) methods typically map dialogue context to speech in an end-to-end manner. Such implicit modeling makes contextual style decisions difficult to supervise, while the entanglement of style, timbre, and content often leads to weak instruction-following and severe timbre drift across turns. To overcome these limitations, we propose Interactive TTS, a dynamic, style-adaptive framework for contextually appropriate and speaker-consistent speech generation. Interactive TTS decouples the process by explicitly modeling contextual style decisions as executable instructions. To bridge the gap between style decisions and speech generation, we introduce Iterative Rejection Sampling Fine-Tuning (Iterative RSFT) and Context-Aware Direct Preference Optimization (CADPO), which significantly enhance instruction-following and align the generated speech with conversational contexts. Extensive experiments demonstrate that Interactive TTS outperforms state-of-the-art models on VStyle and SpeechParaling-Bench. Demo is available at https://wjtian-wonderful.github.io/InteractiveTTS/
State-of-the-art text-to-speech (TTS) models achieve impressive naturalness and expressiveness, yet fine-grained, disentangled control over speaking styles remains challenging. In professional scenarios such as film dubbing, game voice acting, and video content generation, users often need to modify a specific style category, such as emotion, age, or gender, while preserving all others. Existing style-controllable TTS methods typically rely on either text-described styles or speech-reference style transfer, making it difficult to jointly control explicit semantic attributes and preserve subtle, text-undescribed prosodic details. We propose AutoSIFT, a controllable speech generation framework for category-level style editing. AutoSIFT decomposes speaking style into known text-describable categories and unknown residual styles that capture non-verbal prosody and speaker-specific nuances. It consists of a generalized Style Disentangler, which extracts category-aware style prototypes from reference speech, and an Arbitrary Style Infiller, which selectively infills unspecified style categories from the reference. By replacing only text-specified style categories while preserving residual speech-derived styles, AutoSIFT enables natural, expressive, and highly customizable speech generation.
Haowei Lou, Junda Wu, Chengkai Huang +4
UNSW Sydney · University of California San Diego · Macquarie University +1
We propose GLASS, a framework for composable acoustic style control in zero-shot autoregressive text-to-speech (TTS) that learns controls from post-generation rewards rather than style labels. In zero-shot TTS, a speaker prompt often entangles speaker identity with prosodic attributes such as speaking rate and pitch, making it difficult to change style without changing the prompt itself. GLASS instead treats each acoustic attribute as a reward-defined control direction. For each control axis, GLASS freezes the TTS backbone and trains one lightweight LoRA adapter with Group Relative Policy Optimization (GRPO), using speech-token length and mean F0 as style rewards and WER as an intelligibility anchor. Because each control is represented as a LoRA weight update, independently trained adapters can be swapped, interpolated, and composed through linear LoRA arithmetic without retraining the backbone. Experiments on speaking rate and pitch control show targeted style shifts while preserving naturalness, speaker similarity, and intelligibility, and demonstrate smooth interpolation and multi-axis composition across independently trained adapters.
Jaehoon Kang, Yejin Lee, Kyuhong Shim
Department of Artificial Intelligence, Sungkyunkwan University, Korea