Unlocking Fine-Grained and Within-Utterance Speaking Style Control in Prompt-Based Text-to-Speech Models

Jaehoon Kang; Yejin Lee; Yoonji Park; Kyuhong Shim

cs.CLApr 9, 2026

Unlocking Fine-Grained and Within-Utterance Speaking Style Control in Prompt-Based Text-to-Speech Models

Authors: Jaehoon Kang, Yejin Lee, Yoonji Park, Kyuhong Shim

Organizations: Department of Artificial Intelligence, Sungkyunkwan University, Korea · Department of Computer Science and Engineering, Sungkyunkwan University, Korea

Abstract

While prompt-based text-to-speech (TTS) models enable natural language-driven speaking style control, they often provide limited fine-grained control and apply a single global style across an utterance. This restricts practical use cases that require continuous style attribute interpolation across utterances and time-varying style transitions within a single utterance. In this paper, we propose novel techniques to achieve both capabilities in existing prompt-based TTS models. For inter-utterance style interpolation, we compute direction vectors between contrastive style prompts in the embedding space and perform simple interpolation, enabling smooth transitions between style characteristics. For intra-utterance style transition, we first identify a strong attention bias toward early tokens in autoregressive TTS decoders, causing the initial audio realization to dominate subsequent generation. To mitigate this effect, we introduce KV-cache swapping and sliding-window attention masking. Experiments demonstrate that our proposed inter-utterance interpolation achieves a 99-100% success rate in gender conversion, up to 36 Hz pitch variation, and up to 1.6 syllables-per-second speed change. Our intra-utterance transition maintains a speaker similarity of 0.81-0.91 and achieves perceptual smoothness scores of 3.48-4.48.

Figures & tables

Figure 1: Overview of our training-free style control methods. (A) Inter-utterance style interpolation provides continuous control between source and target styles by adjusting the strength factor $\alpha$ . (B) Intra-utterance style transition unlocks style transitions within a single utterance during generation.

Panel of Figure 2 — Figure 2: Inter-utterance Style Interpolation. (a) Given a source-target style prompt pair, we extract their embeddings $\mathbf{e}^{(s)}$ and $\mathbf{e}^{(t)}$ from the text encoder and compute the direction vector $\mathbf{d}=\mathbf{e}^{(t)}-\mathbf{e}^{(s)}$ . By varying the interpolation strength $\alpha$ , we generate new style embeddings to continuously control the speaking style attributes. (b) UMAP visualization of the embedding space shows that interpolated embeddings (colored points with $\alpha\in\{0.4,0.8,1.2,1.6\}$ ) form a smooth trajectory between source (male) and target (female) style representations.

Figure 3: Inter-utterance style interpolation results. (a) Gender conversion success rate, (b) average F0 change for pitch control, (c) average speaking rate change for speed control. All attributes show monotonic changes with

\alpha

in the positive range, demonstrating continuous controllability.

Attribute	Direction	Success	$\Delta$ Metric	SIM
Gender	Female $\rightarrow$ Male	99.0%	–	–
Gender	Male $\rightarrow$ Female	100.0%	–	–
Pitch	High $\rightarrow$ Low	96.3%	$-$ 36.1 Hz	0.76
Pitch	Low $\rightarrow$ High	93.0%	$+$ 35.8 Hz	0.78
Speed	Quick $\rightarrow$ Slow	94.2%	$-$ 1.4 SPS	0.84
Speed	Slow $\rightarrow$ Quick	95.7%	$+$ 1.6 SPS	0.84

Table 1: Objective evaluation of inter-utterance style interpolation.

Attr.	Direction	Style Change ( $\alpha$ )			MOS
Attr.	Direction	$0.5$	$1.0$	$2.0$	MOS
Gender	Female $\rightarrow$ Male	0.09	0.12	1.74	4.25
Gender	Male $\rightarrow$ Female	0.18	0.62	2.00	4.40
Pitch	High $\rightarrow$ Low	0.95	1.09	1.71	4.26
Pitch	Low $\rightarrow$ High	0.68	1.18	1.62	4.29
Speed	Quick $\rightarrow$ Slow	0.26	0.75	1.62	3.99

Table 2: Subjective evaluation of inter-utterance style interpolation.

Figure 4: Cross-attention weights between style text tokens (rows) and generated audio tokens (columns) across multiple decoder layers. Attention weights are actively updated during early generation, then remain nearly constant throughout subsequent generation.

Figure 5: Intra-utterance style transition. To convert speaking style mid-generation, we combine multiple techniques to avoid style self-referencing: (a) prepare an initial KV-cache derived from the target style prompt, and (b) swap the original KV-cache and style embeddings, together with (c) sliding-window attention masking.

Attribute	Direction	Window Size	$\Delta$ Metric	SIM	Trans. (%)	Smoothness
Pitch	High $\rightarrow$ Low	256	$-$ 12.4 Hz	0.81	96.2	4.20
		384	$-$ 8.97 Hz	0.85	73.1	3.79
		512	$-$ 10.9 Hz	0.87	80.0	4.20
		Full	$-$ 11.5 Hz	0.90	55.6	4.00
	Low $\rightarrow$ High	256	$+$ 27.4 Hz	0.84	96.2	3.48
		384	$+$ 19.6 Hz	0.87	73.1	3.79

Table 3: Intra-utterance style transition results.

\Delta

Metric indicates the average pitch (Hz) or speed (SPS) difference between the first and last 3-second segments. SIM indicates speaker similarity between the two segments. Trans. indicates the percentage of samples where listeners perceived a style transition. Smoothness rates the naturalness of style transition on a 1–5 scale. Results are reported using a KV-cache buffer size of

k=48

Attribute	Direction	Style diff	SIM
Pitch	High $\rightarrow$ Low	$-$ 4.3 Hz	0.92
Pitch	Low $\rightarrow$ High	$-$ 2.40 Hz	0.92
Speed	Quick $\rightarrow$ Slow	$-$ 0.23 SPS	0.93
Speed	Slow $\rightarrow$ Quick	$-$ 0.27 SPS	0.91

Table 4: Style transition results when only replacing the style embedding without KV-cache swap.

Figure 6: Ablation study on window size and KV-cache region size ( $n=n_{\text{text}}+k$ ) for High $\rightarrow$ Low pitch conversion.

Figure 7: Inter-utterance style interpolation results when shifting all tokens. Results are comparable to attribute-only interpolation (Figure 3), demonstrating that manipulating only attribute tokens is sufficient for effective style control.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Evaluation interface for inter-utterance style interpolation. Participants compared source and converted audio, rating both the style conversion effectiveness (Q1) and audio quality (Q2).

Figure 9: Evaluation interface for intra-utterance style transition. Participants evaluated whether they detected the intended style transition (Q1) and rated the naturalness of the transition (Q2).

Attribute	Direction	Success (%)	$\Delta$ Metric	SIM
Gender	Female $\rightarrow$ Male	99.3	–	–
Gender	Male $\rightarrow$ Female	100	–	–
Pitch	High $\rightarrow$ Low	91.3	$-32.4$ Hz	0.79
Pitch	Low $\rightarrow$ High	93.5	$+35.4$ Hz	0.79
Speed	Fast $\rightarrow$ Slow	95.5	$-1.5$ SPS	0.85
Speed	Slow $\rightarrow$ Fast	95.75	$+1.7$ SPS	0.85

Appendix

Table 5: Full vector interpolation results at

\alpha=2.0

(attribute tokens) and

\beta=1.0

(non-attribute tokens).

Figure 10: Attention weight variance across audio generation positions for different decoder layers. During early generation, variance fluctuates as the model actively updates attention weights to establish style characteristics. Afterward, variance becomes stable, indicating fixed attention patterns.

Attribute	Prompt
Gender	“A male voice speaks moderate at a medium pitch with monotone modulation and a clean quality.”
Gender	“A female voice speaks moderate at a medium pitch with monotone modulation and a clean quality.”
Pitch	“A male voice speaks normally at a high pitch and a clean quality.”
Pitch	“A male voice speaks normally at a low pitch and a clean quality.”
Speed	“A male voice speaks quickly at a normal pitch and a clean quality.”
	“A male voice speaks slowly at a normal pitch and a clean quality.”

Appendix

Table 6: Style prompts used in experiments. The attribute keywords (underlined) are the tokens where direction vectors are computed and applied.

TTS Synthesis

Text-to-speech synthesis converts written language into intelligible, natural-sounding speech with control over voice, prosody, style, and timing.

See all 275 papers

Interactive TTS: Dynamic Speaking Style Adaptation for Expressive Speech Synthesis
Sep 22, 2026Wenjie Tian, Kangxiang Xia, Jingbin Hu +7TTS Synthesis Direct Preference Optimization
Alibaba ATH Token Foundry
AutoSIFT: Automatic Style Sifting for Controllable Speech Generation with Arbitrary Style Infilling
Jul 14, 2026Haowei Lou, Junda Wu, Chengkai Huang +4TTS Synthesis Representation Disentanglement
UNSW Sydney · University of California San Diego · Macquarie University +1
GLASS: GRPO-Trained LoRA for Acoustic Style Steering in Zero-Shot Text-to-Speech
Jun 4, 2026Jaehoon Kang, Yejin Lee, Kyuhong ShimTTS Synthesis Adapter Tuning
Department of Artificial Intelligence, Sungkyunkwan University, Korea

Sep 22, 2026J/K move · Enter open · S save

Interactive TTS: Dynamic Speaking Style Adaptation for Expressive Speech Synthesis

Wenjie Tian, Kangxiang Xia, Jingbin Hu, Xinfa Zhu+6

Alibaba ATH Token Foundry

TTS Synthesis Direct Preference Optimization Controllable Speech Generation Expressive Speech Synthesis

Open paper AskPDF arXiv

Dynamic speaking style adaptation in multi-turn multimodal interaction remains a major challenge for text-to-speech (TTS) systems. Existing context-aware TTS (CTTS) methods typically map dialogue context to speech in an end-to-end manner. Such implicit modeling makes contextual style decisions difficult to supervise, while the entanglement of style, timbre, and content often leads to weak instruction-following and severe timbre drift across turns. To overcome these limitations, we propose Interactive TTS, a dynamic, style-adaptive framework for contextually appropriate and speaker-consistent speech generation. Interactive TTS decouples the process by explicitly modeling contextual style decisions as executable instructions. To bridge the gap between style decisions and speech generation, we introduce Iterative Rejection Sampling Fine-Tuning (Iterative RSFT) and Context-Aware Direct Preference Optimization (CADPO), which significantly enhance instruction-following and align the generated speech with conversational contexts. Extensive experiments demonstrate that Interactive TTS outperforms state-of-the-art models on VStyle and SpeechParaling-Bench. Demo is available at https://wjtian-wonderful.github.io/InteractiveTTS/

Abstract

TTS Synthesis

Explore similar work

Interactive TTS: Dynamic Speaking Style Adaptation for Expressive Speech Synthesis

AutoSIFT: Automatic Style Sifting for Controllable Speech Generation with Arbitrary Style Infilling

GLASS: GRPO-Trained LoRA for Acoustic Style Steering in Zero-Shot Text-to-Speech