Steering intonational tunes in text-to-speech synthesis

Juan Pablo Vigneaux; Jennifer Cole

cs.SDOct 19, 2026

Steering intonational tunes in text-to-speech synthesis

Authors: Juan Pablo Vigneaux, Jennifer Cole

Organizations: Northwestern University

Abstract

Text-to-speech systems built on residual vector quantization often generate audio in two stages: a transformer backbone emits the first level of each frame's token stack, and a smaller depth model generates the rest autoregressively. We ask which components of such a model are responsible for prosodic intonation, more precisely for generating the English nuclear tune (for instance, the falling pitch melody that distinguishes an assertion from a question's rising melody), and whether interventions on those components can be used to control intonation. Prompt-based control mostly fails; only a final question mark is a reliable lever. Interchange interventions in CSM and S2 locate the tune: the backbone's hidden state carries it to the waveform through the depth model, and the first level of the stack carries little or none of it. Steering the backbone state with a difference-of-means vector computed on new material generated with a final period or question mark moves the tune, but the push toward an assertion degrades the voice. Steering vectors computed instead from the backbone states associated with human imitations of theoretically-specified tunes, whose text carries no punctuation, and applied inside the generation loop, move most recordings to the other condition in both systems (even when fitted without the speaker being steered), at a small cost in predicted naturalness. These imitation-derived vectors are aligned with the one obtained from the two punctuated conditions far beyond a shuffled-label null, so the backbone state of a punctuated prompt already represents intonation. The steered recordings broadly become falling or rising, according to the intent, but the edge tone moves more coherently than the pitch accent, and the target tune cannot be selected precisely beyond this broad tendency.

Figures & tables

Figure 1: Normalized displacement toward the source condition when levels of the token stack are swapped over the final word between a matched assertion and question, by range of levels and by read-out (CSM, ten-speaker set,

90

pairs, paired-bootstrap intervals;

1

means the swapped recording lands on the source condition’s mean). The last group swaps levels

1

–

2

over the word’s first frames ( pre ) instead of its last (§ D.2 ).

Intervention	$\Delta_{\Phi}$	rising, A / I	WER	$\Delta_{\Phi}$	rising, A / I	WER
	CSM ( $90$ pairs)			Fish S2 ( $90$ pairs)
Unmodified	—	$17$ / $93$	$0.257$	—	$17$ / $98$	$0.130$
Swap levels $1$ – $K{-}1$	$0.99$ $[0.97,1.01]$	$90$ / $18$	$0.328$	$0.91$ $[0.82,0.99]$	$91$ / $17$	$0.194$
Swap level $0$	$0.00$ $[0.00,0.00]$	$17$ / $93$	$0.266$	$0.09$ $[0.00,0.21]$	$18$ / $92$	$0.228$
Re-run, own $\bm{h}_{t}$ and $q^{0}_{t}$	$0.01$ $[-0.03,0.03]$	$21$ / $92$	$0.336$	$0.00$ $[-0.02,0.02]$	$13$ / $96$	$0.144$
Source $\bm{h}_{t}$ , own $q^{0}_{t}$	$0.98$ $[0.94,1.02]$	$90$ / $17$	$0.348$	$0.91$ $[0.82,0.98]$	$90$ / $19$	$0.165$

Table 1: Interventions in both systems.

\Delta_{\Phi}

: normalized displacement toward the source condition, mean over the three read-outs and both directions, with the range over those six cells. Rising : per cent of assertion ( A ) and question ( I ) targets whose final-word excursion exceeds

+0.38

ERB afterwards. WER: word error rate. Middle blocks: the depth model re-run with

\bm{h}_{t}

and

q^{0}_{t}

from the target (own) or the source. Steering rows: the offline edit of ( 3 ) at

\alpha=4

gaps; ‡ the probe direction’s most effective strength.

Figure 2: Steering along the tune vector in the loop, CSM,

90

recordings per strength: assertions pushed toward a rise (top), questions toward a fall (bottom). Left: acoustic measures over the final word in units of the source condition’s s.d., centred on the unsteered run ( null ); rightmost, the unmodified target condition, so a strength is read as a position between source and target. Right: nearest human tune shape (§ 3 ), with the target condition’s own distribution at the right; grey, too little voicing (§ G ). Shaded: the strength chosen. S2: Figure 15 ; offline: Figure 12 (CSM) and 13 (S2).

Figure 3:

F_{0}

trajectories of every generated recording, coloured by the nearest of the five human tune clusters (Figure 17 ), in scaled ERB, each voice scaled by its own unsteered recordings as the clusters were scaled by speaker; nothing is fitted to the generated material. Bold: the mean of each colour, with its share of the panel; words with too little voicing are left out. Columns: assertions steered toward a rise, questions toward a fall; rows: unsteered, steered.

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

	CSM ( $180$ )			S2 ( $180$ )
read-out	agreement	AUC	ARI	agreement	AUC	ARI
excursion	$0.84$	$0.90$	$0.46$	$0.84$	$0.96$	$0.47$
velocity	$0.84$	$0.88$	$0.46$	$0.84$	$0.96$	$0.47$
cluster	$0.84$	$0.90$	$0.46$	$0.94$	$0.97$	$0.77$

Appendix

Table 2: The three read-outs on the unsteered ten-speaker sets, scored against the design labels with the score’s sign at zero. Agreement : fraction of recordings put in the right condition; AUC: area under the ROC curve; ARI: adjusted Rand index of the partition.

Final-word interval	mean $\|\Delta\|$	max $\|\Delta\|$	$r$	sign agreement
displaced $-80$ ms	$0.482$	$1.662$	$0.930$	$0.893$
displaced $-40$ ms	$0.233$	$1.191$	$0.981$	$0.940$
displaced $+40$ ms	$0.140$	$1.082$	$0.992$	$0.988$
displaced $+80$ ms	$0.182$	$1.569$	$0.985$	$0.976$
no alignment (asserted interval)	$0.218$	$1.555$	$0.980$	$0.988$

Appendix

Table 3: Excursion under perturbed and absent final-word boundaries, against the excursion under the forced alignment, in ERB. The condition separation is

2.409

ERB under the forced alignment,

2.046

and

2.565

under

\mp 40

ms,

1.560

and

2.565

under

\mp 80

ms, and

2.555

with no alignment.

Figure 4: Each style tag’s mean change against the untagged utterance of the same voice and sentence, over the ten speakers, in three measures of the final word: its excursion (grey), its highest pitch (purple) and its lowest pitch (green), taken as the

95

th and

5

th percentiles of its voiced contour so that one stray pitch point cannot set them. Each has its

95\%

interval (dark) and the Bonferroni interval over the eighteen tags (light); the dashed lines mark the widest

95\%

interval of the excursion. Below, the question mark’s effect on the excursion in the same ten voices, paired by voice, sentence and iteration.

Figure 5: Every style tag in the ten voices. One panel per tag, the untagged control first; each panel draws one recording from each of the ten speakers, all prompted with that speaker’s own eight-tune voice prompt. Each contour is centred on its own mean, so that shape is compared across voices whose registers differ. A speaker is shown on one carrier sentence, rotated across speakers and the same in all of that speaker’s panels, so that within a speaker only the tag changes. Blue: the panel’s mean where at least half of the recordings are voiced; dashed: the control’s mean, repeated in every panel; top right: how many of the ten rise past

+0.38

ERB.

Figure 6: Where the unsteered conditions differ on the final word, ten-speaker sets (

90

recordings per condition and system). Left:

F_{0}

of the first and last fifth of the word’s voiced contour, one point per recording, with each condition’s mean drawn and printed. Right: the separation of the condition means in ERB and as Cohen’s

d

Figure 7: The final-word length of the two recordings of each matched pair in frames, over the

90

CSM pairs of the ten-speaker set.

Figure 10: Reconstruction from levels

0

–

k

, CSM. Left: the conditions’ separation by the excursion and by an unsupervised cluster index against the design labels. At

k=0

the reconstruction carries no voiced material, so neither measure exists. Right: word error rate against depth, with the unmodified recordings’ rate marked.

Figure 11: One matched pair of the CSM ten-speaker set: the target, the target with level $0$ taken from the source, the target with levels $1$ – $31$ taken from the source, and the source. Log-mel spectrograms; dashed lines bound the final word.

CSM, final word of two sentences	$n$	WER	target’s name	source’s name
unmodified	$180$	$0.26$	$0.93$	—
swap level $0$ (codec)	$180$	$0.20$	$0.97$	$0.00$
swap levels $1$ – $31$ (codec)	$180$	$0.50$	$0.00$	$0.77$
depth model: own $\bm{h}_{t}$ , source’s $q^{0}_{t}$	$180$	$0.55$	$0.06$	$0.07$
depth model: source’s $\bm{h}_{t}$ , own $q^{0}_{t}$	$180$	$0.47$	$0.13$	$0.02$
depth model: source’s $\bm{h}_{t}$ and $q^{0}_{t}$	$180$	$0.50$	$0.02$	$0.76$

Appendix

Table 4: The cross-sentence control, CSM ten-speaker set. Target’s name , source’s name : share of transcripts containing the final word of the recording modified or of the one supplying the tokens (the recognizer’s spellings, e.g. Madeleine , counted).

	pooled over the final word		backbone state $\bm{h}_{t}$ , by region
	accuracy	AUC	region	accuracy	AUC
level $0$ codewords	$0.522$	$0.559$	before the final word	$0.661$	$0.760$
level $1$	$0.528$	$0.571$	final-word onset	$0.556$	$0.690$
level $2$	$0.650$	$0.703$	final word	$0.861$	$0.927$
level $3$	$0.589$	$0.603$	final-word offset	$0.839$	$0.904$
level $7$	$0.467$	$0.451$	final frame	$0.844$	$0.929$

Appendix

Table 5: Linear decodability of the condition, CSM ten-speaker set (

180

recordings), leave-one-carrier-sentence-out. The permutation null has mean accuracy

0.47

–

0.52

and maximum

0.54

–

0.64

across cells, so values below about

0.64

are not distinguishable from chance.

Figure 12: Steering along the punctuation vector

\hat{\bm{w}}

offline, CSM, ten-speaker set. One row per direction of the push: assertions pushed toward a rise (top) and questions toward a fall (bottom),

90

recordings per strength. Left: what the edit does to the voice. Each violin is one acoustic measure over the final word — excursion, offset, voiced fraction, harmonics-to-noise ratio — in units of the source condition’s standard deviation, centred on the source’s own re-run without an edit ( null ); the rightmost group is the unmodified target condition. A strength is therefore read as a position between the source and the target. Right: the nearest of the five human tune shapes for the same recordings (§ 3 ), with the target condition’s own distribution at the right; grey, words with too little voicing for a contour (§ G ). The strength axis is in gaps ( 3 ). The edit is made before the normalization,

\mathrm{norm}(\bm{x}_{t}+\alpha g\hat{\bm{w}})

, the form used in the rest of the paper; the S2 version is Figure 13 .

		I $\to$ A				A $\to$ I
	Edit	$\alpha$	offset	voiced	HNR	$\alpha$	offset	voiced	HNR
CSM	natural target		$4.69$	$0.86$	$15.3$		$6.58$	$0.93$	$17.7$
	depth re-run only	$0$	$6.58$	$0.92$	$15.5$	$0$	$4.70$	$0.85$	$13.1$
	after the norm	$8$	$4.68$	$0.74$	$9.7$	$10$	$6.54$	$0.91$	$14.1$
	after the norm, $q^{0}_{t}$ redrawn	$8$	$4.65$	$0.74$	$9.9$	$10$	$6.75$	$0.92$	$14.3$
	before the norm	$30$	$4.65$	$0.72$	$9.4$	$40$	$7.04$	$0.92$	$15.1$

Appendix

Table 6: Voice quality of the offline edits at matched offset, ten-speaker set,

90

targets per direction. Offset : mean offset level of the steered word (ERB); voiced : voiced fraction; HNR in dB. Each row is the grid value of

\alpha

(state units) whose mean offset is closest to the natural target; one gap is about

3.1

state units after the normalization in CSM and

2.3

in S2, and before it

10.6

in CSM and

166

in S2 with the two coordinates out; natural target gives that condition’s unmodified values, and depth re-run only the steered condition’s recordings re-run without an edit.

Figure 13: Steering along the punctuation vector offline in S2, drawn as Figure 12 .

system	push	vector	flip rate	voiced fraction	HNR (dB)
CSM	rise, $2.5$ gaps	held out	$0.60$ $[0.50,0.69]$	$0.97$	$17.7$
		in-sample	$0.56$ $[0.48,0.64]$	$0.98$	$18.1$
	fall, $1.5$ gaps	held out	$0.60$ $[0.53,0.67]$	$0.90$	$14.0$
		in-sample	$0.56$ $[0.46,0.66]$	$0.89$	$14.3$
S2	rise, $2.5$ gaps	held out	$0.76$ $[0.67,0.83]$	$0.92$	$19.6$
		in-sample	$0.72$ $[0.62,0.82]$	$0.92$	$19.0$

Appendix

Table 7: Leave-one-speaker-out steering: flip rates with

95\%

intervals bootstrapped over speakers, and median voice quality of the steered final words,

90

targets per cell. Held out : each speaker steered with a vector fitted on the other nine; in-sample : the vector fitted on all ten.

system	push	$1.5$	$2.5$	$3.5$	$4.5$
CSM	toward a rise	$0.48$	$\mathbf{0.30}$	$0.43$	$0.76$
CSM	toward a fall	$\mathbf{0.37}$	$0.89$	$1.50$	$1.95$
S2	toward a rise	$0.57$	$\mathbf{0.38}$	$0.54$	$0.70$
S2	toward a fall	$\mathbf{0.48}$	$0.82$	$1.14$	$1.45$

Appendix

Table 8: The strength criterion

D

, the mean over excursion, offset, voiced fraction and HNR of the Wasserstein distance between the steered recordings and the target condition’s unmodified recordings, in units of the unsteered runs’ standard deviation, by strength in gaps;

90

recordings per cell. Bold: the minimum, the strength read.

CSM
intervention	direction	$1.5$	$2.5$	$3.5$	$4.5$
tune	A $\to$ I	$0.36$ $[0.22,0.49]$	$0.56$ $[0.48,0.64]$	$0.51$ $[0.42,0.60]$	$0.64$ $[0.52,0.76]$
	I $\to$ A	$0.56$ $[0.45,0.67]$	$0.82$ $[0.77,0.88]$	$0.81$ $[0.71,0.90]$	$0.76$ $[0.69,0.82]$
edge	A $\to$ I	$0.30$ $[0.21,0.39]$	$0.56$ $[0.49,0.62]$	$0.53$ $[0.45,0.61]$	$0.36$ $[0.29,0.43]$
	I $\to$ A	$0.63$ $[0.53,0.74]$	$0.78$ $[0.74,0.83]$	$0.69$ $[0.59,0.80]$	$0.58$ $[0.51,0.67]$
accent	A $\to$ I	$0.16$ $[0.09,0.23]$	$0.19$ $[0.09,0.31]$	$0.30$ $[0.23,0.38]$	$0.42$ $[0.35,0.48]$

Appendix

Table 9: Flip rates by intervention and strength in gaps, with

95\%

intervals bootstrapped over speakers;

90

recordings per cell.

Figure 14: The pilot: mean final-word contours of CSM’s two conditions in a single voice (

84

recordings), centred on the mean of all contours (left) and on each contour’s own mean (right), with individual recordings faint.

system	unsteered condition	hhl / hhh	hll / hlh	lhh	llh / lhl	lll	too little voicing
CSM	assertions ( A )	$0.033$	$0.233$	$0.100$	$0.122$	$\mathbf{0.478}$	$0.033$
CSM	questions ( I )	$0.300$	$0.022$	$\mathbf{0.400}$	$0.256$	$0.000$	$0.022$
S2	assertions ( A )	$0.011$	$0.144$	$0.033$	$0.211$	$\mathbf{0.589}$	$0.011$
S2	questions ( I )	$0.344$	$0.011$	$\mathbf{0.578}$	$0.067$	$0.000$	$0.000$

Appendix

Table 10: The nearest human tune shape of the unsteered ten-speaker recordings, as shares of all

90

recordings per condition; the last column counts words with too little voicing for a contour (§ G ).

Figure 15: Steering along the tune vector inside the loop in S2, drawn as Figure 2 . The push to a rise reaches the high plateau rather than lhh , the most frequent shape of S2’s own questions (

0.58

), which the steered set holds at most

0.12

of, and less with strength.

Figure 16: Flip rate against quality, over every steered recording of each strength of the tune vector in the loop (numbers: strength in gaps;

0

: the unsteered run). Top: median harmonicity of the final word. Bottom: mean UTMOS of the utterance. Dashed: the target condition’s own recordings.

Push to a rise, $2.5$ gaps
system	subset	$n$	voiced fraction	HNR (dB)	offset (ERB)
CSM	flipped	$49$	$0.951$ $[0.926,0.972]$	$19.5$ $[17.5,21.0]$	$7.40$ $[6.66,7.99]$
CSM	target (unmodified)	$90$	$0.927$ $[0.901,0.951]$	$17.7$ $[16.4,19.1]$	$6.58$ $[5.80,7.35]$
S2	flipped	$65$	$0.887$ $[0.868,0.903]$	$19.4$ $[17.5,21.1]$	$7.25$ $[6.43,8.09]$
S2	target (unmodified)	$90$	$0.888$ $[0.877,0.900]$	$17.8$ $[16.2,19.8]$	$6.63$ $[5.87,7.33]$
Push to a fall, $1.5$ gaps

Appendix

Table 11: Voice quality of the steered recordings that flipped, against the target condition, with

95\%

intervals bootstrapped over speakers.

Figure 17: The five nuclear-tune shapes of this corpus, reproducing Figure 5 of Cole et al. (2023) on the corpus recordings: one point per speaker and model tune, speaker-scaled ERB over normalized time. lhh alone, lll alone, lhl with llh , hlh with hll , and hhl with hhh . These centroids are what Figure 3 labels generated recordings against.

Figure 18: CSM’s unsteered recordings, assertions and questions pooled, as each labelling groups them: one panel per class, recordings faint (dashed: assertions; solid: questions), the class mean bold, and the human centroid dashed black; all in the centroids’ scaled ERB. Each panel’s title gives its recordings by condition and its spread , the mean squared distance to the class mean; within is the member-weighted spread as a share of the total. Words with too little voicing are left out.

Figure 19: S2’s unsteered recordings as each labelling groups them, drawn as Figure 18 .

Figure 20: The nearest human centroid (rows) against the five-class probe (columns), in counts, on the unsteered recordings (left) and on those steered along the tune vector at the strengths the paper reads (right: assertions to a rise at $2.5$ gaps, questions to a fall at $1.5$ ), with class totals in brackets. The last row, below the rule, holds the words with too little voicing for a contour, which the probe places in one of its five classes; agreement is the diagonal’s share of the voiced recordings above the rule.

Figure 21: The hlh pilot, CSM. Left: the eight per-tune templates, in speaker-scaled ERB over the final word. Middle: the unsteered run of the

43

assertions nearest lll , each contour in the colour of its nearest template, bold lines the mean of each colour; right: the same recordings steered toward hlh at

2.5

gaps. Percentages: the share of the panel’s voiced recordings nearest each template; words with too little voicing are left out.

$\alpha$ (gaps)	$n$	hhh	hhl	hlh	hll	lhh	lhl	llh	lll	excursion
$0$	$43$	$0$	$0$	$0$	$0$	$0$	$2$	$0$	$98$	$-0.57$
$1.5$	$43$	$5$	$5$	$2$	$14$	$5$	$19$	$2$	$47$	$-0.21$
$2.5$	$43$	$12$	$2$	$26$	$16$	$5$	$5$	$9$	$26$	$-0.33$
$3.5$	$43$	$21$	$56$	$2$	$2$	$5$	$5$	$5$	$5$	$0.59$
$4.5$	$40$	$25$	$68$	$5$	$2$	$0$	$0$	$0$	$0$	$0.66$
$6$	$41$	$15$	$68$	$0$	$5$	$0$	$0$	$0$	$0$	$0.29$

Appendix

Table 12: CSM assertions nearest lll , steered toward hlh : per cent of recordings nearest each per-tune template, and mean excursion (ERB).

0

: the unsteered in-loop run.

n

counts recordings that were generated and aligned; at

6

and

8

gaps,

12\%

and

14\%

of words have too little voicing for a contour.

Figure 22: Steering along the edge vector inside the loop, CSM, drawn as Figure 2 .

Figure 23: Recordings steered along the edge vector labelled by the nearest human tune cluster, drawn as Figure 3 .

Figure 24: Steering along the accent vector inside the loop, CSM, drawn as Figure 2 .

Figure 25: Recordings steered along the accent vector labelled by the nearest human tune cluster, drawn as Figure 3 .

TTS Synthesis

Text-to-speech synthesis converts written language into intelligible, natural-sounding speech with control over voice, prosody, style, and timing.

See all 275 papers

EmphTTS: an emphasis-control TTS with reinforcement learning
Sep 23, 2026Zirui Li, Rech Silas, Lauri Juvela +2TTS Synthesis Controllable Speech Generation
Department of Information and Communications Engineering, Aalto University, Espoo, Finland
Unlocking Fine-Grained and Within-Utterance Speaking Style Control in Prompt-Based Text-to-Speech Models
Apr 9, 2026Jaehoon Kang, Yejin Lee, Yoonji Park +1TTS Synthesis Controllable Speech Generation
Department of Artificial Intelligence, Sungkyunkwan University, Korea · Department of Computer Science and Engineering, Sungkyunkwan University, Korea
Prompt-Consistency Inference for Zero-Shot Flow-Matching Text-to-Speech Models
Oct 3, 2026Vasily Zadorozhnyy, Can Goksen, Kazuhito Koishida +1TTS Synthesis Zero-Shot TTS
Applied Sciences Group, Microsoft Corporation, Redmond, USA

Sep 23, 2026J/K move · Enter open · S save

EmphTTS: an emphasis-control TTS with reinforcement learning

Zirui Li, Rech Silas, Lauri Juvela, Tom Backstrom+1

Department of Information and Communications Engineering, Aalto University, Espoo, Finland

TTS Synthesis Controllable Speech Generation Speech Prosody

Open paper AskPDF arXiv

Generating controllable and human-like emphasis remains an open challenge in text-to-speech, even when explicit emphasis control signals are provided in the text input, limiting the communicative accuracy of synthetic speech in real-world applications. Reinforcement learning has recently shown promise for post-training TTS systems to align with human preference, yet existing methods have not been applied to word-level prosodic control. We present EmphTTS, a non-autoregressive TTS system that applies Group Relative Policy Optimization (GRPO) to the duration predictor with an emphasis localization reward, enabling direct optimization for word-level emphasis. Evaluations show that EmphTTS achieves the best emphasis controllability and performs the best in emphasis objective evaluation. In subjective preference tests, EmphTTS is significantly preferred over synthetic groundtruth and most baselines. Ablation studies show that GRPO improves emphasis realization beyond supervised-finetuning-based duration modeling and simple speaking-rate adjustment, while alleviating the mismatch between the independently trained duration predictor and TTS model.

Abstract

TTS Synthesis

Explore similar work

EmphTTS: an emphasis-control TTS with reinforcement learning

Unlocking Fine-Grained and Within-Utterance Speaking Style Control in Prompt-Based Text-to-Speech Models

Prompt-Consistency Inference for Zero-Shot Flow-Matching Text-to-Speech Models