What Makes a Good Audio Prompt for Zero-Shot TTS? Learning Prompt Selection from Synthesis Outcomes

Arnon Turetzky; Yossi Adi

cs.SDOct 29, 2026

What Makes a Good Audio Prompt for Zero-Shot TTS? Learning Prompt Selection from Synthesis Outcomes

Authors: Arnon Turetzky, Yossi Adi

Organizations: The Hebrew University of Jerusalem

Abstract

Zero-shot text-to-speech (TTS) models clone a target voice from a short reference clip, commonly referred to as the prompt. In practice, the prompt has to be cut from a longer recording of the speaker, so which part to use must be chosen. To understand what makes a good prompt, we analyze hundreds of thousands of outputs synthesized by four TTS models from candidate prompts of YouTube recordings. The choice of prompt substantially changes intelligibility, naturalness and speaker similarity. Standard predicted-quality scores capture mainly the naturalness part but miss most of the intelligibility gain. We find that the models largely agree on which prompt yields the highest measured speaker similarity, but much less on which prompt yields intelligible speech. We propose a small network that learns from the outputs a TTS model produced to rank a recording's candidate prompts from their audio alone. Without synthesizing anything at selection time, it picks prompts that lower the word error rate on held-out recordings by a third (Qwen3-TTS) to two fifths (F5-TTS) relative to a randomly chosen prompt, more than predicted quality achieves and nearly as much as synthesizing a dozen test sentences from every candidate first.

Figures & tables

Figure 1: Our audio selector, a small network trained on a TTS model’s synthesis outcomes, picks prompts that lower WER more than choosing by predicted quality, without synthesizing anything. WER gain over a random prompt (pp, higher is better) on 400 held-out test recordings. Probe selection uses 12 syntheses per candidate, and Qwen3-TTS 1.7B uses the 0.6B selector and descriptor. Error bars: 95% intervals. Details in § 7 and Appendix D .

selector	$\Delta$ WER	$\Delta$ fail.	$\Delta$ UTMOS	$\Delta$ DNSMOS	$\Delta$ Audiobox	$\Delta$ sim.
UTMOS rule	0.83	0.81	0.095	0.024	0.035	-0.006
DNSMOS rule	0.33	0.54	0.038 ‡	0.032	0.019	0.027 †
best single descriptor (transcript confidence)	0.96	0.55	-0.002 ‡	0.006 ‡	-0.010 ‡	0.017 †
descriptor ranker	1.00	0.41	0.069	0.028	0.033	-0.007
audio selector (ours)	1.36 †	0.87	0.050 ‡	0.014	0.022	-0.002
+ UTMOS head ( $w=0.25$ , § 5 )	1.33 †	0.67	0.060 ‡	0.019	0.038	-0.004

Table 1: Test results for Qwen3-TTS 0.6B (400 held-out recordings, all 19 candidates of a recording in one selection).

\Delta

: improvement over a randomly chosen candidate of the same recording (

\uparrow

better; for WER and failures, the reduction in pp); Audiobox is the held-out judge. † / ‡ : better/worse than the UTMOS rule (interval of the difference above/below zero).

		random prompt				$\Delta$ with probe selection ( $\uparrow$ better)				ICC
model	prompt	WER	fail.	UTMOS	sim.	WER	UTMOS	sim.	Audiobox	WER	UTMOS	sim.
Qwen3-TTS	3 s	4.31	3.05	3.81	0.556	1.42	0.190	0.077	0.253	0.26	0.26	0.50
	5 s	4.16	2.24	3.80	0.604	1.21	0.152	0.052	0.176	0.22	0.25	0.38
	10 s	4.01	1.64	3.78	0.650	1.10	0.097	0.023	0.104	0.15	0.20	0.22
Chatterbox	3 s	2.82	0.43	3.77	0.522	0.24	0.203	0.098	0.330	0.10	0.44	0.70
	5 s	2.77	0.49	3.79	0.573	0.20	0.148	0.064	0.219	0.07	0.34	0.57

Table 2: How much the prompt matters (400 training recordings, candidates of one prompt length). With a random prompt: average over a recording’s candidates, WER and failures in % (failure: WER of at least 50% or a runaway output of 25 s or longer).

\Delta

: improvement by probe selection, in pp of WER or in metric units (UTMOS 1–5, Audiobox 1–10, similarity as a cosine). ICC: share of the output variance for one target sentence that is due to the prompt rather than the seed.

Figure 2: Intelligibility gain against naturalness gain on the test (Qwen3-TTS 0.6B), by UTMOS (left) and by the held-out Audiobox judge (right). The curve adds the audio selector’s UTMOS head with weight

w

; probe selection’s naturalness gains lie above the plotted range.

Figure 3: What probing costs (Qwen3-TTS 0.6B, test): WER gain over a random prompt of probe selection with 1–12 syntheses per candidate (19–228 per recording), with intervals, against the audio selector and the UTMOS rule, which synthesize nothing.

variant (cross-validation)	WER	$\Delta$	descr. ranker
final recipe	1.25		0.90
pointwise loss	1.18	-0.07	0.90
labels: prompt’s UTMOS	0.63	-0.62	0.55
Whisper-large-v3 encoder	1.22	-0.03	0.90
CLAP encoder	0.42	-0.82	0.90

Table 3: Selected ablations by cross-validation on the training recordings (mean of three seeds): WER gain over a random prompt (pp), its change from the final recipe, and the descriptor ranker trained with the same labels on the same folds. All ablations in Appendix B .

Figure 4: Left: transfer between TTS models, as the WER gain of a selector trained on one model (rows) for another (columns), in percent of the second model’s probe-selection gain. Right: do two models prefer the same prompts? Within-recording Spearman correlation between the candidates’ speaker similarity (above the diagonal) and WER (below) under two models, averaged over the 400 test recordings and corrected for how consistently each model ranks the candidates between two halves of its own outputs (split-half reliability; uncorrected values in Appendix D ).

prompt property	Qwen 0.6B	Qwen 1.7B	Chat- terbox	F5- TTS
UTMOS	+0.04 †	+0.05 †	+0.05 †	+0.09 †
typical voice	-0.00	-0.01	+0.05 †	+0.07 †
ends in silence	+0.16 †	+0.16 †	+0.01	+0.11 †
transcript confidence	+0.10 †	+0.10 †	+0.03 †	+0.10 †

Table 4: Prompt properties and intelligibility, per TTS model: within-recording Spearman correlation between a property of the prompt and the intelligibility of its outputs (fewer word errors = higher), averaged over the 400 test recordings. Typical voice: speaker embedding close to the recording’s mean; ends in silence: length of the trailing silence; transcript confidence: Whisper’s mean token log-probability. † : interval excludes zero. All 21 descriptors in Appendix A .

selector	WER	fail.	UTMOS	sim.	clean
UTMOS rule	0.21	-0.05	0.550	0.113	79%
audio selector	0.63	0.26	0.349	0.070	31%
+ UTMOS head	0.66	0.26	0.415	0.080	39%
probe selection	0.42	0.18	0.570	0.165

Table 5: Candidates differing in noise (LibriSpeech, Qwen3-TTS 1.7B, all lengths in one selection): gain over a random prompt, WER and failures in pp, and the share of picks that are the clean version of a prompt (by chance: 14%, one of seven conditions); the UTMOS head is added with

w=0.25

. The selectors were trained on YouTube outcomes of Qwen3-TTS 0.6B.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

set	recordings	prompts (3/5/10 s)	outputs per model	TTS models
train	400	3993 / 2389 / 1170	113,280	Qwen3-TTS, Chatterbox, F5-TTS
train	544	5429 / 3249 / 1588	153,990	Qwen3-TTS, Chatterbox, F5-TTS
dev	200	2000 / 1194 / 583	56,655	Qwen3-TTS, Chatterbox
test	200	1994 / 1194 / 591	56,685	Qwen3-TTS, Chatterbox, F5-TTS
test	200	1997 / 1194 / 588	56,685	Qwen3-TTS, Chatterbox, F5-TTS

Appendix

Table 6: Recording sets (disjoint YODAS shards): two training sets, a development set and two test sets. Outputs per model: every candidate prompt

\times

5 target sentences

\times

3 seeds.

descriptor	Qwen3-TTS 0.6B	Qwen3-TTS 1.7B	Chatterbox	F5-TTS
UTMOS	+0.04 †	+0.05 †	+0.05 †	+0.09 †
DNSMOS overall	+0.02	-0.01	+0.05 †	+0.09 †
DNSMOS signal	+0.01	+0.01	+0.05 †	+0.08 †
DNSMOS background	+0.01	-0.01	+0.05 †	+0.06 †
duration	+0.00	-0.03 †	+0.04 †	+0.04 †
words per second	-0.04 †	-0.01	-0.01	+0.00

Appendix

Table 7: All 21 prompt descriptors (Table 4 ): within-recording Spearman correlation with the prompt’s intelligibility (fewer word errors = higher), averaged over the 400 test recordings. † : interval excludes zero.

quantity	value
audio selector $-$ UTMOS rule (pre-specified)	$+0.54$ $[0.14,0.93]$
same, training seeds 1 and 2 (same recipe and data)	$+0.53$ $[-0.06,1.03]$ ; $+0.38$ $[-0.24,0.90]$
selecting by the mean-WER head instead of the WER rank (not pre-specified)	1.38
failed outputs excluded: audio selector, UTMOS rule	0.77, 0.25
wav2vec 2.0 as WER judge: audio selector, UTMOS rule	1.59, 0.88
prompt-transcript disagreement (Whisper vs. wav2vec 2.0) of the pick: audio selector, UTMOS rule, all candidates	0.23, 0.26, 0.30

Appendix

Table 8: Further test results for Qwen3-TTS 0.6B (and F5-TTS where noted).

candidates	audio selector	descriptor ranker	UTMOS rule	audio $-$ rule
all 19 candidates	1.36	1.00	0.83	+0.54 †
only 3 s prompts	1.37	1.34	0.80	+0.57 †
only 5 s prompts	0.90	0.73	0.27	+0.63 †
only 10 s prompts	0.64	0.46	0.06	+0.58
equal numbers of each length	1.20	0.92	0.34	+0.86 †

Appendix

Table 9: WER gain over a random prompt on the test (Qwen3-TTS 0.6B) for different candidate sets: all 19 candidates of a recording (10, 6 and 3 prompts of 3, 5 and 10 s), candidates of one length only, and equal numbers of each length (30 random draws). † : audio selector better than the UTMOS rule.

variant (cross-validation)	WER	change	descriptor ranker
audio selector (final recipe)	1.25		0.90
single model (no ensemble)	1.20	-0.05	0.93
pointwise loss	1.18	-0.07	0.90
no comparison with the recording’s mean	1.27	+0.03	–
attention across candidates	1.29	+0.04	–
+ transcript stream	1.15	-0.10	–

Appendix

Table 10: All ablations by cross-validation on the training recordings (folds keep same-voice recordings together; mean of three seeds): WER gain over a random prompt, its change from the final recipe (differences below about 0.1 are within the spread between seeds), and the descriptor ranker trained with the same labels on the same folds (–: the variant does not concern it).

Qwen3-TTS 0.6B
rule	WER	failures	UTMOS	DNSMOS	Audiobox	similarity
UTMOS rule	0.37	0.33	0.078	0.022	0.046	0.029
DNSMOS rule	0.18	0.43	0.065	0.033	0.034	0.023
Chatterbox
UTMOS rule	0.12	-0.10	0.085	0.028	0.053	0.024
DNSMOS rule	0.18	0.01	0.071	0.040	0.056	0.020

Appendix

Table 11: Quality rules on 400 training recordings, among the 3 s candidates: gain over a random prompt (probe selection in Table 2 ).

TTS model	UTMOS rule	DNSMOS rule	best descriptor	audio selector	probe selection
Qwen 0.6B	0.83	0.33	0.96	1.36 a,b	1.35
Qwen 1.7B ∗	0.28	-0.25	0.44	1.49 a,b,c	1.86
Chatterbox	0.03	-0.01	0.13	0.25 b	0.30
F5-TTS	0.97	1.19	0.66	2.64 a,b,c	3.00

Appendix

Table 12: WER gain over a random prompt on the 400 test recordings (Figure 1 ). The best descriptor is chosen per model on its training recordings ( ∗ : Qwen3-TTS 1.7B has no training outcomes and uses the 0.6B choice and selector). Superscripts on the selector: better than the UTMOS rule (a), the DNSMOS rule (b) or the best descriptor (c).

	UTMOS		Audiobox		similarity
TTS model, selector	selector	rule	selector	rule	selector	rule
Qwen3-TTS 1.7B, selector trained on 0.6B	0.054 ‡	0.092	0.011	0.038	-0.000	-0.005
Chatterbox, own selector	0.046	0.069	0.029	0.048	0.005 †	-0.008
Chatterbox, selector trained on Qwen 0.6B	0.039 ‡	0.069	-0.003 ‡	0.048	-0.005	-0.008
F5-TTS, own selector	0.052 ‡	0.103	0.030	0.045	0.000	-0.009
F5-TTS, selector trained on Qwen 0.6B	0.021 ‡	0.103	0.006	0.045	-0.001	-0.009

Appendix

Table 13: Other TTS models on the 400 test recordings: gain over a random prompt of the audio selector and the UTMOS rule on naturalness and similarity (WER in Table 12 and § D ). † / ‡ : selector better/worse than the rule.

Zero-Shot TTS

Zero-shot text-to-speech (TTS) generates speech in a new voice from brief reference audio without speaker-specific training.

See all 62 papers

Long-Prompt Zero-Shot TTS: Benchmarking Prosody Reproduction Beyond Voice Cloning
Oct 14, 2026Dekun Chen, Renjie Xu, Yingda Shen +2Zero-Shot TTS Speech Prosody
The Chinese University of Hong Kong, Shenzhen · Amphion Technology Co., Ltd.
VoiceTTA: Enhancing Zero-Shot Text-to-Speech via Reinforcement Learning-Based Test-Time Adaptation
Jun 25, 2026Tianxin Xie, Chenxing Li, Dong Yu +1Test-Time Adaptation Reinforcement Learning
The Hong Kong University of Science and Technology (Guangzhou) · Tencent
Prompt-Consistency Inference for Zero-Shot Flow-Matching Text-to-Speech Models
Oct 3, 2026Vasily Zadorozhnyy, Can Goksen, Kazuhito Koishida +1TTS Synthesis Zero-Shot TTS
Applied Sciences Group, Microsoft Corporation, Redmond, USA

Oct 14, 2026J/K move · Enter open · S save

Long-Prompt Zero-Shot TTS: Benchmarking Prosody Reproduction Beyond Voice Cloning

Dekun Chen, Renjie Xu, Yingda Shen, Qinke Ni+1

The Chinese University of Hong Kong, Shenzhen · Amphion Technology Co., Ltd.

Zero-Shot TTS Speech Prosody TTS Evaluation Speech Generation Evaluation

Open paper AskPDF arXiv

Zero-shot text-to-speech (TTS) typically conditions on a short speech prompt, which conveys timbre but provides only limited evidence of a speaker's prosodic patterns. We formulate Long-Prompt Zero-Shot TTS, a task that uses long speech prompts at inference time to reproduce speaker-specific prosody without target-speaker fine-tuning. We introduce LP-TTS-Bench, a bilingual benchmark of 37 speakers with 63.95 prompt hours and 2,728 held-out test utterances. Beyond intelligibility and timbre similarity, LP-TTS-Bench evaluates timing, prominence, intonation, and model-based prosody metrics, enabling multidimensional evaluation of prosody reproduction. As a reference system, we develop LPTTS, whose hierarchical Long-Prompt Encoder compresses long speech prompts into a fixed number of continuous prosody vectors. Experiments show improvements in several prosodic dimensions over a short-prompt baseline, with overall gains as prompt duration increases up to approximately 10 minutes and saturation at longer durations.

Abstract

Zero-Shot TTS

Explore similar work

Long-Prompt Zero-Shot TTS: Benchmarking Prosody Reproduction Beyond Voice Cloning

VoiceTTA: Enhancing Zero-Shot Text-to-Speech via Reinforcement Learning-Based Test-Time Adaptation

Prompt-Consistency Inference for Zero-Shot Flow-Matching Text-to-Speech Models