Long-Prompt Zero-Shot TTS: Benchmarking Prosody Reproduction Beyond Voice Cloning

Dekun Chen; Renjie Xu; Yingda Shen; Qinke Ni; Zhizheng Wu

cs.SDOct 14, 2026

Long-Prompt Zero-Shot TTS: Benchmarking Prosody Reproduction Beyond Voice Cloning

Authors: Dekun Chen, Renjie Xu, Yingda Shen, Qinke Ni, Zhizheng Wu

Organizations: The Chinese University of Hong Kong, Shenzhen · Amphion Technology Co., Ltd.

Abstract

Zero-shot text-to-speech (TTS) typically conditions on a short speech prompt, which conveys timbre but provides only limited evidence of a speaker's prosodic patterns. We formulate Long-Prompt Zero-Shot TTS, a task that uses long speech prompts at inference time to reproduce speaker-specific prosody without target-speaker fine-tuning. We introduce LP-TTS-Bench, a bilingual benchmark of 37 speakers with 63.95 prompt hours and 2,728 held-out test utterances. Beyond intelligibility and timbre similarity, LP-TTS-Bench evaluates timing, prominence, intonation, and model-based prosody metrics, enabling multidimensional evaluation of prosody reproduction. As a reference system, we develop LPTTS, whose hierarchical Long-Prompt Encoder compresses long speech prompts into a fixed number of continuous prosody vectors. Experiments show improvements in several prosodic dimensions over a short-prompt baseline, with overall gains as prompt duration increases up to approximately 10 minutes and saturation at longer durations.

Figures & tables

Figure 1: Illustration of Long-Prompt Zero-Shot TTS . Compared with a short prompt, a long prompt provides richer evidence of speaker-specific prosody, enabling the model to reproduce both timbre and prosody without target-speaker fine-tuning.

Figure 2: Overview of the LPTTS system. The Long-Prompt Encoder extracts global and target-conditioned prosodic representations from long speech prompt and injects them into Qwen3TTS by gated cross-attention. Timbre conditioning is provided by the native speaker encoder.

Chinese
Model	Target Spk. SFT?	Conventional		Timing		Prominence	Intonation		Prosody model		Subjective
Model		WER/CER $\downarrow$	SIM-O $\uparrow$	SRD $\downarrow$	PD $\downarrow$	PCC $\uparrow$	IC $\uparrow$	PRD $\downarrow$	MPM-SIM $\uparrow$	AutoPCP $\uparrow$	P-MOS $\uparrow$
Qwen3TTS-Base	$\times$	0.73	0.56	0.155	0.074	0.389	0.437	5.017	0.721	3.128	2.941
Qwen3TTS-SFT	$\checkmark$	1.78	0.57	0.139	0.065	0.395	0.463	4.764	0.733	3.216	3.471
LPTTS-Base	$\times$	1.29	0.58	0.155	0.069	0.387	0.437	5.013	0.728	3.181	3.235
LPTTS	$\times$	1.03	0.57	0.137	0.067	0.410	0.474	4.532	0.734	3.148	3.529

Table 1: Main results on LP-TTS-Bench. WER/CER is reported in percent. Arrows indicate the preferred direction. The checkmark indicates target-speaker fine-tuning. Bold and underlining denote the best and second-best results within each language, respectively; tied values share the same formatting.

Figure 3: Effect of prompt length on intelligibility, speaker similarity, and prosody reproduction for Chinese. English results show similar trends and are omitted due to space limitations.

Feature	$\Delta$ [95% CI]	Closer Rate
Pitch	$-0.054$ [ $-0.061$ , $-0.040$ ]	60.2%
Energy	$-0.047$ [ $-0.060$ , $-0.036$ ]	56.4%
Timing/pause	$-0.065$ [ $-0.082$ , $-0.063$ ]	59.9%
Overall	$-0.059$ [ $-0.067$ , $-0.053$ ]	62.6%

Table 2: Relationship between textual and prosodic similarity. Negative

\Delta

favors textually similar prompts.

Zero-Shot TTS

Zero-shot text-to-speech (TTS) generates speech in a new voice from brief reference audio without speaker-specific training.

See all 59 papers

Dynamic Prosody Prediction in LLM-based TTS for Improving Speaker Similarity
Jun 13, 2026Zhenwei Mou, Liping Chen, Yajun Hu +3TTS Synthesis Speech Prosody
University of Science and Technology of China, Anhui, China · iFLYTEK, Anhui, China
VoiceTTA: Enhancing Zero-Shot Text-to-Speech via Reinforcement Learning-Based Test-Time Adaptation
Jun 25, 2026Tianxin Xie, Chenxing Li, Dong Yu +1Test-Time Adaptation Reinforcement Learning
The Hong Kong University of Science and Technology (Guangzhou) · Tencent
Taming Long-form Text-to-Speech
Sep 15, 2026Rongxiang Wang, Berkin Durmus, Aysegul Orhon +2TTS Synthesis TTS Evaluation
Argmax, Inc. · University of Virginia · Bilkent University

Jun 13, 2026J/K move · Enter open · S save

Dynamic Prosody Prediction in LLM-based TTS for Improving Speaker Similarity

Zhenwei Mou, Liping Chen, Yajun Hu, Zhen-Hua Ling+2

University of Science and Technology of China, Anhui, China · iFLYTEK, Anhui, China

TTS Synthesis Speech Prosody Voice Cloning

Open paper AskPDF arXiv

Personalized text-to-speech (TTS) aims to clone the target speaker in the synthesized speech, imitating both the voice and speaking style. Current large language model (LLM)-based TTS methods ignore the style-specific prosodic patterns in generated speech, resulting in deficient style learning and thus limiting speaker similarity in synthesized speech. To this end, we investigate the prosody learning conditioned on the synthesized speech, and propose to predict the prosody of the current syllable based on previously predicted speech. Experimental results obtained on three datasets demonstrated the efficacy of the proposed dynamic prosody prediction method in enhancing the prosody learning capability, thereby improving the speaker similarity of the generated speech. Audio samples are available at https://muzw.github.io/dynapros/.

Abstract

Zero-Shot TTS

Explore similar work

Dynamic Prosody Prediction in LLM-based TTS for Improving Speaker Similarity

VoiceTTA: Enhancing Zero-Shot Text-to-Speech via Reinforcement Learning-Based Test-Time Adaptation

Taming Long-form Text-to-Speech