Reliable evaluation of modern zero-shot text-to-speech (TTS) models remains challenging. Subjective tests are costly and hard to reproduce, while objective metrics often saturate, failing to distinguish SOTA systems. To address this, we propose Iterate to Differentiate (I2D), an evaluation framework that recursively synthesizes speech using the model's own outputs as references. Higher-quality models exhibit greater resilience to the distributional shift induced by iterative synthesis, resulting in slower performance degradation. I2D exploits this differential degradation to amplify performance gaps and reveal robustness. By aggregating objective metrics across iterations, I2D improves discriminability and alignment with human judgments, increasing system-level SRCC from 0.118 to 0.464 for UTMOSv2. Experiments on 11 models across Chinese, English, and emotion datasets demonstrate that I2D enables more reliable automated evaluation for zero-shot TTS.
Figures & tables
Figure 1: The overall workflow of our evaluation. The dashed arrows indicate that, after the first synthesis, the reference wav and reference text are updated using the target wav and target text. The objective metrics EMO_F1 are applied only to the Emotion dataset.
Type
Metrics / Dimensions
Objective
WER/CER
SIM
DNSMOS
UTMOSv2
Emotion F1
Subjective
Content Accuracy
Table 1: Objective evaluation metrics and subjective evaluation dimensions
Figure 2: SRCC between objective metrics and subjective dimensions at utterance and system levels (1st and 10th iterations). Utterance-level SRCC is calculated from all individual sample scores, while system-level is derived from model rankings. Specifically, we compare SIM with Spk. Consistency, 1-CER with Content Acc., and UTMOSv2/DNSMOS with Naturalness.
Figure 3: Bar chart of objective metrics for all models on Chinese dataset at the 1st and 10th iterations. The dashed lines indicate the values for real audio.
Architecture
Model
AR
FireRedTTS2 [ 22 ]
Qwen3-TTS [ 4 ]
VoxCPM1.5 [ 34 ]
NAR
F5-TTS [ 20 ]
MaskGCT [ 21 ]
Hybrid
CosyVoice [ 35 ]
Table 2: Evaluated TTS Models and Architectures
Score Method
SIM
1−CER
Iter1(baseline)
0.6818
0.5103
Mean
0.6818
0.4829
LWA
0.6818
0.5194
EWA
0.7273
0.4282
AUC
0.6818
0.4282
Score Method
UTMOSv2
DNSMOS
Table 3: System-level SRCC between aggregated metrics and human evaluation on first-iteration synthesized speech. The correlation of objective metrics computed on first-iteration outputs is included as the baseline.
TTS Models
Objective Metrics (en/zh)
Subjective Metrics (iter1 / iter10)
WER / CER ↓
SIM ↑
UTMOSv2 ↑
Content Acc. ↑
Spk. Consistency ↑
Naturalness ↑
CosyVoice
8.88 / 11.18
19.34 / 23.70
2.98 / 2.76
4.75 / 3.25
3.48 / 1.01
3.93 / 1.60
CosyVoice2
2.77 / 1.51
61.34 / 56.88
3.80 / 3.22
4.84 / 4.62
4.45 / 1.53
4.08 / 2.95
CosyVoice3
2.36 / 1.44
66.88 / 62.14
3.82 / 3.33
4.80 / 4.64
4.55 / 1.90
4.04 / 3.18
CosyVoice3-RL
2.27 / 0.90
67.32 / 62.17
3.83 / 3.33
4.84 / 4.73
4.53 / 1.98
4.07 / 3.16
F5-TTS
17.40 / 5.33
48.74 / 49.23
3.09 / 2.92
4.79 / 2.78
4.49 / 1.05
4.10 / 1.43
Table 4: Objective and subjective evaluation results of different TTS models. Objective metrics are aggregated using Mean Score (Mean) and reported as en/zh . Subjective metrics are reported at the first and tenth iterations (iter1 / iter10). Bold denotes the best result in each sub-column. Underline denotes the second-best result in each sub-column.
Figure 4: System-level SRCC between aggregated UTMOSv2 scores and human ratings of first-iteration speech, computed under different maximum iteration numbers.
TTS Models
Angry
Happy
Sad
Weighted Avg
GT
75.10
85.70
78.20
79.70
CosyVoice
30.85
30.21
57.29
39.57
CosyVoice2
50.69
54.93
47.19
50.94
CosyVoice3
59.69
60.91
46.30
55.65
CosyVoice3-RL
63.87
53.51
34.00
50.47
F5-TTS
52.47
56.14
34.88
47.83
Table 5: Emotion classification F1-score (%) aggregated using the Mean Score strategy on the emotion dataset for different TTS models.
Figure 5: Cross-model iterative evaluation curves of CosyVoice3-RL and F5-TTS. Solid lines denote the original trajectories without reference swapping, while dashed lines represent the trajectories after exchanging reference audio at the 6th iteration.
Recently, zero-shot text-to-speech (TTS) has enabled high-fidelity and expressive speech synthesis, but it often fails to imitate unseen speaking styles from uncommon scenarios (e.g., crosstalk, dialects). Moreover, fine-tuning pretrained models requires large, high-quality datasets, limiting rapid personalization. We propose VoiceTTA, a reinforcement learning-based test-time adaptation (TTA) method that improves voice imitation of pretrained zero-shot TTS models. VoiceTTA introduces two style rewards based on coefficient-of-variation differences of F0 and energy, combined with speaker similarity and intelligibility (WER from a pretrained Whisper model), and optimizes learnable prefixes via group relative preference optimization (GRPO) in a flow matching-based model at inference time. Extensive experiments demonstrate substantial improvements on uncommon speech prompts, outperforming state-of-the-art baselines. Audio samples are available at https://voicetta.pages.dev/
Tianxin Xie, Chenxing Li, Dong Yu +1
The Hong Kong University of Science and Technology (Guangzhou) · Tencent
Voice reconstruction using Text-to-Speech (TTS) offers a communication method for people with speech disorders, which aims to retain their speaker identity while improving intelligibility. Previous work generally relies on Mean Opinion Score (MOS) to evaluate naturalness and speaker similarity, but this has limited sensitivity and reliability. We propose an evaluation framework with subjective and objective components. Subjectively, we evaluate perceived intelligibility and speaker identity using Best Worst Scaling (BWS) with situational framing. Objectively, we demonstrate that standard measures fail to predict reconstruction success for highly unintelligible speakers, so we introduce a novel dual-reference distributional measure to assess the trade-off between intelligibility and speaker identity. By evaluating the output of 17 zero-shot TTS systems for 193 speakers, we show that our framework provides a reliable and task-aligned approach for assessing voice reconstruction.
Ariadna Sanchez, Christoph Minixhofer, Korin Richmond +3
The Centre for Speech Technology Research, University of Edinburgh, UK
Zero-shot text-to-speech (TTS) typically conditions on a short speech prompt, which conveys timbre but provides only limited evidence of a speaker's prosodic patterns. We formulate Long-Prompt Zero-Shot TTS, a task that uses long speech prompts at inference time to reproduce speaker-specific prosody without target-speaker fine-tuning. We introduce LP-TTS-Bench, a bilingual benchmark of 37 speakers with 63.95 prompt hours and 2,728 held-out test utterances. Beyond intelligibility and timbre similarity, LP-TTS-Bench evaluates timing, prominence, intonation, and model-based prosody metrics, enabling multidimensional evaluation of prosody reproduction. As a reference system, we develop LPTTS, whose hierarchical Long-Prompt Encoder compresses long speech prompts into a fixed number of continuous prosody vectors. Experiments show improvements in several prosodic dimensions over a short-prompt baseline, with overall gains as prompt duration increases up to approximately 10 minutes and saturation at longer durations.
Dekun Chen, Renjie Xu, Yingda Shen +2
The Chinese University of Hong Kong, Shenzhen · Amphion Technology Co., Ltd.