Whisper generation is constrained by the difficulty of data collection. Because whispered speech has low acoustic amplitude, high-fidelity recording is challenging. In this paper, we introduce WhispSynth, a large-scale multilingual corpus constructed via a novel high-fidelity generative framework. Specifically, we propose a pipeline integrating Differentiable Digital Signal Processing (DDSP)-based pitch-free method with Text-to-Speech (TTS) models. This framework refines a comprehensive collection of resources, including our newly constructed WhispNJU dataset, into 118 hours of high-fidelity whispered speech from 479 speakers. Unlike standard synthetic or noisy real data, our data engine faithfully preserves source vocal timbre and linguistic content while ensuring acoustic consistency, providing a robust foundation for text-to-whisper research. Experimental results demonstrate that WhispSynth exhibits significantly higher quality than existing corpora. Moreover, our CosyWhisper, tuned with WhispSynth, achieves speech naturalness on par with ground-truth samples. The official implementation and related resources are available at https://github.com/tan90xx/cosywhisper.
Figures & tables
Figure 1: Different Dynamic Range. Whispers exhibit a significantly lower sound pressure level compared to normal speech, even when the linguistic content is identical.
Dataset
Source
Sample Rate
Size ( h )
Avg ( s )
Language
Speaker
P?
CC Type
UTVE-I
Zhang and Hansen (2010)
44100
<1
N.A.
en
12
N
N.A.
UTVE-II
Ghaffarzadegan et al. (2014)
44100
1
N.A.
en
112
N
N.A.
AVWD
Zhou et al. (2019)
44100
<2.44
N.A.
zh
10
Y
N.A.
Whi-spe
Grozdić et al. (2012)
22050
<5
N.A.
sr
10
Y
N.A.
AV-Whisper
Tran et al. (2013)
48000
<10
N.A.
en
11
Y
N.A.
CIAIR
Kawaguchi et al. (2002)
16000
15
N.A.
ja
123
Y
N.A.
Table 1: Overview of datasets used in this work, with the highlighted (gray) rows indicating all obtainable real whisper corpora included in our study (up to 2025-01-05). Entries include data duration (Size), average utterance length (Avg), language, speaker count, whether the paired normal segments are available (P?), and license type (CC Type), “Mixed” indicates multiple license terms or sample rate, “N.A.” denotes unknown or unspecified.
Figure 2: Visualization of the WhispSynth’s generation pipeline. We apply CosyVoice3 and DDSP Model to generate pitch-free voice.
en
zh
# Train
# Test
# Train
# Test
File Count
20694
4770
32704
7256
Size (h)
29.62
6.91
65.86
15.25
Avg (s)
5.15
5.21
7.25
7.57
Speaker Count
78M 87F
15M 15F
206M 194F
26M 25F
Table 2: The statistics of the WhispReal dataset train/test distribution.
Dataset
Source
Language
Naturalness ↑
Intelligibility ↓
F0 ↓
DNSMOS
UTMOS
CER / WER (%)
VTR(%)
wTIMIT
Li-Li et al. (2005)
en
2.76
1.31
50.99
0.69
CHAINs ( subset )
Cummins et al. (2006)
en
2.76
1.48
11.55
0.95
Expresso ( subset )
Nguyen et al. (2023)
en
3.24
1.50
7.77
0.58
EARs ( subset )
Richter et al. (2024)
en
3.43
2.01
6.31
0.30
Whisper40
Yang and Zhou (2024)
zh
2.61
1.27
72.80
0.99
Table 3: Objective evaluation results on open-sourced and our proposed datasets. Comprehensive evaluation on whisper speech conversion and synthesis. Quality : Subjective naturalness (DNSMOS/UTMOS ↑ ). Intelligibility : Character/Word Error Rate ( ↓ ). Pitch : Voiced Time Ratio (VTR) ( ↓ ).
Figure 3: Results of the listening test. Whisper-likeness Mean Opinion Score (W-MOS) based on 20 participants visualized in a standard box plot.
Dataset
Naturalness ↑
Intelligibility ↓
Timbre ↑
F0 ↓
DNSMOS
UTMOS
CER / WER (%)
Cosine
VTR(%)
WhispReal
2.93
1.33
28.3/46.5
0.75
0.77
WhispSynth
3.08
1.48
12.8/29.2
0.80
0.70
Table 5: Performance comparison of our CosyWhisper fine-tuned on WhispReal versus WhispSynth.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: User interface of the listening test.
Language
Method
Evaluation Metrics
W-MOS ↑
UTMOS ↑
DNSMOS ↑
MCD ↓
CER ↓
SpkSim ↑
VTR ↓
Korean
CosyVoice3
3.35 ± 1.70
1.72
2.84
62.07
27.21
0.50
0.83
CosyWhisper (Ours)
3.96 ± 1.04
1.49
2.84
67.67
23.32
0.53
0.75
Japanese
CosyVoice3
2.88 ± 1.36
1.49
2.88
61.41
39.42
0.54
0.96
CosyWhisper (Ours)
4.03 ± 0.58
1.38
2.89
69.33
36.22
0.52
0.82
Appendix
Table 6: Evaluation of whisper speech synthesis on Korean and Japanese.
Figure 5: Sample spectrograms from two datasets. (a-f) English samples from the CHAINs Dataset. (g-l) Chinese samples from the WhispNJU Dataset. (a,g) CosyWhisper; (b,h) CosyVoice3 Finetuned on WhispReal; (c,i) CosyVoice3 Repeat; (d,j) WhispSynth; (e,k) WhispReal; (f,l) Normal Speech.