Automatic Speech Recognition (ASR) performance is heavily dependent on the availability of large-scale, high-quality datasets. For low-resource languages, existing open-source ASR datasets often suffer from insufficient quality and inconsistent annotation, hindering the development of robust models. To address these challenges, we propose a novel and generalizable data aggregation and preprocessing pipeline designed to construct high-quality ASR datasets from diverse, potentially noisy, open-source sources. Our pipeline incorporates rigorous processing steps to ensure data diversity, balance, and the inclusion of crucial features like word-level timestamps. We demonstrate the effectiveness of our methodology by applying it to Vietnamese, resulting in a unified, high-quality 500-hour dataset that provides a foundation for training and evaluating state-of-the-art Vietnamese ASR systems. Our project page is available at https://github.com/qualcomm-ai-research/PhoASR.
Figures & tables
Figure 1: Overview of our pipeline for building a high-quality ASR dataset from multiple sources. The figure uses an English example for illustration. We apply this pipeline to Vietnamese. But, it is adaptable to other languages.
Dataset
Sample Rate (Hz)
Speech Type
Domain
Text Length
Duration
License
VSV-1100
16000
Spontaneous
General
3–43 | 14
2–8 | 4
Apache 2.0
viVoice
24000
Spontaneous
General
1–60 | 16
1–19 | 4
CC-BY-NC-SA 4.0
BUD500
16000
Spontaneous
General
6–32 | 10
1–8 | 3
CC-BY-NC-SA 4.0
VLSP2020
16000
Spontaneous
Unknown
1–132 | 17
1–30 | 5
Custom
Vietnam-Celeb
16000
Spontaneous
Interviews, Shows
W/o transcripts
1–30 | 8
Custom
ViMD
44100
Spontaneous
Broadcast News
6–132 | 62
2–30 | 19
CC-BY-NC-ND 4.0
Table 1: Characteristics of the source datasets. "Text Length" and "Duration" are accounted for "words" and "seconds", respectively, with "min-max | mean" values. All datasets permit research use.
Dataset
Original
Sampled
PhoASR
Train.
Valid.
Test
PhoASR-3100h
VSV-1100
1144.53
48.85
25.69
25.69
N/A
N/A
1144.52
viVoice
1016.97
89.41
33.42
33.42
N/A
N/A
997.07
BUD500
461.90
45.24
20.61
20.55
0.03
0.03
456.58
VLSP2020
261.83
261.83
205.87
196.48
2.06
7.33
258.91
Vietnam-Celeb
187.37
98.95
37.37
37.37
N/A
N/A
62.58
ViMD
102.56
97.57
61.60
53.13
6.54
1.93
98.10
Table 2: Dataset sizes (in hours) at different stages of the pipeline and the final split distribution. "Train." and "Valid." refer to the Training and Validation splits of PhoASR, respectively. N/A indicates that the validation or test split is not available for a given dataset. "PhoASR-3100h" denotes the extended training set, which combines minimally processed data with the high-quality data from the PhoASR training split (see details in Section 4.2 ).
Region
Training
Validation
Test
Total
North
266.17
11.48
8.52
286.17
South
138.09
4.01
4.37
146.47
Central
64.34
3.21
2.48
70.03
Total
468.60
18.70
15.37
502.67
Table 3: Our dataset distribution by region and split. From this point onward, ‘64h’ and ‘469h’ refer to 64.34 hours and 468.6 hours of audio , respectively.
Dataset
North
Central
South
Overall
O-WER
64h-north
19.89
26.16
20.85
21.08
64h-central
21.66
25.40
20.68
22.16
64h-south
21.57
26.16
19.17
21.72
192h
15.88
19.86
15.81
16.53
N-WER
64h-north
16.32
22.46
17.22
17.54
64h-central
17.62
22.10
16.99
18.27
Table 4: WER scores (%) for regional subsets when fine-tuning whisper-small with 40 training epochs.
Ratio
O-WER ↓
N-WER ↓
F1 ↑
IoU ↑
0%
19.21
15.79
75.83
49.26
25%
22.66
19.23
76.42
49.65
50%
20.40
16.84
76.72
51.25
75%
22.16
18.38
79.45
54.84
100%
21.88
17.59
80.61
57.21
Table 5: Performance results when fine-tuning whisper-small with different timestamp ratios (%) on 100h training subset for 40 training epochs.
Model
O-WER ↓
N-WER ↓
F1 ↑
IoU ↑
whisper-small (Pre-trained)
70.16
64.07
-
-
ChunkFormer-large-vi (25K)
32.43
6.89
-
-
PhoWhisper-small (844h)
33.90
8.97
-
-
wav2vec2 (469h)
51.15
14.10
-
-
PhoASR-whisper-small-469h
12.46
8.69
76.40
55.62
PhoASR-whisper-small-3100h
11.70
8.20
83.68
57.39
Table 6: Obtained scores for different models: "PhoASR-whisper-small-469h" trained for 40 epochs; "PhoASR-whisper-small-3100h" trained for 15 epochs. See Appendix B for results on each component dataset. ChunkFormer was trained on an internal non-public dataset of 25K audio hours for 200 epochs. PhoWhisper was trained on a dataset of 844 audio hours, in which 586 hours are private data.
Metric
Model (Training Hours)
BUD500
VLSP2020
ViMD
LSVSC
FOSD
VietMed-L
VIVOS
CMV-vi-14
Overall
O-WER
ChunkFormer-large-vi (25K)
18.22
38.25
13.13
14.16
18.9
14.63
15.38
22.44
32.43
PhoWhisper-small (844h)
19.98
39.77
13.95
15.32
21.0
16.58
16.54
23.87
33.90
wav2vec2 (469h)
21.76
60.89
17.44
19.45
28.79
22.23
33.45
31.04
51.15
PhoASR-whisper-small-469h
8.27
14.25
8.75
6.72
7.73
14.41
5.92
9.18
12.46
PhoASR-whisper-small-3100h
4.22
13.64
8.08
6.4
5.53
15.46
3.06
7.02
11.7
N-WER
ChunkFormer-large-vi (25K)
3.03
8.71
4.46
1.19
0.79
5.95
1.37
2.97
6.89
Table 7: O-WER (%), N-WER (%), F1 (%), IoU (%) for different models on each dataset within our refined test set.