Modern ASR systems are typically trained on large-scale pseudo-labeled, in-the-wild data spanning multiple domains. While such heterogeneous data benefit generalist models designed for broad deployment, they pose challenges for specialist models targeting specific domains: specialist models lack the capacity to learn from all available data, and one must pay closer attention to addressing the mismatch between training and test conditions. In this work, we study targeted data selection as a strategy to address these challenges, selecting relevant subsets from 100k hours of in-the-wild training data to optimize performance on target domains. We represent speech samples using embeddings that capture complementary characteristic--speaker attributes, phonetic content, and semantic meaning--and analyze how relevance and diversity along these axes when performing data selection affect downstream ASR performance. Our experiments with CTC-based Conformer models show that training on a strategically selected 5% subset can exceed the performance of models trained on the full dataset by up to 36.8% relative WER reduction on target domains.
Figures & tables
Type
Dataset
Train (h)
Dev (h)
Test (h)
Source
Granary English
102,458
–
–
Target
LibriSpeech
961
10.5
10.7
Target
CommonVoice English
1,594
27.1
26.9
Target
TED-LIUM
452
1.6
2.6
Table 1: Summary of source and target datasets used in our experiments. Granary serves as the large-scale, in-the-wild, training source data. The target datasets—LibriSpeech, CommonVoice, and TED-LIUM—cover a wide range of domains. Dataset sizes (in hours) are reported when available.
Model
Training Data
LS-clean
LS-other
CV
TED-LIUM
Conformer-Small
LS-960
5.2
12.2
45.4
18.9
CommonVoice
27.8
39.4
32.7
43.8
TED-LIUM
13.5
24.5
41.3
9.4
Granary (Full)
12.5
23.2
36.6
11.0
Granary (Random 5%)
12.5
23.5
37.1
10.7
Δ% (Full → Random 5%)
+0.0%
+1.3%
+1.4%
-2.7%
Table 2: WER (%) of Conformer models trained on different source datasets. The models are evaluated on LibriSpeech (clean/other), CommonVoice (CV), and TED-LIUM test sets. Results are reported for both Conformer-Large and Conformer-Small architectures. “Granary (Random 5%)” denotes training on a randomly selected 5% subset of Granary, while “Granary (Full)” uses the complete dataset. Lower WER indicates better performance. Best results are shown in bold , and second-best results are underlined .
Subset
Model
Approach
Embedding
LS-clean
LS-other
CV
TED-LIUM
Full
Conformer-Small
–
–
12.5
23.2
36.6
11.0
5%
Conformer-Small
Random (Baseline)
–
12.5
23.5
37.1
10.7
+ Fine-Tune
–
14.1
25.2
33.6
17.8
Duration (Baseline)
–
13.2
23.8
36.5
10.8
MMR
Speaker
10.8
20.9
34.4
9.6
WavLM
10.3
20.6
33.6
9.2
Table 3: WER (%) of Conformer-Small models trained on Granary using different 5% data selection strategies. Results are reported on LibriSpeech (clean/other), CommonVoice (CV), and TED-LIUM test sets. “Fusion” indicates the use of a multi-embedding subset selection strategy that combines all three embedding types, as described in Section 3.3 . Lower WER indicates better performance. Best results are shown in bold , and second-best results are underlined .
Subset
Approach
Embedding
LS-clean
LS-other
CV
TED-LIUM
Full
–
–
12.5
23.2
36.6
11.0
5%
Single-dataset
Speaker
10.8
20.9
34.4
9.6
Multi-dataset (Avg.)
11.8
21.9
37.9
9.6
Multi-dataset (Max)
12.6
22.1
33.8
10.3
Single-dataset
WavLM
10.3
20.6
33.6
9.2
Multi-dataset (Avg.)
10.8
20.8
37.9
9.8
Table 4: WER (%) comparing single-dataset and multi-dataset subset selection strategies using different embedding types. Single-dataset selection targets each evaluation domain independently, while multi-dataset strategies select a unified 5% subset that targets all domains simultaneously, using either average (Avg.) or maximum (Max) aggregation of relevance scores (Section 3.4 ). Models are Conformer-Small trained on 5% selected from Granary. Lower WER indicates better performance. Best results are shown in bold , and second-best results are underlined .
Model
Subset ( % )
LS-clean
LS-other
CV
TED-LIUM
Conformer-Small
100
12.5
23.2
36.6
11.0
50
12.1
22.7
35.7
10.7
20
12.3
22.8
36.5
11.0
10
12.5
23.1
36.7
10.8
5
12.5
23.1
36.7
11.2
1
12.9
24.2
38.6
11.5
Table 5: WER (%) of Conformer models trained on different randomly sampled subsets of Granary (1–100%) evaluated on LibriSpeech (clean/other), CommonVoice, and TED-LIUM test sets. Lower WER indicates better performance.
Figure 1: Validation loss curves for Conformer-Large and Conformer-Small models trained on the Granary dataset with the LibriSpeech-clean validation set as a target. Models are compared across three training data conditions: the full dataset, a random 5% subset, and an MMR-selected 5% subset. The MMR selection strategy achieves a lower validation loss than using the full dataset with only 5% .
Embedding
λ
LS-clean
LS-other
CV
TED-LIUM
Speaker
0.0
14.0
23.6
35.8
11.3
0.7
10.8
20.9
34.4
9.6
1.0
10.1
20.8
34.3
9.4
WavLM
0.0
12.9
22.2
35.3
11.4
0.7
10.3
20.6
33.6
9.2
1.0
9.6
20.7
35.0
9.2
Table 6: WER (%) on LibriSpeech (clean/other), CommonVoice (CV), and TED-LIUM test sets for different embeddings and MMR trade-off parameter values ( λ ) as described in Equation 3.2 . Models are Conformer-Small trained on 5% selected from Granary. Lower WER indicates better performance. Best results are shown in bold , and second-best results are underlined .
Source
Target
Accuracy (%)
WavLM
WavLM
76.6
Speaker
21.3
SBERT
5.9
Speaker
WavLM
18.8
Speaker
71.0
SBERT
3.7
Table 7: Cross-embedding predictability using logistic regression on 100 -cluster assignments. Accuracy is reported in percentage. Accuracy for a random classifier is around 1%. Best results are boldfaced .
MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Shanghai Jiao Tong University · School of Intelligence Science and Technology, Nanjing University, Suzhou, China