Modern ASR systems are typically trained on large-scale pseudo-labeled, in-the-wild data spanning multiple domains. While such heterogeneous data benefit generalist models designed for broad deployment, they pose challenges for specialist models targeting specific domains: specialist models lack the capacity to learn from all available data, and one must pay closer attention to addressing the mismatch between training and test conditions. In this work, we study targeted data selection as a strategy to address these challenges, selecting relevant subsets from 100k hours of in-the-wild training data to optimize performance on target domains. We represent speech samples using embeddings that capture complementary characteristic--speaker attributes, phonetic content, and semantic meaning--and analyze how relevance and diversity along these axes when performing data selection affect downstream ASR performance. Our experiments with CTC-based Conformer models show that training on a strategically selected 5% subset can exceed the performance of models trained on the full dataset by up to 36.8% relative WER reduction on target domains.
Figures & tables
Type
Dataset
Train (h)
Dev (h)
Test (h)
Source
Granary English
102,458
–
–
Target
LibriSpeech
961
10.5
10.7
Target
CommonVoice English
1,594
27.1
26.9
Target
TED-LIUM
452
1.6
2.6
Table 1: Summary of source and target datasets used in our experiments. Granary serves as the large-scale, in-the-wild, training source data. The target datasets—LibriSpeech, CommonVoice, and TED-LIUM—cover a wide range of domains. Dataset sizes (in hours) are reported when available.
Model
Training Data
LS-clean
LS-other
CV
TED-LIUM
Conformer-Small
LS-960
5.2
12.2
45.4
18.9
CommonVoice
27.8
39.4
32.7
43.8
TED-LIUM
13.5
24.5
41.3
9.4
Granary (Full)
12.5
23.2
36.6
11.0
Granary (Random 5%)
12.5
23.5
37.1
10.7
Δ% (Full → Random 5%)
+0.0%
+1.3%
+1.4%
-2.7%
Table 2: WER (%) of Conformer models trained on different source datasets. The models are evaluated on LibriSpeech (clean/other), CommonVoice (CV), and TED-LIUM test sets. Results are reported for both Conformer-Large and Conformer-Small architectures. “Granary (Random 5%)” denotes training on a randomly selected 5% subset of Granary, while “Granary (Full)” uses the complete dataset. Lower WER indicates better performance. Best results are shown in bold , and second-best results are underlined .
Subset
Model
Approach
Embedding
LS-clean
LS-other
CV
TED-LIUM
Full
Conformer-Small
–
–
12.5
23.2
36.6
11.0
5%
Conformer-Small
Random (Baseline)
–
12.5
23.5
37.1
10.7
+ Fine-Tune
–
14.1
25.2
33.6
17.8
Duration (Baseline)
–
13.2
23.8
36.5
10.8
MMR
Speaker
10.8
20.9
34.4
9.6
WavLM
10.3
20.6
33.6
9.2
Table 3: WER (%) of Conformer-Small models trained on Granary using different 5% data selection strategies. Results are reported on LibriSpeech (clean/other), CommonVoice (CV), and TED-LIUM test sets. “Fusion” indicates the use of a multi-embedding subset selection strategy that combines all three embedding types, as described in Section 3.3 . Lower WER indicates better performance. Best results are shown in bold , and second-best results are underlined .
Subset
Approach
Embedding
LS-clean
LS-other
CV
TED-LIUM
Full
–
–
12.5
23.2
36.6
11.0
5%
Single-dataset
Speaker
10.8
20.9
34.4
9.6
Multi-dataset (Avg.)
11.8
21.9
37.9
9.6
Multi-dataset (Max)
12.6
22.1
33.8
10.3
Single-dataset
WavLM
10.3
20.6
33.6
9.2
Multi-dataset (Avg.)
10.8
20.8
37.9
9.8
Table 4: WER (%) comparing single-dataset and multi-dataset subset selection strategies using different embedding types. Single-dataset selection targets each evaluation domain independently, while multi-dataset strategies select a unified 5% subset that targets all domains simultaneously, using either average (Avg.) or maximum (Max) aggregation of relevance scores (Section 3.4 ). Models are Conformer-Small trained on 5% selected from Granary. Lower WER indicates better performance. Best results are shown in bold , and second-best results are underlined .
Model
Subset ( % )
LS-clean
LS-other
CV
TED-LIUM
Conformer-Small
100
12.5
23.2
36.6
11.0
50
12.1
22.7
35.7
10.7
20
12.3
22.8
36.5
11.0
10
12.5
23.1
36.7
10.8
5
12.5
23.1
36.7
11.2
1
12.9
24.2
38.6
11.5
Table 5: WER (%) of Conformer models trained on different randomly sampled subsets of Granary (1–100%) evaluated on LibriSpeech (clean/other), CommonVoice, and TED-LIUM test sets. Lower WER indicates better performance.
Figure 1: Validation loss curves for Conformer-Large and Conformer-Small models trained on the Granary dataset with the LibriSpeech-clean validation set as a target. Models are compared across three training data conditions: the full dataset, a random 5% subset, and an MMR-selected 5% subset. The MMR selection strategy achieves a lower validation loss than using the full dataset with only 5% .
Embedding
λ
LS-clean
LS-other
CV
TED-LIUM
Speaker
0.0
14.0
23.6
35.8
11.3
0.7
10.8
20.9
34.4
9.6
1.0
10.1
20.8
34.3
9.4
WavLM
0.0
12.9
22.2
35.3
11.4
0.7
10.3
20.6
33.6
9.2
1.0
9.6
20.7
35.0
9.2
Table 6: WER (%) on LibriSpeech (clean/other), CommonVoice (CV), and TED-LIUM test sets for different embeddings and MMR trade-off parameter values ( λ ) as described in Equation 3.2 . Models are Conformer-Small trained on 5% selected from Granary. Lower WER indicates better performance. Best results are shown in bold , and second-best results are underlined .
Source
Target
Accuracy (%)
WavLM
WavLM
76.6
Speaker
21.3
SBERT
5.9
Speaker
WavLM
18.8
Speaker
71.0
SBERT
3.7
Table 7: Cross-embedding predictability using logistic regression on 100 -cluster assignments. Accuracy is reported in percentage. Accuracy for a random classifier is around 1%. Best results are boldfaced .
End-to-end speech-to-text retrieval is becoming an important capability for voice agents, allowing spoken queries to retrieve relevant documents without intermediate transcription. Training these retrievers typically depends on large collections of spoken query-document pairs that are costly to acquire and difficult to extend across domains and languages. We introduce \ours, which replaces such task-specific supervision with speech-transcript pairs from standard ASR corpora. A lightweight adapter projects the output of a frozen speech encoder into the retrieval space of a frozen pretrained text embedding model. \ours is competitive with cascaded ASR-then-retrieve systems while reducing inference latency, and outperforms existing publicly available speech embedding models. It also transfers to text-to-speech and speech-to-speech retrieval without additional training. Controlled analyses validate the speech-transcript alignment paradigm, demonstrating robustness to reduced training data and consistent gains from larger corpora. Further ablations of loss functions and trainable modules establish practical guidelines for the framework. Model weights and codes are to be released.
Sangkwon Park, Dongwook Lee, Youngho Cho +3
ECE, Seoul National University · IPAI, Seoul National University · University of Seoul +1
Prompt-based contextual ASR requires selecting a compact hotword subset from large dynamic inventories, yet existing pre-selection methods rely either on discrete ASR hypotheses that may discard acoustic evidence or on separately trained retrieval models. We propose Viterbi-guided Selection of Hotwords (VISH), a training-free, plug-and-play framework that directly verifies candidate-specific acoustic support over frame-level CTC posteriors. VISH formulates pre-selection as constrained posterior search; M-VISH makes large-list verification scalable through frame-synchronous shared-prefix Viterbi search with bounded top-K pruning, while Bi-MVISH exploits complementary backward evidence for robust verification. Across five benchmarks, M-VISH achieves up to 11.7% absolute improvement in Best-F1 over a strong learned-retrieval baseline and reduces candidate-search latency by 75.2% at 2,000 candidates relative to hypothesis-based filtering. These results demonstrate the effectiveness of posterior-level verification for accurate and scalable hotword pre-selection without additional model training.
Xiaoyu Gu, Yu Xi, Chencheng Wang +4
MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Shanghai Jiao Tong University · School of Intelligence Science and Technology, Nanjing University, Suzhou, China
Conventional end-to-end automatic speech recognition (ASR) systems rely on paired speech-text data for domain adaptation. Recent LLM-based ASR architectures connect a speech encoder to a large language model via a projection module, enabling adaptation with text-only data. However, this introduces a modality gap, as the LLM is not exposed to the noisy representations produced by the speech projector. We investigate whether small amounts of speech can mitigate this mismatch. We compare three strategies: text-only adaptation, paired speech-text adaptation, and mixed batching (MB), which combines both. Experiments in in-domain and out-of-domain settings show that even limited speech consistently improves performance. Notably, MB using only 10% of the target-domain (less than 4 hours) speech achieves word error rates comparable to, or better than, conventional ASR fine-tuning with the full dataset, indicating that small amounts of speech provide a strong modality-alignment signal.