Recognition of domain-specific and low-frequency terms remains challenging for automatic speech recognition (ASR). Although contextual biasing can improve their recognition, directly providing a large terminology dictionary introduces many irrelevant biasing terms. Retrieval-based contextual biasing addresses this issue by selecting candidate terms from an external dictionary, but querying many recognized words requires numerous dictionary lookups and may yield poorly targeted candidates. We propose a training-free contextual ASR framework in which a pretrained speech large language model (SpeechLLM) jointly generates an ASR hypothesis and localizes error spans likely to involve domain-specific terms. Only the localized spans are used to retrieve phonologically similar terms from an external terminology dictionary. The same SpeechLLM then re-recognizes the audio conditioned on the first-pass hypothesis and the retrieved terms, without task-specific model training. To assess applicability across domains, we evaluate the framework on medical, air traffic control, and financial speech. The proposed method substantially reduces dictionary queries while improving the recall and ranking of relevant terminology candidates and second-pass ASR performance across all three domains.
Figures & tables
Figure 1: Overview of the proposed training-free contextual ASR framework with error-aware selective terminology retrieval.
Task: Transcribe the input speech and identify short domain-term error spans in the generated transcript.
Domain guidance: The target domain is medical speech. Relevant terminology includes medications, diseases, anatomy, clinical procedures, laboratory tests, and medical abbreviations.
Selection guidance: Select a span only when it is both domain-relevant and likely to be incorrect. Do not select a span solely because it contains a domain-specific term. Use acoustic evidence and the generated transcript to identify misrecognized, weakly supported, or acoustically unclear regions. Return short spans corresponding to the ASR-side error expression.
ICL format: Each demonstration consists of an input speech signal and a structured output containing its ASR hypothesis and domain-term error spans. The demonstrations include both error-containing and error-free cases.
Table 1: Prompt example for joint ASR and domain-term error localization on MedSyn.
Table 3
Method
Spans / Utt.
Error Precision
Error Recall
General Recall
Domain Recall
Span-selection baselines
All ASR 1-grams
13.37
8.1
92.8
83.6
99.1
Random span
0.99
11.5
10.6
11.0
10.3
NE extraction
1.72
29.5
63.6
37.0
81.7
Confidence span
1.15
49.4
50.6
28.9
65.3
SpeechLLM input configurations
Table 4: Domain-term error localization performance on MedSyn. All metrics except Spans / Utt. are reported in percent.
Method
Queries / Utt.
Reduction
R@1
R@10
R@50
MRR@10
All ASR 1-grams
13.37
0.0
20.7
53.2
63.7
31.9
Random span
0.99
92.6
3.7
6.1
8.4
4.4
NE extraction
1.72
87.1
30.9
52.6
64.4
38.1
Confidence span
1.15
91.4
31.3
45.4
54.3
35.8
Ours (Joint ICL)
0.99
92.6
41.4
61.8
73.1
47.9
Table 5: ANE retrieval performance on MedSyn. All metrics except Queries / Utt. are reported in percent.
Top-10
Top-50
Method
WER
B-WER
U-WER
WER
B-WER
U-WER
Baseline ASR
7.30
41.43
2.80
7.30
41.43
2.80
All ASR 1-grams
5.83
28.83
2.80
5.99
30.42
2.77
Random span
7.12
39.95
2.79
7.09
39.81
2.78
NE extraction
5.89
29.71
2.75
6.01
30.80
2.74
Confidence span
6.08
31.40
2.75
6.22
32.52
2.75
Table 6: Second-pass ASR performance on MedSyn (%).
Dataset
Method
Queries / Utt.
R@10
MRR@10
WER
B-WER
ATCOSIM
Baseline ASR
14.81
46.40
All ASR 1-grams
5.89
66.6
34.5
13.13
32.42
Random span
1.08
28.1
17.1
14.08
41.00
NE extraction
1.51
61.5
37.5
13.06
31.97
Confidence span
0.97
34.3
22.9
13.68
36.93
Ours (Joint ICL)
1.08
69.6
46.1
12.88
30.65
Table 7: Cross-domain ANE retrieval and second-pass ASR performance with top-10 retrieved candidates. All metrics except Queries / Utt. are reported in percent.
Recognizing new and rare words - named entities, acronyms, domain specific special words, and other items scarce in training data - remains a key challenge for automatic speech recognition (ASR). We compare two strategies for this: context biasing methods, where an ASR model is extended such that during inference a word list can be supplied, and speech large language models (LLMs) prompted with context directly. We evaluate two context biasing methods based on Whisper against three speech LLMs across read and non-read speech, reporting biased, unbiased, and overall word error rate (WER). The context biasing methods cut biased WER by up to 88% relative while leaving other words largely unaffected. Speech LLMs excel on read speech but generalize less well to non-read speech, and prove sensitive to distractor count and prompt word order. We characterize the resulting trade-offs to guide method selection.
Christian Huber, Alexander Waibel
Interactive Systems Lab · Karlsruhe Institute of Technology · Karlsruhe, Germany +2
Speech recognition in specialized domains requires leveraging contextual or topical information to improve the recognition of domain-specific entities. Speech Large Language Models (Speech LLMs) have substantially advanced speech understanding and reasoning capabilities, making context-aware speech recognition possible without predefined bias lists. In this paper, we propose LaSR (Latent Speech Reasoning), a novel training paradigm featuring a context-aware reasoning trajectory that leverages the latent reasoning process. Instead of generating explicit intermediate tokens, LaSR aligns chain-of-thought (CoT) supervision around the acoustic feature region of the target word, and introduces latent reasoning periods for context information grounding and transcriptional transition. Furthermore, to effectively benchmark context-aware speech recognition, we propose Spoken Darwin-Science, a large-scale corpus focusing on academic terminologies. Preliminary experiments on Fun-Audio-Chat demonstrate that LaSR significantly improves terminology recognition without introducing additional latency and consistently outperforms standard supervised fine-tuning baselines. Our findings highlight the potential of latent reasoning in building efficient, context-aware speech assistants.
Contextual automatic speech recognition (ASR) with Speech-LLMs is typically trained with oracle conversation history, but relies on error-prone history at inference, causing a train-test mismatch in the context channel that we term contextual exposure bias. We propose a unified training framework to improve robustness under realistic histories: (i) Teacher Error Knowledge by using Whisper large-v3 hypotheses as training-time history, (ii) Context Dropout to regularize over-reliance on history, and (iii) Direct Preference Optimization (DPO) on curated failure cases. Experiments on TED-LIUM 3 (in-domain) and zero-shot LibriSpeech (out-of-domain) show consistent gains under predicted-history decoding. With a two-utterance history as context, SFT with Whisper hypotheses reduce WER from 5.59% (oracle-history training) to 5.47%, and DPO further improves to 5.17%. Under irrelevant-context attacks, DPO yields the smallest degradation (5.17% -> 5.63%), indicating improved robustness to misleading context. Our code and models are published on https://github.com/XYGuo1996/Contextual_Speech_LLMs.
Xiaoyong Guo, Nanjie Li, Zijie Zeng +4
School of Computer Science and Technology, Xinjiang University, Urumqi, China · Timekettle · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Urumqi, China