Transcribing domain-specific entities and rare proper nouns remains a major challenge in automatic speech recognition (ASR). In this paper, we propose BaLEEN (Biasing with Latent Encoded Entities), a lightweight, hypernetwork-based framework for dynamic contextual adaptation without fine-tuning the underlying ASR model. BaLEEN encodes variable-length contextual keywords using a pretrained language model, compresses them into a fixed sequence of latent vectors via a Perceiver bottleneck, and injects context-dependent bias vectors directly into the intermediate encoder representations of the ASR model. Because both the language model and the backbone ASR model remain entirely frozen during training, BaLEEN operates as a plug-and-play adapter that incurs zero computational overhead at inference time when context biases are precomputed. We evaluate our method on a CTC-based ASR model using a Wikipedia-derived corpus with annotated named entities and synthetic speech. Experimental results demonstrate that BaLEEN reduces keyword miss rate by 8.7% on the test set relative to the unbiased baseline while simultaneously improving overall word error rate by 21% and character error rate by 28%.
Figures & tables
Figure 1: Our proposed methods for context biasing. The linear injection layer is instantiated for each target ASR layer.
Figure 2: Synthetic dataset construction flow.
Architecture
WER (↓)
CER (↓)
KMR (↓)
Baseline (no FT)
9.53
2.74
40.66
Baseline (XAttn + logits)
11.47
3.29
35.59
Baseline (XAttn + logits), DC
11.16
2.58
35.26
Linear + hidden (late)
7.97
2.06
36.37
Linear + hidden (midlate)
7.86
2.06
36.15
Linear + hidden (uniform)
7.86
2.04
35.67
Table 1: Evaluation results on the validation set. All metrics are reported in percentages (%). DC stands for dynamic context with probabilistically mixed hard-negative keywords.
Architecture
WER (↓)
CER (↓)
KMR (↓)
Baseline (no FT)
10.36
3.04
43.34
Baseline (XAttn + logits)
11.19
3.21
38.59
Baseline (XAttn + logits), DC
13.91
5.12
42.32
Linear + hidden (late)
9.51
2.63
45.92
Linear + hidden (midlate)
9.16
2.49
44.41
Linear + hidden (uniform)
8.70
2.36
41.32
Table 2: Evaluation results on the test set. All metrics are reported in percentages (%).
Recognizing new and rare words - named entities, acronyms, domain specific special words, and other items scarce in training data - remains a key challenge for automatic speech recognition (ASR). We compare two strategies for this: context biasing methods, where an ASR model is extended such that during inference a word list can be supplied, and speech large language models (LLMs) prompted with context directly. We evaluate two context biasing methods based on Whisper against three speech LLMs across read and non-read speech, reporting biased, unbiased, and overall word error rate (WER). The context biasing methods cut biased WER by up to 88% relative while leaving other words largely unaffected. Speech LLMs excel on read speech but generalize less well to non-read speech, and prove sensitive to distractor count and prompt word order. We characterize the resulting trade-offs to guide method selection.
Christian Huber, Alexander Waibel
Interactive Systems Lab · Karlsruhe Institute of Technology · Karlsruhe, Germany +2
Contextual biasing seeks to integrate external knowledge into automatic speech recognition (ASR) systems to accurately recognize domain-specific entities. In this paper, we propose COALA (Contextualized ASR Leveraging Biasing Scoring), a robust framework designed to enhance speech-augmented language models (SLMs) in complex multi-entity scenarios. Considering the inherent context-window limitations of SLMs, identifying relevant target entities from a large-scale biasing list is crucial for effective recognition. To this end, COALA maps SLM latent representations into a specialized discriminative space to quantify the matching intensity between audio segments and candidate entities. Furthermore, we address the training collapse in prior study when handling multi-target utterances-where multiple rare words co-occur. Experimental results on the LibriSpeech benchmark demonstrate that COALA consistently achieves superior contextual biasing performance across various biasing list scales.
Recognizing entity phrases remains a critical challenge for speech large language models. Existing prompting methods lack an explicit decoding-time biasing weight, limiting their controllability. Generative error correction methods can introduce hallucinated over-corrections. To address these limitations, we propose LOGIC (logit-space integration for contextual biasing), a robust framework operating directly in the logit space. By decoupling context injection from input processing, LOGIC enables explicit control over the biasing strength. Extensive experiments with an open-source speech large language model across 11 locales demonstrate that LOGIC achieves an average 9% relative reduction in entity word error rate, with an average false alarm rate increase of 0.3% and a 2.8% relative runtime overhead. When combined with prompting, LOGIC can reduce entity word error rate by 5% relative to the prompt-only method.