CLAR: CIF-Localized Alignment for Retrieval-Augmented Speech LLM-Based Contextual ASR

Shangkun Huang; Huan Shen; Wei Zou; Yunzhang Chen

cs.SDMar 26, 2026

CLAR: CIF-Localized Alignment for Retrieval-Augmented Speech LLM-Based Contextual ASR

Authors: Shangkun Huang, Huan Shen, Wei Zou, Yunzhang Chen

Organizations: BRVoice Team, Bairong, Inc., China

Abstract

Speech LLM-based ASR often struggles with named entities and long-tail words due to strong internal language-model priors. Retrieval-augmented biasing can help, but its effectiveness depends on accurate hotword localization in full-utterance speech under weak supervision. We propose CLAR, a dual-encoder speech-text retriever that uses Continuous Integrate-and-Fire (CIF) to learn monotonic token-level alignments without timestamps. With length-aware localized matching, CLAR anchors short-entity acoustic cues and reduces representation dilution and attention drift. The retriever is trained with a multi-granularity objective combining global and local segment-level contrastive losses and a CIF quantity constraint. At inference, top-ranked hotwords are injected as contextual prompts for the Speech LLM, improving recognition without shallow fusion. Experiments show that CLAR significantly improves hotword retrieval and reduces both CER and B-WER against strong contextual ASR baselines.

Figures & tables

Figure 1: CLAR retrieval-augmented contextual ASR: CIF-Localized Alignment retriever for localized hotword matching and prompt-based decoding with a Speech LLM.

Figure 2: The CLAR retriever with dual encoders, CIF-based monotonic alignment, and length-aware localized matching for hotword retrieval.

Algorithm 1: CIF-Localized Alignment CLAR for hotword retrieval
Input: speech features $\mathbf{X}$ , hotword candidates $\{\mathbf{y}_{j}\}_{j=1}^{M}$ , candidate token lengths $L_{j}$ , and top- $K$ .
Output: ranked hotword list $\mathcal{C}_{K}$ .
1. Encode speech and predict CIF alignment: $\mathbf{H}\leftarrow\text{SpeechEnc}(\mathbf{X})$ , $\boldsymbol{\alpha}\leftarrow\text{CIFPred}(\mathbf{H})$ , and obtain token boundaries $\{(b_{s},e_{s})\}_{s=1}^{S}$ .
2. Project and normalize embeddings: $\mathbf{a}_{t}\leftarrow\text{Norm}(f_{a}(\mathbf{h}_{t}))$ and $\mathbf{e}_{j}\leftarrow\text{Norm}(f_{t}(\text{TextEnc}(\mathbf{y}_{j})))$ .
3. Compute frame–candidate similarity: $S_{t,j}\leftarrow\tau\,\mathbf{a}_{t}^{\top}\mathbf{e}_{j}$ .
4. For each candidate $j$ , perform length-aware localized scoring: $\text{score}_{j}(s)=\frac{1}{e_{s+L_{j}-1}-b_{s}+1}\sum_{t=b_{s}}^{e_{s+L_{j}-1}}S_{t,j}$ , and set $\hat{r}_{j}\leftarrow\max_{s}\text{score}_{j}(s)$ .

Table 3

Loss Terms			Test-AISHELL-1-NE			Dev-AISHELL-1-NE
$\mathcal{L}_{\text{global}}$	$\mathcal{L}_{\text{local}}$	$\mathcal{L}_{\text{cif}}$	R@1	R@5	R@10	R@1	R@5	R@10
✓	$\times$	$\times$	33.29	54.70	62.62	29.99	47.53	58.85
$\times$	✓	✓	96.29	98.89	99.38	95.43	98.88	99.25
✓	✓	✓	97.03	99.75	99.75	95.73	98.88	99.48

Table 1: Ablation of loss components for hotword retrieval recall (%) on AISHELL-1-NE.

Dataset	SeACo-Paraformer [ 24 ]	GLCLAP [ 10 ]	CLAR
Test Aishell-1 NT	96.00	96.96	97.03

Table 2: Comparison of F1 scores (%) on the test Test-AISHELL-1-NE dataset.

Figure 3: Case-study similarity map for hotword localization.

Test-AISHELL-1-NE
Distractor Setting	CER $\downarrow$	B-WER $\downarrow$	U-CER $\downarrow$
Vanilla Step-Audio2-mini	$1.86$	$12.92$	$0.80$
NA	$1.82$	$13.03$	$0.76$
GT	$0.82$	$2.34$	$0.78$
$N=226$ (R1)	$1.36$	$5.97$	$1.03$
$N=400$	$0.92$	$2.78$	$0.83$

Table 3: Impact of distractor-list size on contextual ASR (%) on AISHELL-1-NE.

Model	CER $\downarrow$	B-WER $\downarrow$
SeACo-Paraformer [ 24 ]	$2.20$	$6.71$
CFL [ 2 ]	$1.51$	$6.03$
PAC [ 25 ]	$1.13$	$3.07$
Ours	0.92	2.78

Table 4: Comparison with baseline contextual ASR models (%) on AISHELL-1-NE.

Explore similar work

Training-Free Contextual ASR via SpeechLLM-Based Error-Aware Selective Retrieval
Sep 25, 2026Natsuo Yamashita, Ai Nemoto, Ryosuke Koichi +1Automatic Speech Recognition
Hitachi, Ltd., Research & Development Group, Japan · Hitachi Advanced Systems Corporation, Japan
SSEmb: Training a Generalizable Speech Retrieval Model Using Only Automatic Speech Recognition Data
Oct 13, 2026Sangkwon Park, Dongwook Lee, Youngho Cho +3Audio-Text Retrieval
ECE, Seoul National University · IPAI, Seoul National University · University of Seoul +1
How to Recognize New Words: A Comparison Between Context Biasing Methods and Speech LLMs
Aug 6, 2026Christian Huber, Alexander WaibelASR Evaluation Automatic Speech Recognition
Interactive Systems Lab · Karlsruhe Institute of Technology · Karlsruhe, Germany +2

Sep 25, 2026J/K move · Enter open · S save

Training-Free Contextual ASR via SpeechLLM-Based Error-Aware Selective Retrieval

Natsuo Yamashita, Ai Nemoto, Ryosuke Koichi, Masaaki Yamamoto

Hitachi, Ltd., Research & Development Group, Japan · Hitachi Advanced Systems Corporation, Japan

Automatic Speech Recognition

Open paper AskPDF arXiv

Recognition of domain-specific and low-frequency terms remains challenging for automatic speech recognition (ASR). Although contextual biasing can improve their recognition, directly providing a large terminology dictionary introduces many irrelevant biasing terms. Retrieval-based contextual biasing addresses this issue by selecting candidate terms from an external dictionary, but querying many recognized words requires numerous dictionary lookups and may yield poorly targeted candidates. We propose a training-free contextual ASR framework in which a pretrained speech large language model (SpeechLLM) jointly generates an ASR hypothesis and localizes error spans likely to involve domain-specific terms. Only the localized spans are used to retrieve phonologically similar terms from an external terminology dictionary. The same SpeechLLM then re-recognizes the audio conditioned on the first-pass hypothesis and the retrieved terms, without task-specific model training. To assess applicability across domains, we evaluate the framework on medical, air traffic control, and financial speech. The proposed method substantially reduces dictionary queries while improving the recall and ranking of relevant terminology candidates and second-pass ASR performance across all three domains.

Abstract

Explore similar work

Training-Free Contextual ASR via SpeechLLM-Based Error-Aware Selective Retrieval

SSEmb: Training a Generalizable Speech Retrieval Model Using Only Automatic Speech Recognition Data

How to Recognize New Words: A Comparison Between Context Biasing Methods and Speech LLMs