Despite extensions to speech inputs, effectively leveraging the rich knowledge and contextual understanding of large language models (LLMs) in automatic speech recognition (ASR) remains non-trivial, as the task primarily involves direct speech-to-text mapping. To address this, this paper proposes chain-of-thought ASR (CoT-ASR), which constructs a reasoning chain that enables LLMs to first analyze the input speech and generate contextual analysis, thereby fully exploiting their generative capabilities. With this contextual reasoning, CoT-ASR then performs more informed speech recognition and completes both reasoning and transcription in a single pass. Moreover, CoT-ASR naturally supports user-guided transcription: while designed to self-generate reasoning, it can also seamlessly incorporate user-provided context to guide transcription, further extending ASR functionality. To reduce the modality gap, this paper introduces a CTC-guided Modality Adapter, which uses CTC non-blank token probabilities to weight LLM embeddings, efficiently aligning speech encoder outputs with the LLM's textual latent space. Experiments show that, compared to standard LLM-based ASR, CoT-ASR achieves a relative reduction of 8.7% in word error rate (WER) and 16.9% in entity error rate (EER).
Figures & tables
Figure 1: Illustration of CoT-ASR. The text prompt is a fixed template for the ASR task. The output sequence comprises contextual analysis and transcription, marked by the <CONTEXT> and <TRANSCRIPT> tags, respectively. The reasoning-based contextual analysis is highlighted in red, while the subsequent transcription is marked in green. Both are produced sequentially in a single one-pass generation.
Figure 2: Comparison between CoT-ASR and standard LLM-based ASR. During generation, CoT-ASR first performs a contextual reasoning analysis, which then guides the subsequent transcription. This example illustrates how CoT-ASR leverages the rich knowledge of LLMs to ultimately improve transcription quality.
Figure 3: Illustration of the CTC-guided Modality Adapter. The linear output layer projects the encoder outputs to the CTC vocabulary size including the blank token, while the linear layer maps the encoder outputs to the LLM hidden dimension. ⊗ denotes matrix multiplication and ⊕ denotes addition. Gate denotes a frame-wise scalar from the linear projection and sigmoid that modulates the residual contribution.
Metric
Test Set
Phi4MM-style
CoT-ASR
Baseline
WER
LibriSpeech test-clean
2.41
2.20
LibriSpeech test-other
5.10
4.82
FLEURS
3.85
3.57
EER
Consumer Goods
15.45
13.46
Nutrition
7.36
6.66
Table 1: WER ( ↓ ) results on English ASR test sets from public datasets, including LibriSpeech (test-clean and test-other) and FLEURS, along with EER ( ↓ ) results on a series of in-house English ASR evaluation sets spanning diverse domains. Note that the training data used for the models constructed in this paper do not include these public datasets.
Test Set
CoT-ASR
CoT-ASR
with User Context
Consumer Goods
13.46
9.03
Nutrition
6.66
5.24
Gaming
14.53
12.99
Wellness
11.51
8.13
Patient History
3.51
3.31
Table 2: EER ( ↓ ) results on a series of in-house ASR evaluation sets, comparing CoT-ASR using its self-generated contextual reasoning (the default setting) with scenarios guided by user-provided context.
Test Set
CoT-ASR
CoT-ASR
with Linear Adapter
Consumer Goods
13.46
14.93
Nutrition
6.66
6.80
Gaming
14.53
16.52
Wellness
11.51
12.87
Patient History
3.51
4.11
Table 3: EER ( ↓ ) results on a series of in-house ASR evaluation sets across multiple domains, comparing CoT-ASR with the proposed CTC-guided Modality Adapter (the default setting) against a linear-projection-based adapter (with activation function).
LLM
Phi4
Whisper
Gemma 3n
Qwen3
Qwen2.5
Voxtral
Size
3.8B
1.5B
8B
30B
7B
3B
Test Set
Phi4MM-style Baseline
CoT-ASR
Whisper- Large-v3
Gemma 3n E4B IT
Qwen3-Omni- 30B-A3B-Instruct
Qwen2.5- Omni-7B
Voxtral-Mini -3B-2507
Consumer.
15.45
13.46
14.32
22.66
15.02
23.79
14.32
Nutrition
7.36
6.66
7.13
11.77
6.63
14.72
7.36
Gaming
17.42
14.53
15.14
23.59
15.01
28.68
14.15
Wellness
14.27
11.51
11.88
26.15
11.51
23.76
13.31
Table 4: EER ( ↓ ) results on a series of in-house ASR evaluation sets across multiple domains, comparing CoT-ASR with several open-source foundation models, including Whisper-large-v3, Gemma 3n E4B IT, Qwen3-Omni-30B-A3B-Instruct, Qwen2.5-Omni-7B, and Voxtral-Mini-3B-2507. Note that the comparison is not well controlled and is provided for context, as differences in LLM backbones and training data prevent direct comparison. The models built in this paper were trained on 38k hours of data, whereas the open-source models are trained on substantially larger datasets, for example, Whisper-large-v3 used approximately 5M hours of training data. In all cases, speech serves as the sole input, without any additional context provided.
Model
test-clean
test-other
WER
B-WER
U-WER
WER
B-WER
U-WER
CoT-ASR
2.20
9.31
1.29
4.82
20.01
3.02
Phi4MM-style Baseline
2.41
10.41
1.40
5.10
21.33
3.18
+Bias List (Size=100)
2.42
8.84
1.60
5.56
18.47
4.03
+Bias List (Size=500)
5.99
10.83
5.25
13.80
22.51
12.78
Table 5: WER / B-WER (biased WER) / U-WER (unbiased WER) ( ↓ ) results on LibriSpeech test sets, comparing CoT-ASR with Phi4MM-style baseline using contextual biasing. Note that this comparison is not well-controlled, as CoT-ASR does not receive the biasing list as additional input.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Prompt Content
< INSTRUCTION >
You are a High-Fidelity Context Synthesizer. Your task is to generate a < CONTEXT > block based on the provided < GROUND_TRUTH > text.
This context will be placed before the ASR transcript to guide a Speech-LLM. It must act as a semantic "warm-up" by defining the environment, the speaker’s intent, and the specific entities or examples mentioned.
CORE ATTRIBUTES OF THE CONTEXT:
1. Scenario Identification: Start by identifying the format (e.g., educational lecture, formal briefing, casual discussion).
2. Specific Weaving: Incorporate unique examples, technical terms, or key names found in the text (e.g., if the text mentions "sugar" or "Sherman", the context must include them) to ground the model.
Appendix
Table 6: Prompt used in this paper to construct reasoning-augmented ASR data.
Test Set
Phi4MM-style Baseline
CoT-ASR
Consumer Goods
7.38
6.47
Nutrition
5.00
4.67
Gaming
8.56
8.03
Wellness
9.13
8.31
Patient History
2.42
2.08
Pharmacy
2.54
2.22
Appendix
Table 7: WER ( ↓ ) results on a series of in-house English ASR evaluations sets.
Test Set
0% error
10% error
25% error
50% error
Consumer Goods
9.03
11.02
11.46
12.76
Nutrition
5.24
5.44
5.67
6.07
Gaming
12.99
13.70
14.50
14.24
Wellness
8.13
8.50
9.70
10.65
Patient History
3.31
3.54
3.47
3.58
Pharmacy
3.11
3.31
3.94
4.38
Appendix
Table 8: Robustness of CoT-ASR to simulated errors in user-provided context. EER (%) is reported under varying error rates.
Test Set
CoT-ASR
w/ Linear Adapter
w/ CTC Compressor
w/ SSR
Consumer Goods
13.46
14.93
21.70
21.44
Nutrition
6.66
6.80
11.57
12.23
Gaming
14.53
16.52
24.29
25.12
Wellness
11.51
12.87
19.12
18.37
Patient History
3.51
4.11
7.03
7.16
Pharmacy
5.97
6.66
11.57
10.88
Appendix
Table 9: EER ( ↓ ) results on on in-house ASR evaluation sets for CoT-ASR with different modality adapters.
Speech recognition in specialized domains requires leveraging contextual or topical information to improve the recognition of domain-specific entities. Speech Large Language Models (Speech LLMs) have substantially advanced speech understanding and reasoning capabilities, making context-aware speech recognition possible without predefined bias lists. In this paper, we propose LaSR (Latent Speech Reasoning), a novel training paradigm featuring a context-aware reasoning trajectory that leverages the latent reasoning process. Instead of generating explicit intermediate tokens, LaSR aligns chain-of-thought (CoT) supervision around the acoustic feature region of the target word, and introduces latent reasoning periods for context information grounding and transcriptional transition. Furthermore, to effectively benchmark context-aware speech recognition, we propose Spoken Darwin-Science, a large-scale corpus focusing on academic terminologies. Preliminary experiments on Fun-Audio-Chat demonstrate that LaSR significantly improves terminology recognition without introducing additional latency and consistently outperforms standard supervised fine-tuning baselines. Our findings highlight the potential of latent reasoning in building efficient, context-aware speech assistants.
Standard LLM-based speech recognition systems typically process utterances in isolation, limiting their ability to leverage conversational context. In this work, we study whether multimodal context from prior turns improves LLM-based ASR and how to represent that context efficiently. We find that, after supervised multi-turn training, conversational context mainly helps with the recognition of contextual entities. However, conditioning on raw context is expensive because the prior-turn audio token sequence grows rapidly with conversation length. To address this, we propose Abstract Compression, which replaces the audio portion of prior turns with a fixed number of learned latent tokens while retaining corresponding transcripts explicitly. On both in-domain and out-of-domain test sets, the compressed model recovers part of the gains of raw-context conditioning with a smaller prior-turn audio footprint. We also provide targeted analyses of the compression setup and its trade-offs.
Recognizing entity phrases remains a critical challenge for speech large language models. Existing prompting methods lack an explicit decoding-time biasing weight, limiting their controllability. Generative error correction methods can introduce hallucinated over-corrections. To address these limitations, we propose LOGIC (logit-space integration for contextual biasing), a robust framework operating directly in the logit space. By decoupling context injection from input processing, LOGIC enables explicit control over the biasing strength. Extensive experiments with an open-source speech large language model across 11 locales demonstrate that LOGIC achieves an average 9% relative reduction in entity word error rate, with an average false alarm rate increase of 0.3% and a 2.8% relative runtime overhead. When combined with prompting, LOGIC can reduce entity word error rate by 5% relative to the prompt-only method.