Recent automatic speech recognition (ASR) systems increasingly integrate large language models (LLMs) to leverage their semantic knowledge, either externally through logit fusion or internally through warm initialization. However, how to effectively combine these two strategies remains underexplored. In this work, we refine warm-initialized LLM-based ASR models by leveraging their own pre-adaptation base LLMs, focusing on LoRA-adapted settings where the base LLM is preserved. To achieve this, we propose Hybrid Search, a targeted correction strategy motivated by two observations. First, interaction features that characterize the relationship between LLM-based ASR hidden states and base-LLM hidden states provide informative signals about a token's degree of semantic dependence. Second, selectively refining targeted tokens with high semantic dependence improves ASR performance far beyond naive global LLM-correction methods including rescoring and late fusion. Our analysis suggests that, even after semantic knowledge transfer through warm initialization, LLM-based ASR models can still leverage their base LLM to further improve inference-time performance.
Figures & tables
Figure 1: ASR-LLM self-correction framework. (a) Section 3 : Features are extracted from hidden-state interactions between the ASR-LLM and the LLM. (b) Section 4 : These features govern a gating mechanism that decides whether to incorporate LLM log-probabilities into prediction.
Figure 2: Top: Average values and standard deviations of AngleFeat(l,i) and MagFeat(l,i) across tokens reveal distinct layer-wise phases. Bottom: Hidden-state interaction feature distributions for named-entity and non-named-entity tokens. Intermediate-layer aggregation reveals a clear distribution shift, while the late-layer features at layer 31 exhibit a further shift toward lower cosine similarity and higher relative norm for named-entity tokens.
Figure 3: Visualization of our proposed Hybrid Search. When features lie in the selected regions (red border), beam search is initiated (shaded region), backtracking to the beginning-of-word token. Upon reaching a consensus non-targeted beginning-of-word token ( "_not" ), beams are rescored and decoding reverts to greedy search. The symbol ∅ denotes subsequent generated tokens that are omitted.
Experiment
Decoding Passes
Relative Cost
Greedy Search
One-Pass
1×
+ Hybrid Search
+ (Decoding)
One-Pass
1.4×∗
Beam Search
One-Pass
5×
+ Hybrid Search
+ (Error Correction)
Two-Pass
–
Table 1: Hybrid Search for Decoding provides an efficient alternative within the family of beam-search methods, whereas Hybrid Search for Error Correction requires a second pass but further improves upon Beam Search (Table 2 ).
Open ASR Leaderboard
Common Voice
Avg.
AMI
Earn.
Giga.
Libri.
SPGI.
TED.
Vox.
Word Error Rate
(A) GER ⇒ Parakeet Chen et al. (2023)
22.91
–
12.10
4.84
3.98
6.09
7.49
–
–
(B) Neko Qwen1.5-MoE Lin et al. (2025)
12.60
11.82
9.95
2.32
1.94
3.20
5.80
–
–
(C) Speech-Hands ⇌ Parakeet Wan et al. (2026)
11.20
–
11.10
3.18
2.16
4.37
6.02
–
–
Best of LLM-corrective (A–C) (Beam Search)
11.20
11.82
9.95
2.32
1.94
3.20
5.80
–
–
Table 2: WER and NE-ER across the Open ASR Leaderboard and Common Voice (in %). Top: Phi-4-Multimodal Beam Search outperforms the best prior LLM-corrective methods that incorporate beam search internally on most benchmarks. Middle and Bottom: Hybrid Search applied to Greedy Search provides a cost-effective alternative to Beam Search, recovering 25% of its WER improvement and 96% of its NE-ER improvement. When applied to Beam Search, Hybrid Search further reduces average NE-ER by 3.3% while maintaining comparable WER. In contrast, naive refinement methods such as rescoring and late fusion noticeably degrade WER. Values below the refinement results report two-sided exact McNemar test p -values against the Beam Search baseline; significant results ( p<0.05 ) are shown in teal and non-significant results in gray.
Figure 4: Relative WER and NE-ER improvements across test sets, measured against beam search. Dashed lines connect Rescoring and Hybrid Search results, showing that Hybrid Search substantially improves WER while maintaining NE-ER accuracy.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Named-Entity Distribution Shifts of Granite-4.0-1B-Speech
Figure 6: Generalization of the targeted region derived from the YODAS dataset. Most datasets exhibit a similar distribution shift. Under the targeted region selection derived from the YODAS dataset, named entities are effectively concentrated.
Figure 7: Layer-wise heatmaps for Common Voice showing AngleFeat values. The text reads: “Nebel also had philosopher Mortimer Adler and James Randi on frequently.” Named entities “Nebel”, “Mortimer”, “Adler”, “James”, and “Randi” are highlighted by the red box. Some of the named-entity tokens exhibit higher AngleFeat values.
Figure 8: Layer-wise heatmaps for Common Voice showing MagFeat values. The text reads: “Nebel also had philosopher Mortimer Adler and James Randi on frequently.” Named entities “Nebel”, “Mortimer”, “Adler”, “James”, and “Randi” are highlighted by the red box. Some of the named-entity tokens exhibit higher MagFeat values.
Automatic speech recognition (ASR) systems, trained on paired speech-text data, have been improved by leveraging language models (LMs) trained on text-only data. LM fusion methods such as shallow fusion and density ratio are well-established methods that incorporate external LMs during ASR decoding. However, they incur additional computational costs due to LM inference, which is particularly problematic for recent larger LMs. In this study, we propose incorporating external LMs via model merging. This method integrates the LMs directly into the parameters of an LLM-based ASR model, requiring no additional computational cost at inference. We formulate domain extension and transfer via arithmetic operations on LoRA parameters. Experimental evaluations were conducted for the domain adaptation of LLM-based ASR trained on CSJ and LibriSpeech. We show that our LM merging consistently improved the ASR performance in the target domains, without degrading inference speed or memory footprint.
Hayato Futami, Tatsuya Kawahara
Graduate School of Informatics, Kyoto University, Japan
Large language models can post-edit ASR by using linguistic context to compare N-best hypotheses and rewrite the transcript when needed, often improving WER without retraining the recognizer. However, prompt-only correction is brittle: it may over-edit when top-1 is correct and hallucinate when the reference is absent, and often needs very large models. We propose oracle-guided fine-tuning that turns N-best lists into supervision via three curated pools: corrections (reference in the list but not top-1) to teach selection, regularizers (top-1 correct) to discourage unnecessary rewrites, and hard negatives (reference absent) to suppress hallucination. Fine-tuning on this data provides more effective supervision than random subset fine-tuning for learning a hybrid select-or-rewrite policy, with the clearest gains from around 3B parameters on the proprietary dataset but less conclusive results on CommonVoice, a divergence suggesting that public read-speech text may overlap with LLM pretraining and mask genuine post-editing gains.
Shiran Liu, Gaohui Xu, Sergio Burdisso +3
Idiap Research Institute, Martigny, Switzerland · KTH Royal Institute of Technology, Stockholm, Sweden · Brno University of Technology, Brno, Czech Republic
Recent efforts to extend large language models (LLMs) to speech inputs typically rely on cascaded ASR-LLM pipelines, end-to-end speech-language models, or bridge/distillation-based adaptation. While these routes respectively reuse strong pretrained components, enable native speech-language interaction, or offer lightweight adaptation, they often suffer from transcript-interface latency, costly multimodal training, or sequential speech-language coupling. To address these limitations, we present AuRA, a method that distills audio encoding capability into the LLM. Specifically, AuRA feeds the same speech input to an ASR encoder (as a teacher) and a LoRA-adapted LLM (as a student) through a lightweight audio embedding layer, and uses layer-wise distillation to align the student's hidden states with corresponding teacher representations, thereby internalizing speech representations into lightweight LLM-side adaptations. Compared with cascaded and serial bridge methods, AuRA enables tighter speech-language joint modeling and efficient parallel end-to-end inference, while also reusing pretrained speech and language models rather than requiring large-scale multimodal training. On multiple speech-language benchmarks, AuRA consistently outperforms cascaded systems, speech-to-LLM adaptation baselines, and large-scale speech-language and multimodal models in both effectiveness and efficiency.