Listen to the Latents: Self-Correcting Speech Recognition in Large Audio Language Models Through Hidden-State Interactions

Chan-Jan Hsu; Jaeyeon Kim; Chao-Han Huck Yang; Shinji Watanabe; Hung-yi Lee; Carlos Busso

cs.CLAug 31, 2026

Listen to the Latents: Self-Correcting Speech Recognition in Large Audio Language Models Through Hidden-State Interactions

Authors: Chan-Jan Hsu, Jaeyeon Kim, Chao-Han Huck Yang, Shinji Watanabe, Hung-yi Lee, Carlos Busso

Organizations: Carnegie Mellon University · NVIDIA · Work done at NVIDIA · National Taiwan University

Abstract

Recent automatic speech recognition (ASR) systems increasingly integrate large language models (LLMs) to leverage their semantic knowledge, either externally through logit fusion or internally through warm initialization. However, how to effectively combine these two strategies remains underexplored. In this work, we refine warm-initialized LLM-based ASR models by leveraging their own pre-adaptation base LLMs, focusing on LoRA-adapted settings where the base LLM is preserved. To achieve this, we propose Hybrid Search, a targeted correction strategy motivated by two observations. First, interaction features that characterize the relationship between LLM-based ASR hidden states and base-LLM hidden states provide informative signals about a token's degree of semantic dependence. Second, selectively refining targeted tokens with high semantic dependence improves ASR performance far beyond naive global LLM-correction methods including rescoring and late fusion. Our analysis suggests that, even after semantic knowledge transfer through warm initialization, LLM-based ASR models can still leverage their base LLM to further improve inference-time performance.

Figures & tables

Figure 1: ASR-LLM self-correction framework. (a) Section 3 : Features are extracted from hidden-state interactions between the ASR-LLM and the LLM. (b) Section 4 : These features govern a gating mechanism that decides whether to incorporate LLM log-probabilities into prediction.

Panel of Figure 2 — Figure 2: Top: Average values and standard deviations of $\mathrm{AngleFeat}(l,i)$ and $\mathrm{MagFeat}(l,i)$ across tokens reveal distinct layer-wise phases. Bottom: Hidden-state interaction feature distributions for named-entity and non-named-entity tokens. Intermediate-layer aggregation reveals a clear distribution shift, while the late-layer features at layer 31 exhibit a further shift toward lower cosine similarity and higher relative norm for named-entity tokens.

Panel of Figure 3 — Figure 3: Visualization of our proposed Hybrid Search. When features lie in the selected regions (red border), beam search is initiated (shaded region), backtracking to the beginning-of-word token. Upon reaching a consensus non-targeted beginning-of-word token ( "_not" ), beams are rescored and decoding reverts to greedy search. The symbol $\varnothing$ denotes subsequent generated tokens that are omitted.

Experiment	Decoding Passes	Relative Cost
Greedy Search	One-Pass	$1\times$
+ Hybrid Search
+ (Decoding)	One-Pass	$1.4\times^{*}$
Beam Search	One-Pass	$5\times$
+ Hybrid Search
+ (Error Correction)	Two-Pass	–

Table 1: Hybrid Search for Decoding provides an efficient alternative within the family of beam-search methods, whereas Hybrid Search for Error Correction requires a second pass but further improves upon Beam Search (Table 2 ).

	Open ASR Leaderboard							Common Voice	Avg.
	AMI	Earn.	Giga.	Libri.	SPGI.	TED.	Vox.	Common Voice	Avg.
Word Error Rate
(A) GER $\Rightarrow$ Parakeet Chen et al. (2023)	22.91	–	12.10	4.84	3.98	6.09	7.49	–	–
(B) Neko Qwen1.5-MoE Lin et al. (2025)	12.60	11.82	9.95	2.32	1.94	3.20	5.80	–	–
(C) Speech-Hands $\rightleftharpoons$ Parakeet Wan et al. (2026)	11.20	–	11.10	3.18	2.16	4.37	6.02	–	–
Best of LLM-corrective (A–C) (Beam Search)	11.20	11.82	9.95	2.32	1.94	3.20	5.80	–	–

Table 2: WER and NE-ER across the Open ASR Leaderboard and Common Voice (in %). Top: Phi-4-Multimodal Beam Search outperforms the best prior LLM-corrective methods that incorporate beam search internally on most benchmarks. Middle and Bottom: Hybrid Search applied to Greedy Search provides a cost-effective alternative to Beam Search, recovering 25% of its WER improvement and 96% of its NE-ER improvement. When applied to Beam Search, Hybrid Search further reduces average NE-ER by 3.3% while maintaining comparable WER. In contrast, naive refinement methods such as rescoring and late fusion noticeably degrade WER. Values below the refinement results report two-sided exact McNemar test

p

-values against the Beam Search baseline; significant results (

p<0.05

) are shown in teal and non-significant results in gray.

Figure 4: Relative WER and NE-ER improvements across test sets, measured against beam search. Dashed lines connect Rescoring and Hybrid Search results, showing that Hybrid Search substantially improves WER while maintaining NE-ER accuracy.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Named-Entity Distribution Shifts of Granite-4.0-1B-Speech

Panel of Figure 6 — Figure 6: Generalization of the targeted region derived from the YODAS dataset. Most datasets exhibit a similar distribution shift. Under the targeted region selection derived from the YODAS dataset, named entities are effectively concentrated.

Figure 7: Layer-wise heatmaps for Common Voice showing AngleFeat values. The text reads: “Nebel also had philosopher Mortimer Adler and James Randi on frequently.” Named entities “Nebel”, “Mortimer”, “Adler”, “James”, and “Randi” are highlighted by the red box. Some of the named-entity tokens exhibit higher AngleFeat values.

Figure 8: Layer-wise heatmaps for Common Voice showing MagFeat values. The text reads: “Nebel also had philosopher Mortimer Adler and James Randi on frequently.” Named entities “Nebel”, “Mortimer”, “Adler”, “James”, and “Randi” are highlighted by the red box. Some of the named-entity tokens exhibit higher MagFeat values.

Automatic Speech Recognition

Automatic speech recognition converts spoken language into text across speakers, languages, domains, and recording conditions.

See all 459 papers

Merging the Knowledge of LLMs for Automatic Speech Recognition
Sep 14, 2026Hayato Futami, Tatsuya KawaharaDomain Adaptation for ASR Automatic Speech Recognition
Graduate School of Informatics, Kyoto University, Japan
Select or Rewrite: Oracle-Guided Fine-Tuning for Generative ASR Post-Editing from N-Best Hypotheses
Oct 10, 2026Shiran Liu, Gaohui Xu, Sergio Burdisso +3Language Model Post-Training Fine-Tuning
Idiap Research Institute, Martigny, Switzerland · KTH Royal Institute of Technology, Stockholm, Sweden · Brno University of Technology, Brno, Czech Republic
AuRA: Internalizing Audio Understanding into LLMs as LoRA
Jun 9, 2026Bo Cheng, Lei Shi, Zhanyu Ma +5Spoken Language Understanding Audio Understanding
Meituan · Jilin University

Sep 14, 2026J/K move · Enter open · S save

Merging the Knowledge of LLMs for Automatic Speech Recognition

Hayato Futami, Tatsuya Kawahara

Graduate School of Informatics, Kyoto University, Japan

Domain Adaptation for ASR Automatic Speech Recognition Domain Adaptation for LLMs Language Model Merging

Open paper AskPDF arXiv

Automatic speech recognition (ASR) systems, trained on paired speech-text data, have been improved by leveraging language models (LMs) trained on text-only data. LM fusion methods such as shallow fusion and density ratio are well-established methods that incorporate external LMs during ASR decoding. However, they incur additional computational costs due to LM inference, which is particularly problematic for recent larger LMs. In this study, we propose incorporating external LMs via model merging. This method integrates the LMs directly into the parameters of an LLM-based ASR model, requiring no additional computational cost at inference. We formulate domain extension and transfer via arithmetic operations on LoRA parameters. Experimental evaluations were conducted for the domain adaptation of LLM-based ASR trained on CSJ and LibriSpeech. We show that our LM merging consistently improved the ASR performance in the target domains, without degrading inference speed or memory footprint.

Abstract

Automatic Speech Recognition

Explore similar work

Merging the Knowledge of LLMs for Automatic Speech Recognition

Select or Rewrite: Oracle-Guided Fine-Tuning for Generative ASR Post-Editing from N-Best Hypotheses

AuRA: Internalizing Audio Understanding into LLMs as LoRA