cs.CLNov 18, 2026

Induction Head Origins of SLAM-ASR Speech Recognition Models

Authors: Artem Fedorchenko, Tanel Alumäe

Organizations: Tallinn University of Technology, Estonia

Abstract

What makes a small trained map from encoder states to the embedding space of a frozen language model sufficient for competitive speech recognition? The premise of the SLAM-ASR architecture is that a frozen language model possesses linguistic knowledge rich enough to perform the recognition task. This assumption, while reasonable, provides little insight into the mechanisms underlying these models. We therefore seek the concrete internal structures on which they rely. This work shows that SLAM-ASR models rely on induction heads of the frozen language model: the same heads that copy repeated token sequences in text are also important for mapping speech embeddings to transcript tokens. Moreover, we find pairs of heads that behave as a constrained form of differential attention, cancelling attention noise and sharpening attention onto the previously generated word. Finally, we show how induction heads could be used to improve transcription quality under noisy audio.

Figures & tables

Explore similar work

CardsList
  1. Zero-Shot Spoken Language Understanding with SLAM-ASR: What Do Monolingual ASR-Trained Linear Projectors Enable?

    Oct 14, 2026Edwin Arkel Rios, Sneh Sourabh, Jocelyn Zaruma +3Spoken Language UnderstandingSpeech Language Models

  2. In-Context Learning in Speech Language Models: Analyzing the Role of Acoustic Features, Linguistic Structure, and Induction Heads

    Apr 7, 2026Charlotte Pouw, Hosein Mohebbi, Afra Alishahi +1In-Context LearningSpeech Language Models

  3. On the Role of Encoder Depth: Pruning Whisper and LoRA Fine-Tuning in SLAM-ASR

    Mar 30, 2026Ganesh Pavan Kartikeya Bharadwaj Kolluri, Michael Kampouridis, Ravi ShekharNeural Network PruningSpeech Language Models