Induction Head Origins of SLAM-ASR Speech Recognition Models
Organizations: Tallinn University of Technology, Estonia
Abstract
What makes a small trained map from encoder states to the embedding space of a frozen language model sufficient for competitive speech recognition? The premise of the SLAM-ASR architecture is that a frozen language model possesses linguistic knowledge rich enough to perform the recognition task. This assumption, while reasonable, provides little insight into the mechanisms underlying these models. We therefore seek the concrete internal structures on which they rely. This work shows that SLAM-ASR models rely on induction heads of the frozen language model: the same heads that copy repeated token sequences in text are also important for mapping speech embeddings to transcript tokens. Moreover, we find pairs of heads that behave as a constrained form of differential attention, cancelling attention noise and sharpening attention onto the previously generated word. Finally, we show how induction heads could be used to improve transcription quality under noisy audio.
Figures & tables
| decoder | intact | blocked | random |
|---|---|---|---|
| Llama | 3.3–3.6 | 7.1–8.3 | 3.3–3.6 |
| Gemma | 3.8–4.0 | 23.2–53.5 | 4.0–4.2 |
| Ministral | 3.0–3.4 | 3.9–5.5 | 3.1–3.4 |
| WER, blocked | ||||||
|---|---|---|---|---|---|---|
| decoder layer: | eff. | both | ||||
| Llama 8: H8, H6 | 0.72–0.74 | 0.78–0.82 | 0.52–0.57 | 0.64–0.67 | 45–56 | 3.8–7.2 |
| Gemma 8: H1, H0 | 0.66–0.67 | 0.92–0.94 | 0.74–0.76 | 0.80–0.81 | 58–149 | 15–39 |
| Ministral 14: H0, H2 | 0.78–0.79 | 0.73–0.74 | 0.78–0.80 | 0.80–0.81 | 5.1–13.8 | 3.3–4.2 |
| babble 0 dB | noisy (mean) | |||||
|---|---|---|---|---|---|---|
| decoder | scaled | clean | WER | WER | ||
| Llama | none | 4.6 | 89.0 | 40.4 | 41.9 | 15.0 |
| 4.7 | 81.8 | 32.1 | 39.9 | 12.4 | ||
| random | 4.5 | 89.9 | 41.3 | 42.4 | 15.5 | |
| same layers | 4.8 | 111.8 | 63.6 | 50.8 | 23.8 | |
| Gemma | none | 4.7 | 148.6 | 94.9 | 66.9 | 37.9 |
Automatic Speech Recognition
Automatic speech recognition converts spoken language into text across speakers, languages, domains, and recording conditions.
See all 507 papers