cs.CLApr 7, 2026

In-Context Learning in Speech Language Models: Analyzing the Role of Acoustic Features, Linguistic Structure, and Induction Heads

Authors: Charlotte Pouw, Hosein Mohebbi, Afra Alishahi, Willem Zuidema

Organizations: ILLC, University of Amsterdam · CSAI, Tilburg University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

In-Context Learning (ICL) has been extensively studied in text-only Language Models, but remains largely unexplored in the speech domain. Here, we investigate how linguistic and acoustic features affect ICL in Speech Language Models. We focus on the Text-to-Speech (TTS) task, which allows us to analyze ICL from two angles: (1) how accurately the model infers the task from the demonstrations (i.e., generating the correct spoken content), and (2) to what extent the model mimics the acoustic characteristics of the demonstration speech in its output. We find that speaking rate strongly affects ICL performance and is also mimicked in the output, whereas pitch range and intensity have little impact on performance and are not consistently reproduced. Finally, we investigate the role of induction heads in speech-based ICL and show that these heads play a causal role: ablating the top-k induction heads completely removes the model's ICL ability, mirroring findings from text-based ICL.

Figures & tables

Figure 1: Our experimental setup: we construct ICL prompts and apply different linguistic and acoustic manipulations. We then (A) evaluate SpiritLM’s performance on the TTS task under each manipulation, and (B) measure the acoustic characteristics of the output speech.
ConditionDegree of OverlapDemonstrationTarget
Core—the boy judged the adulta secretary forgets a cousin
Lexical Overlapnouns onlythe secretary judged the cousina secretary forgets a cousin
main verb onlythe boy forgot the adult
function words onlya boy judges an adult
all wordsa secretary forgets a cousin
Semantic Similaritynouns onlythe employee judged the unclea secretary forgets a cousin
Table 1: Illustrative examples of linguistic overlap conditions. Examples are shown for sentences with active syntax; the same conditions apply to the passive sentences. The overlapping linguistic elements between the demonstration and target sentences are underlined . Note that verb tense deliberately only agrees between demonstration and target in the function words only and all words condition.
Figure 2: ICL performance of SpiritLM given demonstrations synthesized by KokoroTTS. (A) : performance on the Core condition across 1-5 demonstrations; (B) : performance on the Core condition, separated by the syntactic structure of the target and the demonstration; (C) : performance under various types of linguistic overlap between demonstration and target; (D) : performance under various acoustic manipulations of the demonstrations. Shaded regions/error bars indicate 95% confidence intervals.
Figure 3: Acoustic features of SpiritLM speech outputs given different demonstration types. Top row: demonstrations are synthesized using KokoroTTS; Bottom row: demonstrations are synthesized using SpeechT5. Shaded regions indicate 95% confidence intervals.
Figure 4: Identifying and ablating induction heads in SpiritLM.
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Word Error Rate of different Whisper model sizes on various speech types.
Figure 6: ICL performance of SpiritLM given demonstrations synthesized by SpeechT5. (A) : performance on the Core condition across 1-5 demonstrations; (B) : performance on the Core condition, separated by the syntactic structure of the target and the demonstration; (C) : performance under various types of linguistic overlap between demonstration and target; (D) : performance under various acoustic manipulations of the demonstrations. Shaded regions/error bars indicate 95% confidence intervals.
Figure 7: Prefix-matching scores for each head in SpiritLM on sequences of random tokens (text tokens only, speech tokens only, or interleaved text and speech tokens). The results of Llama-2-7B (the text-only backbone of SpiritLM) are added as a reference.
Figure 8: Attention to prefix-matching and non-prefix-matching tokens (speech vs. text) by different head groups, across three different demonstration types.
More in this topic

In-Context Learning

In-context learning studies how models adapt to new tasks from examples or instructions supplied at inference time, without changing their weights.

See all 404 papers

Explore similar work

CardsList

  1. Do Spoken Language Models Hear Speech as They Read Text? Bridging Structural Gaps Between Speech and Text

    Aug 24, 2026Hyeonyu Kim, Hwayeon Kim, Youngwon Choi +2Speech Language ModelsSpoken Language Understanding

    Maum AI Inc. · KAIST · Atmanity Inc.

  2. Interleaved Speech Language Models Latently Work In Text

    Jun 21, 2026Talia Sternberg, Gallil Maimon, Yossi AdiLLM InterpretabilitySpeech Language Models

    The Hebrew University of Jerusalem

  3. Interpreting and Steering a Text-to-Speech Language Model with Sparse Autoencoders

    Jun 8, 2026Nikita Koriagin, Georgii Aparin, Nikita Balagansky +1TTS SynthesisLanguage Model Steering

    ♣T-Tech · ♦AI Foundation and Algorithm Lab