In-Context Learning (ICL) has been extensively studied in text-only Language Models, but remains largely unexplored in the speech domain. Here, we investigate how linguistic and acoustic features affect ICL in Speech Language Models. We focus on the Text-to-Speech (TTS) task, which allows us to analyze ICL from two angles: (1) how accurately the model infers the task from the demonstrations (i.e., generating the correct spoken content), and (2) to what extent the model mimics the acoustic characteristics of the demonstration speech in its output. We find that speaking rate strongly affects ICL performance and is also mimicked in the output, whereas pitch range and intensity have little impact on performance and are not consistently reproduced. Finally, we investigate the role of induction heads in speech-based ICL and show that these heads play a causal role: ablating the top-k induction heads completely removes the model's ICL ability, mirroring findings from text-based ICL.
Figures & tables
Figure 1: Our experimental setup: we construct ICL prompts and apply different linguistic and acoustic manipulations. We then (A) evaluate SpiritLM’s performance on the TTS task under each manipulation, and (B) measure the acoustic characteristics of the output speech.
Condition
Degree of Overlap
Demonstration
Target
Core
—
the boy judged the adult
a secretary forgets a cousin
Lexical Overlap
nouns only
the secretary judged the cousin
a secretary forgets a cousin
main verb only
the boy forgot the adult
function words only
a boy judges an adult
all words
a secretary forgets a cousin
Semantic Similarity
nouns only
the employee judged the uncle
a secretary forgets a cousin
Table 1: Illustrative examples of linguistic overlap conditions. Examples are shown for sentences with active syntax; the same conditions apply to the passive sentences. The overlapping linguistic elements between the demonstration and target sentences are underlined . Note that verb tense deliberately only agrees between demonstration and target in the function words only and all words condition.
Figure 2: ICL performance of SpiritLM given demonstrations synthesized by KokoroTTS. (A) : performance on the Core condition across 1-5 demonstrations; (B) : performance on the Core condition, separated by the syntactic structure of the target and the demonstration; (C) : performance under various types of linguistic overlap between demonstration and target; (D) : performance under various acoustic manipulations of the demonstrations. Shaded regions/error bars indicate 95% confidence intervals.
Figure 3: Acoustic features of SpiritLM speech outputs given different demonstration types. Top row: demonstrations are synthesized using KokoroTTS; Bottom row: demonstrations are synthesized using SpeechT5. Shaded regions indicate 95% confidence intervals.
Figure 4: Identifying and ablating induction heads in SpiritLM.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Word Error Rate of different Whisper model sizes on various speech types.
Figure 6: ICL performance of SpiritLM given demonstrations synthesized by SpeechT5. (A) : performance on the Core condition across 1-5 demonstrations; (B) : performance on the Core condition, separated by the syntactic structure of the target and the demonstration; (C) : performance under various types of linguistic overlap between demonstration and target; (D) : performance under various acoustic manipulations of the demonstrations. Shaded regions/error bars indicate 95% confidence intervals.
Figure 7: Prefix-matching scores for each head in SpiritLM on sequences of random tokens (text tokens only, speech tokens only, or interleaved text and speech tokens). The results of Llama-2-7B (the text-only backbone of SpiritLM) are added as a reference.
Figure 8: Attention to prefix-matching and non-prefix-matching tokens (speech vs. text) by different head groups, across three different demonstration types.
Spoken Language Models (SLMs) generate textual responses directly from speech, offering an alternative to cascaded systems. Despite recent advances, existing SLMs still exhibit weaker instruction-following behavior and limited generalization across diverse tasks compared to text-based language models. Our analysis shows that speech and text representations in current SLMs remain weakly aligned despite strong downstream performance, indicating that structural differences between continuous, temporally varying speech and discrete text remain insufficiently addressed. To address this, we propose a simple framework that decouples length mismatch from semantic alignment and encourages closer correspondence between speech and text representations. Experiments across multiple benchmarks demonstrate competitive performance against strong baselines, underscoring the importance of explicitly addressing structural differences between speech and text in SLM training. Our code is publicly available at https://github.com/jaykim9870/Do_SLMs_Hear_Speech_as_They_Read_Text.
Speech language models (SLMs) have been extensively studied, with the common paradigm incorporating text data and pre-trained text LMs. A leading approach is speech-text interleaving in which models are trained over sequences containing both speech and text tokens, aiming to boost even speech-only capabilities. Yet the way these two modalities interact in the model latent space remains unclear. In this work, we analyze interleaved speech-text LMs from different model families and sizes through the scope of the logit lens to provide such insight. We reveal that these models go through an implicit transcription phase in which the text token of the spoken word becomes decodable in intermediate layers, despite not being trained for speech recognition. The transcription of the word appears as one of the top candidate words for as much as 77% of the data. Following this stage, the models proceed to predict the next word in the text space before transforming back to the speech domain. We finally analyze the role of interleaving data, and initializing from text LMs in eliciting this behavior, as well as seeing how this correlates with spoken knowledge abilities. Our analysis sheds light on the internal mechanisms underlying the relationship between speech and text modalities and could shape SLM optimization.
Language models increasingly serve as the backbone of text-to-speech (TTS) systems, yet we understand little about the representations they build when text and generated speech tokens share a single residual stream. We train BatchTopK sparse autoencoders on the LM backbone of CosyVoice3 and introduce a modality-aware auto-interp pipeline that labels each feature from where it fires-text-prefix context, 1-second speech clips, or both. The recovered features are interpretable, spanning phonemes, laughter, accent prompts and speaker gender. Steering through the SAE latent space shows these features are causal rather than merely descriptive: targeted interventions raise laughter probability from 0.02 to 0.79, flip perceived speaker gender, and control speech rate while preserving spoken content. SAE features thus serve both as interpretability objects and as control directions for TTS synthesis.
Nikita Koriagin, Georgii Aparin, Nikita Balagansky +1