Distilling Text LMs into Speech-to-Speech LMs by Matching Internal Text Predictions

Ju-Chieh Chou; Jiawei Zhou; Karen Livescu

cs.CLOct 12, 2026

Distilling Text LMs into Speech-to-Speech LMs by Matching Internal Text Predictions

Authors: Ju-Chieh Chou, Jiawei Zhou, Karen Livescu

Organizations: TTI-Chicago · Stony Brook University

Abstract

Text language models (LMs) acquire broad linguistic and factual capabilities through large-scale pretraining, while comparable capabilities remain difficult to learn from speech alone. Recent work has shown that knowledge distillation from pretrained text LMs offers a promising way to transfer their linguistic and factual capabilities to speech models. However, existing distillation approaches have largely focused on speech-input, text-output models. This idea has yet to be explored for {\it speech-to-speech} models, which operate on both speech input and speech output. In this work we introduce a speech-to-speech pretraining method that creates an internal text interface for knowledge transfer while preserving the speech-only input and output. The model first consumes speech context without its transcription, switches to text prediction, and then uses the predicted text to lead future speech generation word by word. It also provides an interface for knowledge distillation: the internal text predictions can be directly distilled from the base text LM. To the best of our knowledge, this is the first application of text LM distillation to a speech-to-speech LM. Under matched training conditions, the proposed sequence construction substantially improves the generative perplexity of spoken continuations over speech-only and interleaved speech-text models, approaching the performance of a text-leading inner-monologue model with external ASR prompt transcription. More importantly, distillation improves factual performance under both text and speech evaluation, indicating better retention of knowledge from the base text language model.

Figures & tables

Figure 1: Our TGSLM architecture. Given a speech context, the model generates subsequent words sequentially, with each text word followed by its corresponding speech segments. In TGSLM-distill, the text-token predictions are additionally trained to match those of a frozen text LM.

Model	Speech-only	Text-guided
	prompt	generation
Text leading	✗	✓
Interleaved	✓	✗
TGSLM	✓	✓

Table 1: How text is used at inference in SLMs.

Speech-only baselines (no text guidance)
Model	Direction	Params	Data	GenPPL $\downarrow$	BLEU $\downarrow$	Spk $\uparrow$	UTMOS $\uparrow$
Ground truth	–	–	–	43.8	6.0	0.74	2.82
Speech-only	S $\rightarrow$ S	0.8B	21.3k h	151.7	4.3	0.42	2.56
Interleaved	S $\rightarrow$ S	0.8B	21.3k h	164.1	5.4	0.37	2.36
TGSLM variants
TGSLM	S $\rightarrow$ (T, S)	0.8B	21.3k h	58.9	3.9	0.36	2.41

Table 2: Generation-based metrics on the Emilia test set: genPPL, bigram auto-BLEU (BLEU), speaker similarity (Spk), and UTMOS-v2 (UTMOS) ( Baba et al., 2024 ) ; see Sec. 4.3 for details. For the Whisper prompt transcript row in text-leading references, the model is conditioned on a transcript of the speech prompt produced by Whisper small.en . TGSLM-Large-distill uses

\lambda=0.5

with text replay. All rows are evaluated on the same subset: a shared subset of 1,325 prompts (of 1,500) with non-empty continuations from every model, including external models in Appendix A.6 , covering 4,957 of 7,500 continuations.

Figure 2: Ablation of text replay and distillation weight

\lambda

. TGSLM uses neither. TGSLM + replay adds text replay only (

\lambda=0

); TGSLM-distill adds both. Left: Pass@5 (%) on Known-1000. Right: accuracy (%) on spoken (sSC) and text (tSC) StoryCloze ( Hassid et al., 2023 ) .

Model	Direction	Params	Data	Pass@1	Pass@5
Text-LM reference
Qwen3-0.6B backbone ( Yang et al., 2025 )	T $\rightarrow$ T	0.6B	–	46.8	77.9
Qwen3-1.7B backbone ( Yang et al., 2025 )	T $\rightarrow$ T	1.7B	–	60.8	86.6
Our models
TGSLM-distill	S $\rightarrow$ (T, S)	0.8B	45k h	9.1	22.4
TGSLM-Large-distill	S $\rightarrow$ (T, S)	1.9B	135k h	16.8	37.9

Table 3: Known-1000 evaluation. Pass@

K

(%) measures whether the answer appears in at least one of

K

sampled continuations. Direction S

\rightarrow

(T, S) denotes a speech prompt followed by a text-guided speech continuation. TGSLM-Large-distill uses a distillation weight of

\lambda=0.5

and text replay.

Our models
Model	Happy	Sad	Angry	Macro
Original prefix only	$64.6\,{\color[rgb]{0.7,0.7,0.7}\pm 9.4}$	$92.7\,{\color[rgb]{0.7,0.7,0.7}\pm 4.7}$	$77.1\,{\color[rgb]{0.7,0.7,0.7}\pm 8.3}$	$78.1\,{\color[rgb]{0.7,0.7,0.7}\pm 4.3}$
Speech-only	$54.2\,{\color[rgb]{0.7,0.7,0.7}\pm 5.2}$	$76.9\,{\color[rgb]{0.7,0.7,0.7}\pm 4.3}$	$36.7\,{\color[rgb]{0.7,0.7,0.7}\pm 4.5}$	$55.9\,{\color[rgb]{0.7,0.7,0.7}\pm 2.7}$
TGSLM ( $\lambda=0$ )	$57.7\,{\color[rgb]{0.7,0.7,0.7}\pm 4.9}$	$72.1\,{\color[rgb]{0.7,0.7,0.7}\pm 4.6}$	$43.3\,{\color[rgb]{0.7,0.7,0.7}\pm 5.0}$	$57.7\,{\color[rgb]{0.7,0.7,0.7}\pm 2.8}$
TGSLM-distill ( $\lambda=0.5$ )	$60.0\,{\color[rgb]{0.7,0.7,0.7}\pm 4.4}$	$77.3\,{\color[rgb]{0.7,0.7,0.7}\pm 4.3}$	$34.8\,{\color[rgb]{0.7,0.7,0.7}\pm 4.7}$	$57.4\,{\color[rgb]{0.7,0.7,0.7}\pm 2.6}$
TGSLM-Large-distill ( $\lambda=0.5$ )	$56.0\,{\color[rgb]{0.7,0.7,0.7}\pm 4.6}$	$75.4\,{\color[rgb]{0.7,0.7,0.7}\pm 4.3}$	$43.5\,{\color[rgb]{0.7,0.7,0.7}\pm 5.4}$	$58.3\,{\color[rgb]{0.7,0.7,0.7}\pm 2.6}$

Table 4: Gemini-Emo three-way emotion classification accuracy (%, higher is better) on continuation using emotion2vec-plus-large ( Ma et al., 2024 ) . We report 95% percentile bootstrap confidence intervals from 1000 resamples of the 96 triplets.

\pm h

denotes the half-width interval. Generation failures and empty continuations are counted as incorrect.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Shared prefix	I noticed that the bread in the kitchen
Emotion	suffix
Happy	has baked into the most wonderful golden loaf!
Sad	has turned stale and completely moldy already today.
Angry	was left uncovered on the counter again!

Appendix

Table 5: An example from Gemini-Emo ( item_01 ). Each sentence combines the shared prefix with the corresponding suffix and is synthesized in the indicated emotion. Evaluation uses only the cropped spoken prefix, whose words are identical across emotions.

Prompt purpose	Model	Prompt template
Text generation	gemini-3.7 -flash	Create exactly 25 English speech-continuation stimulus items. For every item: write one shared prefix of 7–12 words as a natural, affect-neutral, syntactically unfinished conversational turn; write emotion-specific 5–14-word suffixes that complete that prefix; form each carrier as PREFIX + ‘‘ ’’ + SUFFIX ; use varied, ordinary ASR-friendly language; and return the supplied item schema. The generation ran in batches through 500 items.
Shared TTS interface	gemini-3.1 -flash- tts-preview voice: Schedar	Synthesize speech for the transcript below. AUDIO PROFILE: Use one consistent adult speaker identity and a standard American English accent. SCENE: This is a natural conversation with one familiar listener in a quiet room. DIRECTOR’S NOTES: <EMOTION-SPECIFIC DIRECTOR’S NOTES> Speak the transcript exactly once and verbatim. Do not read these directions. Do not add, remove, or replace words. Do not laugh, cry, sigh, gasp, whisper, shout, or add nonverbal sounds. TRANSCRIPT: <TRANSCRIPT>
Happy director’s notes		Convey a small, private happiness, as if sharing good news with someone you trust. Keep an easy moderate pace and soft-to-medium energy. Let a subtle smile warm the tone and add light upward inflections. Sound understated but unmistakably pleased, not breathy, cute, or theatrical. Apply these cues from the first word, including the neutral lead-in. Keep the performance conversational, coherent, and moderately expressive rather than exaggerated.
Sad director’s notes		Convey strong genuine sorrow while trying to remain composed. Use a soft voice, slow measured pacing, lowered resonance, and pronounced downward intonation. Sound deeply sad but never cry, sob, or become theatrical. Apply these cues from the first word, including the neutral lead-in. Keep the performance conversational, coherent, and moderately expressive rather than exaggerated.
Angry director’s notes		Sound clearly angry but fully controlled. Use a firm tense tone, clipped articulation, strong stress, deliberate pacing, and moderately high energy. Address the listener directly without shouting. Apply these cues from the first word, including the neutral lead-in. Keep the performance conversational, coherent, and moderately expressive rather than exaggerated.

Appendix

Table 6: Gemini-Emo language-model and TTS prompt specification. The shared TTS row applies to all three emotion prompts. Each prompt inserts one emotion-specific director instruction between the scene and transcript sections.

Hyperparameter	Architecture ablations	Distillation ablations	TGSLM-Large-distill
Training
Backbone	Qwen3-0.6B-Base	Qwen3-0.6B-Base	Qwen3-1.7B-Base
Optimizer updates	40,000; 50,000 for model with text replay	100,000	785,052 (665,000 + annealing)
Audio budget (s/GPU/update)	480	540	240
GPUs	4	4	4
Peak learning rate	$5\times 10^{-4}$	$5\times 10^{-4}$	$3\times 10^{-4}$

Appendix

Table 7: Training hyperparameters and decoding parameters.

Speech-only baselines (no text guidance)
Model	Direction	Params	Data	GenPPL $\downarrow$	BLEU $\downarrow$	Spk $\uparrow$	UTMOS $\uparrow$
Ground truth	–	–	–	43.8	6.0	0.74	2.82
Speech-only	S $\rightarrow$ S	0.8B	21.3k h	151.7	4.3	0.42	2.56
Interleaved	S $\rightarrow$ S	0.8B	21.3k h	164.1	5.4	0.37	2.36
TGSLM variants
TGSLM	S $\rightarrow$ (T, S)	0.8B	21.3k h	58.9	3.9	0.36	2.41

Appendix

Table 8: Generation-based metrics on the Emilia test set: genPPL, bigram auto-BLEU (BLEU), speaker similarity (Spk), and UTMOS-v2 (UTMOS) ( Baba et al., 2024 ) ; see Sec. 4.3 for details. For the Whisper prompt transcript row in text-leading references, the model is conditioned on a transcript of the speech prompt produced by Whisper small.en . All rows are evaluated on the same subset: a shared subset of 1,325 prompts (of 1,500) with non-empty continuations from every model, covering 4,957 of 7,500 continuations. TGSLM-Large-distill uses

\lambda=0.5

with text replay.

Model	Direction	Params	Data	Pass@1	Pass@5
Text-LM reference
Qwen3-0.6B backbone ( Yang et al., 2025 )	T $\rightarrow$ T	0.6B	–	46.8	77.9
Qwen3-1.7B backbone ( Yang et al., 2025 )	T $\rightarrow$ T	1.7B	–	60.8	86.6
Our models
TGSLM-distill	S $\rightarrow$ (T, S)	0.8B	45k h	9.1	22.4
TGSLM-Large-distill	S $\rightarrow$ (T, S)	1.9B	135k h	16.8	37.9

Appendix

Table 9: Known-1000 evaluation. Pass@

K

(%) measures whether the answer appears in at least one of

K

sampled continuations. Direction S

\rightarrow

(T, S) denotes a speech prompt followed by a text-guided speech continuation. TGSLM-Large-distill uses a distillation weight of

\lambda=0.5

and text replay. Post-trained models are shown in gray because they are not directly comparable to pretrained models.

Speech Language Models

Speech language models connect spoken audio with large language models for understanding, transcription, translation, and real-time dialogue.

See all 301 papers

Apr 1, 2026

Adapting Text LLMs to Speech via Multimodal Depth Up-Scaling

Adapting pre-trained text Large Language Models (LLMs) into Speech Language Models (Speech LMs) via continual pretraining on speech data is promising, but often degrades the original text capabilities. We propose Multimodal Depth Upscaling, an extension of an emerging strategy in continual LLM pre-training, where new transformer layers are inserted into a frozen text LLM and only the added layers are trained on speech data. Experiments with SmolLM2-360M and SmolLM2-1.7B on 48k hours of English Automatic Speech Recognition (ASR) data show that depth up-scaling achieves ASR comparable to full fine-tuning while causing far less text degradation than both full fine-tuning and Low-Rank Adaptation (LoRA). We further show that incorporating E-Branchformer, an architecture designed for speech recognition, as the inserted layers achieves ASR that matches or surpasses full fine-tuning on the larger model while reducing text degradation by over 75% with 60% fewer trainable parameters.

Kazuki Yano, Jun Suzuki, Shinji Watanabe

Tohoku University, Japan · Carnegie Mellon University, USA

Speech Language Models Automatic Speech Recognition Catastrophic Forgetting

Ask PDF

Aug 24, 2026

Do Spoken Language Models Hear Speech as They Read Text? Bridging Structural Gaps Between Speech and Text

Spoken Language Models (SLMs) generate textual responses directly from speech, offering an alternative to cascaded systems. Despite recent advances, existing SLMs still exhibit weaker instruction-following behavior and limited generalization across diverse tasks compared to text-based language models. Our analysis shows that speech and text representations in current SLMs remain weakly aligned despite strong downstream performance, indicating that structural differences between continuous, temporally varying speech and discrete text remain insufficiently addressed. To address this, we propose a simple framework that decouples length mismatch from semantic alignment and encourages closer correspondence between speech and text representations. Experiments across multiple benchmarks demonstrate competitive performance against strong baselines, underscoring the importance of explicitly addressing structural differences between speech and text in SLM training. Our code is publicly available at https://github.com/jaykim9870/Do_SLMs_Hear_Speech_as_They_Read_Text.

Hyeonyu Kim, Hwayeon Kim, Youngwon Choi +2

Maum AI Inc. · KAIST · Atmanity Inc.

Speech Language Models Spoken Language Understanding

Ask PDF

Mar 7, 2026

Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervision

Speech Large Language Models (LLMs) that understand and follow instructions in many languages are useful for real-world interaction, but are difficult to train with supervised fine-tuning, requiring large, task-specific speech corpora. While recent distillation-based approaches train performant English-only Speech LLMs using only annotated ASR data by aligning text and speech using only a lightweight projector, these models under-perform when scaled to multilingual settings due to language interference in the shared projector. We address this by introducing language-aware distillation using a query bank and a gating network that selects or mixes query tokens using a Q-Former projector. Our approach shows gains of 14% over matched multilingual distillation baselines on instruction following. We further synthesize Audio-MLQA, a multilingual spoken QA benchmark built on MLQA with high-quality TTS questions. Our best model improves over existing Speech LLM baselines by 32% on Audio-MLQA.

Shreyas Gopal, Donghang Wu, Ashutosh Anshul +5

College of Computing and Data Science, Nanyang Technological University, Singapore · AI Singapore, National University of Singapore · Institute for Infocomm Research (I2R), A∗STAR, Singapore

Speech Language Models Language Model Distillation Instruction Following

Ask PDF

Abstract

Speech Language Models

Explore similar work

Adapting Text LLMs to Speech via Multimodal Depth Up-Scaling

Do Spoken Language Models Hear Speech as They Read Text? Bridging Structural Gaps Between Speech and Text

Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervision