Adapting Text LLMs to Speech via Multimodal Depth Up-Scaling

Kazuki Yano; Jun Suzuki; Shinji Watanabe

cs.CLApr 1, 2026

Adapting Text LLMs to Speech via Multimodal Depth Up-Scaling

Authors: Kazuki Yano, Jun Suzuki, Shinji Watanabe

Organizations: Tohoku University, Japan · Carnegie Mellon University, USA

Abstract

Adapting pre-trained text Large Language Models (LLMs) into Speech Language Models (Speech LMs) via continual pretraining on speech data is promising, but often degrades the original text capabilities. We propose Multimodal Depth Upscaling, an extension of an emerging strategy in continual LLM pre-training, where new transformer layers are inserted into a frozen text LLM and only the added layers are trained on speech data. Experiments with SmolLM2-360M and SmolLM2-1.7B on 48k hours of English Automatic Speech Recognition (ASR) data show that depth up-scaling achieves ASR comparable to full fine-tuning while causing far less text degradation than both full fine-tuning and Low-Rank Adaptation (LoRA). We further show that incorporating E-Branchformer, an architecture designed for speech recognition, as the inserted layers achieves ASR that matches or surpasses full fine-tuning on the larger model while reducing text degradation by over 75% with 60% fewer trainable parameters.

Figures & tables

Figure 1: Proposed multimodal depth up-scaling strategy for Speech LM adaptation: trainable E-Branchformer blocks are interleaved with frozen transformer blocks to acquire speech capabilities while preserving pre-trained text representations.

Model	Method	Position	Trainable	WER $\downarrow$		Text $\uparrow$
Model	Method	Position	Trainable	clean	other	Avg	$\Delta$
360M	pre-trained	–	–	–	–	56.8	0.0
	Full fine-tuning	–	360M	2.7	6.0	34.6	-22.2
	LoRA	–	200M	11.3	17.3	34.9	-22.0
	Depth Up-scaling	Interleaved	200M	3.1	6.7	54.9	-1.9
		Sandwich	200M	3.3	7.0	53.1	-3.8

Table 1: Results on SmolLM2-360M and SmolLM2-1.7B. WER (%) on LibriSpeech (lower is better). Text task accuracy is the average of 8 benchmarks (%, higher is better).

\Delta

denotes the change from the pre-trained model. Depth up-scaling results are with added layers kept at inference. Dropping added layers recovers the pre-trained performance (

\Delta=0.0

) by design. Italic = full fine-tuning (updates all parameters). Bold = best among depth up-scaling.

Model	Added layer type	Trainable	WER $\downarrow$		Text $\uparrow$
Model	Added layer type	Trainable	clean	other	Avg	$\Delta$
360M	Full fine-tuning	360M	2.7	6.0	34.6	-22.2
	Standard transformer	200M	3.1	6.7	54.9	-1.9
	E-Branchformer	220M	2.9	6.2	54.6	-2.2
1.7B	Full fine-tuning	1.87B	2.3	5.6	36.5	-32.6
	Standard transformer	0.66B	2.4	5.5	60.8	-8.3

Table 2: Effect of added layer architecture ( Interleaved placement) and function preserving initialization. Top: architecture comparison. Bottom: initialization comparison for E-Branchformer on SmolLM2-1.7B.

\mathbf{W}_{\mathrm{Merge}}

is the merge projection (§ 3.3 ).

Task	Prompt	Depth Up-scaling	Full FT
Translation	Translate the sentence into French	Une indifférence froide, claire, régnait dans l’âme.	in in in in in in …
Simplification	Rewrite into simple, modern English	A chilly indifference pervaded his soul.	in in in in in in …
Summarization	Summarize in one clear sentence	The man was indifferent to everything.	in in in in in in …

Table 3: Zero-shot prompt-based task outputs on SmolLM2-1.7B ( Interleaved , E-Branchformer). Full FT = full fine-tuning, Depth = multimodal depth up-scaling. Input speech: “ A cold, lucid indifference reigned in his soul. ”

Speech Language Models

Speech language models connect spoken audio with large language models for understanding, transcription, translation, and real-time dialogue.

See all 301 papers

Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM
May 7, 2026Wenqian Cui, Xiao-Hui Li, Daxin Tan +2Audio Understanding Speech Prosody
The Chinese University of Hong Kong · Huawei Technologies
FastSLM: Hierarchical Temporal Abstraction for Efficient Long-Form Speech Adaptation
Jan 8, 2026Junseok Lee, Chang-Jae ChunMultimodal Large Language Models Speech Processing
OKESTRO · Sejong University
Distilling Text LMs into Speech-to-Speech LMs by Matching Internal Text Predictions
Oct 12, 2026Ju-Chieh Chou, Jiawei Zhou, Karen LivescuSpeech Language Models Language Model Distillation
TTI-Chicago · Stony Brook University

May 7, 2026J/K move · Enter open · S save

Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM

Wenqian Cui, Xiao-Hui Li, Daxin Tan, Qiyong Zheng+1

The Chinese University of Hong Kong · Huawei Technologies

Audio Understanding Speech Prosody Speech Language Models

Open paper AskPDF arXiv

Speech large language models (SLMs) are typically built from text large language model (TLM) checkpoints, yet they still suffer from a substantial modality gap. Prior work has mainly attempted to reduce this gap from the output side by making speech generation more text-like, but the gap remains. We argue that the key remaining bottleneck lies on the input side. We propose TextPro-SLM, an SLM that makes spoken input more closely resemble that of a prosody-aware text LLM. TextPro-SLM combines WhisperPro, a unified speech encoder that produces synchronized text tokens and prosody embeddings, with an LLM backbone trained to preserve the semantic capabilities of the original TLM while learning paralinguistic understanding. Experiments show that TextPro-SLM achieves the lowest modality gap among leading SLMs at both 3B and 7B scales, while also delivering strong overall performance on paralinguistic understanding tasks. These gains are achieved with only roughly 1,000 hours of LLM training audio, suggesting that reducing the modality gap from the input side is both effective and data-efficient.

Abstract

Speech Language Models

Explore similar work

Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM

FastSLM: Hierarchical Temporal Abstraction for Efficient Long-Form Speech Adaptation

Distilling Text LMs into Speech-to-Speech LMs by Matching Internal Text Predictions