DirectSpeech2LLM: A Simple End-to-End Framework to Mitigate Prompt Overfitting in Speech-LLMs

Hemant Yadav; Sunayana Sitaram; Roger Zimmermann; Rajiv Ratn Shah

cs.CLOct 6, 2026

DirectSpeech2LLM: A Simple End-to-End Framework to Mitigate Prompt Overfitting in Speech-LLMs

Authors: Hemant Yadav, Sunayana Sitaram, Roger Zimmermann, Rajiv Ratn Shah

Organizations: IIIT Delhi, India · Microsoft Research, India · National University of Singapore

Abstract

Speech-LLMs often exhibit prompt overfitting, where models solely trained on automatic speech recognition (ASR) instruction fail to generalize to new instructions such as speech translation and continue to behave primarily as ASR system. We propose DirectSpeech2LLM, a simple end-to-end framework that preserves the instruction-following ability of the LLM on unseen tasks when conditioned on speech. It computes distance-based CTC loss over the frozen LLM embedding matrix and uses greedy CTC labels to derive geometrically and temporally aligned speech embeddings respectively as an input to the LLM. Trained solely on 960 hours of LibriSpeech ASR data, DirectSpeech2LLM outperforms the cascaded system on ASR (seen task) and generalizes zero-shot to speech translation and emotion recognition (two unseen tasks), closely matching the cascaded system upper bound on these two new instructions despite seeing neither during training. We also find that geometric alignment strength plays a smaller role than previously assumed, as our modified CTC loss is shown to provide sufficient implicit geometric grounding without requiring an explicit regression loss. Results are consistent across two LLM families and scale with both more training data and model capacity.

Figures & tables

Figure 1 : Overview of DirectSpeech2LLM . Given an input speech signal, the SFM produces frame representations $s_{t}\in\mathbb{R}^{d}$ , where $d$ matches the dimensionality of the LLM input embeddings $Q$ . Logits are computed as negative squared distances ( $Z_{t,v}$ ) followed by log-softmax and CTC loss ( $\mathcal{L}_{\text{ctc}}$ ) is applied. Next, the resulting greedy CTC predicted labels are then used to downsample SFM output embeddings via mean pooling. The LLM is fed with the resulting updated sequence embeddings from SFM and is trained with cross-entropy loss ( $\mathcal{L}_{\text{llm}}$ ). The reader should keep in mind that the LLM-IES serves only as a geometric anchor for the SFM-OES, without being replaced or discretized with the LLM input embeddings. Commitment loss is not shown in the figure for clarity.

Cascaded (CTC + LLM) (50k steps)
Model	LLM (Learnable)	Train data duration (hrs)	ASR (Librispeech)	ST (CoVoST2)		SQA (IEMOCAP)
			WER $\downarrow$	BLEU $\uparrow$		ACC $\uparrow$
			t-clean / t-other	en-de	en-zh	Emotion
Whisper + LLM [ 11 ]	LLama-7B	-	2.7 / 5.2	18.2	-	-
Ours ( Backbone )	Phi3.1-3.8B	1k	3.13 / 5.12	19.37	17.52	44.76 (IFR = 0.99)
Ours ( Backbone )	Qwen3-4B	1k	2.35 / 4.57	19.88	34.03	42.98 (IFR = 1.00)

Table 1 : Models marked with SFT uses supervised finetuning for that task. DirectSpeech2LLM shows near perfect instruction following on two unseen tasks, ST and ER as performance approaches to the upper bound of cascade system showing LLM’s generalization ability to new instructions is preserved. IFR, as explained in [ 13 ] is shown for SQA task. [(

\rightarrow

) means continued training after 38k hours.]

Hparams	ASR (WER $\downarrow$ )			ST (BLEU $\uparrow$ )
	Cascaded	end-to-end			Cascaded
	CTC $\rightarrow$ +LLM	CTC $\;/\;$ W (ith) B (lank) $\;/\;$ N (o) B (lank)		WB $\;/\;$ NB	CTC + LLM
	Libri-other (in-domain)		VP (out-of-domain))	CoVoST2 (en-de)
Stage 1: Modified -CTC (20k steps)
$\rightarrow$ $\mathcal{L}_{\text{ctc}}$	4.88 $\rightarrow$ 4.86	- / - / 100.00	16.14 / - / 100.00	- / 0.00	19.89
$\rightarrow$ $+$ $\mathcal{L}_{\text{commit}}$	4.94 $\rightarrow$ 4.85	- / - / 16.05	16.18 / - / 26.43	- / 10.44	19.92

Table 2 : Ablation study of DirectSpeech2LLM on Stage 1 and Stage 2. Assume Qwen3-4B model and LibriSpeech training data (1k) unless mentioned otherwise. Results compare a cascaded interface (CTC + LLM) with an end-to-end interface. WB and NB corresponds to With blank and No blank setup in training and inference both. Different rows show training setup and columns show inference setup. Hparams (rows) denote the effect of different losses, extended training steps, additional speech data (CV) or different LLM backbone. For stage 2 results, a fair comparison to Linear-CTC loss is Modified-CTC loss

-\ \mathcal{L}_{\text{commit}}

. Gray rows show the best results for each stage. Row marked with (

\ast

) is the closest equivalent to AlignFormer minus the input ordering trick.

	en-es	en-de
Wav2Prompt (CIF)	13.8	-
$-$ MSE	5.4	-
DirectSpeech2LLM (Modified CTC)	-	19.97
$-$ $\mathcal{L}_{\text{commit}}$	-	19.69

Table 3 : The BLEU scores show the sensitivity to the MSE loss in Wav2Prompt or commitment loss in DirectSpeech2LLM. Both are regression loss for geometric alignment.

Panel of Figure 2 — Figure 2 : Validation commitment loss of DirectSpeech2LLM with and without commitment loss.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Tasks	WB	WB decoding_E	NB	CTC + LLM
Libri-clean	2.4	2.95	2.8	2.39
Libri-other	4.39	5.73	5.51	4.61
en-de	21.45	21.61	21.68	20.29
en-zh	34.20	33.91	35.23	34.98
Emotion	40.89	39.35	41.53	43.55

Appendix

Table 4 : Multitask training setup with Librispeech and en-de training data without commitment loss.

Duration	WB	NB	CTC
1 min	.62	0.62	0.62
5 min	2.97	2.6	3.71
10 min	14.26	4.22	5.08
20 min	38.48	33.12	12.37

Appendix

Table 5 : Longform speech inference results.

Panel of Figure 3 — Figure 3 : Left: Stage 2 LLM WER vs LLM backbone shows training longer would improve performance further. Right: Commitment loss vs LLM backbone.

Figure 4 : TBP method overview. The authors of Byte Latent Transformer [ 47 ] use dynamic boundaries at step 2, while TBP uses static tokenizer dependent boundaries.

-	-	-	t-clean / t-other	en-de	en-zh	Emotion
end-to-end
Ours	Qwen-3-4B	1k	2.21 (2.05) / 4.28 (4.13)	19.97	33.74	40.16 (IFR = 1.00)
Ours $-$ $\mathcal{L}_{\text{commit}}$			2.13 (1.96) / 4.09 (3.93)	19.69	32.86	39.52 (IFR = 1.00)
Ours $+$ TBP			1.96 (1.81) / 4.06 (3.90)	19.79	32.74	40.97 (IFR = 1.00)

Appendix

Table 6 : Extension to Table 1 . Numbers with underline shows the best result for each test set. The improvements except on the ASR task are marginal. Overall, the results suggest that input tokenization has no substantial effect on the LLM’s task-solving ability. Values in parentheses report ASR results computed using the Whisper EnglishTextNormalizer compared to mms normalization.

Stage 2
Hparams	ASR (WER $\downarrow$ )			ST (BLEU $\uparrow$ )
	Cascaded	end-to-end			Cascaded
	CTC $\rightarrow$ +LLM	CTC $\;/\;$ W (ith) B (lank) $\;/\;$ N (o) B (lank)		WB $\;/\;$ NB	CTC + LLM
	Libri-other (in-domain)		VP (out-of-domain))	CoVoST2 (en-de)
Ours	4.60 $\rightarrow$ 4.57	- / 4.28 / 5.83	15.95 / 13.86 / 15.09	5.80 / 19.97 (49.4/26.1/15.1/9.1)	19.88 (48.8/25.7/14.8/8.9)
Ours_TBP	4.38 $\rightarrow$ 4.38	- / 4.06 / 4.43	15.74 / 13.89 / 14.42	11.44 / 19.79 (49.0/25.8/15.0/9.0)	20.10 (49.3/26.2/15.2/9.1)

Appendix

Table 7 : Extension to Table 2 . Numbers with underline shows the best result for each test set.

Figure 5 : Updated DirectSpeech2LLM_TBP method.

Z_{t,v}=-\left(\lVert s_{t}\rVert_{2}^{2}-2s_{t}^{\top}q_{v}+\lVert q_{v}\rVert_{2}^{2}\right)

Appendix

Algorithm 1 DirectSpeech2LLM method

Panel of Figure 6 — Figure 6 : Prompt template used for Stage 2 training and inference with Qwen3, including task-specific prompts for ASR, speech translation, and emotion recognition.

Stage 1 – Failure: Formatting
Real: i’d recommend him to you instead of blackstone thanks laughed kenneth
CTC: i’d recommend him to you instead of blackstone thanks laupped kenneth
LLM (NB): i d e r c o m e n d h i m t o y o u i n s t e a d o f b a c k s t o n e t h a n s l a l p e d k e n n e t h
Stage 2 – Failure: Formatting / Translation
Real: if they tried to run they were hit from behind if they stood still they were clubbed carefully
CTC: if they tried to run they were hit from behind if they stood still they were clubbed carefully

Appendix

Table 8 : Dominant failure cases of DirectSpeech2LLM on the ASR task when using the NB decoding strategy. Stage 1 formatting failures are expected as we ask the LLM to repeat. Even after stage 2 LLM finetuning, the formatting failures of limited vocabulary of speech tokenizer are still visible, very rare. We also found degradation in instruction following as sometimes, instead of repeating, the llm starts translating.

Base comparison
Model	Vocab	WER (%)
Phi-3.1-3.8B	$V_{\text{llm}}$	0.91
Qwen-3-4B	$V_{\text{llm}}$	0.02
Phi-3.1-3.8B	$V_{\text{ctc}}$	6.91
Qwen-3-4B	$V_{\text{ctc}}$	4.40
Qwen scaling under $V_{\text{ctc}}$

Appendix

Table 9 : Effect of using a

\sim

1k subset of the LLM vocabulary for tokenizing SFM-OES. When evaluated on LibriSpeech test-other using ground-truth transcripts as input to the ASR prompt. Performance degrades substantially under

V_{\text{ctc}}

, with Phi-3.1-3.8B showing higher sensitivity than Qwen-3-4B. No clear scaling trend is observed across Qwen models. The reader should keep in mind that inflated WER is because of formatting errors as shown in Table 8 in the Appendix. Furthermore, these errors to a large extent are resolved after finetuning.

Explore similar work

Unlocking Speech-Text Compositional Powers: Instruction-Following Speech Language Models without Instruction Tuning
Jul 2, 2026Congrui Du, Yang Zhang, Kaizhi Qian +1Speech Language Models Instruction Tuning
University of California, Santa Barbara, USA · MIT-IBM Computing Research Lab, IBM Research, USA
LLM can Read Spectrogram: Encoder-free Speech-Language Modeling
Jun 8, 2026Ruchao Fan, Yiming Wang, Yuxuan Hu +6Automatic Speech Recognition Spectral Features
Microsoft, USA
Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMs
Jun 8, 2026Ming-Hao Hsu, Yuxuan Hu, Shujie Liu +3Speech Language Models Token Embeddings
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China. · Microsoft Research, Redmond, WA, USA. · Microsoft Research Asia, Hong Kong, China.

Jul 2, 2026, cs.CLJ/K move · Enter open · S save

Unlocking Speech-Text Compositional Powers: Instruction-Following Speech Language Models without Instruction Tuning

Congrui Du, Yang Zhang, Kaizhi Qian, Shiyu Chang

University of California, Santa Barbara, USA · MIT-IBM Computing Research Lab, IBM Research, USA

Speech Language Models Instruction Tuning Multimodal Continual Instruction Tuning

Open paper AskPDF arXiv

Instruction tuning for speech language models (SLMs) is substantially more challenging than for text-based large language models (LLMs), as it requires learning a new modality and a wide range of speech-specific instructions in addition to those supported by text LLMs. Existing SLM training approaches largely replicate the text LLM training paradigm by synthesizing large-scale speech pre-training and instruction-tuning datasets. However, this strategy is difficult to scale, since speech sequences are significantly longer than text sequences. In this paper, we propose SpeechCombine, an instruction-following speech language model trained without any instruction tuning, using only a single round of speech pre-training on 30k hours of data. Starting from a text LLM base model, we perform continuous pre-training on speech utterances to obtain a speech-adapted model, and then directly combine its weights with the weight difference between the instruction-tuned and base versions of the text LLM. Our results show that this simple combination strategy not only preserves the knowledge and capabilities of the original text LLM, but also effectively transfers them to the speech domain. These findings suggest a new direction for SLM training that avoids reliance on massive speech data.

Abstract

Explore similar work

Unlocking Speech-Text Compositional Powers: Instruction-Following Speech Language Models without Instruction Tuning

LLM can Read Spectrogram: Encoder-free Speech-Language Modeling

Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMs