Extending large language models (LLMs) to the speech domain has recently gained significant attention. A typical approach connects a pretrained LLM with an audio encoder through a projection module and trains the resulting model on large-scale, task-specific instruction-tuning datasets. However, curating such instruction-tuning data for specific requirements is time-consuming, and models trained in this manner often generalize poorly to unseen tasks. In this work, we first formulate that the strongest generalization of a speech-LLM is achieved when it is trained with Self-Generated Instruction-Free Tuning (SIFT), in which supervision signals are generated by a frozen LLM using textual representations of speech as input. Our proposed SIFT paradigm eliminates the need for collecting task-specific question-answer pairs and yields the theoretically best generalization to unseen tasks. Building upon this paradigm, we introduce AZeroS (Auden Zero-instruction-tuned Speech-LLM), which is trained on speech-text pairs derived from publicly available corpora, including approximately 25,000 hours of speech with ASR transcripts and 3,000 hours of speech with paralinguistic labels. Built upon Qwen2.5-7B-Instruct, the model updates only two lightweight projection modules (23.8 million parameters each), while keeping both the LLM and audio encoders frozen. Despite the minimal training cost and modest data scale, AZeroS achieves state-of-the-art performance on both semantic and paralinguistic benchmarks, including VoiceBench, AIR-Bench Foundation (Speech), and AIR-Bench Chat (Speech).
Figures & tables
Figure 1: Overview of the proposed AzeroS framework built upon a frozen Qwen2.5-7B-Instruct backbone. The system couples a Self-Generated Data Construction pipeline (left) with a two-stage Self-Generated Instruction-Free Tuning (SIFT) training procedure (right). In Stage 1 (Semantic) , the LLM generates targets from speech transcripts to train a projector on the semantic TTA Liu et al. (2025) encoder. In Stage 2 (Semantic + Paralinguistic) , we augment inputs with metadata (e.g., emotion, gender, age); the resulting rich responses train a second projector on a paralinguistic Auden-Voice Huo et al. (2025) encoder. Together, this progressive SIFT paradigm extends the model’s capability from purely semantic to joint semantic–paralinguistic understanding without relying on task-specific instructions.
Method
Oracle Info h~(x) (Goal 1)
Instruction I (Goal 2)
Limitations/Characteristics
AudioChatLlama
Transcript Only (e.g., The weather is great! )
Null ( ∅ )
Semantic only alignment , ignores other info.
DESTA-2/2.5
Transcript + Attributes Tags (e.g., The weather is great! (Gender: Female, Emotion: Happy…))
Required (e.g., ”What can you hear from the audio?” )
Still requires instruction I ; Partial alignment, requires ASR text appended at inference.
SIT (Fallback)
Full Info , either in attributes format (e.g., The weather is great! (Gender: Female, Emotion: Happy…) ) or natural description (e.g., A woman says happily:”The weather is great!”… )
Required (e.g., Describe all the information you can hear. )
Elicit all info in rich response y ; Still relies on instruction I , may introduce instruction bias.
SIFT (Optimal)
Null ( ∅ )
Ideal optimal case No instruction bias. Generalized to any task.
Table 1: Unified Comparison of Alignment Strategies. We frame prior works as constrained special cases within our paradigm: AudioChatLlama Fathullah et al. (2024) is limited by information sufficiency (transcript only), while DESTA-2/2.5 Lu et al. (2024) ; Lu et al. (2025) relies on rigid instructions and auxiliary ASR inputs. SIFT overcomes these bottlenecks by satisfying both Information Maximization and Self-Elicit goals, achieving pure instruction-free alignment.
Figure 2: Illustration of semantic-only instruction-tuning configurations: (a) TSIT s , (b) SIT s , and (c) SIFT s . Examples are selected from an utterance in the IEMOCAP dataset Busso et al. (2008) , with the transcript “what shall we do if they suddenly walk in on us” . ( […] indicates truncated text. Detailed examples are in Appendix A .)
Figure 3: Illustration of semantic and paralinguistic tuning configurations: (a) SIT sp and (b) SIFT sp . Examples are selected from an utterance in the IEMOCAP dataset Busso et al. (2008) , with the transcript ”what shall we do if they suddenly walk in on us” and paralinguistic metadata (gender: female, emotion: happy) . ( […] indicates truncated text. Detailed examples are in Appendix A .)
VoiceBench
AIR-Bench (Speech)
Method
Alpaca Eval
Comm Eval
Wild Voice
SD- QA
BBH
Adv Bench
IF Eval
OBQA
MMSU
Overall
Gender
Emotion
Age
LID
Entity
Intent
Avg
Chat
Stage 1: semantic-only (on Data-S)
SIFT s
4.44
4.22
3.95
59.22
56.10
99.23
62.63
73.19
60.08
73.63
30.47
46.40
34.60
76.20
72.40
81.80
56.98
7.22
Stage 2: semantic + paralinguistic (on Data-SP)
SIT sp
4.38
4.14
3.86
57.60
55.40
98.65
60.01
71.87
58.85
72.22
74.85
73.10
57.20
85.30
72.90
87.20
75.09
8.19
SIFT sp
4.41
4.08
3.81
59.13
55.80
99.04
61.10
70.33
59.01
72.27
83.18
74.60
64.00
85.50
72.70
86.10
77.68
8.21
Table 4: Full results on VoiceBench and AIR-Bench, comparing multi-stage training configurations for comprehensive semantic + paralinguistic alignment , trained on both Data-S and Data-SP .
Figure 4: Comparison of tuning configurations for the Dialog-LLM: (a) SIT ssp and (b) SIFT ssp . By injecting a system prompt to enforce natural conversation, the LLM’s default behavior changes: it tends to prioritize conciseness over detail, leading to the loss of paralinguistic information in the SIFT targets (as seen in (b)). Explicit instructions (SIT) are required to force the model to verbalize these details (a). ( […] indicates truncated text. Detailed examples are in Appendix A .)
VoiceBench
AIR-Bench (Speech)
Method
Alpaca Eval
Comm Eval
Wild Voice
SD- QA
BBH
Adv Bench
IF Eval
OBQA
MMSU
Overall
Gender
Emotion
Age
LID
Entity
Intent
Avg
Chat
SIFT sp
4.41
4.08
3.81
59.13
55.80
99.04
61.10
70.33
59.01
72.27
83.18
74.60
64.00
85.50
72.70
86.10
77.68
8.21
SIFT ssp
4.37
4.01
3.77
58.77
57.80
96.92
61.88
65.27
53.97
70.85
78.06
71.95
53.40
88.20
70.90
88.30
75.14
8.09
SIFT ssp + SIT ssp
4.40
4.14
3.80
58.23
55.20
98.46
60.95
67.69
55.47
71.42
82.03
68.15
67.00
88.70
73.70
87.50
77.85
8.17
Table 7: Comparison results on VoiceBench and AIR-Bench among SIFT sp , SIFT ssp and SIFT ssp + SIT ssp . Performance scores lower than expected are marked by underlines.
Figure 5: Comparison of intrinsic response behaviors between the two alignment targets at inference time (without system prompts). The ssp model (First Replier), trained with SIFT+SIT, acts as a Speech Dialog Role , providing natural and concise human-like responses. In contrast, the sp model (Second Replier), trained with SIFT, acts as a Speech Understanding Role , providing detailed analytical descriptions.
Instruction tuning for speech language models (SLMs) is substantially more challenging than for text-based large language models (LLMs), as it requires learning a new modality and a wide range of speech-specific instructions in addition to those supported by text LLMs. Existing SLM training approaches largely replicate the text LLM training paradigm by synthesizing large-scale speech pre-training and instruction-tuning datasets. However, this strategy is difficult to scale, since speech sequences are significantly longer than text sequences. In this paper, we propose SpeechCombine, an instruction-following speech language model trained without any instruction tuning, using only a single round of speech pre-training on 30k hours of data. Starting from a text LLM base model, we perform continuous pre-training on speech utterances to obtain a speech-adapted model, and then directly combine its weights with the weight difference between the instruction-tuned and base versions of the text LLM. Our results show that this simple combination strategy not only preserves the knowledge and capabilities of the original text LLM, but also effectively transfers them to the speech domain. These findings suggest a new direction for SLM training that avoids reliance on massive speech data.
Congrui Du, Yang Zhang, Kaizhi Qian +1
University of California, Santa Barbara, USA · MIT-IBM Computing Research Lab, IBM Research, USA
Speech-LLMs often exhibit prompt overfitting, where models solely trained on automatic speech recognition (ASR) instruction fail to generalize to new instructions such as speech translation and continue to behave primarily as ASR system. We propose DirectSpeech2LLM, a simple end-to-end framework that preserves the instruction-following ability of the LLM on unseen tasks when conditioned on speech. It computes distance-based CTC loss over the frozen LLM embedding matrix and uses greedy CTC labels to derive geometrically and temporally aligned speech embeddings respectively as an input to the LLM. Trained solely on 960 hours of LibriSpeech ASR data, DirectSpeech2LLM outperforms the cascaded system on ASR (seen task) and generalizes zero-shot to speech translation and emotion recognition (two unseen tasks), closely matching the cascaded system upper bound on these two new instructions despite seeing neither during training. We also find that geometric alignment strength plays a smaller role than previously assumed, as our modified CTC loss is shown to provide sufficient implicit geometric grounding without requiring an explicit regression loss. Results are consistent across two LLM families and scale with both more training data and model capacity.
Hemant Yadav, Sunayana Sitaram, Roger Zimmermann +1
IIIT Delhi, India · Microsoft Research, India · National University of Singapore
While Large Language Models excel in natural language processing, efficiently extending their capabilities to spoken input remains a significant challenge. Existing methods for building SpeechLLMs often rely on computationally expensive full-model fine-tuning, or employ parameter-efficient projectors that suffer from inefficient token sequence lengths and costly full-model supervision. In this paper, we introduce Aligned Continuous Integrate-and-Fire, a highly efficient framework for zero-shot speech processing. Our method dynamically compresses continuous acoustic frames into the exact discrete token length of the target text utilizing explicit Dynamic Time Warping alignments. This allows our initial training stage to establish a robust acoustic-to-semantic bridge using lightweight distance metrics, entirely bypassing the computationally expensive LLM forward pass. For subsequent fine-tuning, we propose a memory-efficient knowledge distillation objective that targets a single LLM layer, performing competitively with full-model cross-entropy training at a fraction of the computational cost. Through extensive evaluations on Automatic Speech Recognition and Speech Translation, we demonstrate that our method achieves superior performance compared to prior parameter-efficient baselines.
Abderrahmane Issam, Yusuf Can Semerci, Jan Scholtes +1
Department of Advanced Computing Sciences Maastricht University