Large language models show that simple autoregressive training can yield scalable and coherent generation, but extending this paradigm to speech remains challenging due to the entanglement of semantic and acoustic information. Most existing speech language models rely on text supervision, hierarchical token streams, or complex hybrid architectures, departing from the single-stream generative pretraining paradigm that has proven effective in text. In this work, we introduce WavSLM, a speech language model trained by quantizing and distilling self-supervised WavLM representations into a single codebook and optimizing an autoregressive next-chunk prediction objective. WavSLM jointly models semantic and acoustic information within a single token stream without text supervision or text pretraining. Despite its simplicity, it achieves competitive performance on consistency benchmarks and speech generation while using fewer parameters, less training data, and supporting streaming inference.
Figures & tables
Figure 1: WavSLM architecture. Raw speech is processed by FocalCodec-Stream \textcolor blue(blue), which includes the feature extractor and lower WavLM layers, followed by a compressor, quantizer, decompressor, and decoder to produce a low-bitrate, single-stream sequence of discrete tokens. The decompressor converts tokens back into continuous features that are compatible with the upper WavLM layers \textcolor red(red). These layers are used as a causal speech language modeling backbone, with a lightweight language modeling head on top. The model is trained with a next-chunk prediction objective, jointly predicting C=4 tokens at each autoregressive step.
Model
Params
Textually Pretrained
Codebooks
Acoustic Consistency
Alignment
Spoken Content
Avg ↑
Sent. ↑
Spk. ↑
Gend. ↑
Sent. ↑
sWUGGY ↑
sBLiMP ↑
tSC ↑
Human [ 14 , 36 ]
–
–
–
97.2
91.5
98.6
93.3
–
–
90.2
–
Large-Scale Baselines TWIST 1.3B [ 14 , 36 ]
1.3B
✓
1 × 500
61.5
69.0
69.5
53.0
72.7
57.0
70.6
64.8
TWIST 7B [ 14 , 36 ]
7.0B
✓
1 × 500
61.5
71.0
70.0
51.5
74.5
59.2
76.4
66.3
SpiRit LM [ 18 , 36 ]
7.0B
✓
1 × 501
54.5
69.5
67.0
48.0
69.0
58.3
82.9
64.2
SpiRit LM Expr. [ 18 , 36 ]
7.0B
✓
[501, 64, 100]
73.5
81.0
85.0
52.0
65.0
54.2
75.4
69.4
Table 1: Likelihood-based evaluation. Best and second-best results are highlighted. Human scores are boxed when they outperform all models.
Model
Params
Textually Pretrained
Codebooks
UTMOS ↑
Sim ↑
PPL ↓
RTF ↑
LLaMA-Mimi 1.3B [ 24 ]
1.3B
✓
4 × 2048
3.57
91.3
153
2.0
LLaMA-Mimi 8B [ 24 ]
8.0B
✓
4 × 2048
3.56
91.5
122
1.1
WavSLM-2k
305M
✗
1 × 2048
3.72
91.8
161
5.9
WavSLM-4k
307M
✗
1 × 4096
3.69
91.6
162
5.8
WavSLM-65k
370M
✗
1 × 65536
3.66
91.3
210
5.8
Table 2: Generation-based evaluation. Best and second-best results are highlighted.
Window Size
Chunk Size
Acoustic Consistency
Alignment
Spoken Content
Avg ↑
Generation
Sent. ↑
Spk. ↑
Gend. ↑
Sent. ↑
sWUGGY ↑
sBLiMP ↑
tSC ↑
UTMOS ↑
Sim ↑
PPL ↓
RTF ↑
512
4
75.0
88.5
90.5
51.5
63.7
53.9
63.3
69.5
3.69
91.6
162
5.8
1024
4
76.0
87.5
90.0
47.0
64.8
54.7
66.7
69.5
3.69
91.7
151
5.8
2048
4
74.5
88.5
88.5
49.0
64.6
54.7
66.9
69.5
3.70
91.7
148
5.8
512
8
78.5
86.0
83.0
48.5
63.7
55.5
64.8
68.6
2.92
90.0
174
10.9
512
16
71.5
84.5
81.5
49.0
57.5
54.7
62.7
65.9
1.97
86.5
181
16.4
Table 3: Effect of window and chunk size for WavSLM-4k. Best and second-best results are highlighted.
Scaling Multimodal Large Language Models (MLLMs) to long-form speech is bottlenecked by the explosive growth of input tokens. Existing speech-language models project high-frame-rate acoustic features directly into the LLM input space, making long-context processing computationally prohibitive. Unlike images or videos, speech lacks spatial redundancy, making extreme token compression particularly challenging. To address this limitation, we propose FastSLM, a token-efficient architecture featuring the Hierarchical Temporal Abstractor (HTA), which progressively distills acoustic features across multiple temporal scales. HTA achieves an extreme compression rate of 1.67 tokens per second (97% reduction) while preserving essential linguistic information for downstream speech-language understanding. Experimental results demonstrate that FastSLM achieves competitive performance across diverse speech-language tasks while requiring substantially fewer speech tokens and FLOPs than existing speech-language models. The source code and model checkpoints are available at https://github.com/Lee-junseok1025/FastSLM.
Spoken Language Models (SLMs) generate textual responses directly from speech, offering an alternative to cascaded systems. Despite recent advances, existing SLMs still exhibit weaker instruction-following behavior and limited generalization across diverse tasks compared to text-based language models. Our analysis shows that speech and text representations in current SLMs remain weakly aligned despite strong downstream performance, indicating that structural differences between continuous, temporally varying speech and discrete text remain insufficiently addressed. To address this, we propose a simple framework that decouples length mismatch from semantic alignment and encourages closer correspondence between speech and text representations. Experiments across multiple benchmarks demonstrate competitive performance against strong baselines, underscoring the importance of explicitly addressing structural differences between speech and text in SLM training. Our code is publicly available at https://github.com/jaykim9870/Do_SLMs_Hear_Speech_as_They_Read_Text.
While Large Multimodal Models excel in comprehension, high-throughput inference engines lack native support for multimodal generation. This is severe in Speech Language Models, where generating multi-layered audio tokens via decoupled AR+NAR or synchronous Multi-Token Prediction (MTP) with delay-pattern interleaving conflicts with standard single-stream loops. We present a vLLM-based inference pipeline for unified speech understanding and generation. We extend autoregressive decoding to natively execute delay-pattern de-interleaving and coordinated multi-stream sampling, integrating an on-GPU acoustic decoder for end-to-end waveform synthesis. Crucially, we overcome the shared intuition that Classifier-Free Guidance (CFG) halves throughput. By co-scheduling paired conditional and unconditional requests within a continuous batch, our CFG implementation sustains 80% of non-CFG throughput, absorbing dual-request and logit merging overheads. We open-source our framework.