Generative spoken language models (SLMs) offer a route to learning linguistic structure purely from speech, yet they lag far behind text-based models in both capability and data scaling efficiency. We identify two bottlenecks underlying this gap: the limited linguistic quality of existing speech units and the reliance on a single stream of discrete units to represent speech. Multiple codebooks are widely used to encode acoustic features but rarely to represent speech semantics. We present GSLM2, which learns and models complementary properties of speech at multiple levels of abstraction. Its encoder, SpidR 2.0, uses self-supervised learning with differentiable prediction heads to induce high-quality discrete units at multiple levels. A multi-stream SLM then jointly models these units using a shared temporal backbone, stream-specific predictors, and an autoregressive head that captures cross-stream dependencies. Across training data sizes and model scales, up to 6M hours and 6.5B parameters, GSLM2 consistently improves linguistic competence while scaling more efficiently with data. As a result, it substantially narrows the gap between spoken and text-based LMs; across read and conversational speech domains, we shrink the gap by about two orders of magnitude on sWuggy, sBLIMP, and tSC, while also outperforming competitive speech-text systems on all three tasks. We open-source the training code and model checkpoints at https://github.com/facebookresearch/gslm2.
Figures & tables
Figure 1: Overview of GSLM2, comprising SpidR 2.0 and multi-stream SLM. Left : SpidR 2.0 extracts discrete speech units from multiple encoder layers, which are jointly modeled by the multi-stream SLM. Right : Scaling performance of SLMs and a text LM trained on varying amounts of Libri-Light data, measured by the average accuracy across thes Wuggy, sBLIMP, and tSC tasks.
Benchmark
Metric
Capability
Evaluation
ZeroSpeech’21
sWuggy(-all)
Lexical
Real word vs. phonotactically matched nonword.
sBLIMP
Syntax
Grammatical vs. minimally different ungrammatical sentence.
StoryCloze (SC)
Topic SC (tSC)
Semantics
Correct ending vs. random ending from another story.
Spoken SC (sSC)
Semantics
Correct ending vs. adversarial, superficially plausible ending.
LongTail-Swap
WordSwap (WS)
Semantics
Context-appropriate word vs. frequency- and POS-matched substitution.
InflectionSwap (IS)
Morphosyntax
Context-appropriate inflection vs. another form of the same lemma.
Table 1: Spoken-language evaluation suite. Metrics described with “vs.” use likelihood-based paired comparisons, while the others assess prompted generation.
Figure 3
Training Regime
Model
#Params
#Data † [h]
sWuggy
sBLIMP
tSC
sSC
Speech-only training
GSLM2 (ours; Libri-Light)
300M
6K
76.3
67.2
78.9
52.5
1.2B
60K
79.3
71.8
84.5
57.6
GSLM2 (ours; Conversational Speech)
6.5B
6M
80.7
72.9
90.5
63.2
GSLM ( Lakhotia et al., 2021 )
100M
6K
64.8
54.2
66.6
53.3
AudioLM ( Borsos et al., 2023 )
300M
60K
71.5
64.7
–
–
Cuervo & Marxer (2024)
823M
160K
–
61.3
75.4
–
Table 3: Comparison of GSLM2 with prior SLMs under different training regimes. Prior results are from cited papers unless marked ‡ (our evaluation). Bold indicates the best overall result; underlining indicates the best prior baseline in each regime. † The amount of speech data used for training.
Model
τ^
ASR-PPL ( ↓ )
LLMJ ( ↑ )
600 h / 90M
Text LM
1.22
690.0 ± 12.9
1.88 ± 0.02
SpidR SLM
1.31
670.5 ± 0 9.4
1.73 ± 0.02
SpidR 2.0 SLM
1.34
626.5 ± 14.3
1.78 ± 0.02
GSLM2
1.49
583.5 ± 12.1
1.80 ± 0.02
6K h / 300M
Text LM
1.21
331.0 ± 0 6.5
2.30 ± 0.03
SpidR SLM
1.39
407.2 ± 0 9.7
2.20 ± 0.02
Table 4: Prompt-conditioned continuation results. τ^ is the decoding temperature calibrated with ground-truth repetition statistics.
ABX ( ↓ )
MAP ( ↑ )
Model
l∗
within
across
Avg.
l∗
Score
DinoSR
5
4.58
6.02
5.30
11
69.49
SpidR
6
3.53
4.31
3.92
5
63.42
SpidR 2.0
10
2.54
3.13
2.83
7
69.58
Table 5: ABX and MAP results for DinoSR, SpidR, and SpidR 2.0, reported at the best layer l∗ for each metric.
Figure 3: Layer-wise ABX, MAP, and single-stream SLM results. SLMs have 90M parameters and are trained on 6K hours of Libri-Light. The horizontal axis denotes the layer index l . Error bars for SpidR and SpidR 2.0 indicate the standard deviation across eight encoder-training seeds.
Appendix figures & tables38 assets
Supplementary material from the paper’s appendix.
Appendix
Corpus
Language(s)
Component
Training data
LibriSpeech
English
Speech encoder
960 h
Libri-Light
English
SLM
600 h / 6K h / 60K h
Duration predictor
600 h
Libriheavy
English
Text LM
Transcripts aligned with Libri-Light
LJ Speech
English
Unit vocoder
24 h
VoxPopuli
En, Fr, De
Speech encoder
6K h per language
Appendix
Table 6: Training corpora, languages, and data scales for GSLM components and text LMs.
Encoder
Language model
Hyperparameter
90M
90M
300M
1.2B
2.5B
6.5B
Transformer layers
12
12
24
24
32
32
Hidden dimension
768
768
1,024
2,048
2,560
4,096
FFN dimension
3,072
3,072
4,096
8,192
10,240
16,384
Attention heads
12
12
16
32
32
32
Appendix
Table 7: Transformer backbone configurations for the speech encoders and spoken/text LMs. Scale labels denote nominal backbone sizes, and Table 8 reports the exact total parameter counts for the LMs.
LM backbone scale
Model
90M
300M
1.2B
2.5B
6.5B
Text LM
92.9M
312.8M
1.23B
2.54B
6.49B
SpidR SLM
87.0M
304.9M
1.21B
2.52B
6.45B
SpidR 2.0 SLM
87.0M
304.9M
1.21B
2.52B
6.45B
GSLM2
96.3M
315.5M
1.23B
2.54B
6.48B
Appendix
Table 8: Total parameter counts for the LMs in the main comparison (Table 2 and Figure 2 ). Each count includes the Transformer backbone and its associated input and output modules.
Data
Model
#Params
Peak learning rate
Steps
Global batch (units/tokens)
Context length
#GPUs
600 h
Text LM
90M
10−2
10k
81,920
2,048
8 DDP
SLM
90M
10−2
15k
655,360
2,048
8 DDP
GSLM2
90M
10−2
15k
1,310,720
4,096
32 DDP
6K h
Text LM
300M
5.0×10−3
20k
491,520
2,048
8 DDP
SLM
300M (90M)
5.0×10−3(10−2)
25k
655,360
2,048
8 DDP
GSLM2
300M (90M)
5.0×10−3(10−2)
25k
1,310,720
4,096
32 DDP
Appendix
Table 9: Training configurations for text LMs, single-stream SLMs, and GSLM2 across data and model scales. Parenthesized values correspond to the parenthesized model size.
Compute (FLOPs)
Scale
Model
Training
Inference / audio s
600 h / 90M
Text LM
5.26×1017
9.52×108
SpidR 2.0 SLM
6.13×1018
3.22×109
GSLM2
2.34×1019
1.11×1010
6K h / 300M
Text LM
2.10×1019
3.17×109
SpidR 2.0 SLM
3.47×1019
1.09×1010
Appendix
Table 10: Training and inference compute across model scales. Training FLOPs include both forward and backward passes, whereas inference FLOPs correspond to a forward pass per second of audio. Inference FLOPs of text LMs assume 4.45 BPE tokens/s, measured on Libriheavy transcriptions.
Figure 13
Figure 6: Error scaling on conversational speech, both axes logarithmic. Each marker is a trained model; lines are the fitted power law error=aN−γ , whose slope is −γ . Fitted exponents γ are given in each panel; a larger γ means error falls faster as training data grows.
Figure 7: Quality-diversity sweeps for temperature calibration. Each point represents an evaluated temperature, and endpoint labels indicate the temperature range of each sweep. The dashed line at NRS=1 indicates the calibration target, where generated continuations match the repetition level of ground-truth continuations under combined unigram-bigram auto-BLEU. Values above and below one indicate greater and lower repetition, respectively. Results are averaged over ten random runs. The oracle is computed directly from the ground-truth continuations.
Figure 16
ZS21 ( ↑ )
SC ( ↑ )
ID
Model
sWuggy
sBLIMP
tSC
A0
GSLM2
76.3
67.2
78.9
A1
↪ w/o self-distillation ( Ldistill→Lce )
68.5
57.3
70.1
A2
↪ w/o single-stream loss ( β=0.0 )
67.9
55.8
65.4
A3
↪ Halve context length & batch size
75.7
66.8
78.3
A4
SpidR 2.0 SLM
73.9
65.1
74.7
Appendix
Table 11: Ablations of GSLM2 (Libri-Light 6K h; 300M parameters) on the primary evaluation metrics, including sWuggy, sBLIMP, and tSC. A1 and A3 modify the full model ( A0 ), while A2 builds on A1 . Values are accuracies (%; chance level: 50%).
Stream-specific head
Generative head
Order j
Layer π(j)
sWuggy
sBLIMP
tSC
sWuggy
sBLIMP
tSC
1
10
70.1
62.8
77.2
57.2
53.6
59.3
2
5
71.7
64.8
73.7
61.8
53.5
55.9
3
6
72.1
64.7
77.6
57.5
52.0
53.2
4
9
71.5
64.0
79.3
60.9
52.2
61.8
5
8
72.3
63.0
77.7
55.6
50.8
51.8
Appendix
Table 12: Performance of GSLM2 (Libri-Light 6K h; 300M parameters) using different heads for scoring. Stream-specific heads score unit-change positions, whereas the generative head scores all retained positions. Individual rows select one stream’s score while retaining all input streams; “All streams” averages per-stream scores before making paired comparisons. Scores are accuracies (%; chance level: 50%).
ZS21 ( ↑ )
SC ( ↑ )
Model
Unit layers
sWuggy
sBLIMP
tSC
SpidR 2.0 SLM
7
73.9
65.1
74.7
GSLM2
5–10
76.3
67.2
78.9
7 (repeated)
73.7
65.3
74.4
7 (sampled)
69.9
61.0
68.4
Appendix
Table 13: Effect of layer diversity. All GSLM2 variants use six streams and cross-stream deduplication. The repeated-layer control copies the layer-7 argmax sequence across streams; the sampled control independently samples units from the same layer’s categorical distributions for each stream. Scores are accuracies (%; chance level: 50%).
ZS21 ( ↑ )
SC ( ↑ )
Model
#Params
sWuggy
sBLIMP
tSC
sSC
Ours
GSLM2 (Libri-Light 60K h)
1.2B
79.3
71.8
84.5
57.6
GSLM2 (Conversational Speech 6M h)
6.5B
80.7
72.9
90.5
63.2
Open-weight speech LLMs
Gemma 4 ( Gemma Team, 2026 )
4B
72.6
67.7
88.6
80.4
Appendix
Table 14: Results for open-weight and proprietary speech-capable LLMs on ZeroSpeech’21 (ZS21) and StoryCloze (SC). Speech LLMs use prompted forced-choice evaluation. GSLM2 is included as a speech-only reference evaluated using likelihood-based scoring. Scores are accuracies (%; chance level: 50%).
Figure 10: LT-Swap accuracy on WordSwap (WS), InflectionSwap (IS), and AgreementSwap (AS) by word-frequency bin at the three Libri-Light training scales. Chance level: 50%.
ASR-PPL ( ↓ )
LLMJ ( ↑ )
Model
τ=1.0
τ=0.8
τ=1.0
τ=0.8
Oracle
Text LM
55.2
4.55
SpidR SLM
60.1
4.54
SpidR 2.0 SLM
70.1
4.41
GSLM2
59.5
4.53
600 h / 90M
Text LM
229.1 ± 5.8
0 76.8 ± 2.4
2.08 ± 0.01
2.22 ± 0.02
Appendix
Table 15: Speech-continuation performance at fixed decoding temperatures τ∈{1.0,0.8} , complementing the results with calibrated temperatures in Table 4 . Oracle rows evaluate ground-truth continuations.
Language
Model
sWuggy
sBLIMP
tSC
English
Text LM
77.0
63.9
76.8
SpidR SLM
60.3
53.3
69.3
SpidR 2.0 SLM
62.9
55.3
69.6
GSLM2
65.6
57.2
72.7
French
Text LM
–
95.0
75.7
SpidR SLM
–
58.8
63.9
Appendix
Table 16: Results for SLMs trained on 6K-hour English, French, and German data from VoxPopuli. Text LMs serve as toplines, which are trained using ASR transcriptions. Scores are accuracies (%; chance level: 50%).
ZS21 ( ↑ )
SC ( ↑ )
Model
Training Input
Evaluation Input
(s)Wuggy
(s)BLIMP
tSC
Text LM
Ground-truth text
B1 : Ground-truth text
79.5
66.8
84.8
B2 : ASR transcript of speech
70.6
71.0
75.4
ASR transcripts
B3 : Ground-truth text
81.2
66.9
85.1
B4 : ASR transcript of speech
71.5
69.1
77.4
GSLM2
Speech
Speech
76.3
67.2
78.9
Appendix
Table 17: Performance of Text LMs and GSLM2 (Libri-Light 6K h; 300M parameters) under different training and evaluation conditions. Text LMs are trained on either ground-truth or ASR transcripts. In the cascaded setting, spoken stimuli are transcribed and the resulting transcripts are scored by the Text LM. GSLM2 is trained and evaluated directly on speech. Scores are accuracies (%; chance level: 50%).
Figure 11: LT-Swap accuracy by word-frequency bin, comparing cascaded ASR-LM system ( B4 in Table 17 ) with Text LM and GSLM2. Chance level: 50%.
Figure 12: Layerwise codebook quality on ZeroSpeech’21 dev-clean . DinoSR uses the official released checkpoint; SpidR and SpidR 2.0 show mean ± standard deviation over eight training seeds. Purity and PNMI are in %.
Figure 13: Phone content of the discrete units. Conditional distribution P(phone∣code) on LibriSpeech dev-clean , computed using forced alignments at 10 ms and discrete codes at 20 ms. Rows are the 40 phone labels ordered by corpus frequency ( SIL first); columns are the active codewords only (203 in (a) SpidR and 128 in (b) SpidR 2.0), grouped by their dominant phone in row order and sorted within each group by the conditional probability of that phone. Concentration on a single phone within each column indicates high phone purity; the column ordering arranges these peaks into a staircase pattern.
Variant
Best layer l∗
l=l∗
l=5
l=10
SpidR (reproduced)
6
5.16 ± 1.37
5.25 ± 0.76
10.90 ± 2.97
+ shared heads
6
4.82 ± 0.57
5.75 ± 0.74
6.78 ± 1.02
+ regularization
9
4.12 ± 0.34
8.75 ± 0.75
4.29 ± 0.42
+ normalizations
10
3.58 ± 0.31
5.88 ± 0.27
3.58 ± 0.31
+ LR=2.0×10−4 ( SpidR 2.0 )
10
2.88 ± 0.12
4.73 ± 0.22
2.88 ± 0.12
Appendix
Table 18: ABX ablation results (%; lower is better), averaged over dev-clean and dev-other and over within- and across-speaker conditions. Modifications are cumulative. Values are mean ± standard deviation across eight seeds of encoder training. For each variant, l∗ denotes the layer with the lowest task-averaged ABX error. l=5 and l=10 provide fixed-layer comparisons.
Variant
Best layer
Average
dev-clean
dev-other
SpidR (reproduced)
4
58.3 ± 2.5
65.2 ± 3.0
51.4 ± 3.9
+ shared heads
5
61.3 ± 3.4
66.2 ± 5.4
56.4 ± 4.1
+ regularization
7
68.0 ± 1.6
73.3 ± 2.2
62.7 ± 2.4
+ normalizations
8
72.3 ± 0.3
77.0 ± 0.3
67.5 ± 0.4
+ LR=2.0×10−4 ( SpidR 2.0 )
7
69.2 ± 0.3
75.3 ± 0.4
63.1 ± 0.5
Appendix
Table 19: MAP ablation results (%; higher is better). Modifications are cumulative. Values are mean ± standard deviation across eight seeds of encoder training. For each variant, all scores are reported at the layer with the highest average MAP score.
Variant
Best layer
Average
sWuggy
sBLIMP
tSC
SpidR (reproduced)
5
63.9 ± 0.6
66.4 ± 1.4
56.4 ± 1.0
68.9 ± 0.8
+ shared heads
5
58.3 ± 0.7
59.9 ± 1.4
51.6 ± 0.8
63.3 ± 1.3
+ regularization
7
66.7 ± 0.9
67.5 ± 2.3
60.4 ± 1.0
72.3 ± 0.9
+ normalizations
8
69.1 ± 0.7
70.1 ± 1.6
63.0 ± 1.2
74.1 ± 0.7
+ LR=2.0×10−4 ( SpidR 2.0 )
7
70.1 ± 0.7
72.1 ± 1.2
64.0 ± 1.2
74.2 ± 1.3
Appendix
Table 20: Ablation results for downstream SLM metrics: sWuggy, sBLIMP, and tSC (%; higher is better). Modifications are cumulative. The SLM has 90M parameters and is trained on 6K hours of Libri-Light. Values are mean ± standard deviation across eight seeds of encoder training. For each variant, all scores are reported at the layer that maximizes average accuracy across the three metrics.
Figure 14: Layerwise codebook metrics for the cumulative ablations from SpidR to SpidR 2.0, evaluated on ZeroSpeech’21 dev-clean . Panels show the same metrics as Figure 12 . Results are mean ± standard deviation across eight seeds.
ABX ( ↓ )
MAP ( ↑ )
Within
Across
Variant
l∗
cln
oth
cln
oth
Avg.
l∗
cln
oth
Avg.
SpidR 2.0 (ME step, CoE constant)
10
2.33
2.74
2.56
3.70
2.83
7
75.6
63.6
69.6
A : ME constant, CoE constant
8
2.61
3.00
2.87
3.93
3.10
7
75.4
63.3
69.4
B : ME step, CoE step
10
2.57
3.18
2.79
4.10
3.16
7
72.1
60.0
66.0
C : Baseline + student reg. ( 1× )
9
2.36
2.77
2.58
3.73
2.86
6
75.2
62.2
68.7
Appendix
Table 21: Effect of entropy regularization on ABX and MAP (%) for continuous representations from SpidR 2.0. All variants share the same base recipe as SpidR 2.0 and differ only in the regularization schedule/target. For each variant, ABX and MAP are reported at their respective best layers l∗ , selected by task-averaged performance.
Figure 15: Layerwise codebook metrics on ZeroSpeech’21 dev-clean for the regularization setups in Appendix C.2.4 . Panels show the same metrics as Figure 12 . All runs use the SpidR 2.0 recipe and seed 0, differing only in regularization. The baseline reduces the ME weight λ from 0.1 to 0.001 after 20k steps and keeps the CoE weight μ=0.1 . A keeps both weights at 0.1 ; B reduces both to 0.001 . C and D regularize both teacher and student using full and half baseline weights, respectively. Purity and PNMI are in %.
Figure 16: Training dynamics of SpidR 2.0. We report the mean (line), as well as the minimum and maximum for each training step (shaded area) across eight seeds. Top row: SSL loss and task-averaged ABX error at layer 10. Bottom row: predictive confidence, El,t[maxcpl,t[c]] , and codebook perplexity, El[2Hc(Et[pl,t])] , for the teacher (solid) and student (dash-dotted).
Model
Layer
F1 ( ↑ )
R-val ( ↑ )
PER ( ↓ )
ABX ( ↓ )
PNMI ( ↑ )
Dev languages
DinoSR (LibriSpeech)
5
61.1
48.9
82.4
10.8
49.0
SpidR (LibriSpeech)
5
62.5
50.4
79.1
11.8
52.8
SpidR (VP19)
5
63.0
44.8
82.8
8.7
55.5
SpidR 2.0 (LibriSpeech)
6
58.8
61.6
65.2
9.1
48.6
SpidR 2.0 (VP19)
6
64.3
65.3
61.1
8.9
50.0
Appendix
Table 22: DiscoPhon cross-lingual evaluation. All metrics are reported at each model’s best-PNMI layer, selected by averaging PNMI across dev and test languages. Scores are averaged over languages within each split and expressed in %. ABX is computed on continuous triphone representations and averaged over within-/across-speaker conditions. Dev languages: German, Swahili, Tamil, Thai, Turkish, Ukrainian. Test languages: Basque, English, French, Japanese, Mandarin, Wolof.
Recognition
Detection
Semantics
Paraling.
Model
ASR ( ↓ )
PR ( ↓ )
KS ( ↑ )
QbE ( ↑ )
IC ( ↑ )
SF-F1 ( ↑ )
SF-CER ( ↓ )
ER ( ↑ )
DinoSR
4.78
3.66
96.17
4.99
98.63
88.85
23.66
66.15
SpidR
7.23
5.17
95.39
8.02
95.28
86.92
27.54
65.52
SpidR 2.0
6.61
4.34
96.27
7.68
97.94
86.29
27.29
64.03
WavLM ( Base )
6.21
4.84
96.79
8.70
98.63
89.38
22.86
65.94
WavLM Large
3.44
3.06
97.86
8.86
99.31
92.21
18.36
70.62
Appendix
Table 23: Downstream evaluation on SUPERB. For DinoSR, SpidR, and SpidR 2.0, we evaluate five learning rates for each trained downstream probe and report the test score of the run with the best development-set performance. For WavLM, we report the scores from the official SUPERB leaderboard. ASR reports word error rate without LM decoding; PR reports phone error rate. KS, IC, and ER report accuracy; SF reports slot-type F1 and slot-value character error rate (CER). QbE reports maximum term weighted value, scaled by 100.
Figure 17: Learned layer-mixing weights for the SUPERB probes. Softmax-normalized weights combine intermediate encoder representations into the features used by each downstream probe.
Model
Unit type
Layer
k
ABX ( ↓ )
sWuggy ( ↑ )
sBLIMP ( ↑ )
tSC ( ↑ )
SpidR
k -means
6
256
7.20
71.0
59.1
70.2
Codes
6
203
6.31
69.8
58.1
70.1
SpidR 2.0
k -means
8
256
6.63
72.5
61.6
73.6
k -means
8
129
6.15
71.8
62.7
73.7
k -means (LS-960)
7
128
6.75
72.4
62.6
74.1
Codes
7
128
5.56
73.2
65.6
75.8
Appendix
Table 24: Discrete ABX and SLM results for SpidR and SpidR 2.0. Codes denotes units obtained from the prediction heads; k -means denotes clustering of continuous encoder representations. For learned codes, k denotes the number of active codes; for k -means, it denotes the number of clusters. k -means is fitted on LibriSpeech train-clean-100 , except for LS-960, which uses the full 960-hour corpus. Results are reported at the layer with the highest average development score across sWuggy, sBLIMP, and tSC. ABX errors are averaged over dev-clean and dev-other and over within- and across-speaker conditions.
Figure 18: Cross-layer unit-stream similarity. Pairwise NMI between unit streams extracted from encoder layers 5-12 of (a) SpidR and (b) SpidR 2.0, computed on LibriSpeech dev-clean and averaged over eight encoder training seeds. Since the matrices are symmetric, only entries below the diagonal are displayed; the diagonal (NMI =1 ) and upper triangle are greyed out.
ABX ( ↓ )
MAP ( ↑ )
Within
Across
Model
Params [M]
l
cln
oth
cln
oth
Avg.
l
cln
oth
Avg.
Self-supervised (masked / predictive)
data2vec 2.0 Large ( Baevski et al., 2023 )
313
2
6.20
5.92
6.92
7.75
6.70
12
76.1
72.6
74.4
data2vec ( Baevski et al., 2022 )
93
4
4.41
5.51
5.12
7.40
5.61
10
72.5
66.9
69.7
SpidR 2.0
93
10
2.33
2.74
2.56
3.70
2.83
7
75.6
63.6
69.6
Appendix
Table 25: ABX and MAP (%) for SpidR 2.0 and publicly available speech encoders on the ZeroSpeech’21 development set. Each metric is reported at the encoder layer l that achieves its best average score. ABX is evaluated within and across speakers; cln / oth denote the clean/other subsets. Rows are ordered by decreasing average MAP within each group. In the first model group, bold and underlined scores indicate the best and second-best averages for each metric, respectively.
Figure 19: Pearson correlations between proxy metrics (rows) and target SLM metrics (columns: sWuggy, sBLIMP, tSC, and their average, denoted avg. SLM) across the progressive-ablation models from SpidR to SpidR 2.0 (Appendix C.2.3 ). Each point represents a model-layer pair (layers 5-12), with scores averaged over eight encoder training seeds (i.e., 40 points per panel). Dashed lines show OLS fits, and each panel reports Pearson’s r . Two-sided p -values for correlations with avg. SLM are shown to the right of each row, with rows ordered by increasing p -value (i.e., the most strongly correlated metric at the top).
Figure 20: Pearson correlations between proxy metrics (rows) and target SLM metrics (columns: sWuggy, sBLIMP, tSC, and their average, denoted avg. SLM) within SpidR 2.0, across eight training seeds and eight encoder layers (5-12). Each point represents a layer-seed pair (i.e., 64 points per subplot). Dashed lines show OLS fits, and each panel reports Pearson’s r . Two-sided p -values for correlations with avg. SLM are shown to the right of each row, with rows ordered by increasing p -value (i.e., the most strongly correlated metric at the top). The observed correlations primarily reflect variation across layers rather than across seeds.
Oracle duration
Predicted duration
Model
WER ( ↓ )
UTMOS ( ↑ )
WER ( ↓ )
UTMOS ( ↑ )
Ground truth
1.11%
4.15
–
–
SpidR
1.91%
2.99
2.08%
3.02
SpidR 2.0
3.35%
3.00
3.76%
3.08
GSLM2
2.00%
3.49
2.13%
3.51
Appendix
Table 26: Speech resynthesis quality using oracle and predicted unit durations. Oracle durations are extracted from the ground-truth utterance, while predicted durations are estimated from the deduplicated unit sequence using the corresponding duration predictor.
ZeroSpeech’21 ( ↑ )
StoryCloze ( ↑ )
LongTail-Swap ( ↑ )
Model
sWuggy
sBLIMP
tSC
sSC
WS
IS
AS
600 h / 90M
Text LM
73.4 [72.9, 73.8]
61.7 [61.3, 62.0]
75.4 [73.4, 77.3]
53.4 [51.2, 55.7]
73.5 [72.9, 74.0]
80.6 [79.9, 81.3]
62.2 [61.3, 63.2]
SpidR SLM
67.4 [67.0, 67.7]
54.2 [53.9, 54.6]
65.4 [63.2, 67.7]
52.4 [50.1, 54.6]
54.3 [53.6, 55.0]
57.9 [57.0, 59.1]
54.2 [53.4, 55.3]
SpidR 2.0 SLM
69.2 [68.9, 69.6]
57.7 [57.4, 58.1]
68.1 [66.0, 70.1]
52.7 [50.4, 54.9]
58.0 [57.3, 58.6]
63.5 [62.8, 64.8]
55.0 [54.1, 56.1]
GSLM2
71.5 [71.1, 71.8]
59.8 [59.4, 60.1]
74.1 [72.1, 76.1]
52.0 [49.9, 54.4]
62.4 [61.8, 63.1]
68.2 [67.5, 69.4]
59.1 [58.1, 60.1]
6K h / 300M
Text LM
79.5 [79.1, 79.9]
66.8 [66.4, 67.1]
84.8 [83.2, 86.4]
52.2 [49.9, 54.4]
80.1 [79.7, 80.5]
90.9 [90.7, 91.1]
79.1 [78.8, 79.4]
Appendix
Table 27: Accuracy (%) and corresponding 95% bootstrap confidence intervals for the main results reported in Table 2 .
Spoken Language Models (SLMs) generate textual responses directly from speech, offering an alternative to cascaded systems. Despite recent advances, existing SLMs still exhibit weaker instruction-following behavior and limited generalization across diverse tasks compared to text-based language models. Our analysis shows that speech and text representations in current SLMs remain weakly aligned despite strong downstream performance, indicating that structural differences between continuous, temporally varying speech and discrete text remain insufficiently addressed. To address this, we propose a simple framework that decouples length mismatch from semantic alignment and encourages closer correspondence between speech and text representations. Experiments across multiple benchmarks demonstrate competitive performance against strong baselines, underscoring the importance of explicitly addressing structural differences between speech and text in SLM training. Our code is publicly available at https://github.com/jaykim9870/Do_SLMs_Hear_Speech_as_They_Read_Text.
Large language models show that simple autoregressive training can yield scalable and coherent generation, but extending this paradigm to speech remains challenging due to the entanglement of semantic and acoustic information. Most existing speech language models rely on text supervision, hierarchical token streams, or complex hybrid architectures, departing from the single-stream generative pretraining paradigm that has proven effective in text. In this work, we introduce WavSLM, a speech language model trained by quantizing and distilling self-supervised WavLM representations into a single codebook and optimizing an autoregressive next-chunk prediction objective. WavSLM jointly models semantic and acoustic information within a single token stream without text supervision or text pretraining. Despite its simplicity, it achieves competitive performance on consistency benchmarks and speech generation while using fewer parameters, less training data, and supporting streaming inference.
Luca Della Libera, Cem Subakan, Mirco Ravanelli
Concordia University · Mila-Quebec AI Institute · Universit´e Laval
Generative Spoken Language Modeling (GSLM) enables text-free speech modeling by training language models (LMs) using discrete speech representations instead of textual transcription. In this paper, we investigate the performance of GSLM on speech synthesis and continuation using discrete speech representations with varying bitrates. We segment speech representations with fixed widths and train K-means models in multiple cluster sizes, resulting in various bitrate settings. We demonstrate that intelligible and natural speech can be synthesized at lower bitrate settings than the baseline. Furthermore, speech continuation quality remains stable at lower bitrates across multiple metrics, suggesting that the conventional GSLM setting may be redundant for effective speech generation. Although LLM-based metrics show higher correlation with human subjective score than conventional metrics, it remains low, highlighting the need for more stable automatic evaluation methods.