GSLM2: Multi-Stream Speech Representation Learning and Generative Spoken Language Modeling
Organizations: Meta Superintelligence Labs · Université Paris Dauphine-PSL · AMI Labs · Doctolib ·
Abstract
Generative spoken language models (SLMs) offer a route to learning linguistic structure purely from speech, yet they lag far behind text-based models in both capability and data scaling efficiency. We identify two bottlenecks underlying this gap: the limited linguistic quality of existing speech units and the reliance on a single stream of discrete units to represent speech. Multiple codebooks are widely used to encode acoustic features but rarely to represent speech semantics. We present GSLM2, which learns and models complementary properties of speech at multiple levels of abstraction. Its encoder, SpidR 2.0, uses self-supervised learning with differentiable prediction heads to induce high-quality discrete units at multiple levels. A multi-stream SLM then jointly models these units using a shared temporal backbone, stream-specific predictors, and an autoregressive head that captures cross-stream dependencies. Across training data sizes and model scales, up to 6M hours and 6.5B parameters, GSLM2 consistently improves linguistic competence while scaling more efficiently with data. As a result, it substantially narrows the gap between spoken and text-based LMs; across read and conversational speech domains, we shrink the gap by about two orders of magnitude on sWuggy, sBLIMP, and tSC, while also outperforming competitive speech-text systems on all three tasks. We open-source the training code and model checkpoints at https://github.com/facebookresearch/gslm2.
Figures & tables
| Benchmark | Metric | Capability | Evaluation |
| ZeroSpeech’21 | sWuggy(-all) | Lexical | Real word vs. phonotactically matched nonword. |
| sBLIMP | Syntax | Grammatical vs. minimally different ungrammatical sentence. | |
| StoryCloze (SC) | Topic SC (tSC) | Semantics | Correct ending vs. random ending from another story. |
| Spoken SC (sSC) | Semantics | Correct ending vs. adversarial, superficially plausible ending. | |
| LongTail-Swap | WordSwap (WS) | Semantics | Context-appropriate word vs. frequency- and POS-matched substitution. |
| InflectionSwap (IS) | Morphosyntax | Context-appropriate inflection vs. another form of the same lemma. |
| Training Regime | Model | #Params | #Data † [h] | sWuggy | sBLIMP | tSC | sSC |
| Speech-only training | GSLM2 (ours; Libri-Light) | 300M | 6K | 76.3 | 67.2 | 78.9 | 52.5 |
| 1.2B | 60K | 79.3 | 71.8 | 84.5 | 57.6 | ||
| GSLM2 (ours; Conversational Speech) | 6.5B | 6M | 80.7 | 72.9 | 90.5 | 63.2 | |
| GSLM ( Lakhotia et al., 2021 ) | 100M | 6K | 64.8 | 54.2 | 66.6 | 53.3 | |
| AudioLM ( Borsos et al., 2023 ) | 300M | 60K | 71.5 | 64.7 | – | – | |
| Cuervo & Marxer (2024) | 823M | 160K | – | 61.3 | 75.4 | – |
| Model | ASR-PPL ( ) | LLMJ ( ) | ||
| 600 h / 90M | Text LM | 1.22 | 690.0 12.9 | 1.88 0.02 |
| SpidR SLM | 1.31 | 670.5 0 9.4 | 1.73 0.02 | |
| SpidR 2.0 SLM | 1.34 | 626.5 14.3 | 1.78 0.02 | |
| GSLM2 | 1.49 | 583.5 12.1 | 1.80 0.02 | |
| 6K h / 300M | Text LM | 1.21 | 331.0 0 6.5 | 2.30 0.03 |
| SpidR SLM | 1.39 | 407.2 0 9.7 | 2.20 0.02 | |
| ABX ( ) | MAP ( ) | |||||
| Model | within | across | Avg. | Score | ||
| DinoSR | 5 | 4.58 | 6.02 | 5.30 | 11 | 69.49 |
| SpidR | 6 | 3.53 | 4.31 | 3.92 | 5 | 63.42 |
| SpidR 2.0 | 10 | 2.54 | 3.13 | 2.83 | 7 | 69.58 |
Appendix figures & tables38 assets
Supplementary material from the paper’s appendix.
Appendix
| Corpus | Language(s) | Component | Training data |
| LibriSpeech | English | Speech encoder | 960 h |
| Libri-Light | English | SLM | 600 h / 6K h / 60K h |
| Duration predictor | 600 h | ||
| Libriheavy | English | Text LM | Transcripts aligned with Libri-Light |
| LJ Speech | English | Unit vocoder | 24 h |
| VoxPopuli | En, Fr, De | Speech encoder | 6K h per language |
| Encoder | Language model | |||||
| Hyperparameter | 90M | 90M | 300M | 1.2B | 2.5B | 6.5B |
| Transformer layers | 12 | 12 | 24 | 24 | 32 | 32 |
| Hidden dimension | 768 | 768 | 1,024 | 2,048 | 2,560 | 4,096 |
| FFN dimension | 3,072 | 3,072 | 4,096 | 8,192 | 10,240 | 16,384 |
| Attention heads | 12 | 12 | 16 | 32 | 32 | 32 |
| LM backbone scale | |||||
| Model | 90M | 300M | 1.2B | 2.5B | 6.5B |
| Text LM | 92.9M | 312.8M | 1.23B | 2.54B | 6.49B |
| SpidR SLM | 87.0M | 304.9M | 1.21B | 2.52B | 6.45B |
| SpidR 2.0 SLM | 87.0M | 304.9M | 1.21B | 2.52B | 6.45B |
| GSLM2 | 96.3M | 315.5M | 1.23B | 2.54B | 6.48B |
| Data | Model | #Params | Peak learning rate | Steps | Global batch (units/tokens) | Context length | #GPUs |
| 600 h | Text LM | 90M | 10k | 8 DDP | |||
| SLM | 90M | 15k | 8 DDP | ||||
| GSLM2 | 90M | 15k | 32 DDP | ||||
| 6K h | Text LM | 300M | 20k | 8 DDP | |||
| SLM | 300M (90M) | 25k | 8 DDP | ||||
| GSLM2 | 300M (90M) | 25k | 32 DDP |
| Compute (FLOPs) | |||
| Scale | Model | Training | Inference / audio s |
| 600 h / 90M | Text LM | ||
| SpidR 2.0 SLM | |||
| GSLM2 | |||
| 6K h / 300M | Text LM | ||
| SpidR 2.0 SLM | |||
| ZS21 ( ) | SC ( ) | |||
| ID | Model | sWuggy | sBLIMP | tSC |
| A0 | GSLM2 | 76.3 | 67.2 | 78.9 |
| A1 | w/o self-distillation ( ) | 68.5 | 57.3 | 70.1 |
| A2 | w/o single-stream loss ( ) | 67.9 | 55.8 | 65.4 |
| A3 | Halve context length & batch size | 75.7 | 66.8 | 78.3 |
| A4 | SpidR 2.0 SLM | 73.9 | 65.1 | 74.7 |
| Stream-specific head | Generative head | ||||||
| Order | Layer | sWuggy | sBLIMP | tSC | sWuggy | sBLIMP | tSC |
| 1 | 10 | 70.1 | 62.8 | 77.2 | 57.2 | 53.6 | 59.3 |
| 2 | 5 | 71.7 | 64.8 | 73.7 | 61.8 | 53.5 | 55.9 |
| 3 | 6 | 72.1 | 64.7 | 77.6 | 57.5 | 52.0 | 53.2 |
| 4 | 9 | 71.5 | 64.0 | 79.3 | 60.9 | 52.2 | 61.8 |
| 5 | 8 | 72.3 | 63.0 | 77.7 | 55.6 | 50.8 | 51.8 |
| ZS21 ( ) | SC ( ) | |||
| Model | Unit layers | sWuggy | sBLIMP | tSC |
| SpidR 2.0 SLM | 7 | 73.9 | 65.1 | 74.7 |
| GSLM2 | 5–10 | 76.3 | 67.2 | 78.9 |
| 7 (repeated) | 73.7 | 65.3 | 74.4 | |
| 7 (sampled) | 69.9 | 61.0 | 68.4 | |
| ZS21 ( ) | SC ( ) | ||||
| Model | #Params | sWuggy | sBLIMP | tSC | sSC |
| Ours | |||||
| GSLM2 (Libri-Light 60K h) | 1.2B | 79.3 | 71.8 | 84.5 | 57.6 |
| GSLM2 (Conversational Speech 6M h) | 6.5B | 80.7 | 72.9 | 90.5 | 63.2 |
| Open-weight speech LLMs | |||||
| Gemma 4 ( Gemma Team, 2026 ) | 4B | 72.6 | 67.7 | 88.6 | 80.4 |
| ASR-PPL ( ) | LLMJ ( ) | ||||
| Model | |||||
| Oracle | Text LM | 55.2 | 4.55 | ||
| SpidR SLM | 60.1 | 4.54 | |||
| SpidR 2.0 SLM | 70.1 | 4.41 | |||
| GSLM2 | 59.5 | 4.53 | |||
| 600 h / 90M | Text LM | 229.1 5.8 | 0 76.8 2.4 | 2.08 0.01 | 2.22 0.02 |
| Language | Model | sWuggy | sBLIMP | tSC |
| English | Text LM | 77.0 | 63.9 | 76.8 |
| SpidR SLM | 60.3 | 53.3 | 69.3 | |
| SpidR 2.0 SLM | 62.9 | 55.3 | 69.6 | |
| GSLM2 | 65.6 | 57.2 | 72.7 | |
| French | Text LM | – | 95.0 | 75.7 |
| SpidR SLM | – | 58.8 | 63.9 |
| ZS21 ( ) | SC ( ) | ||||
| Model | Training Input | Evaluation Input | (s)Wuggy | (s)BLIMP | tSC |
| Text LM | Ground-truth text | B1 : Ground-truth text | 79.5 | 66.8 | 84.8 |
| B2 : ASR transcript of speech | 70.6 | 71.0 | 75.4 | ||
| ASR transcripts | B3 : Ground-truth text | 81.2 | 66.9 | 85.1 | |
| B4 : ASR transcript of speech | 71.5 | 69.1 | 77.4 | ||
| GSLM2 | Speech | Speech | 76.3 | 67.2 | 78.9 |
| Variant | Best layer | |||
| SpidR (reproduced) | 6 | 5.16 1.37 | 5.25 0.76 | 10.90 2.97 |
| + shared heads | 6 | 4.82 0.57 | 5.75 0.74 | 6.78 1.02 |
| + regularization | 9 | 4.12 0.34 | 8.75 0.75 | 4.29 0.42 |
| + normalizations | 10 | 3.58 0.31 | 5.88 0.27 | 3.58 0.31 |
| + ( SpidR 2.0 ) | 10 | 2.88 0.12 | 4.73 0.22 | 2.88 0.12 |
| Variant | Best layer | Average | dev-clean | dev-other |
| SpidR (reproduced) | 4 | 58.3 2.5 | 65.2 3.0 | 51.4 3.9 |
| + shared heads | 5 | 61.3 3.4 | 66.2 5.4 | 56.4 4.1 |
| + regularization | 7 | 68.0 1.6 | 73.3 2.2 | 62.7 2.4 |
| + normalizations | 8 | 72.3 0.3 | 77.0 0.3 | 67.5 0.4 |
| + ( SpidR 2.0 ) | 7 | 69.2 0.3 | 75.3 0.4 | 63.1 0.5 |
| Variant | Best layer | Average | sWuggy | sBLIMP | tSC |
| SpidR (reproduced) | 5 | 63.9 0.6 | 66.4 1.4 | 56.4 1.0 | 68.9 0.8 |
| + shared heads | 5 | 58.3 0.7 | 59.9 1.4 | 51.6 0.8 | 63.3 1.3 |
| + regularization | 7 | 66.7 0.9 | 67.5 2.3 | 60.4 1.0 | 72.3 0.9 |
| + normalizations | 8 | 69.1 0.7 | 70.1 1.6 | 63.0 1.2 | 74.1 0.7 |
| + ( SpidR 2.0 ) | 7 | 70.1 0.7 | 72.1 1.2 | 64.0 1.2 | 74.2 1.3 |
| ABX ( ) | MAP ( ) | |||||||||
| Within | Across | |||||||||
| Variant | cln | oth | cln | oth | Avg. | cln | oth | Avg. | ||
| SpidR 2.0 (ME step, CoE constant) | 10 | 2.33 | 2.74 | 2.56 | 3.70 | 2.83 | 7 | 75.6 | 63.6 | 69.6 |
| A : ME constant, CoE constant | 8 | 2.61 | 3.00 | 2.87 | 3.93 | 3.10 | 7 | 75.4 | 63.3 | 69.4 |
| B : ME step, CoE step | 10 | 2.57 | 3.18 | 2.79 | 4.10 | 3.16 | 7 | 72.1 | 60.0 | 66.0 |
| C : Baseline student reg. ( ) | 9 | 2.36 | 2.77 | 2.58 | 3.73 | 2.86 | 6 | 75.2 | 62.2 | 68.7 |
| Model | Layer | F1 ( ) | R-val ( ) | PER ( ) | ABX ( ) | PNMI ( ) |
| Dev languages | ||||||
| DinoSR (LibriSpeech) | 5 | 61.1 | 48.9 | 82.4 | 10.8 | 49.0 |
| SpidR (LibriSpeech) | 5 | 62.5 | 50.4 | 79.1 | 11.8 | 52.8 |
| SpidR (VP19) | 5 | 63.0 | 44.8 | 82.8 | 8.7 | 55.5 |
| SpidR 2.0 (LibriSpeech) | 6 | 58.8 | 61.6 | 65.2 | 9.1 | 48.6 |
| SpidR 2.0 (VP19) | 6 | 64.3 | 65.3 | 61.1 | 8.9 | 50.0 |
| Recognition | Detection | Semantics | Paraling. | |||||
| Model | ASR ( ) | PR ( ) | KS ( ) | QbE ( ) | IC ( ) | SF-F1 ( ) | SF-CER ( ) | ER ( ) |
| DinoSR | 4.78 | 3.66 | 96.17 | 4.99 | 98.63 | 88.85 | 23.66 | 66.15 |
| SpidR | 7.23 | 5.17 | 95.39 | 8.02 | 95.28 | 86.92 | 27.54 | 65.52 |
| SpidR 2.0 | 6.61 | 4.34 | 96.27 | 7.68 | 97.94 | 86.29 | 27.29 | 64.03 |
| WavLM ( Base ) | 6.21 | 4.84 | 96.79 | 8.70 | 98.63 | 89.38 | 22.86 | 65.94 |
| WavLM Large | 3.44 | 3.06 | 97.86 | 8.86 | 99.31 | 92.21 | 18.36 | 70.62 |
| Model | Unit type | Layer | ABX ( ) | sWuggy ( ) | sBLIMP ( ) | tSC ( ) | |
| SpidR | -means | 6 | 256 | 7.20 | 71.0 | 59.1 | 70.2 |
| Codes | 6 | 203 | 6.31 | 69.8 | 58.1 | 70.1 | |
| SpidR 2.0 | -means | 8 | 256 | 6.63 | 72.5 | 61.6 | 73.6 |
| -means | 8 | 129 | 6.15 | 71.8 | 62.7 | 73.7 | |
| -means (LS-960) | 7 | 128 | 6.75 | 72.4 | 62.6 | 74.1 | |
| Codes | 7 | 128 | 5.56 | 73.2 | 65.6 | 75.8 |
| ABX ( ) | MAP ( ) | ||||||||||
| Within | Across | ||||||||||
| Model | Params [M] | cln | oth | cln | oth | Avg. | cln | oth | Avg. | ||
| Self-supervised (masked / predictive) | |||||||||||
| data2vec 2.0 Large ( Baevski et al., 2023 ) | 313 | 2 | 6.20 | 5.92 | 6.92 | 7.75 | 6.70 | 12 | 76.1 | 72.6 | 74.4 |
| data2vec ( Baevski et al., 2022 ) | 93 | 4 | 4.41 | 5.51 | 5.12 | 7.40 | 5.61 | 10 | 72.5 | 66.9 | 69.7 |
| SpidR 2.0 | 93 | 10 | 2.33 | 2.74 | 2.56 | 3.70 | 2.83 | 7 | 75.6 | 63.6 | 69.6 |
| Oracle duration | Predicted duration | |||
| Model | WER ( ) | UTMOS ( ) | WER ( ) | UTMOS ( ) |
| Ground truth | 1.11% | 4.15 | – | – |
| SpidR | 1.91% | 2.99 | 2.08% | 3.02 |
| SpidR 2.0 | 3.35% | 3.00 | 3.76% | 3.08 |
| GSLM2 | 2.00% | 3.49 | 2.13% | 3.51 |
| ZeroSpeech’21 ( ) | StoryCloze ( ) | LongTail-Swap ( ) | ||||||
| Model | sWuggy | sBLIMP | tSC | sSC | WS | IS | AS | |
| 600 h / 90M | Text LM | 73.4 [72.9, 73.8] | 61.7 [61.3, 62.0] | 75.4 [73.4, 77.3] | 53.4 [51.2, 55.7] | 73.5 [72.9, 74.0] | 80.6 [79.9, 81.3] | 62.2 [61.3, 63.2] |
| SpidR SLM | 67.4 [67.0, 67.7] | 54.2 [53.9, 54.6] | 65.4 [63.2, 67.7] | 52.4 [50.1, 54.6] | 54.3 [53.6, 55.0] | 57.9 [57.0, 59.1] | 54.2 [53.4, 55.3] | |
| SpidR 2.0 SLM | 69.2 [68.9, 69.6] | 57.7 [57.4, 58.1] | 68.1 [66.0, 70.1] | 52.7 [50.4, 54.9] | 58.0 [57.3, 58.6] | 63.5 [62.8, 64.8] | 55.0 [54.1, 56.1] | |
| GSLM2 | 71.5 [71.1, 71.8] | 59.8 [59.4, 60.1] | 74.1 [72.1, 76.1] | 52.0 [49.9, 54.4] | 62.4 [61.8, 63.1] | 68.2 [67.5, 69.4] | 59.1 [58.1, 60.1] | |
| 6K h / 300M | Text LM | 79.5 [79.1, 79.9] | 66.8 [66.4, 67.1] | 84.8 [83.2, 86.4] | 52.2 [49.9, 54.4] | 80.1 [79.7, 80.5] | 90.9 [90.7, 91.1] | 79.1 [78.8, 79.4] |
Speech Language Models
Speech language models connect spoken audio with large language models for understanding, transcription, translation, and real-time dialogue.
See all 327 papers