cs.CLApr 2, 2026
SaveTracking the emergence of linguistic structure in self-supervised models learning from speech
Organizations: Institute for Logic, Language and Computation, University of Amsterdam, The Netherlands · Centre for Language Studies, Radboud University, The Netherlands · Cognitive Science and Artificial Intelligence, Tilburg University, The Netherlands
Abstract
Self-supervised speech models learn effective representations of spoken language, which have been shown to reflect various aspects of linguistic structure. But when does such structure emerge in model training? We study the encoding of a wide range of linguistic structures, across layers and intermediate checkpoints of six Wav2Vec2 and HuBERT models trained on spoken Dutch. We find that different levels of linguistic structure show notably distinct layerwise patterns as well as learning trajectories, which can partially be explained by differences in their degree of abstraction from the acoustic signal and the timescale at which information from the input is integrated. Moreover, we find that the level at which pre-training objectives are defined strongly affects both the layerwise organization and the learning trajectories of linguistic structures, with greater parallelism induced by higher-order prediction tasks (i.e. iteratively refined pseudo-labels).
Figures & tables
Figure 1: (A) We train a set of six models on the same dataset, consisting of 831 hours of Dutch speech recordings. (B) We explore how results vary between model architectures with minimal differences in training set-up (Wav2Vec2, HuBERT-I1, HuBERT-I2). (C) We probe each model’s internal representations for nine types of linguistic structure, which differ in their degrees of abstraction from the acoustic signal and in their timescales of information integration. We compare results for each structure across model layers as well as training steps.
Figure 2: Example outputs for two probing techniques applied to different layers of a Wav2Vec2 model at 100K training steps, with accompanying scores. Data points in the top figures visualize the top 3 LDA directions for syllable type test samples, colored by syllable type. Yellow lines in the bottom row visualize model-reconstrcuted dependency links for one test sentence; black lines indicate the true dependency structure.
Figure 3: Layerwise scores for all representational probes (rows) and three speech SSL models (columns). Grey dashed lines indicate baseline scores extracted from a Wav2Vec2 model trained on non-speech acoustic scenes. Shading indicates the std. dev. over 5 folds.
Figure 4: Learning trajectories for all representational probes, across training checkpoints of one Wav2Vec2 model (seed 2). Left: best-layer score for each checkpoint, with fitted sigmoid curves for the speech-trained model; grey dots indicate non-speech baseline model scores. Right: all layerwise scores across training.
Figure 5: Normalized parametric curves 5 visualizing the learning trajectories for all probe scores across training checkpoints, in all six models. Stars indicate the training step where 95% of the maximum observed score across training is achieved; these steps are again marked in the rightmost plots, with different structure levels on separate rows. Different levels of linguistic structure consistently show distinct learning dynamics across S3M architectures and model seeds, with increased parallelism between levels for HuBERT-I2 models compared to HuBERT-I1 and Wav2Vec2.
Figure 6: Relative change in best-layer scores when further training Wav2Vec2 and HuBERT-I2 up to 200K training steps. Note that scores are on different scales across probes; these results are only displayed together to comprehensively visualize relative changes between checkpoints for each probe, not to compare absolute scores between probes. The five lines for each probe connect scores for the same folds of the analysis data.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1: Pre-training loss curves (train & validation loss) for all models trained for this study. Checkpoints additionally validated for their downstream performance on automatic speech recognition (see Table S1 ) are marked with grey vertical lines.
| Test set | Step | HuBERT-I2 (1) | HuBERT-I2 (2) | Wav2Vec2 (1) | Wav2Vec2 (2) |
| IFADV | 100 | 0.97 | 0.98 | 0.97 | 0.97 |
| (dialogues) | 1000 | 0.98 | 0.97 | 0.98 | 0.98 |
| 10000 | 0.86 | 0.86 | 0.91 | 0.91 | |
| 100000 | 0.67 | 0.65 | 0.71 | 0.70 | |
| 200000 | 0.68 | 0.66 | 0.69 | 0.69 | |
| NBest | 100 | 0.83 | 0.85 | 0.83 | 0.82 |
Table S1: Word-error rate (WER) results for the HuBERT-I2 (1 & 2) and Wav2Vec2-NL (1 & 2) models pre-trained for this study, after finetuning intermediate checkpoints for Dutch automatic speech recognition (using the CGN-O component from the Spoken Dutch corpus, Schuurman et al., 2003 ); evaluated across several test sets: IFADV ( van Son et al., 2008 ) ; the NBest benchmark ( Vaessen et al., 2025 ) ; Dutch CommonVoice ( Ardila et al., 2020 ) ; Multilingual LibriSpeech ( Pratap et al., 2020 ) ; and a held-out part of CGN-O. Step refers to the pre-training step. Bold values mark the lowest WER per model and test set.
Figure S2: Layerwise scores for all representational probes (columns) and three speech SSL models (rows). Grey dashed lines indicate baseline scores extracted from a Wav2Vec2 model trained on non-speech acoustic scenes. Shading indicates the std. dev. over 5 folds. Note that scores are on different scales across probes and displayed together only to demonstrate the variety in relative layerwise peak locations, not absolute performance.
Figure S3: Control analyses for the syntactic structure probe: grey markers show alignment between model-reconstructed dependency parses and linear-sequential dependency structures (where each word is only connected to its neighbors); blue markers show alignment to the sentences’ true dependency structures. In all speech-trained models, alignment to the true dependency parse is (eventually) higher than alignment to the sequential parse. This pattern is reversed for the model trained on non-speech acoustic scenes. Top row: best-layer scores across training checkpoints. Bottom row: layerwise scores for the checkpoint at 100k training steps. The dashed line indicates the mean UUAS score quantifying similarity between the true and sequential dependency structures themselves.
| Analysis | Number of categories | Data samples per category | Total samples | Data split constraints |
| Phone clustering | 37 phones (see Table S8 ) | 50 | 1850 | 80/20% train-test splits |
| Syllable form clustering | 100 syllable forms (see Table S9 ) | 20 | 2000 | 80/20% train-test splits |
| Syllable type clustering | 20 types (see Table S9 ) | 100 (5 syllable forms per type, with 20 samples each) | 2000 | 80/20% train-test splits, syllable forms do not overlap between train & test |
| Word form clustering | 250 word forms | 10 | 2500 | 80/20% train-test splits |
| Part-of-speech clustering | 4 part-of-speech categories | 150 (50 word forms per category, with 3 samples each) | 600 | 80/20% train-test splits, word forms do not overlap between train & test |
Table S2: Dataset details for all clustering probes. For each analysis with number of categories , we train LDA projections to an dimensional subspace (following default configurations in the scikit-learn ( Pedregosa et al., 2011 ) library); i.e. phone LDA projections consist of 36 discriminative directions, etc.
| Summary statistics: Word form analysis data | |
| Total word forms | 250 |
| Number of samples per word form | 10 |
| Total samples | 2500 |
| Numer of word forms per POS category | 109 nouns, 44 adverbs, 25 adjectives, 21 verbs, 17 adpositions, 15 pronouns, 19 other |
| Duration (ms) | Mean: 538, Std.dev: 220 |
| Length (number of phones) | Mean: 5.94, Std.dev: 2.22 |
Table S3: Summary statistics on words used for word form clustering analyses. For our semantic alignment analyses, we randomly sampled 60 nouns, 20 adverbs, 20 adjectives, and 20 verbs out of the available word forms in this dataset, including all samples for each form (1000 samples in total).
| Summary statistics: Homophone analysis data | |
| Total number of triplets (A, B, X) | 2326 |
| Total number of unique words | 4104 |
| Word duration (ms) | Mean: 591, Std.dev.: 210 |
| Word length (number of phones) | Mean: 6.51, Std.dev: 2.02 |
Table S4: Summary statistics on words used for the homophone disambiguation analyses.
| Summary statistics: Part-of-speech analysis data | |
| Number of word forms per POS category | 50 nouns, 50 adverbs, 50 adjectives, 50 verbs |
| Number of samples per word form | 3 |
| Total samples | 600 |
| Duration (ms) | Mean: 434, Std.dev: 165 |
| Length (number of phones) | Mean: 4.85, Std.dev: 1.79 |
| Zipf frequency ( Keuleers et al., 2010 ) | Mean: 5.01, Std.dev: 0.91 |
Table S5: Summary statistics on words used for the part-of-speech clustering analyses. The words used for part-of-speech clustering are sampled from the sentence dataset used for the syntactic structural probe ( Table S6 )
| Summary statistics: Sentence analysis data | |
| Total number of sentences | 2406 |
| Number of audiobooks that sentences were sourced from | 10 |
| Sentence duration (s) | Mean: 4.15, Std.dev: 2.45 |
| Sentence length (number of words) | Mean: 10.6, Std.dev: 5.13 |
| Word duration (ms) | Mean: 325, Std.dev: 188 |
| Word dependency depth | Mean: 1.73, Std.dev: 1.07 |
Table S6: Summary statistics on sentences used for the syntactic structural probe analyses.
| Analysis | Feature space | Feature details |
| Acoustic alignment | Mel-frequency cepstral coefficients (MFCCs) | 20-dimensional, extracted using the librosa library ( McFee et al., 2015 ) , mean-pooled over time |
| Semantic alignment | Fasttext embeddings | 300-dimensional, Fasttext model ( Bojanowski et al., 2017 ) for Dutch released by Mikolov et al. (2018) , accessed from the HuggingFace Hub through the transformers library ( Wolf et al., 2020 ) |
Table S7: Details on the feature spaces used for the acoustic and semantic alignment analyses. Acoustic alignment was computed on the same set of 1850 samples used for phone clustering (see Table S2 ).
| Broad phon. category | Fine phon. category | Included phones |
| consonant | plosive | \textipa p, b, t, d, k, g |
| fricative | \textipa f, v, s, z, S, x, G, h | |
| nasal | \textipa m, n, N | |
| approximant | \textipa V, l, r, j | |
| vowel | long | \textipa i:, y:, e:, ø:, a:, o:, u: |
| short | \textipa I, E, A, O, 0 |
Table S8: Dutch phones included in our phone-level analyses, along with their broad and fine category.
| Syllable type | Syllable forms |
| CV | \textipa d@, t@, x@, b@, l@ |
| CVC | \textipa v@r, d@r, l@k, k@r, t@x |
| CVV | \textipa na:, tu:, ma:, re:, le: |
| CVVC | \textipa hEit, vo:r, Va:r, Ve:r, hœys |
| VC | \textipa Op, Af, Om, In, Ax |
| CVCC | \textipa k@nt, d@rs, r@nt, V@rk, G@nt |
Table S9: Included syllable forms for each syllable type, ordered by frequency of syllable type as estimated from the Spoken Dutch Corpus ( Zuidema, 2009 ) . We sample all syllable forms from multi-syllabic words to avoid confounding effects of word form clustering on the syllable-level analyses.
More in this topic
Self-Supervised Speech Representation Learning
Self-Supervised Speech Representation Learning trains models on unlabeled audio to extract features useful for speech tasks and linguistic analysis.
See all 101 papers