Tracking the emergence of linguistic structure in self-supervised models learning from speech
Authors: Marianne de Heer Kloots, Martijn Bentum, Hosein Mohebbi, Charlotte Pouw, Gaofei Shen, Willem Zuidema
Organizations: Institute for Logic, Language and Computation, University of Amsterdam, The Netherlands · Centre for Language Studies, Radboud University, The Netherlands · Cognitive Science and Artificial Intelligence, Tilburg University, The Netherlands
Self-supervised speech models learn effective representations of spoken language, which have been shown to reflect various aspects of linguistic structure. But when does such structure emerge in model training? We study the encoding of a wide range of linguistic structures, across layers and intermediate checkpoints of six Wav2Vec2 and HuBERT models trained on spoken Dutch. We find that different levels of linguistic structure show notably distinct layerwise patterns as well as learning trajectories, which can partially be explained by differences in their degree of abstraction from the acoustic signal and the timescale at which information from the input is integrated. Moreover, we find that the level at which pre-training objectives are defined strongly affects both the layerwise organization and the learning trajectories of linguistic structures, with greater parallelism induced by higher-order prediction tasks (i.e. iteratively refined pseudo-labels).
Figures & tables
Figure 1: (A) We train a set of six models on the same dataset, consisting of 831 hours of Dutch speech recordings. (B) We explore how results vary between model architectures with minimal differences in training set-up (Wav2Vec2, HuBERT-I1, HuBERT-I2). (C) We probe each model’s internal representations for nine types of linguistic structure, which differ in their degrees of abstraction from the acoustic signal and in their timescales of information integration. We compare results for each structure across model layers as well as training steps.
Figure 2: Example outputs for two probing techniques applied to different layers of a Wav2Vec2 model at 100K training steps, with accompanying scores. Data points in the top figures visualize the top 3 LDA directions for syllable type test samples, colored by syllable type. Yellow lines in the bottom row visualize model-reconstrcuted dependency links for one test sentence; black lines indicate the true dependency structure.
Figure 3: Layerwise scores for all representational probes (rows) and three speech SSL models (columns). Grey dashed lines indicate baseline scores extracted from a Wav2Vec2 model trained on non-speech acoustic scenes. Shading indicates the std. dev. over 5 folds.
Figure 4: Learning trajectories for all representational probes, across training checkpoints of one Wav2Vec2 model (seed 2). Left: best-layer score for each checkpoint, with fitted sigmoid curves for the speech-trained model; grey dots indicate non-speech baseline model scores. Right: all layerwise scores across training.
Figure 5: Normalized parametric curves 5 visualizing the learning trajectories for all probe scores across training checkpoints, in all six models. Stars indicate the training step where 95% of the maximum observed score across training is achieved; these steps are again marked in the rightmost plots, with different structure levels on separate rows. Different levels of linguistic structure consistently show distinct learning dynamics across S3M architectures and model seeds, with increased parallelism between levels for HuBERT-I2 models compared to HuBERT-I1 and Wav2Vec2.
Figure 6: Relative change in best-layer scores when further training Wav2Vec2 and HuBERT-I2 up to 200K training steps. Note that scores are on different scales across probes; these results are only displayed together to comprehensively visualize relative changes between checkpoints for each probe, not to compare absolute scores between probes. The five lines for each probe connect scores for the same folds of the analysis data.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1: Pre-training loss curves (train & validation loss) for all models trained for this study. Checkpoints additionally validated for their downstream performance on automatic speech recognition (see Table S1 ) are marked with grey vertical lines.
Test set
Step
HuBERT-I2 (1)
HuBERT-I2 (2)
Wav2Vec2 (1)
Wav2Vec2 (2)
IFADV
100
0.97
0.98
0.97
0.97
(dialogues)
1000
0.98
0.97
0.98
0.98
10000
0.86
0.86
0.91
0.91
100000
0.67
0.65
0.71
0.70
200000
0.68
0.66
0.69
0.69
NBest
100
0.83
0.85
0.83
0.82
Appendix
Table S1: Word-error rate (WER) results for the HuBERT-I2 (1 & 2) and Wav2Vec2-NL (1 & 2) models pre-trained for this study, after finetuning intermediate checkpoints for Dutch automatic speech recognition (using the CGN-O component from the Spoken Dutch corpus, Schuurman et al., 2003 ); evaluated across several test sets: IFADV ( van Son et al., 2008 ) ; the NBest benchmark ( Vaessen et al., 2025 ) ; Dutch CommonVoice ( Ardila et al., 2020 ) ; Multilingual LibriSpeech ( Pratap et al., 2020 ) ; and a held-out part of CGN-O. Step refers to the pre-training step. Bold values mark the lowest WER per model and test set.
Figure S2: Layerwise scores for all representational probes (columns) and three speech SSL models (rows). Grey dashed lines indicate baseline scores extracted from a Wav2Vec2 model trained on non-speech acoustic scenes. Shading indicates the std. dev. over 5 folds. Note that scores are on different scales across probes and displayed together only to demonstrate the variety in relative layerwise peak locations, not absolute performance.
Figure S3: Control analyses for the syntactic structure probe: grey markers show alignment between model-reconstructed dependency parses and linear-sequential dependency structures (where each word is only connected to its neighbors); blue markers show alignment to the sentences’ true dependency structures. In all speech-trained models, alignment to the true dependency parse is (eventually) higher than alignment to the sequential parse. This pattern is reversed for the model trained on non-speech acoustic scenes. Top row: best-layer scores across training checkpoints. Bottom row: layerwise scores for the checkpoint at 100k training steps. The dashed line indicates the mean UUAS score quantifying similarity between the true and sequential dependency structures themselves.
Analysis
Number of categories
Data samples per category
Total samples
Data split constraints
Phone clustering
37 phones (see Table S8 )
50
1850
80/20% train-test splits
Syllable form clustering
100 syllable forms (see Table S9 )
20
2000
80/20% train-test splits
Syllable type clustering
20 types (see Table S9 )
100 (5 syllable forms per type, with 20 samples each)
2000
80/20% train-test splits, syllable forms do not overlap between train & test
Word form clustering
250 word forms
10
2500
80/20% train-test splits
Part-of-speech clustering
4 part-of-speech categories
150 (50 word forms per category, with 3 samples each)
600
80/20% train-test splits, word forms do not overlap between train & test
Appendix
Table S2: Dataset details for all clustering probes. For each analysis with number of categories N , we train LDA projections to an N−1 dimensional subspace (following default configurations in the scikit-learn ( Pedregosa et al., 2011 ) library); i.e. phone LDA projections consist of 36 discriminative directions, etc.
Table S3: Summary statistics on words used for word form clustering analyses. For our semantic alignment analyses, we randomly sampled 60 nouns, 20 adverbs, 20 adjectives, and 20 verbs out of the available word forms in this dataset, including all samples for each form (1000 samples in total).
Summary statistics: Homophone analysis data
Total number of triplets (A, B, X)
2326
Total number of unique words
4104
Word duration (ms)
Mean: 591, Std.dev.: 210
Word length (number of phones)
Mean: 6.51, Std.dev: 2.02
Appendix
Table S4: Summary statistics on words used for the homophone disambiguation analyses.
Summary statistics: Part-of-speech analysis data
Number of word forms per POS category
50 nouns, 50 adverbs, 50 adjectives, 50 verbs
Number of samples per word form
3
Total samples
600
Duration (ms)
Mean: 434, Std.dev: 165
Length (number of phones)
Mean: 4.85, Std.dev: 1.79
Zipf frequency ( Keuleers et al., 2010 )
Mean: 5.01, Std.dev: 0.91
Appendix
Table S5: Summary statistics on words used for the part-of-speech clustering analyses. The words used for part-of-speech clustering are sampled from the sentence dataset used for the syntactic structural probe ( Table S6 )
Summary statistics: Sentence analysis data
Total number of sentences
2406
Number of audiobooks that sentences were sourced from
10
Sentence duration (s)
Mean: 4.15, Std.dev: 2.45
Sentence length (number of words)
Mean: 10.6, Std.dev: 5.13
Word duration (ms)
Mean: 325, Std.dev: 188
Word dependency depth
Mean: 1.73, Std.dev: 1.07
Appendix
Table S6: Summary statistics on sentences used for the syntactic structural probe analyses.
Analysis
Feature space
Feature details
Acoustic alignment
Mel-frequency cepstral coefficients (MFCCs)
20-dimensional, extracted using the librosa library ( McFee et al., 2015 ) , mean-pooled over time
Semantic alignment
Fasttext embeddings
300-dimensional, Fasttext model ( Bojanowski et al., 2017 ) for Dutch released by Mikolov et al. (2018) , accessed from the HuggingFace Hub through the transformers library ( Wolf et al., 2020 )
Appendix
Table S7: Details on the feature spaces used for the acoustic and semantic alignment analyses. Acoustic alignment was computed on the same set of 1850 samples used for phone clustering (see Table S2 ).
Broad phon. category
Fine phon. category
Included phones
consonant
plosive
\textipa p, b, t, d, k, g
fricative
\textipa f, v, s, z, S, x, G, h
nasal
\textipa m, n, N
approximant
\textipa V, l, r, j
vowel
long
\textipa i:, y:, e:, ø:, a:, o:, u:
short
\textipa I, E, A, O, 0
Appendix
Table S8: Dutch phones included in our phone-level analyses, along with their broad and fine category.
Syllable type
Syllable forms
CV
\textipa d@, t@, x@, b@, l@
CVC
\textipa v@r, d@r, l@k, k@r, t@x
CVV
\textipa na:, tu:, ma:, re:, le:
CVVC
\textipa hEit, vo:r, Va:r, Ve:r, hœys
VC
\textipa Op, Af, Om, In, Ax
CVCC
\textipa k@nt, d@rs, r@nt, V@rk, G@nt
Appendix
Table S9: Included syllable forms for each syllable type, ordered by frequency of syllable type as estimated from the Spoken Dutch Corpus ( Zuidema, 2009 ) . We sample all syllable forms from multi-syllabic words to avoid confounding effects of word form clustering on the syllable-level analyses.
This study examines the extent to which the wav2vec2.0 architecture exhibits evidence of compensation for phonological context. We conducted a pseudo-replication of a perceptional compensation experiment on Mandarin Chinese tones, and compared the embedding similarities and probing classifier outputs between a purely self-supervised pre-trained model and a model fine-tuned for Mandarin ASR. No evidence of compensation was found in the embedding similarities of the purely pre-trained model. Probing classifiers showed some evidence of compensation in addition to the expected layer-wise improvements in categorization, but failed to replicate human performance on isolated test syllables. Our findings contrast with previous reports of sensitivity to phonological structure emerging through pre-training alone, and suggest that supervised objectives may be necessary to encourage the abstraction of at least some types of phonological regularities.
James Kirby, Ioana Krehan, Michele Gubian
Institute for Phonetics and Speech Processing, LMU Munich, Germany
Self-supervised speech foundation models are now used in a wide array of downstream applications, including traditional speech recognition and as the basis for tokens in speech-aware language models. Attempts to understand their usefulness have largely focused on probing their representations' ability to discriminate phonemes and words. However, discriminative ability for words need not imply specialized representation of words per se. Good discrimination of words may be explained by good encoding of word form (phonemes) rather than form-independent word representations encoding identity or syntactic/semantic properties. By partialling out phoneme information using residualization, we show that, in later layers, HuBERT and wav2vec 2.0 do in general learn representations which encode words with reasonable fidelity independently of local phonetic content. We show that this simple approach to disentanglement can enhance higher-order linguistic information in word discovery tasks.
Multilingual self-supervised speech models can benefit from sharing information across languages, but under a matched total pretraining data budget they still fall short of monolingual models. We show that strengthening the model's ability to discriminate languages during pretraining reduces and, on some measures, closes this multilingual gap on continuous phonetic and higher-level linguistic measures, while preserving substantial cross-language sharing. Using a controlled English/French HuBERT setting, we test two interventions which strengthen language discrimination: an auxiliary language classifier and per-language k-means targets. Across interventions, continuous-feature phone discrimination error (phone-ABX, lower is better) decreases from 11.6% in the bilingual baseline to 10.4% (monolingual: 10.8%), while lexical performance (sWUGGY, higher is better) increases from 52.1% to 56.7% (monolingual: 58.5%) and prosodic performance (ProsAudit, lexical subtask, higher is better) from 68.9% to 72.9% (monolingual: 72.6%). Across HuBERT training stages, the strongest gains on most linguistic measures occur when language discrimination is introduced in the first iteration, whereas later or repeated interventions yield smaller improvements and are accompanied by increased language-wise segregation. These results support a causal role for language discrimination in reducing the additional cost of multilingual learning.