Self-supervised speech models (S3Ms) encode a rich hierarchy of acoustic, prosodic, and phonetic information, yet when and where it emerges during pretraining remains unclear. We train frozen-encoder probes on every Transformer layer across pretraining checkpoints of two matched models: HuBERT-Base, whose stage 2 is a separately initialized model trained on new pseudo-label targets, and single-objective wav2vec 2.0-Base. Our five tasks span an acoustic-to-abstract gradient: absolute pitch (F0), speaker-relative pitch, lexical stress, phoneme boundaries, and 38-way phoneme identity. Acoustic-prosodic cues and phone boundaries are decodable from the earliest checkpoints and barely shift across layers, whereas phoneme identity is constructed during pretraining, re-localizing to upper-middle layers under HuBERT's stage-2 targets at a matched update count. Acquisition order therefore tracks abstraction from the acoustic surface, not the segmental/suprasegmental distinction, extending to prosody an account established for phonetic, lexical and syntactic structure; the no-switch wav2vec 2.0 shows no comparable re-localization.
Figures & tables
Fig. 1 : Overview of our probing setup. We extract features at each learning checkpoint (A) and Transformer layer in S3Ms (B) and use them for downstream tasks. The encoders that produce the input features are frozen during probe training.
Task
Dataset
Duration
Pitch
PTDB-TUG
9.6 hours
F0
PTDB-TUG
9.6 hours
Phoneme boundary
TIMIT
4.3 hours
Phoneme
TIMIT
4.3 hours
Lexical Stress
Prosody-ABX
4.6 minutes
TABLE I : Datasets & duration time
Fig. 2 : Probe accuracy for speaker-relative pitch. x : log-spaced cumulative updates; for HuBERT, 0–250k is stage 1 and 250k–650k the separately initialized stage 2; wav2vec 2.0 has no stage split.
Fig. 3 : F0 regression performance ( R2 ). Axis as in Fig. 2 .
Fig. 4 : Probe accuracy for phoneme boundary detection. Axis as in Fig. 2 .
Fig. 5 : Probe accuracy for phoneme task. Axis as in Fig. 2 .
Fig. 6 : Per-phoneme-class F1 across pretraining. Axis as in Fig. 2 .
Fig. 7 : Probe accuracy for stress ABX. Axis as in Fig. 2 .
Transformer-based self-supervised speech models (S3Ms) are often described as contextualized, yet what this entails remains unclear. Here, we focus on how a single frame-level S3M representation can encode phones and their surrounding context. Prior work has shown that S3Ms represent phones compositionally; for example, phonological vectors such as voicing, bilabiality, and nasality vectors are superposed in the S3M representation of [m]. We extend this view by proposing that phonological information from a sequence of neighboring phones is also compositionally encoded in a single frame, such that vectors corresponding to previous, current, and next phones are superposed within a single frame-level representation. We show that this structure has several properties, including orthogonality between relative positions, and emergence of implicit phonetic boundaries. Together, our findings advance our understanding of context-dependent S3M representations.
Kwanghee Choi, Eunjung Yeo, Cheol Jun Cho +2
UT Austin · UC Berkeley · Carnegie Mellon University
Self-supervised speech models learn effective representations of spoken language, which have been shown to reflect various aspects of linguistic structure. But when does such structure emerge in model training? We study the encoding of a wide range of linguistic structures, across layers and intermediate checkpoints of six Wav2Vec2 and HuBERT models trained on spoken Dutch. We find that different levels of linguistic structure show notably distinct layerwise patterns as well as learning trajectories, which can partially be explained by differences in their degree of abstraction from the acoustic signal and the timescale at which information from the input is integrated. Moreover, we find that the level at which pre-training objectives are defined strongly affects both the layerwise organization and the learning trajectories of linguistic structures, with greater parallelism induced by higher-order prediction tasks (i.e. iteratively refined pseudo-labels).
Marianne de Heer Kloots, Martijn Bentum, Hosein Mohebbi +3
Institute for Logic, Language and Computation, University of Amsterdam, The Netherlands · Centre for Language Studies, Radboud University, The Netherlands · Cognitive Science and Artificial Intelligence, Tilburg University, The Netherlands
Self-supervised speech foundation models are now used in a wide array of downstream applications, including traditional speech recognition and as the basis for tokens in speech-aware language models. Attempts to understand their usefulness have largely focused on probing their representations' ability to discriminate phonemes and words. However, discriminative ability for words need not imply specialized representation of words per se. Good discrimination of words may be explained by good encoding of word form (phonemes) rather than form-independent word representations encoding identity or syntactic/semantic properties. By partialling out phoneme information using residualization, we show that, in later layers, HuBERT and wav2vec 2.0 do in general learn representations which encode words with reasonable fidelity independently of local phonetic content. We show that this simple approach to disentanglement can enhance higher-order linguistic information in word discovery tasks.