Understanding When and Where Self-Supervised Speech Models Learn Phonemes and Prosodic Cues through Layer-Wise and Time-Resolved Probing
Organizations: Division of Information Science, Nara Institute of Science and Technology, Japan
Abstract
Self-supervised speech models (S3Ms) encode a rich hierarchy of acoustic, prosodic, and phonetic information, yet when and where it emerges during pretraining remains unclear. We train frozen-encoder probes on every Transformer layer across pretraining checkpoints of two matched models: HuBERT-Base, whose stage 2 is a separately initialized model trained on new pseudo-label targets, and single-objective wav2vec 2.0-Base. Our five tasks span an acoustic-to-abstract gradient: absolute pitch (F0), speaker-relative pitch, lexical stress, phoneme boundaries, and 38-way phoneme identity. Acoustic-prosodic cues and phone boundaries are decodable from the earliest checkpoints and barely shift across layers, whereas phoneme identity is constructed during pretraining, re-localizing to upper-middle layers under HuBERT's stage-2 targets at a matched update count. Acquisition order therefore tracks abstraction from the acoustic surface, not the segmental/suprasegmental distinction, extending to prosody an account established for phonetic, lexical and syntactic structure; the no-switch wav2vec 2.0 shows no comparable re-localization.
Figures & tables
| Task | Dataset | Duration |
|---|---|---|
| Pitch | PTDB-TUG | 9.6 hours |
| F0 | PTDB-TUG | 9.6 hours |
| Phoneme boundary | TIMIT | 4.3 hours |
| Phoneme | TIMIT | 4.3 hours |
| Lexical Stress | Prosody-ABX | 4.6 minutes |
Self-Supervised Speech Representation Learning
Self-Supervised Speech Representation Learning trains models on unlabeled audio to extract features useful for speech tasks and linguistic analysis.
See all 107 papers