cs.CLNov 8, 2026

Understanding When and Where Self-Supervised Speech Models Learn Phonemes and Prosodic Cues through Layer-Wise and Time-Resolved Probing

Authors: Masaru Tanibata, Shun Takahashi, Hiroki Ouchi, Sakriani Sakti

Organizations: Division of Information Science, Nara Institute of Science and Technology, Japan

Abstract

Self-supervised speech models (S3Ms) encode a rich hierarchy of acoustic, prosodic, and phonetic information, yet when and where it emerges during pretraining remains unclear. We train frozen-encoder probes on every Transformer layer across pretraining checkpoints of two matched models: HuBERT-Base, whose stage 2 is a separately initialized model trained on new pseudo-label targets, and single-objective wav2vec 2.0-Base. Our five tasks span an acoustic-to-abstract gradient: absolute pitch (F0), speaker-relative pitch, lexical stress, phoneme boundaries, and 38-way phoneme identity. Acoustic-prosodic cues and phone boundaries are decodable from the earliest checkpoints and barely shift across layers, whereas phoneme identity is constructed during pretraining, re-localizing to upper-middle layers under HuBERT's stage-2 targets at a matched update count. Acquisition order therefore tracks abstraction from the acoustic surface, not the segmental/suprasegmental distinction, extending to prosody an account established for phonetic, lexical and syntactic structure; the no-switch wav2vec 2.0 shows no comparable re-localization.

Figures & tables

Explore similar work

CardsList
  1. Self-Supervised Speech Models Encode Phonetic Context via Position-dependent Orthogonal Subspaces

    Mar 13, 2026Kwanghee Choi, Eunjung Yeo, Cheol Jun Cho +2Self-Supervised Speech Representation Learning

  2. Tracking the emergence of linguistic structure in self-supervised models learning from speech

    Apr 2, 2026Marianne de Heer Kloots, Martijn Bentum, Hosein Mohebbi +3Self-Supervised Speech Representation LearningSelf-Supervised Pre-Training

  3. Do speech foundation models really learn words?

    Sep 9, 2026Robin Huo, Ewan DunbarRepresentation LearningSpeech Foundation Models