Transformer-based self-supervised speech models (S3Ms) are often described as contextualized, yet what this entails remains unclear. Here, we focus on how a single frame-level S3M representation can encode phones and their surrounding context. Prior work has shown that S3Ms represent phones compositionally; for example, phonological vectors such as voicing, bilabiality, and nasality vectors are superposed in the S3M representation of [m]. We extend this view by proposing that phonological information from a sequence of neighboring phones is also compositionally encoded in a single frame, such that vectors corresponding to previous, current, and next phones are superposed within a single frame-level representation. We show that this structure has several properties, including orthogonality between relative positions, and emergence of implicit phonetic boundaries. Together, our findings advance our understanding of context-dependent S3M representations.
Figures & tables
Figure 1: (a) Each frame-level S3M representation contains phonological vectors from position-sensitive, orthogonal subspaces. (b) Cosine similarity between frame-level representations and phonological vectors ( low , strident ) from the previous, current, and next phone subspaces. Individual frames encode phonological information from the current and neighboring phones with respect to phonetic boundaries.
Figure 2: Phonological analogy success rates comparing mean and center pooling on TIMIT (above) and VoxAngeles (below). Center pooling is on par and often outperforms mean-pooling, indicating that phonological compositionality is present at the level of individual frame representations.
Figure 3: Phonological analogy success rates for probing contextual information encoded in a single frame-level representation of p0 on TIMIT (upper) and VoxAngeles (lower). Center-pooled S3M representations support phonological analogies for the current phone position (0) and its neighbors (±1) .
Figure 4: Phonological analogy success rate for center phone p0 with respect to relative position on TIMIT (upper) and VoxAngeles (lower). WavLM exhibits the widest window, with high success rates within the center phone position 0 , decreasing for ±1 , and near-zero for ±2 . Spectral representations, unlike S3Ms, show nonzero success rates only near the center.
Figure 5: Cosine similarity between phonological vectors extracted from frame-level S3M representations on TIMIT (upper) and VoxAngeles (lower). The structure mirrors that of [ 11 ] , with opposing features showing negative similarity and related features showing positive similarity.
Figure 6: Cosine similarity between phonological vectors associated with different relative phone positions ( −2 to +2 ) from TIMIT (upper) and VoxAngeles (lower). Comparing with Figure 5 , relative similarity structure is preserved within positions. Further, vectors from different positions exhibit substantially lower similarity than those from the same position, implying approximate positional orthogonality.
Figure 7: Average L2 norm of eight phonological vectors per its relative position for each layer on TIMIT (upper) and VoxAngeles (lower). We observe that the average vector size decreases with increasing distance from the center phone.
Figure 8: Observing the level of orthogonality for each layer. We observe the average of absolute values of cosine similarities within- and across-subspace, on TIMIT (upper) and VoxAngeles (lower). Here, we define within- as same natural class (vowel or consonant), and same position ( −2 , −1 , 0 , +1 , or +2 ).
Figure 9: Phonetic segmentation experiment on TIMIT, where frame index 5 denotes the phonetic boundary of TIMIT. Cosine similarity between frame-level S3M representations and position-dependent phonological vectors around the phonetic boundaries. For example, for the boundary between unvoiced [-voi] and voiced [+voi] , left figure (boundary onsets) compares the evidence of current phone being voiced ( cos(v0,r[i]) ; blue) and the following phone being voiced ( cos(v+1,r[i]) ; orange), whereas the right figure (boundary offsets): compares the evidence of current phone being voiced ( cos(v0,r[i]) ; blue) and the preceding phone being voiced ( cos(v−1,r[i]) ; orange).
Figure 10: Cosine similarity between frame-level S3M representations and phonological vectors for relative phone positions ( −2 to +2 ) over a full TIMIT utterance. Staircase-like transitions often align with phonetic boundaries, illustrating position-specific orthogonal phonological subspaces.
Figure 11: Layerwise mask-filling behavior on TIMIT. Higher cosine similarity indicates more accurate recovery of masked representations from surrounding context.
Self-supervised speech models (S3Ms) are known to encode rich phonetic information, yet how this information is structured remains underexplored. We conduct a comprehensive study across 96 languages to analyze the underlying structure of S3M representations, with particular attention to phonological vectors. We first show that there exist linear directions within the model's representation space that correspond to phonological features. We further demonstrate that the scale of these phonological vectors correlate to the degree of acoustic realization of their corresponding phonological features in a continuous manner. For example, the difference between [d] and [t] yields a voicing vector: adding this vector to [p] produces [b], while scaling it results in a continuum of voicing. Together, these findings indicate that S3Ms encode speech using phonologically interpretable and compositional vectors, demonstrating phonological vector arithmetic. All code and interactive demos are available at https://github.com/juice500ml/phonetic-arithmetic .
Phone segmentation and recognition are inherently related tasks, yet modern approaches typically model them separately. We argue that phonetic structure is already latent in the representations of self-supervised speech models (S3Ms), so one only needs to steer them to solve both tasks. We leverage S3M-based Phonological Activation Mapping (SPAM), which maps each S3M representation frame to a vector of phonological feature activations, such as voicing and nasality. On top of SPAM, we introduce two simple but effective lightweight, gradient-descent-free prediction heads: a recognition head and a segmentation head. Our method requires less than a minute of phonetic transcriptions, and the recognition head generalizes to phones unseen during training. Across a diverse range of datasets, our approach attains strong segmentation performance.
Shikhar Bharadwaj, Kwanghee Choi, Stephen McIntosh +8
This study examines the extent to which the wav2vec2.0 architecture exhibits evidence of compensation for phonological context. We conducted a pseudo-replication of a perceptional compensation experiment on Mandarin Chinese tones, and compared the embedding similarities and probing classifier outputs between a purely self-supervised pre-trained model and a model fine-tuned for Mandarin ASR. No evidence of compensation was found in the embedding similarities of the purely pre-trained model. Probing classifiers showed some evidence of compensation in addition to the expected layer-wise improvements in categorization, but failed to replicate human performance on isolated test syllables. Our findings contrast with previous reports of sensitivity to phonological structure emerging through pre-training alone, and suggest that supervised objectives may be necessary to encourage the abstraction of at least some types of phonological regularities.
James Kirby, Ioana Krehan, Michele Gubian
Institute for Phonetics and Speech Processing, LMU Munich, Germany