Transformer-based self-supervised speech models (S3Ms) are often described as contextualized, yet what this entails remains unclear. Here, we focus on how a single frame-level S3M representation can encode phones and their surrounding context. Prior work has shown that S3Ms represent phones compositionally; for example, phonological vectors such as voicing, bilabiality, and nasality vectors are superposed in the S3M representation of [m]. We extend this view by proposing that phonological information from a sequence of neighboring phones is also compositionally encoded in a single frame, such that vectors corresponding to previous, current, and next phones are superposed within a single frame-level representation. We show that this structure has several properties, including orthogonality between relative positions, and emergence of implicit phonetic boundaries. Together, our findings advance our understanding of context-dependent S3M representations.
Figures & tables
Figure 1: (a) Each frame-level S3M representation contains phonological vectors from position-sensitive, orthogonal subspaces. (b) Cosine similarity between frame-level representations and phonological vectors ( low , strident ) from the previous, current, and next phone subspaces. Individual frames encode phonological information from the current and neighboring phones with respect to phonetic boundaries.
Figure 2: Phonological analogy success rates comparing mean and center pooling on TIMIT (above) and VoxAngeles (below). Center pooling is on par and often outperforms mean-pooling, indicating that phonological compositionality is present at the level of individual frame representations.
Figure 3: Phonological analogy success rates for probing contextual information encoded in a single frame-level representation of p0 on TIMIT (upper) and VoxAngeles (lower). Center-pooled S3M representations support phonological analogies for the current phone position (0) and its neighbors (±1) .
Figure 4: Phonological analogy success rate for center phone p0 with respect to relative position on TIMIT (upper) and VoxAngeles (lower). WavLM exhibits the widest window, with high success rates within the center phone position 0 , decreasing for ±1 , and near-zero for ±2 . Spectral representations, unlike S3Ms, show nonzero success rates only near the center.
Figure 5: Cosine similarity between phonological vectors extracted from frame-level S3M representations on TIMIT (upper) and VoxAngeles (lower). The structure mirrors that of [ 11 ] , with opposing features showing negative similarity and related features showing positive similarity.
Figure 6: Cosine similarity between phonological vectors associated with different relative phone positions ( −2 to +2 ) from TIMIT (upper) and VoxAngeles (lower). Comparing with Figure 5 , relative similarity structure is preserved within positions. Further, vectors from different positions exhibit substantially lower similarity than those from the same position, implying approximate positional orthogonality.
Figure 7: Average L2 norm of eight phonological vectors per its relative position for each layer on TIMIT (upper) and VoxAngeles (lower). We observe that the average vector size decreases with increasing distance from the center phone.
Figure 8: Observing the level of orthogonality for each layer. We observe the average of absolute values of cosine similarities within- and across-subspace, on TIMIT (upper) and VoxAngeles (lower). Here, we define within- as same natural class (vowel or consonant), and same position ( −2 , −1 , 0 , +1 , or +2 ).
Figure 9: Phonetic segmentation experiment on TIMIT, where frame index 5 denotes the phonetic boundary of TIMIT. Cosine similarity between frame-level S3M representations and position-dependent phonological vectors around the phonetic boundaries. For example, for the boundary between unvoiced [-voi] and voiced [+voi] , left figure (boundary onsets) compares the evidence of current phone being voiced ( cos(v0,r[i]) ; blue) and the following phone being voiced ( cos(v+1,r[i]) ; orange), whereas the right figure (boundary offsets): compares the evidence of current phone being voiced ( cos(v0,r[i]) ; blue) and the preceding phone being voiced ( cos(v−1,r[i]) ; orange).
Figure 10: Cosine similarity between frame-level S3M representations and phonological vectors for relative phone positions ( −2 to +2 ) over a full TIMIT utterance. Staircase-like transitions often align with phonetic boundaries, illustrating position-specific orthogonal phonological subspaces.
Figure 11: Layerwise mask-filling behavior on TIMIT. Higher cosine similarity indicates more accurate recovery of masked representations from surrounding context.