eess.ASFeb 21, 2026
Save[b] = [d] - [t] + [p]: Self-supervised Speech Models Discover Phonological Vector Arithmetic
Organizations: UT Austin · UC Berkeley · CMU
Abstract
Self-supervised speech models (S3Ms) are known to encode rich phonetic information, yet how this information is structured remains underexplored. We conduct a comprehensive study across 96 languages to analyze the underlying structure of S3M representations, with particular attention to phonological vectors. We first show that there exist linear directions within the model's representation space that correspond to phonological features. We further demonstrate that the scale of these phonological vectors correlate to the degree of acoustic realization of their corresponding phonological features in a continuous manner. For example, the difference between [d] and [t] yields a voicing vector: adding this vector to [p] produces [b], while scaling it results in a continuum of voicing. Together, these findings indicate that S3Ms encode speech using phonologically interpretable and compositional vectors, demonstrating phonological vector arithmetic. All code and interactive demos are available at https://github.com/juice500ml/phonetic-arithmetic .
Figures & tables
Figure 1: Comparing analogies for text and speech. Word representations enable semantic analogies 3 3 3 Word analogies borrowed from Ethayarajh et al. (2019) . , while speech representations enable phonological analogies. Such analogies ( Section 3 ) can be used to control speech synthesis in a phonologically grounded manner ( Section 4 ).
Figure 2: Comparing S3Ms with spectral representations on TIMIT (top) and VoxAngeles (bottom).
Figure 3: Comparing consonant-only and vowel-only quadruplets on TIMIT (top) and VoxAngeles (bottom) for WavLM. Number within the parenthesis denotes the number of quadruplets. We exclude cases where a quadruplet contains both consonants and vowels.
| Phonological feature | High | Low | Back | Round | Nasal | Sonorant | Strident | Voice |
|---|---|---|---|---|---|---|---|---|
| Acoustic measurement | F1 | F1 | F2 | F2 | F1BW | HNR | COG | COG |
| Expected correlation sign | – | + | – | – | – | + | + | – |
Table 1: Summary of phonological features, their associated acoustic measurements, and expected correlation signs. The phonologically expected correlation sign is denoted as + (positive) or – (negative). Five types of acoustic measurements are used: first formant (F1), second formant (F2), first-formant bandwidth (F1BW), harmonic-to-noise ratio (HNR), and center of gravity (COG). Further details are provided in § A.3 .
Figure 4: Comparison between the phonological vector scale and the acoustic measurements ( § A.3 ) on TIMIT. denotes Spearman’s rank correlation coefficient. Blue and orange plots indicate vowels and consonants, respectively. The empirically observed correlation signs match the theoretical expectations shown in Table 1 . Further, plots demonstrate the linearity of phonological vectors, resulting in monotonic (but not necessarily linear) changes in acoustic measurements.
Figure 5: Applying round vector to front vowel [i], where there is no front rounded vowel in English. Orange and blue arrows denote F2 and F3, respectively, which are all decreasing for .
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Comparing pairing consistency score (PCS) of S3Ms and spectral representations using TIMIT (upper) and VoxAngeles (lower).
Figure 10: Comparing success rates for S3Ms with spectral representations using TIMIT. We denote feature and audio slicing as feat and audio, respectively.
Figure 11: Comparing success rates for feature sliced S3Ms with spectral representations using VoxAngeles. We denote feature and audio slicing as feat and audio, respectively.
Figure 12: Comparing averaged similarities for feature sliced S3Ms with audio sliced MFCC on TIMIT (upper) and VoxAngeles (lower). We calculate the 99% CI considering quadruplet-wise cosine averages.
Figure 13: Comparing pre-trained S3M (XLSR-53) and fine-tuned phone recognition models (Wav2vec2Phoneme and MultIPA) on TIMIT.
Figure 14: Phonological feature-wise success rates for consonants on TIMIT using WavLM representations.
Figure 16: Phonological feature-wise success rates for consonants on VoxAngeles using WavLM representations.
Figure 18: PanPhon feature distance-wise success rates on TIMIT using WavLM representations.
Figure 20: Approximating various phonological vectors on TIMIT (upper) and VoxAngeles (lower) with different sample size . Each histogram depicts the distribution of cosine similarities.
Figure 21: Approximating various phonological vectors on TIMIT (upper) and VoxAngeles (lower) using a fixed phone pair. Each histogram depicts the distribution of cosine similarities. Figure 20 is overlaid for comparison.
Figure 22: Cosine similarities between different phonological vectors drawn from TIMIT.
Figure 24: Acoustic measurements of F1, F2, and F3 for rounded (blue) and unrounded (orange) vowels on TIMIT (upper) and VoxAngeles (lower). indicates Spearman’s rank correlation coefficient between the roundness and the acoustic measurements.
Figure 25: Comparing the phonological vector weight with acoustic measurements on TIMIT (upper) VoxAngeles (lower) using WavLM. We observe three measurements, F1, F2, and F3, for the round vector. indicates Spearman’s rank correlation coefficient.
Figure 26: Comparing the phonological vector weight with acoustic measurements on VoxAngeles using WavLM. indicates Spearman’s rank correlation coefficient. Blue and orange plots indicate vowels and consonants, respectively.
Figure 27: Comparing the acoustic measurements of original and synthesized speech ( ) on TIMIT (upper two rows) and VoxAngeles (lower two rows). We use the same range of x-axis from y-axis in Figures 4 and 26 . We observe that the differences are highly centralized to zero, ensuring the stability of resynthesis through the vocoder.
Figure 28: Comparing the phonological vector weight with acoustic measurements on TIMIT (upper two rows) and VoxAngeles (lower two rows) using phonological vectors from audio-sliced MFCC. indicates Spearman’s rank correlation coefficient. Blue and orange plots indicate vowels and consonants, respectively. There is little to no controllability, with the exception of weak correlation on back vowel on VoxAngeles.