Scaling Self-Supervised Speech Models Uncovers Deep Linguistic Relationships: Evidence from the Pacific Cluster

Minu Kim; Hoirin Kim; David R. Mortensen

cs.CLMar 7, 2026

Scaling Self-Supervised Speech Models Uncovers Deep Linguistic Relationships: Evidence from the Pacific Cluster

Authors: Minu Kim, Hoirin Kim, David R. Mortensen

Organizations: Thomas Lord Department of Computer Science, University of Southern California, USA · School of Electrical Engineering, KAIST, Republic of Korea · Language Technologies Institute, Carnegie Mellon University, USA

Abstract

Similarities between language representations derived from Self-Supervised Speech Models (S3Ms) have been observed to primarily reflect geographic proximity or surface typological similarities driven by recent expansion or contact, potentially missing deeper genealogical signals. We investigate how scaling an S3M-based language identification system from 126 to 4,017 languages reshapes this topology, and find a non-linear effect: phylogenetic recovery stays flat up to the 1K scale, but the 4K model undergoes a qualitative shift, resolving both clear lineages and long-term linguistic contact. Most strikingly, a robust Pacific macro-cluster emerges, grouping genealogically unrelated Papuan, Oceanic, and Australian languages, and we trace its driver to a concentrated encoding that captures shared acoustic signatures such as global energy dynamics. These results suggest that massive S3Ms internalize multiple layers of language history, offering a promising perspective for computational phylogenetics and the study of language contact.

Figures & tables

Figure 1: Evaluation data exposure: MMS-LID-1K vs. 4K. Identical seen/unseen status for 91.8% (45/49) of evaluation languages ensures that performance gaps come from increased linguistic diversity rather than direct exposure.

Figure 2: Phylogenetic recovery performance across cluster counts $K\in[2,20]$ for four model scales. Performance plateaus up to 1K, but shifting to 4K results in a leap that peaks at $K=18$ , substantially outperforming all smaller baselines.

Figure 3: 4K model bootstrap consensus dendrogram (1,000 replicates). Branch values indicate support percentages. Red boxes highlight clusters reflecting linguistic contact, including the distinction between Austronesian subgroups A and B. Remarkably, 36 of 37 branches with $>50\%$ confidence align with established phylogenetic or areal groupings.

Figure 4: POA cluster quality across $K$ . Precision plateaus at 0.92 for 128, 256, and 1K models, but the 4K model achieves perfect Precision (1.00), demonstrating a unique capacity to isolate Pacific languages.

Figure 5: PCA projections of the embedding spaces for 1K (left) and 4K (right) models across selected groups. The 4K model exhibits more clearly delineated family-level boundaries and a more distinct spatial separation of the POA cluster compared to the 1K model.

	MMS-LID-1K		MMS-LID-4K
	FDR	Bonf.	FDR	Bonf.
# Sig. Dimensions	257	36	169	25
Feature
Energy dynamic range	18.3	–	18.9	28.0
MFCC 1 std	27.2	11.1	9.5	4.0
MFCC 2 std	17.9	2.8	13.6	12.0

Table 1: Of the total 1,280 dimensions, the first row defines the significant subset used as the denominator for the percentages (%) of Pearson-correlated dimensions per feature. (

p<.05

)

Feature	Direction (POA)	Effect size (Cohen’s $d$ )
Energy dynamic range	Higher	+0.69
MFCC 1 std	Lower	-0.76
MFCC 2 std	Lower	-0.88
MFCC 3 std	Lower	-1.17
MFCC 4 std	Lower	-0.76
Spectral centroid std	Lower	-0.61

Table 2: Significant acoustic differences between POA and Non-POA languages (Mann–Whitney U test). All listed features satisfy

p<.001

and

|d|>0.5

Self-Supervised Speech Representation Learning

Self-Supervised Speech Representation Learning trains models on unlabeled audio to extract features useful for speech tasks and linguistic analysis.

See all 101 papers

[b] = [d] - [t] + [p]: Self-supervised Speech Models Discover Phonological Vector Arithmetic
Feb 21, 2026Kwanghee Choi, Eunjung Yeo, Cheol Jun Cho +2Self-Supervised Speech Representation Learning
UT Austin · UC Berkeley · CMU
Self-Supervised Lexical Representation Learning for Fast, Large-Scale Phylogenetic Inference
Sep 4, 2026Tim WientzekPhylogenetic Inference Self-Supervised Learning
Self-Supervised Speech Models Encode Phonetic Context via Position-dependent Orthogonal Subspaces
Mar 13, 2026Kwanghee Choi, Eunjung Yeo, Cheol Jun Cho +2Self-Supervised Speech Representation Learning
UT Austin · UC Berkeley · Carnegie Mellon University

Feb 21, 2026J/K move · Enter open · S save

[b] = [d] - [t] + [p]: Self-supervised Speech Models Discover Phonological Vector Arithmetic

Kwanghee Choi, Eunjung Yeo, Cheol Jun Cho, David Harwath+1

UT Austin · UC Berkeley · CMU

Self-Supervised Speech Representation Learning

Open paper AskPDF arXiv

Self-supervised speech models (S3Ms) are known to encode rich phonetic information, yet how this information is structured remains underexplored. We conduct a comprehensive study across 96 languages to analyze the underlying structure of S3M representations, with particular attention to phonological vectors. We first show that there exist linear directions within the model's representation space that correspond to phonological features. We further demonstrate that the scale of these phonological vectors correlate to the degree of acoustic realization of their corresponding phonological features in a continuous manner. For example, the difference between [d] and [t] yields a voicing vector: adding this vector to [p] produces [b], while scaling it results in a continuum of voicing. Together, these findings indicate that S3Ms encode speech using phonologically interpretable and compositional vectors, demonstrating phonological vector arithmetic. All code and interactive demos are available at https://github.com/juice500ml/phonetic-arithmetic .

Abstract

Self-Supervised Speech Representation Learning

Explore similar work

[b] = [d] - [t] + [p]: Self-supervised Speech Models Discover Phonological Vector Arithmetic

Self-Supervised Lexical Representation Learning for Fast, Large-Scale Phylogenetic Inference

Self-Supervised Speech Models Encode Phonetic Context via Position-dependent Orthogonal Subspaces