DiscoPhon: Benchmarking the Unsupervised Discovery of Phoneme Inventories With Discrete Speech Units

Maxime Poli; Manel Khentout; Angelo Ortiz Tandazo; Ewan Dunbar; Emmanuel Chemla; Emmanuel Dupoux

cs.CLMar 19, 2026

DiscoPhon: Benchmarking the Unsupervised Discovery of Phoneme Inventories With Discrete Speech Units

Authors: Maxime Poli, Manel Khentout, Angelo Ortiz Tandazo, Ewan Dunbar, Emmanuel Chemla, Emmanuel Dupoux

Organizations: LSCP, ENS, EHESS, CNRS, PSL University, France · University of Toronto, Canada

Abstract

We introduce DiscoPhon, a multilingual benchmark for evaluating unsupervised phoneme discovery from discrete speech units. DiscoPhon covers 6 dev and 6 test languages, chosen to span a wide range of phonemic contrasts. Given only 10 hours of speech in a previously unseen language, systems must produce discrete units that are mapped to a predefined phoneme inventory, through either a many-to-one or a one-to-one assignment. The resulting sequences are evaluated for unit quality, recognition and segmentation. We provide four pretrained multilingual HuBERT and SpidR baselines, and show that phonemic information is available enough in current models for derived units to correlate well with phonemes, though with variations across languages.

Figures & tables

Figure 1 : Overview of the streams in the evaluation pipeline. The model maps a waveform to units $\bm{u}$ , evaluated against the gold phones $\bm{p}$ via $\textnormal{PNMI}(\bm{p},\bm{u})$ . An assignment, here many-to-one, then assigns units to phones $\bm{a}$ , assessed for recognition with $\textnormal{PER}(\bm{p},\bm{a})$ and segmentation with $R\textnormal{-value}(\bm{p},\bm{a})$ and $F_{1}(\bm{p},\bm{a})$ .

	Fricative	Affricate	Plosive	Vibrant	Nasal	Approximant	Monophthong	Diphthong
dev languages
German	\textipa f, \textipa s, \textipa S, \textipa ç, \textipa x, \textipa h	\textipa t͡s	\textipa p, \textipa t, \textipa k, \textipa b, \textipa d, \textipa \textscriptg	\textipa r, \textipa \textfishhookr	\textipa m, \textipa n, \textipa ŋ	\textipa l, \textipa j	\textipa I, \textipa i:, \textipa Y, \textipa y:, \textipa U, \textipa u:, \textipa e:, \textipa ø:	\textipa a \textsubarch I, \textipa a \textsubarch U
	\textipa v, \textipa z						\textipa @, \textipa o:, \textipa E, \textipa E:, \textipa \textrevepsilon , \textipa œ, \textipa O, \textipa a, \textipa a:
Swahili	\textipa f, \textipa T, \textipa s, \textipa S, \textipa h	\textipa t͡S	\textipa p, \textipa t, \textipa k, \textipa b, \textipa d, \textipa \textbardotlessj , \textipa \textscriptg	\textipa \textfishhookr	\textipa m, \textipa n, \textipa \textltailn	\textipa w, \textipa l, \textipa j	\textipa i, \textipa u, \textipa E, \textipa O, \textipa A
	\textipa v, \textipa D, \textipa z, \textipa G
Tamil	\textipa s, \textipa \textrtails	\textipa t͡S, \textipa d͡Z	\textipa p, \textipa \textsubbridge t, \textipa \textrtailt , \textipa k	\textipa \textfishhookr	\textipa m, \textipa n, \textipa \textrtailn , \textipa \textltailn , \textipa ŋ	\textipa V, \textipa l, \textipa \textrtaill , \textipa \textturnrrtail , \textipa j	\textipa i, \textipa i:, \textipa u, \textipa u:, \textipa e, \textipa e:, \textipa o, \textipa o:, \textipa a, \textipa a:

Table 1 : Phonemic categories considered in the benchmark. dev languages: German (41 phonemes), Swahili (29), Tamil (29), Thai (40), Turkish (27), and Ukrainian (35). test languages: Basque (29 phonemes), English (39), French (34), Japanese (42), Mandarin Chinese (42), and Wolof (39). German, English, French, and Wolof data from [ 20 , 24 ] , other languages from [ 25 , 26 ] .

	dev languages					test languages
	PER $\downarrow$	$\bm R$ -value $\uparrow$	$F_{1}$ $\uparrow$	PNMI $\uparrow$	ABX c. $\downarrow$	PER $\downarrow$	$\bm R$ -value $\uparrow$	$F_{1}$ $\uparrow$	PNMI $\uparrow$	ABX c. $\downarrow$
Zero-shot
HuBERT MMS-ulab (L12)	117.25	17.70	58.67	48.32	11.18	126.46	6.99	57.79	51.05	8.72
HuBERT VP-20 (L11)	118.26	19.63	58.87	47.01	10.23	127.02	9.39	58.18	49.49	7.79
SpidR MMS-ulab (L5)	80.50	45.93	$\mathbf{64.77}$	$\mathbf{56.23}$	9.30	84.81	40.99	$\mathbf{65.65}$	$\mathbf{58.58}$	7.37
SpidR VP-20 (L6)	$\mathbf{64.41}$	$\mathbf{58.75}$	61.83	54.51	$\mathbf{8.80}$	$\mathbf{64.48}$	$\mathbf{56.04}$	63.05	58.12	$\mathbf{6.57}$

Table 2 : DiscoPhon baselines many-to-one scores (256 units). Units are mapped to the most probable phoneme for evaluation. Best layer L is chosen to minimize continuous ABX on dev languages. HuBERT units:

K

-means on layer L trained on pretraining data (zero-shot), or finetuning data; SpidR units: prediction heads of layer L. Results (in

\%

) averaged across dev and test languages, resp. ABX on continuous representations averaged between within and across speaker conditions. Best scores in bold .

	dev languages		test languages
	PER $\downarrow$	$\bm R$ -value $\uparrow$	PER $\downarrow$	$\bm R$ -value $\uparrow$
Zero-shot
HuBERT MMS-ulab	271.80	-99.10	264.84	-93.39
HuBERT VP-20	184.11	-27.71	196.58	-37.36
SpidR MMS-ulab	135.64	14.72	142.06	7.13
SpidR VP-20	$\mathbf{113.57}$	$\mathbf{31.53}$	$\mathbf{120.03}$	$\mathbf{24.45}$

Table 3 : DiscoPhon baselines one-to-one scores (

|\mathcal{P}|+1

units). Each phoneme is mapped to a single unit. For each model and language,

K

-means is fitted on pretraining (zero-shot models) or finetuning data. Target layer as in Table 2 , best scores in bold .

Figure 2 : Analysis of recognition errors in a many-to-one evaluation. A : Breakdown of the phone error rate of SpidR VP-20 with 256 units into insertions, substitutions, and deletions for each language, comparing zero-shot and finetuned models. B : Substitution confusion matrix between phoneme class for the finetuned model, averaged across languages. Each row indicates, given a ground-truth class of phonemes, which predicted class it was replaced by (in $\%$ ).

Explore similar work

Language Discrimination Improves Linguistic Learning in Multilingual Speech Models
Sep 27, 2026Maureen de Seyssel, Jie Chi, Zakaria AldenehMultilingual Automatic Speech Recognition Cross-Lingual Consistency
Apple
BiMamba2 Masked Discrete-Unit Prediction for Multilingual Speech Representation for Unsupervised Speech in the Wild Challenge
Sep 23, 2026Prakriti Subedi, Howard Prioleau, Saurav K AryalMamba-Based Models Encoders
Howard University, Washington D.C., USA
Multilingual Phonological Feature Recognition with Self-Supervised Speech Models
May 25, 2026Abner Hernandez, Tomás Arias-Vergara, Daiqi Liu +2Self-Supervised Speech Models Pronunciation Assessment
Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany · GITA Lab. Facultad de Ingenier´ıa. Universidad de Antioquia UdeA, Medell´ın, Colombia

Sep 27, 2026, cs.CLJ/K move · Enter open · S save

Language Discrimination Improves Linguistic Learning in Multilingual Speech Models

Maureen de Seyssel, Jie Chi, Zakaria Aldeneh

Apple

Multilingual Automatic Speech Recognition Cross-Lingual Consistency Linguistics Prosody Dial

Open paper AskPDF arXiv

Multilingual self-supervised speech models can benefit from sharing information across languages, but under a matched total pretraining data budget they still fall short of monolingual models. We show that strengthening the model's ability to discriminate languages during pretraining reduces and, on some measures, closes this multilingual gap on continuous phonetic and higher-level linguistic measures, while preserving substantial cross-language sharing. Using a controlled English/French HuBERT setting, we test two interventions which strengthen language discrimination: an auxiliary language classifier and per-language k-means targets. Across interventions, continuous-feature phone discrimination error (phone-ABX, lower is better) decreases from 11.6% in the bilingual baseline to 10.4% (monolingual: 10.8%), while lexical performance (sWUGGY, higher is better) increases from 52.1% to 56.7% (monolingual: 58.5%) and prosodic performance (ProsAudit, lexical subtask, higher is better) from 68.9% to 72.9% (monolingual: 72.6%). Across HuBERT training stages, the strongest gains on most linguistic measures occur when language discrimination is introduced in the first iteration, whereas later or repeated interventions yield smaller improvements and are accompanied by increased language-wise segregation. These results support a causal role for language discrimination in reducing the additional cost of multilingual learning.

Abstract

Explore similar work

Language Discrimination Improves Linguistic Learning in Multilingual Speech Models

BiMamba2 Masked Discrete-Unit Prediction for Multilingual Speech Representation for Unsupervised Speech in the Wild Challenge

Multilingual Phonological Feature Recognition with Self-Supervised Speech Models