BiMamba2 Masked Discrete-Unit Prediction for Multilingual Speech Representation for Unsupervised Speech in the Wild Challenge

Prakriti Subedi; Howard Prioleau; Saurav K Aryal

cs.SDSep 23, 2026

BiMamba2 Masked Discrete-Unit Prediction for Multilingual Speech Representation for Unsupervised Speech in the Wild Challenge

Authors: Prakriti Subedi, Howard Prioleau, Saurav K Aryal

Organizations: Howard University, Washington D.C., USA

Abstract

We describe our submission to the Unsupervised Speech in the Wild (UPS) Challenge at Interspeech 2026, a bidirectional Mamba-2 (BiMamba2) encoder trained with masked discrete-unit prediction following the HuBERT-style paradigm. The 47.88M-parameter model is trained on 250 hours of speech across 67 languages from the MLCommons Unsupervised People's Speech dataset, with no labeled data. The objective combines masked k-means pseudo-label prediction with language identification supervision and VICReg regularization. On official evaluation, the system achieves an Adjusted Rand Index of 0.735, exceeding four baselines on speaker clustering. Language identification macro-F1 (0.073) and character error rate (0.870) remain below supervised baselines. We analyze a local-official discrepancy in metric scale and checkpoint ranking, highlighting limitations of in-distribution diagnostics for predicting Dynabench probe outcomes.

Figures & tables

en, es, de, pt, ar, fr, it, gl, ne, id, pl, zh, nl, ca, sv, am, ru, hu, hi, eu, ur, jw, ms, tr, ko, el, kn, sa, cy, ta, sw, uk, bs, ro, si, fa, te, sr, so, hr, ja, nn, la, da, tl, hy, vi, sq, pa, sk, ka, yo, be, bn, he, bg, km, br, af, sd, lt, sn, cs, mr, ps, th, ml

Table 1: Training corpus languages (67 total, language codes from UPS metadata), ordered by utterance count descending.

Figure 1: BiMamba2 encoder layer (repeated $n$ times). Given $\mathbf{x}\in\mathbb{R}^{B\times S\times D}$ , the layer computes $\mathbf{y}=\mathrm{SS}(\mathbf{x})+\mathrm{flip}(\mathrm{SS}(\mathrm{flip}(\mathbf{x})))+D\mathbf{x}$ , where $\mathrm{SS}$ is the Mamba2 state-space model and $\mathrm{flip}$ reverses the sequence. The forward and backward SS paths provide a bidirectional representation, while $D\mathbf{x}$ acts as a linear skip connection.

System	Macro-F1 $\uparrow$	CER $\downarrow$	ARI $\uparrow$
Whisper [ 24 ]	0.950	0.790	0.560
HuBERT-large [ 2 ]	0.690	0.570	0.600
XLSR [ 4 ]	0.260	0.900	0.100
wav2vec 2.0 [ 1 ]	0.250	0.980	0.630
Ours (d=512, 8L) †	0.052	0.996	0.291
Ours (d=768, 12L, step 48k)	0.070	0.998	0.710

Table 2: Official UPS Challenge results (Dynabench).

\uparrow

: higher is better;

\downarrow

: lower is better. Best per metric in bold .

\dagger

d{=}512

, 8 layers, batch 64, step 49,000.

Metric	Step 19.5k	Step 48k
Macro-F1	0.342	0.395
ARI	0.225	0.232
Intra-class sim.	0.786	0.747
Inter-class sim.	0.004	$-$ 0.002
Separability ratio	208.98	$-$ 366.84

Table 3: Local holdout validation (45 languages, 6,646 samples). Not official challenge scores. Intra/inter-class similarity computed as mean cosine similarity over mean-pooled embeddings per language. Separability ratio

=

intra-class sim.

\div

inter-class sim.; negative value at step 48k arises from negative mean inter-class cosine similarity.

Explore similar work

DiscoPhon: Benchmarking the Unsupervised Discovery of Phoneme Inventories With Discrete Speech Units
Mar 19, 2026Maxime Poli, Manel Khentout, Angelo Ortiz Tandazo +3Grapheme-To-Phoneme Multilingual Benchmark
LSCP, ENS, EHESS, CNRS, PSL University, France · University of Toronto, Canada
MauBERT: Universal Phonetic Inductive Biases for Few-Shot Acoustic Units Discovery
Dec 22, 2025Angelo Ortiz Tandazo, Manel Khentout, Youssef Benchekroun +2Self-Supervised Speech Models Articulatory
ENS, PSL Research University, EHESS, CNRS, Paris, France · Meta AI Research, France · Univ. Grenoble Alpes, CNRS, Grenoble INP, GIPSA-lab, Grenoble, France
From Monolingual to Multilingual: Evaluating Mamba for ASR in South African Languages
Jul 1, 2026Jesujoba O. Alabi, Julian Herreilers, Badr M. Abdullah +1Multilingual Automatic Speech Recognition Automatic Speech Recognition
Saarland University, Germany · Saigen, South Africa

Mar 19, 2026, cs.CLJ/K move · Enter open · S save

DiscoPhon: Benchmarking the Unsupervised Discovery of Phoneme Inventories With Discrete Speech Units

Maxime Poli, Manel Khentout, Angelo Ortiz Tandazo, Ewan Dunbar+2

LSCP, ENS, EHESS, CNRS, PSL University, France · University of Toronto, Canada

Grapheme-To-Phoneme Multilingual Benchmark Unsupervised

Open paper AskPDF arXiv

We introduce DiscoPhon, a multilingual benchmark for evaluating unsupervised phoneme discovery from discrete speech units. DiscoPhon covers 6 dev and 6 test languages, chosen to span a wide range of phonemic contrasts. Given only 10 hours of speech in a previously unseen language, systems must produce discrete units that are mapped to a predefined phoneme inventory, through either a many-to-one or a one-to-one assignment. The resulting sequences are evaluated for unit quality, recognition and segmentation. We provide four pretrained multilingual HuBERT and SpidR baselines, and show that phonemic information is available enough in current models for derived units to correlate well with phonemes, though with variations across languages.

Abstract

Explore similar work

DiscoPhon: Benchmarking the Unsupervised Discovery of Phoneme Inventories With Discrete Speech Units

MauBERT: Universal Phonetic Inductive Biases for Few-Shot Acoustic Units Discovery

From Monolingual to Multilingual: Evaluating Mamba for ASR in South African Languages