Sequence-Level Unsupervised Training in Speech Recognition: A Theoretical Study

Zijian Yang; Jörg Barkoczi; Ralf Schlüter; Hermann Ney

cs.SDMar 2, 2026

Sequence-Level Unsupervised Training in Speech Recognition: A Theoretical Study

Authors: Zijian Yang, Jörg Barkoczi, Ralf Schlüter, Hermann Ney

Organizations: Human Language Technology and Pattern Recognition, Computer Science Department, RWTH Aachen University, Germany · AppTek GmbH, Germany

Abstract

Unsupervised speech recognition is a task of training a speech recognition model with unpaired data. To determine when and how unsupervised speech recognition can succeed, and how classification error relates to candidate training objectives, we develop a theoretical framework for unsupervised speech recognition grounded in classification error bounds. We introduce two conditions under which unsupervised speech recognition is possible. The necessity of these conditions are also discussed. Under these conditions, we derive a classification error bound for unsupervised speech recognition and validate this bound in simulations. Motivated by this bound, we propose a single-stage sequence-level cross-entropy loss for unsupervised speech recognition.

Figures & tables

Figure 1: Simulation result for sequence-level bound between sequence-level marginal distributions $\sum_{x_{1}^{N}}|pr(x_{1}^{N})-q(x_{1}^{N})|$ and $\overline{D}_{q}$ . The simulation is done with $|\mathcal{X}|=4$ , $|\mathcal{C}|=3$ , $N=3$ , $\|\mathbf{P}_{C}^{+}\|_{1}\leq 2$ . $\mathbf{P}_{C}$ is guaranteed to have full column rank by conditioning that the minimum singular value is larger than 0.01. The grey dots refer to simulation points.

Automatic Speech Recognition

Automatic speech recognition converts spoken language into text across speakers, languages, domains, and recording conditions.

See all 480 papers

BiMamba2 Masked Discrete-Unit Prediction for Multilingual Speech Representation for Unsupervised Speech in the Wild Challenge
Sep 23, 2026Prakriti Subedi, Howard Prioleau, Saurav K AryalSpeech Processing Self-Supervised Speech Representation Learning
Howard University, Washington D.C., USA
Pay Attention to CTC: Fast and Robust Pseudo-Labelling for Unified Speech Recognition
Feb 22, 2026Alexandros Haliassos, Rodrigo Mira, Stavros PetridisConnectionist Temporal Classification Automatic Speech Recognition
NatWest AI Research · Imperial College London
Unsupervised Speech Enhancement via Drifting
Sep 28, 2026Diego Caviedes-Nozal, Liang Xu, Rasmus Kongsgaard Olsson +1Speech Enhancement
GN A/S, Denmark · Victoria University of Wellington, New Zealand

Sep 23, 2026J/K move · Enter open · S save

BiMamba2 Masked Discrete-Unit Prediction for Multilingual Speech Representation for Unsupervised Speech in the Wild Challenge

Prakriti Subedi, Howard Prioleau, Saurav K Aryal

Howard University, Washington D.C., USA

Speech Processing Self-Supervised Speech Representation Learning Discrete Speech Representation Learning

Open paper AskPDF arXiv

We describe our submission to the Unsupervised Speech in the Wild (UPS) Challenge at Interspeech 2026, a bidirectional Mamba-2 (BiMamba2) encoder trained with masked discrete-unit prediction following the HuBERT-style paradigm. The 47.88M-parameter model is trained on 250 hours of speech across 67 languages from the MLCommons Unsupervised People's Speech dataset, with no labeled data. The objective combines masked k-means pseudo-label prediction with language identification supervision and VICReg regularization. On official evaluation, the system achieves an Adjusted Rand Index of 0.735, exceeding four baselines on speaker clustering. Language identification macro-F1 (0.073) and character error rate (0.870) remain below supervised baselines. We analyze a local-official discrepancy in metric scale and checkpoint ranking, highlighting limitations of in-distribution diagnostics for predicting Dynabench probe outcomes.

Abstract

Automatic Speech Recognition

Explore similar work

BiMamba2 Masked Discrete-Unit Prediction for Multilingual Speech Representation for Unsupervised Speech in the Wild Challenge

Pay Attention to CTC: Fast and Robust Pseudo-Labelling for Unified Speech Recognition

Unsupervised Speech Enhancement via Drifting