Benchmarking_Fast_Domain_Adaptation_for_Unsupervised_Speech_Units

Robin San Roman; Manel Khentout; Tu Anh Nguyen; Paul Michel; Yossi Adi; Emmanuel Dupoux

cs.LGAug 27, 2026

Benchmarking_Fast_Domain_Adaptation_for_Unsupervised_Speech_Units

Authors: Robin San Roman, Manel Khentout, Tu Anh Nguyen, Paul Michel, Yossi Adi, Emmanuel Dupoux

Organizations: CoML team, ENS, PSL university, CNRS, EHESS

Abstract

Representation learning has attracted great atten- tion and managed to reach good performances as a pretraining method for downstream tasks or as a first step towards unsu- pervised speech modeling. Yet, little is known about how such methods deal with out-of-domain speech and how could they be adapted in a few shot to new domains. This is important especially for accented speech where one observes a long tail of accents that diverge from the standard ones. We introduce ABX- Accent, a benchmark based on the AESRC dataset that features 10 different accents of English. It includes a small (< 10 hours) unlabelled training set in each of the accents and adaptations of the Zero Resources Challenge ABX evaluation metrics to each of the accents. We illustrate this benchmark with a baseline model that uses adaptive domain normalization to fine tune a pretrained Contrastive Predictive Coding model on the accents. This method is first developed on LibriSpeech using a male/female split. When applied to the new benchmark, the proposed method yields a relative improvement of 23.6% on across-speaker ABX scores on average compared to non adapted models. The data and metrics will be open sourced upon paper acceptance

Figures & tables

Fig. 1: Structure of the ABX-Accent benchmkark. Each of the K accent has a training set, a dev and a test set, each containing different speakers (K=10 accents). To allow for speaker adaptation, the dev and test set provide for each speaker a 2min adaptation set.

Accent	Acronym	Duration	Nb of speakers
American	e-us	9:56h	21 H, 25 F
British	e-uk	15:06h	38 H, 36 F
Canadian	e-ca	8:07h	9 H, 10 F
Chinese	e-ch	8:23h	14 H, 12 F
Indian	e-in	7:34h	8 H, 10 F
Portuguese	e-pt	9:18h	13 H, 15 F

TABLE I: Duration and number of speakers for the Train Set of ABX-Accent. Prefix ”e-” highlights that the data is English speech.

Split /Domain	Old	New	Old	New	Old	New	Old	New
	FT		resamp-FT		DN		DN+resamp-FT
2min 1 spk	9.16	11.82	9.24	11.79	9.06	15.22	9.10	10.8
20min 10 spk	9.31	11.42	9.17	11.15	9.10	13.68	9.15	9.31
2h 60 spk	9.41	10.62	9.05	10.02	9.22	9.76	9.05	8.59
8h 240 spk	9.57	9.74	8.89	9.38	9.09	8.98	8.89	8.19
16h 479 spk	9.56	9.96	9.00	9.33	8.93	8.93	8.34	8.04

TABLE II: ABX scores (%) across speakers for CPC models trained on the Old domain (male or female LibriSpeech) and tested on the Old and New domains (the other sex). Lower scores are better and are averaged across males and females experiments. On average the baseline models only trained on the Old domain give 9.06% and 11.86% on the Old and New test sets respectively. Bold correspond to the best scores for each line and underline, overall. FT: simple fine tuning. resamp-FT: fine tuning with resampling. DN: domain normalization.

	e-us	e-uk	e-ca	e-ch	e-in	e-ja	e-ko	e-pt	e-ru	e-es	Average
MFCC	$37.9$	$37.3$	$35.5$	$32.7$	$33.5$	$36.2$	$32.6$	$35.0$	$35.2$	$35.6$	$35.1\pm 2$
AESRC	$16.1$	$14.9$	$22.4$	$16.0$	$17.1$	$20.1$	$21.9$	$15.5$	$18.6$	$16.3$	$17.9\pm 2$
LS pretrain	$13.1$	$13.2$	$20.3$	$14.2$	$15.4$	$19.4$	$20.6$	$13.7$	$16.5$	$14.7$	$16.1\pm 3$
FT (single)	$12.9$	$12.9$	$20.6$	$13.9$	$16.0$	$10.9$	$20.4$	$13.6$	$13.0$	$12.7$	$14.7\pm 3$
FT (joint)	$\bf 12.3$	$12.4$	$20.1$	$13.4$	$11.7$	$10.5$	$19.9$	$13.4$	$12.2$	$12.1$	$13.8\pm 3$
DN + FT (single)	$13.0$	$13.3$	$20.3$	$12.8$	$\bf 11.0$	$9.8$	$20.0$	$18.2$	$\bf 9.9$	$11.7$	$14.0\pm 4$

TABLE III: ABX score across speakers within domain, on the accented English test set for different domain adaptation methods (FT: fine tuning with resampling, DN: domain normalization). We boldface the best results per columns.

Few-Shot Domain Adaptation

Few-shot domain adaptation transfers models to a new data domain using only a small number of labeled target examples.

See all 53 papers

Contrastive Regularization for Accent-Robust ASR
May 5, 2026Van-Phat Thai, Aradhya Dhruv, Duc-Thinh Pham +1Automatic Speech Recognition Supervised Contrastive Learning
Air Traffic Management Research Institute, Nanyang Technological University, Singapore · Center of AI Research, VinUniversity, Vietnam
AccentCL: Robust Accent Classification with Incremental Expansion
Oct 5, 2026Mu-Ruei Tseng, Waris Quamer, Ghady Nasrallah +1Class-Incremental Learning Audio Classification
Department of Computer Science and Engineering Texas A&M University College Station, TX 77840, USA
Prosodic ABX: A Language-Agnostic Method for Measuring Prosodic Contrast in Speech Representations
Apr 2, 2026Haitong Sun, Stephen McIntosh, Kwanghee Choi +3Speech Prosody Self-Supervised Speech Representation Learning
The University of Tokyo, Japan · University of Texas at Austin, USA

May 5, 2026J/K move · Enter open · S save

Contrastive Regularization for Accent-Robust ASR

Van-Phat Thai, Aradhya Dhruv, Duc-Thinh Pham, Sameer Alam

Air Traffic Management Research Institute, Nanyang Technological University, Singapore · Center of AI Research, VinUniversity, Vietnam

Automatic Speech Recognition Supervised Contrastive Learning Robust ASR

Open paper AskPDF arXiv

ASR systems based on self-supervised acoustic pretraining and CTC fine-tuning achieve strong performance on native speech but remain sensitive to accent variability. We investigate supervised contrastive learning (SupCon) as a lightweight, accent-invariant auxiliary objective for CTC fine-tuning. An utterance-level contrastive loss regularizes encoder representations without architectural modification or explicit accent supervision. Experiments on the L2-ARCTIC benchmark show consistent WER reductions across multiple pretrained encoders, with up to 25 -- 29% relative reduction under unseen-accent evaluation. Analysis using within-transcript cosine dispersion indicates that SupCon promotes more compact and stable representation geometry under accent variability. Overall, SupCon provides an effective and model-agnostic regularization strategy for improving accent robustness.

Abstract

Few-Shot Domain Adaptation

Explore similar work

Contrastive Regularization for Accent-Robust ASR

AccentCL: Robust Accent Classification with Incremental Expansion

Prosodic ABX: A Language-Agnostic Method for Measuring Prosodic Contrast in Speech Representations