Connecting the Dots: Enduring Performance Gaps in ASR Systems Between Standard and Minority English Varieties

Alexander Metzger; Aruna Srivastava; Ruslan Mukhamedvaleev; Eunjung Yeo; Syed Ishtiaque Ahmed; Nina Markl; Sachin Kumar; Farhan Samir

cs.CLOct 13, 2026

Connecting the Dots: Enduring Performance Gaps in ASR Systems Between Standard and Minority English Varieties

Authors: Alexander Metzger, Aruna Srivastava, Ruslan Mukhamedvaleev, Eunjung Yeo, Syed Ishtiaque Ahmed, Nina Markl, Sachin Kumar, Farhan Samir

Organizations: Koel Labs LLC · University of Texas at Austin · University of Toronto · University of Essex ·

Abstract

Automatic Speech Recognition (ASR) system evaluations have consistently revealed performance disparities along lines of race, gender, socioeconomic status, and language variety. While these disparities are well-documented, they are often (implicitly or explicitly) treated as issues that subsequent, larger, and more universal models will correct. We challenge this assumption of steady progress in reducing accent-related performance disparities, conducting a longitudinal evaluation of 16 prominent systems released between 2021-2025 across 8 datasets representing diverse English accent varieties. We find that the performance disparity between Mainstream US English (MUSE) and minority accents is stubborn, not reliably declining as a function of time. Specifically, while systems are reliably performant on MUSE, minority accents are subject to risks of large performance degradations between generations of systems, sometimes worsening by 65% absolute Word Error Rate. We argue that standard ASR benchmarks enact an implicit language policy privileging Mainstream US English varieties. As such, these performance disparities reflect systemic failures of language technology development infrastructure that disproportionately affect underrepresented speaker populations.

Figures & tables

Panel of Figure 1 — Fig. 1 : Longitudinal system performances on 8 datasets. Top panels: Word Error Rates (WERs) for each accent variety. Bottom panels: WER difference between Mainstream US English (MUSE) or Standard Southern British English (SSBE) and the minority accent variety with the highest average WER in the dataset. Bottom panels also show the average performance gap over the time period from the earliest evaluated model (2021-10) to the latest one (2025-11), as well as the correlation between the temporal ordering of the models and the size of the performance gap between the two accent varieties in each evaluation.

Dataset	Background	$\mathbf{MD}$	Rank
L2-Arctic	Vietnamese En.	$\mathbf{11.4\%}$ *	5/7
	MUSE	$3.9\%$	1/7
L2-Arctic-Spontaneous	Korean En.	$\mathbf{9.8\%}$ *	3/7
	MUSE	$7.5\%$	2/7
Speech Accent Archive	Korean En.	$\mathbf{11.7\%}$	4/12
	MUSE	$11.6\%$	3/12

TABLE II: Mean Degradation (MD) and MD rank.

Dataset	Duration (s, IQR)	Samples	Accents	Speaker Demographics
Speech Accent Archive	5.4–7.2	1,238	nld, spa, ita, eng, fra, kor, tur, ara, por, deu, zho, rus	Mean age 33.31 (SD 13.57); started English at 10.53 (SD 8.49); 55.7% male.
L2-Arctic	2.8–4.4	3,599	vie, cmn, hin, spa, ara, kor	Iowa State University students; age 22–43, mean 29 (SD 6.9).
L2-Arctic Spontaneous	6.2–7.3	22	vie, cmn, hin, spa, ara, kor	Same as L2-Arctic.
OpenSLR83	4.6–7.7	17,877	Irish, Midlands, Northern, Scottish, Southern, Welsh	Volunteers 21+; recruited by Google London and Cardiff University.
ALLSTAR	1.5–2.4	42,398	pbr, cct, kor, guj, vie, ctw, run, hin, fra, rus, gis, csp, gre, ind, spa, heb, tur, ger, far, shs, cmn, jpn	Northwestern University students; age 18–41, mean 23.52.

TABLE V: Summary of evaluation datasets. Duration is given in seconds per sample as the interquartile range (IQR). All datasets contain scripted speech except L2-Arctic Spontaneous.

Dataset (Accent)	$\mathbf{\bar{d}_{(m_{i}\to m_{j})}}$	Prediction ( $m_{i}$ )	Prediction ( $m_{j}$ )	Transcript
L2-Arctic (Vietnamese)	12.5%	Whisper Large v2 (2022-12): Without a doubt, some of them have dinner engagements.	MMS 1B All (2023-05): wit dou a doubt some of them hapdiner engagements	without a doubt some of them have dinner engagements
L2-Arctic Spontaneous (Korean)	15.4%	Canary (2025-07): Okay, um I can see skyscrapers tall buildings and I think	Omnilingual 7B (2025-11): i can see skyscripers tall buildings and i think	ok uh i can see skyscrapers tall buildings and i think
OpenSLR-83 (Irish)	5.7%	Google Chirp 3 (2025-10): Your app facilitates or promotes content that includes gratuitous violence or dangerous activities.	Omnilingual 7B (2025-11): your off facilitates or promotes content that includes gratuitous flylence or dangerous activities	Your app facilitates or promotes content that includes gratuitous violence or dangerous activities
SAA (Korean)	18.3%	Whisper Large v2 (2022-12): Please call Stella. Ask her to bring these things with her from the store.	MMS 1B All (2023-05): please costella escr to bring these things with her from the store	please call stella Ask her to bring these things with her from the store.
ALLSTAR-HT1 (Nkore)	6.7%	Whisper Large v1 (2022-09): the salt shaker is empty	Canary (2025-07): the sword sheikah is empty	the salt shaker is empty
ALLSTAR-HT2 (Nkore)	3.1%	Whisper Large v1 (2022-09): he is washing his face with soap	Omnilingual (2025-11): he is washing his face with sore	he is washing his face with soap

TABLE VI: Illustration of degradation from earlier model

m_{i}

to later model

m_{j}

across all of our evaluation datasets.

Demographic Bias in ASR

Demographic bias in automatic speech recognition (ASR) concerns unequal transcription accuracy and user burdens across speaker groups and language varieties.

See all 30 papers

Linguistic Distance Segregates Latent Representations in Automatic Speech Recognition Systems
Aug 31, 2026Ting-Hui Cheng, Line Katrine Harder Clemmensen, Sneha DasAutomatic Speech Recognition
Department of Applied Mathematics and Computer Science Technical University of Denmark
ACES: Accent Subspaces for Coupling, Explanations, and Stress-Testing in Automatic Speech Recognition
Feb 28, 2026Swapnil ParekhDemographic Bias in ASR Automatic Speech Recognition
Intuit
Evaluating Bias in Phoneme-Based Automatic Speech Recognition Systems: An Analysis of IPA Transcription Models
Jun 10, 2026Maneesha Rani Saha, Catherine Bao, Neal PatwariASR Evaluation Demographic Bias in ASR
University of Utah

Aug 31, 2026J/K move · Enter open · S save

Linguistic Distance Segregates Latent Representations in Automatic Speech Recognition Systems

Ting-Hui Cheng, Line Katrine Harder Clemmensen, Sneha Das

Department of Applied Mathematics and Computer Science Technical University of Denmark

Automatic Speech Recognition

Open paper AskPDF arXiv

While automatic speech recognition (ASR) models have achieved remarkable improvements in recent years, performance disparities persist across different speaker populations. One such disparity is for speakers whose first languages (L1) are from families distant from English. This paper investigates the relationship between first language background and English ASR performance. Through empirical analysis, we observe that the correlation between speakers' L1 distance and ASR error rates yields a systematic effect on English Speech, with its strength varying across datasets and models. This association is statistically significant in a follow-up analysis accounting for dataset-level variation in Tweedie mixed-effects models (

p<0.001

across evaluated models). In addition, analysis of the latent space reveals a L1-based spatial segregation across deeper acoustic layers in the majority of evaluated architectures

Abstract

Demographic Bias in ASR

Explore similar work

Linguistic Distance Segregates Latent Representations in Automatic Speech Recognition Systems

ACES: Accent Subspaces for Coupling, Explanations, and Stress-Testing in Automatic Speech Recognition

Evaluating Bias in Phoneme-Based Automatic Speech Recognition Systems: An Analysis of IPA Transcription Models