A Sociolinguistic Analysis of Automatic Speech Recognition Bias in Newcastle English
Authors: Dana Serditova, Kevin Tang
Organizations: University of Regensburg, Faculty of Languages, Literature, and Cultures, Department of English and American Studies, Regensburg, 93040, Bavaria, Germany · Heinrich Heine University Düsseldorf, Faculty of Arts and Humanities, Institute of English and American Studies, Department English Language and Linguistics, Düsseldorf, 40225, North Rhine-Westphalia, Germany · University of Florida, Department of Linguistics, Gainesville, 32611-5454, Florida, U.S.A.
Automatic Speech Recognition (ASR) systems are widely used in everyday communication, education, healthcare, and industry, yet their performance remains uneven across speakers, particularly when dialectal variation diverges from the mainstream accents represented in training data. This study investigates ASR bias through a sociolinguistic analysis of Newcastle English, a regional variety of North-East England that has been shown to challenge current speech recognition technologies. Using spontaneous speech from the Diachronic Electronic Corpus of Tyneside English (DECTE), we evaluate the output of a state-of-the-art commercial ASR system and conduct a fine-grained analysis of more than 3,000 transcription errors. Errors are classified by linguistic domain and examined in relation to social variables including gender, age, and socioeconomic status. In addition, an acoustic case study of selected vowel features demonstrates how gradient phonetic variation contributes directly to misrecognition. The results show that phonological variation accounts for the majority of errors, with recurrent failures linked to dialect-specific features like vowel quality and glottalisation, as well as local vocabulary and non-standard grammatical forms. Error rates also vary across social groups, with higher error frequencies observed for men and for speakers at the extremes of the age spectrum. These findings indicate that ASR errors are not random but socially patterned and can be explained from a sociolinguistic perspective. Thus, the study demonstrates the importance of incorporating sociolinguistic expertise into the evaluation and development of speech technologies and argues that more equitable ASR systems require explicit attention to dialectal variation and community-based speech data.
Figures & tables
Feature Name
Newcastle English
Standard Southern British English
Retention of [h]
Typically retained initially, but dropped among the WC ( Hughes et al., 2013 ) .
Typically retained in all phonological environments.
Lack of dark []
Typically a clear [l] ( Hughes et al., 2013 ) .
Syllable-final dark [].
HappY-tensing
Realised as [i] or [i:] ( Hughes et al., 2013 ) .
More centralised, closer to [I].
foot/strut Split Absence
foot and strut vowels both realised as [U] ( Hughes et al., 2013 ) .
Distinct vowels: foot [U] or [], strut [2].
Vowel Quality in bath and trap
No bath-trap split, no bath retraction, trap pronounced with [a] ( Beal, 2004 ; Hickey, 2015 ) .
bath words have retracted [A:] (e.g., “path” [pA:]). trap pronounced with [æ].
Vowel Quality in fleece and goose
Closer to cardinal vowels in closed syllables ( Mearns, 2015 ) .
Table 1: Summary of salient Newcastle English features: phonetic and phonological features, with Standard Southern British English counterparts for comparison. WC = working class; MC = middle class.
Feature Name
Newcastle English
Standard Southern British English
Unmarked Plurals
“six month”, “three pound” ( Beal, 2004 ) .
Marked with regular plural -s.
Multiple Negation
Present in Newcastle English ( Beal, 1993 ) .
Considered non-standard.
Conflation of Past and Past Participle
“they’ve broke it” ( Beal, 1993 ) .
Distinction maintained (“they’ve broken it”).
Pronouns
Regional forms “yous” (2nd pl.), “wor” (meaning “our"), “us” in object position instead of “me” ( Beal, 1993 ; Beal, 2004 ; Pearce, 2012 ) .
Standard forms “you”, “our” [aU@] or [A:], “me”.
Local Vocabulary
bairn , clamming , etc. ( Hughes et al., 2013 ) .
These words are not used.
Table 2: Summary of salient Newcastle English features: morphosyntactic and lexical features, with Standard Southern British English counterparts for comparison.
Error Type
Count
Percentage
Phonological
1,826
60.8%
Lexical
590
19.6%
Standardisation
294
9.8%
Morphosyntax
235
7.8%
Spelling
21
0.7%
Table 3: Frequency distribution of error types.
Figure 1: Proportion of error types by gender and age.
Figure 2: Proportion of error types by socioeconomic status.
Predictor
β^
SE
z
p
Sig.
(Intercept)
1.955
0.078
25.017
<.001
Lexical
-1.324
0.170
-7.806
<.001
Standardisation
-0.040
0.146
-0.273
.785
Phonological
2.430
0.131
18.513
<.001
Gender: women
0.201
0.153
1.316
.188
Lexical × Women
0.017
0.338
0.050
.960
Table 4: Fixed effects from the generalised linear mixed model (Negative Binomial) predicting error count. The model includes a random intercept for speaker.
Figure 3: Predicted error counts by gender and error type.
Error Subcategory
Count
Percentage
Vowel quality
721
39.5%
Glottalisation/glottal stop
414
22.7%
Monophthongisation
403
22.1%
G-dropping
171
9.4%
Clear /l/
45
2.5%
Phonetic reduction
23
1.3%
Table 5: Frequency distribution of phonological errors by subcategory. Percentages indicate the share of total phonological errors.
Figure 4: Distribution of vowel-related phonological errors by feature (excluding monophthongisation).
Error Subcategory
Count
Percentage
verb paradigm
111
47.4%
pronoun
102
43.6%
tenses
17
7.3%
plurals
4
1.7%
Table 6: Frequency distribution of morphosyntactic errors by subcategory. Percentages indicate the share of total morphosyntactic errors.
Figure 5: Distribution of the most common morphosyntactic errors by feature.
Figure 6: Most frequent lexical items causing ASR errors, log-scaled.
Figure 7: Most frequent lexical errors, distributed by gender. Plotted using scaled F score metric using scattertext ( Kessler, 2017 ) . An interactive version of this figure can be found in the OSF repository.
Figure 8: Relationship between diphthong-specific ASR error counts and speakers’ median absolute DCT1 values for F1 and F2 in face and goat . Points represent individual speakers, coloured by overall ASR error group; black lines show descriptive linear trends with 95% confidence intervals.
Figure 9: Formant trajectory of a speaker with the least number of errors.
Figure 10: Formant trajectories of speakers with the highest number of errors.
Figure 11: Formant trajectories of speakers with the highest number of errors.
Figure 12: Formant trajectories of speakers in the Mid-range error group.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 13: Formant trajectory of a speaker with a high number of errors.
Figure 14: Formant trajectories of speakers with a low number of errors.
Figure 15: Formant trajectories of speakers in the Mid-range error group.
Variable
Category
N
%
Age
16–20
19
23.5
21–40
18
22.2
41–60
22
27.2
61–90
22
27.2
Gender
Men
42
51.9
Women
39
48.1
Appendix
Table 7: Distribution of speakers included in the analysis by age, gender, and socioeconomic status.
Demographic bias in automatic speech recognition (ASR) concerns unequal transcription accuracy and user burdens across speaker groups and language varieties.
Automatic Speech Recognition (ASR) system evaluations have consistently revealed performance disparities along lines of race, gender, socioeconomic status, and language variety. While these disparities are well-documented, they are often (implicitly or explicitly) treated as issues that subsequent, larger, and more universal models will correct. We challenge this assumption of steady progress in reducing accent-related performance disparities, conducting a longitudinal evaluation of 16 prominent systems released between 2021-2025 across 8 datasets representing diverse English accent varieties. We find that the performance disparity between Mainstream US English (MUSE) and minority accents is stubborn, not reliably declining as a function of time. Specifically, while systems are reliably performant on MUSE, minority accents are subject to risks of large performance degradations between generations of systems, sometimes worsening by 65% absolute Word Error Rate. We argue that standard ASR benchmarks enact an implicit language policy privileging Mainstream US English varieties. As such, these performance disparities reflect systemic failures of language technology development infrastructure that disproportionately affect underrepresented speaker populations.
Alexander Metzger, Aruna Srivastava, Ruslan Mukhamedvaleev +5
Koel Labs LLC · University of Texas at Austin · University of Toronto +2
While automatic speech recognition (ASR) models have achieved remarkable improvements in recent years, performance disparities persist across different speaker populations. One such disparity is for speakers whose first languages (L1) are from families distant from English. This paper investigates the relationship between first language background and English ASR performance. Through empirical analysis, we observe that the correlation between speakers' L1 distance and ASR error rates yields a systematic effect on English Speech, with its strength varying across datasets and models. This association is statistically significant in a follow-up analysis accounting for dataset-level variation in Tweedie mixed-effects models (p<0.001 across evaluated models). In addition, analysis of the latent space reveals a L1-based spatial segregation across deeper acoustic layers in the majority of evaluated architectures
Ting-Hui Cheng, Line Katrine Harder Clemmensen, Sneha Das
Department of Applied Mathematics and Computer Science Technical University of Denmark
While modern Automatic Speech Recognition (ASR) systems achieve high accuracy on benchmark corpora, their performance often degrades when there is real-world variability. This work focuses on variability arising due to accented, spontaneous, and domain-specific speech. In particular, we introduce PAper REading DAtaset (PAREDA), a first-of-its-kind multi-accent speech dataset consisting of discussions on academic Natural Language Processing (NLP) papers between speakers with Australian, Indian-English, and Chinese English accents. Each session elicits a spontaneous monologue (a summary of a paper's abstract) and a non-monologue (a question-and-answer session between participants), resulting in a corpus rich with technical jargon and conversational phenomena. We evaluate the performance of SOTA ASR models on PAREDA, analysing the impact of accent mixing and increased speech rate. Our results show that, in the zero-shot setting, models perform worse, confirming the dataset's challenging nature. However, fine-tuning on PAREDA significantly reduces the Word Error Rate (WER), demonstrating that our dataset captures linguistic characteristics often missing from existing corpora. PAREDA serves as a valuable new resource for building and evaluating more robust and inclusive ASR systems for specialised, real-world applications.