A Sociolinguistic Analysis of Automatic Speech Recognition Bias in Newcastle English
Authors: Dana Serditova, Kevin Tang
Organizations: University of Regensburg, Faculty of Languages, Literature, and Cultures, Department of English and American Studies, Regensburg, 93040, Bavaria, Germany · Heinrich Heine University Düsseldorf, Faculty of Arts and Humanities, Institute of English and American Studies, Department English Language and Linguistics, Düsseldorf, 40225, North Rhine-Westphalia, Germany · University of Florida, Department of Linguistics, Gainesville, 32611-5454, Florida, U.S.A.
Automatic Speech Recognition (ASR) systems are widely used in everyday communication, education, healthcare, and industry, yet their performance remains uneven across speakers, particularly when dialectal variation diverges from the mainstream accents represented in training data. This study investigates ASR bias through a sociolinguistic analysis of Newcastle English, a regional variety of North-East England that has been shown to challenge current speech recognition technologies. Using spontaneous speech from the Diachronic Electronic Corpus of Tyneside English (DECTE), we evaluate the output of a state-of-the-art commercial ASR system and conduct a fine-grained analysis of more than 3,000 transcription errors. Errors are classified by linguistic domain and examined in relation to social variables including gender, age, and socioeconomic status. In addition, an acoustic case study of selected vowel features demonstrates how gradient phonetic variation contributes directly to misrecognition. The results show that phonological variation accounts for the majority of errors, with recurrent failures linked to dialect-specific features like vowel quality and glottalisation, as well as local vocabulary and non-standard grammatical forms. Error rates also vary across social groups, with higher error frequencies observed for men and for speakers at the extremes of the age spectrum. These findings indicate that ASR errors are not random but socially patterned and can be explained from a sociolinguistic perspective. Thus, the study demonstrates the importance of incorporating sociolinguistic expertise into the evaluation and development of speech technologies and argues that more equitable ASR systems require explicit attention to dialectal variation and community-based speech data.
Figures & tables
Feature Name
Newcastle English
Standard Southern British English
Retention of [h]
Typically retained initially, but dropped among the WC ( Hughes et al., 2013 ) .
Typically retained in all phonological environments.
Lack of dark []
Typically a clear [l] ( Hughes et al., 2013 ) .
Syllable-final dark [].
HappY-tensing
Realised as [i] or [i:] ( Hughes et al., 2013 ) .
More centralised, closer to [I].
foot/strut Split Absence
foot and strut vowels both realised as [U] ( Hughes et al., 2013 ) .
Distinct vowels: foot [U] or [], strut [2].
Vowel Quality in bath and trap
No bath-trap split, no bath retraction, trap pronounced with [a] ( Beal, 2004 ; Hickey, 2015 ) .
bath words have retracted [A:] (e.g., “path” [pA:]). trap pronounced with [æ].
Vowel Quality in fleece and goose
Closer to cardinal vowels in closed syllables ( Mearns, 2015 ) .
Table 1: Summary of salient Newcastle English features: phonetic and phonological features, with Standard Southern British English counterparts for comparison. WC = working class; MC = middle class.
Feature Name
Newcastle English
Standard Southern British English
Unmarked Plurals
“six month”, “three pound” ( Beal, 2004 ) .
Marked with regular plural -s.
Multiple Negation
Present in Newcastle English ( Beal, 1993 ) .
Considered non-standard.
Conflation of Past and Past Participle
“they’ve broke it” ( Beal, 1993 ) .
Distinction maintained (“they’ve broken it”).
Pronouns
Regional forms “yous” (2nd pl.), “wor” (meaning “our"), “us” in object position instead of “me” ( Beal, 1993 ; Beal, 2004 ; Pearce, 2012 ) .
Standard forms “you”, “our” [aU@] or [A:], “me”.
Local Vocabulary
bairn , clamming , etc. ( Hughes et al., 2013 ) .
These words are not used.
Table 2: Summary of salient Newcastle English features: morphosyntactic and lexical features, with Standard Southern British English counterparts for comparison.
Error Type
Count
Percentage
Phonological
1,826
60.8%
Lexical
590
19.6%
Standardisation
294
9.8%
Morphosyntax
235
7.8%
Spelling
21
0.7%
Table 3: Frequency distribution of error types.
Figure 1: Proportion of error types by gender and age.
Figure 2: Proportion of error types by socioeconomic status.
Predictor
β^
SE
z
p
Sig.
(Intercept)
1.955
0.078
25.017
<.001
Lexical
-1.324
0.170
-7.806
<.001
Standardisation
-0.040
0.146
-0.273
.785
Phonological
2.430
0.131
18.513
<.001
Gender: women
0.201
0.153
1.316
.188
Lexical × Women
0.017
0.338
0.050
.960
Table 4: Fixed effects from the generalised linear mixed model (Negative Binomial) predicting error count. The model includes a random intercept for speaker.
Figure 3: Predicted error counts by gender and error type.
Error Subcategory
Count
Percentage
Vowel quality
721
39.5%
Glottalisation/glottal stop
414
22.7%
Monophthongisation
403
22.1%
G-dropping
171
9.4%
Clear /l/
45
2.5%
Phonetic reduction
23
1.3%
Table 5: Frequency distribution of phonological errors by subcategory. Percentages indicate the share of total phonological errors.
Figure 4: Distribution of vowel-related phonological errors by feature (excluding monophthongisation).
Error Subcategory
Count
Percentage
verb paradigm
111
47.4%
pronoun
102
43.6%
tenses
17
7.3%
plurals
4
1.7%
Table 6: Frequency distribution of morphosyntactic errors by subcategory. Percentages indicate the share of total morphosyntactic errors.
Figure 5: Distribution of the most common morphosyntactic errors by feature.
Figure 6: Most frequent lexical items causing ASR errors, log-scaled.
Figure 7: Most frequent lexical errors, distributed by gender. Plotted using scaled F score metric using scattertext ( Kessler, 2017 ) . An interactive version of this figure can be found in the OSF repository.
Figure 8: Relationship between diphthong-specific ASR error counts and speakers’ median absolute DCT1 values for F1 and F2 in face and goat . Points represent individual speakers, coloured by overall ASR error group; black lines show descriptive linear trends with 95% confidence intervals.
Figure 9: Formant trajectory of a speaker with the least number of errors.
Figure 10: Formant trajectories of speakers with the highest number of errors.
Figure 11: Formant trajectories of speakers with the highest number of errors.
Figure 12: Formant trajectories of speakers in the Mid-range error group.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 13: Formant trajectory of a speaker with a high number of errors.
Figure 14: Formant trajectories of speakers with a low number of errors.
Figure 15: Formant trajectories of speakers in the Mid-range error group.
Variable
Category
N
%
Age
16–20
19
23.5
21–40
18
22.2
41–60
22
27.2
61–90
22
27.2
Gender
Men
42
51.9
Women
39
48.1
Appendix
Table 7: Distribution of speakers included in the analysis by age, gender, and socioeconomic status.
Demographic bias in automatic speech recognition (ASR) concerns unequal transcription accuracy and user burdens across speaker groups and language varieties.