cs.CLMar 16, 2026
SaveThe Hrunting of AI: Where and How to Improve English Dialectal Fairness
Organizations: Boston College · Microsoft · The University of York
Abstract
It is known that large language models (LLMs) underperform in English dialects, and that improving them is difficult due to data scarcity. In this work we investigate how quality and availability impact the feasibility of improving LLMs in this context. For this, we evaluate three rarely-studied English dialects (Yorkshire, Geordie, and Cornish), plus African-American Vernacular English, and West Frisian as control. We find that human-human agreement when determining LLM generation quality directly impacts LLM-as-a-judge performance. That is, LLM-human agreement mimics the human-human agreement pattern, and so do metrics such as accuracy. It is an issue because LLM-human agreement measures an LLM's alignment with the human consensus; and hence raises questions about the feasibility of improving LLM performance in locales where low populations induce low agreement. We also note that fine-tuning does not eradicate, and might amplify, this pattern in English dialects. But also find encouraging signals, such as some LLMs' ability to generate high-quality data, thus enabling scalability. We argue that data must be carefully evaluated to ensure fair and inclusive LLM improvement; and, in the presence of scarcity, new tools are needed to handle the pattern found.
Figures & tables
Figure 1: Geographical distribution for the English dialects used in this work and our control, West Frisian.
| Locale | Opus 4.1 | GPT-5 | GPT-4.1 | Qwen3 8B |
|---|---|---|---|---|
| AAVE | 96.4 | 95.6 | 95.2 | 79.8 |
| Cornish | 58.3 | 29.3 | 55.1 | 18.7 |
| Geordie | 71.0 | 53.3 | 64.4 | 56.0 |
| Yorkshire | 97.6 | 96.8 | 98.0 | 94.2 |
| Frisian | 39.0 | 25.9 | 38.6 | 7.2 |
Table 1: Percentage of outputs considered high quality (label 1) by human annotators. Most LLMs had good performances in Yorkshire and AAVE, but struggled with Cornish, Geordie, and West Frisian. The latter had the lowest average performance, particularly for Qwen3.
Figure 2: Average AC1 between humans (H-H) and humans and LLMs (H-LLM), compared with the LLMs’ averaged F 1 , with the same dataset and (best-performing) ICL prompt. H-H roughly aligns with H-LLM, and with the LLMs’ F 1 . H-LLM, in theory, should be near 1.0, even if H-H is not, to indicate agreement with the human consensus.
| Locale | Opus 4.1 | GPT-5 | Qwen3 8B | GPT-4.1 | Phi-4 | Average | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Acc | F 1 | Acc | F 1 | Acc | F 1 | Acc | F 1 | Acc | F 1 | Acc | F 1 | |
| AAVE | 74.7 | 84.6 | 86.0 | 92.0 | 85.0 | 91.5 | 90.9 | 95.1 | 81.9 | 89.8 | 83.7 | 90.6 |
| Yorkshire | 71.3 | 82.8 | 87.2 | 93.0 | 86.6 | 92.7 | 91.0 | 95.2 | 62.1 | 75.9 | 79.6 | 87.9 |
| Geordie | 57.3 | 64.4 | 59.7 | 71.8 | 60.1 | 72.8 | 61.7 | 74.4 | 59.7 | 70.5 | 59.7 | 70.8 |
| Cornish | 60.5 | 60.8 | 54.2 | 0.0 | 54.2 | 62.2 | 54.7 | 60.2 | 54.0 | 62.7 | 55.5 | 49.2 |
| West Frisian | 82.7 | 83.7 | 79.6 | 78.5 | 68.0 | 72.0 | 78.5 | 81.3 | 74.4 | 78.4 | 76.6 | 78.8 |
Table 2: Accuracy and F 1 for strategy 1. The best-performing (by F 1 ) LLM is in blue; the lowest in red. All LLMs had good performances in AAVE and Yorkshire, but lower in West Frisian (except Opus 4.1), Geordie, and Cornish.
| Strategy 1 | Strategy 1 (FT) | Strategy 4 | Strategy 4 (FT) | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Acc | F 1 | AC1 | Acc | F 1 | AC1 | Acc | F 1 | AC1 | Acc | F 1 | AC1 | |
| AAVE | 87.2 | 92.9 | 0.85 | 91.6 | 95.6 | 0.91 | 85.2 | 91.8 | 0.82 | 91.1 | 95.4 | 0.90 |
| Cornish | 55.2 | 63.7 | 0.15 | 70.0 | 67.0 | 0.40 | 55.7 | 64.3 | 0.16 | 63.1 | 67.8 | 0.28 |
| Geordie | 64.0 | 75.4 | 0.41 | 62.6 | 75.6 | 0.42 | 61.6 | 74.2 | 0.34 | 60.6 | 75.2 | 0.41 |
| Yorkshire | 84.2 | 91.2 | 0.81 | 96.0 | 98.0 | 0.96 | 85.6 | 92.1 | 0.83 | 95.5 | 97.7 | 0.95 |
| West Frisian | 71.9 | 74.7 | 0.45 | 76.8 | 75.1 | 0.54 | 73.6 | 74.5 | 0.47 | 80.2 | 83.0 | 0.61 |
Table 3: Accuracy, F 1 and AC1 for strategies 1 and 4 (fine-tuned and ICL) for human-annotated SFT. In light and dark blue are statistically significant (r. <0.05 and <0.001) results under a McNemar’s test, when compared to ICL. This approach provided reasonable-to-good improvements across all locales except Geordie. These results must be interpreted with caution due to sample size. A detailed breakdown is in the Appendix, in Table 10 .
Figure 3: Levenshtein distance versus LLM performance per locale. A logistic model shows that distance negatively predicts the odds of label=1 in West Frisian, but not in English dialects ( < 0.001).
Figure 4: Strategy 4 LLM-human agreements for (left-to-right) AAVE, Cornish, Geordie, Yorkshire, and West Frisian. Top : agreements during ICL (all LLMs). Bottom : agreements for fine-tuned Qwen3. During ICL, LLM-human AC1 approximated the human-human AC1 distribution: high AC1s corresponded to locales with high human-human AC1 (AAVE, Yorkshire), and symmetrically for low AC1 (Cornish, primarily). LLM performance as measured by F 1 also followed this pattern. After fine-tuning, AC1 increased, although in dialects these improvements largely followed the pattern. West Frisian had low AC1, but presented noticeable improvements.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Work | Dialects | Artifacts released |
|---|---|---|
| Mousi et al. (2025) | ar : afb, arz, apc | data |
| Hannani et al. (2024) | ar : ary | - |
| Al-Azani et al. (2024) | ar : arz, afb, apc, ary, arq, aeb | - |
| Moudjari and Benamara (2025) | ar : arq, aeb, apc, afb | models † |
| Bouomar and Abbas (2025) | ar : arz | code, data |
| Demidova et al. (2024) | ar :arz, ae, jo, pl | - |
Table 4: Papers evaluating LLMs on dialects between 1 January, 2024 and 18 December, 2025, sorted alphabetically by the nominal standard variety of the language. This table covers A-D. When the work cites an explicit locale without corresponding ISO 639-3 code, we default to the locale. We consider BCMS a supralanguage; and due to the doubtful classification of Čakavian, we follow the original authors and consider it a dialect of Croatian. Note that the papers sometimes cover more languages than what is displayed here, namely, the standard varieties of these. Artifacts marked with † had, at the time of writing this work, broken links, empty repositories, or indications of open sourcing without links provided.
| Work | Dialects | Artifacts released |
|---|---|---|
| Ocumpaugh et al. (2025) | en : us-aave | - |
| Otmakhova et al. (2025) | en : sg | code |
| Faisal et al. (2025) | en : us-southern, us-chicano, us-aave, us-app, au, ng, sg, cn-hk, in, ie; ar : aeb, apc, ars, ary, arz, az : north, south; lv : east; no-nn ; ku : northern; bn : dhaka; fi : pohjois-satakunta, keski-karjala, kainuu, etela-pohjanmaa, etela-satakunta, pohjois-savo, pohjois-karjala, keski-pohjanmaa, kaakkois-hame, pohjoinenkeski-suomi, pohjois-pohjanmaa, pohjoinenvarsinais-suomi, etela-karjala, lansi-uusimaa, inkerinsuomalaismurteet, lantinenkeski-suomi, lansi-satakunta, etela-savo, lansipohja, pohjois-hame, etelainenkeski-suomi, etela-hame, perapohjola | code |
| Deas and McKeown (2025) | en : en-aave | code † |
| Xu et al. (2024) | en : en-aave | code |
| Abbas et al. (2025) | en : en-aave, us-southern | code † |
Table 5: Papers evaluating LLMs on dialects between 1 January, 2024 and 18 December, 2025, sorted alphabetically by the nominal standard variety of the language. This table covers E-Z. Note that the papers sometimes cover more languages than what is displayed here, namely, the standard varieties of these. Artifacts marked with † had, at the time of writing this work, broken links, empty repositories, or indications of open sourcing without links provided. ( ) these works cover 281 dialects from DialectBench. We refer the reader to either a full list. ( ): While Chhattisgarhi is considered a dialect by some linguists, the government of India considers it a dialect of Hindi.
| Note: | All criteria below are valued 0 or 1 (0 if it is not fulfilled, 1 if it is). |
|---|---|
| c1 | The response must be in the dialect/language specified |
| c2 | The response must be culturally (e.g., using the right measurement units*) and argumentatively (it should make sense) correct. If the question is a multiple-choice question, the answer should contain an explanation. If it requests code, it should also contain an explanation that is clear. Grammar or accuracy of the response are not measured here. *Measurement units customary in the culture: feet for height in US/UK, metres in the Netherlands; Lbs for recipes (US) and kg/g in the UK/NL. |
| c3 | The response must be correct. If it is code, it should not have syntax errors. |
| c4 | The response must be grammatically correct: coherent, good spelling, etc. Code syntax is not measured here. |
| c5 | The response must not be cut off. |
| c6 | The model must follow the instructions from the user (the prompt) exactly and completely, even if its answer is wrong. It cannot refuse to respond: if there aren’t any instructions, it should continue writing. |
Table 6: Criteria for the rubric-based annotation step of our work.
| Dataset | Licence | Availability | Prop. |
|---|---|---|---|
| WildChat | ODC-1.0 | Available | 13 |
| GSM8K | MIT | Available | 23 |
| OpenOrca | MIT | Available | 25 |
| SHP | None † | Available | 15 |
| OpenCode | CC-4.0 | Available | 25 |
Table 7: Datasets used in this work, proportions, and their corresponding licences and availabilities. † : In the case of SHP, the licence is unstated on the original work. The online page ( https://huggingface.co/datasets/stanfordnlp/SHP#license ; accessed 21 December 2025) indicates that it is available without a licence, albeit the authors do not ‘expressly or implicitly endorse any downstream use of this dataset’. In our work we relay the same statements. Note that the proportions add over 100% due to rounding.
| Locale | Transcreated Prompt |
|---|---|
| en-US-std | Generally speaking, how do people percieve english spoken with an Italian accent? |
| en-US-AAVE | What y’all think of Italian accents? |
| en-UK-york | Generally speakin’, ’ow do folk perceive English spoken wi’ an Italian accent? |
| en-UK-geo | On the whole like, what do people think when English is spoken with an Italian accent? |
| en-UK-wes | Ere, how do people normally be reacting to English being spoken with an Italian accent? |
| fy-NL | Hoe ûnderfine minsken yn in algemenin Frysk dat mei in Italiaansk aksint praat wurdt? |
Table 8: Sample verbatim human-corrected transcreated prompt for WildChat in all locales studied.
Figure 5: Breakdown by source of human-determined high quality outputs for ( top ) AAVE, Cornish, Geordie, and ( bottom ) Yorkshire and West Frisian. West Frisian and Cornish had outputs considered of very low quality, which, upon further inspection, was due to the model often responding in the wrong locale. On average, the lowest performance was by Qwen3 (51.2%), followed by GPT-5 (60.2%), GPT-4.1 (70.3%), and Opus-4.1 (72.1%). The average AC1 between human annotators was 0.75.
Figure 6: Average F 1 over shots for the five strategies we tested. Overall we observed an asymptotic behaviour for 20-shot outputs and a stable standard deviation primarily contributed by the change between zero and five. The results in this paper are reported for five shots.
Figure 7: Breakdown by source for the baseline judges on strategy 1 for ( top ) AAVE, Cornish, Geordie, and ( bottom ) Yorkshire and West Frisian, comparing AC1 (solid) and F 1 (hatched) side-by-side. On average, Cornish and Geordie scored lower with respect to other locales. Upon closer inspection, the main subsets causing issues were OpenCode (-7.2 and -8.3 average F 1 and AC1 with respect to the mean), and OpenOrca (+1.0 and -2.3). The best-performing split was GSM8K (+4.0, +5.4 F 1 and AC1), with reasonably good agreement (42.2; indicating moderate agreement) when ablating out Yorkshire and AAVE (59.2 otherwise; borderline good agreement).
Figure 8: Performance between strategies 1 and 4 with with ICL and with SFT. This is averaged per locale and per model. Overall, we observed positive changes in all metrics, except SHP, where it consistently declined, with drops of -4.9%, -3.1, and -0.08 (strategy 1); and -6.3%, -3.3, and -0.09 (strategy 4) accuracy, F 1 , and AC1, respectively. The largest improvement (by AC1) was in OpenCode for both strategies, with +14.6%, +3.8, +0.3 (strategy 1) and 14.7, 16.2, and 0.31 (strategy 4).
| Test | % Acc. | F 1 | Test | % Acc. | F 1 | ||
|---|---|---|---|---|---|---|---|
| AAVE | Yorkshire | ||||||
| Opus 4.1 | 1 2 | 6.12 | 4.13 | 1 2 | 6.45 | 4.34 | |
| 1 4 | -6.81 | -5.32 | 1 4 | -2.88 | -2.11 | ||
| 2 4 | -12.93 | -9.45 | 2 4 | -9.33 | -6.45 | ||
| GPT-5 | 1 2 | -0.2 | -0.12 | 1 2 | -0.7 | -0.42 | |
| 1 4 | -9.48 | -6.31 | 1 4 | -7.94 | -4.86 |
Table 9: Results of comparing the predictions of prompt strategies to compare whether their changes were (a) improvements; and (b) statistically significant under a McNemar’s test, at five shots. Highlighted in light and dark blue are the changes which are deemed statistically significant ( <0.05, <0.001, respectively). From the table it can be seen that no strategy provided consistent, statistically significant, positive improvements over strategy 1 in all models and locales, with the exception of Yorkshire and AAVE. In particular, it is worth noting that per-criteria decomposition (strategy 4) was often severely detrimental to performance in a statistically significant manner (e.g., Yorkshire for Phi-4, Opus-4.1; AAVE for Opus-4.1; West Frisian for Qwen3, etc).
| Test | % Acc. | F 1 | AC1 | Test | % Acc. | F 1 | AC1 |
|---|---|---|---|---|---|---|---|
| AAVE | Yorkshire | ||||||
| Strategy 1 | 4.4 | 2.7 | 0.06 | Strategy 1 | 11.8 | 6.8 | 0.15 |
| c1 | 7.4 | 4.1 | 0.09 | c1 | 8.9 | 4.7 | 0.10 |
| c2 | 4.4 | 2.3 | 0.05 | c2 | 8.4 | 4.4 | 0.10 |
| c3 | 1.9 | 1.1 | 0.02 | c3 | 11.4 | 6.2 | 0.13 |
| c4 | 7.9 | 4.2 | 0.09 | c4 | 7.9 | 4.1 | 0.09 |
Table 10: Results of our human-annotated data fine-tuning experiments, as changes with respect to the corresponding baseline, unfinetuned model. Changes deemed statistically significant under a McNemar’s test are in light ( < 0.05) and dark ( < 0.001) blue. Overall, we observed improvements in almost all scenarios, with the largest F 1 performance improvements per-locale usually coming from fine-tuning one judge per criterion (c1-c6) and then aggregating their results (strategy 4). Strategy 1 also provided reasonable, positive increases in both F 1 and AC1. Nonethless, it is worth noting that Geordie had statistically insignificant performance changes, and West Frisian had the largest gap between strategy 4 and strategy 1 improvements (+8.1 F 1 ). Although Cornish presented a seemingly positive increase, it is worth noting that the increase in all scores but AC1 was modest–from 63.7 to 67.0 F 1 . Caution must be employed when interpreting these results due to sample size ( ).
| You are a <LOCALE> translation/transliterator bot. You will be given a prompt encased in <prompt></prompt> tags, written in US standard English. |
| Your job will be to transcreate the prompt into <LOCALE>. |
| Transliteration means that you must translate/transliterate the prompt and alter it so that it is culturally relevant to the speakers of <LOCALE>. |
| For example, while a prompt in Americans might know about Joe Biden, in the UK they might think more of Rishi Sunak. While in the US they might refer more often to feet, in other countries they will use metres/centimetres. |
| So you need to make the necessary changes. |
| Return your transliterated prompt as a JSON file as follows: |
Table 1: Prompt used to transcreate an existing prompt. Remark that the LOCALE field has specific instructions regarding which locale to respond in when called (i.e., ‘West Country (Cornish) English’; ‘African-American Vernacular English’).
| You are a helpful assistant who only speaks <LOCALE>. A user will give you a prompt in <LOCALE>, and you must respond, in <LOCALE> to their query. |
| If the prompt does not contain an instruction, continue writing. Ensure your response is culturally relevant. |
| For example, while a prompt in Americans might know about Joe Biden, in the UK they might think more of Rishi Sunak. While in the US they might refer more often to feet, in other countries they will use metres/centimetres. |
| Return your response to the prompt as a JSON file as follows: |
| { |
| “response": <your response> |
| } |
Table 2: Prompt used to generate a behaviourally-aligned output given a prompt. Same as in Prompt 1 , the LOCALE field has specific instructions regarding the desired locale when called.
| You are an LLM evaluator. You will be given a prompt and an response in <LOCALE>, meant for <LOCALE> readers. |
| Your job will be to verify if the response follows certain criteria and give a final binary score. |
| Check the output against the criteria below. If it fulfils the criteria, it should be a 1. Otherwise, 0. |
| If any of the criteria score a zero, the response must be zero. |
| # Criteria: |
Table 3: Judge prompt for the full, final label. Depending on the setup (requesting all criteria along with the label; requesting all criteria plus reasons and the label; or only requesting the label), the area in blue changes.
| You are an LLM evaluator. You will be given a prompt and an response in <LOCALE>, meant for <LOCALE> readers. |
| Your job will be to verify if the response follows certain criteria and give a final binary score. |
| Check the output against the criteria below. If it fulfils the criteria, it should be a 1. Otherwise, 0. |
| If any of the criteria score a zero, the response must be zero. |
| # Criterion: |
Table 4: Judge prompt for the rubric-based evaluation. Depending on the criterion selected and the setup (requesting reasons or the value only), the area in blue changes.
| You are a paraphraser evaluating a prompt and an output for an LLM. |
| You will be given a datapoint (prompt/output), a label, and a list of reasons why that datapoint’s output has that label. |
| Your job will be to return a SIMILAR prompt and output, such that the OUTPUT (1) it matches the list of reasons, and (2) matches the label. |
| The output must match the values in the list of reasons. |
| Here’s the rubric used for these reasons: |
| You will be given a prompt and an response in <LOCALE>, meant for <LOCALE> readers. |
Table 5: Prompt used to generate a new synthetic point based on the rubric given. The user prompt (not pictured) includes the prompt and output to be paraphrased, the rubric decomposition with scores, and the final label.
More in this topic
Linguistic Bias in Language Models
Linguistic bias in language models studies how language varieties, identity cues, and cultural context shape model outputs and judgments.
See all 90 papers