Evaluating the Usage of African-American Vernacular English in Large Language Models

Deja Dunlap; R. Thomas McCoy

cs.CLFeb 25, 2026

Evaluating the Usage of African-American Vernacular English in Large Language Models

Authors: Deja Dunlap, R. Thomas McCoy

Organizations: Yale University, New Haven CT 06511, USA

Abstract

In AI, most evaluations of natural language understanding tasks are conducted in standardized dialects such as Standard American English (SAE). In this work, we investigate how accurately large language models (LLMs) represent African American Vernacular English (AAVE). We analyze three LLMs to compare their usage of AAVE to the usage of humans who natively speak AAVE. We first analyzed interviews from the Corpus of Regional African American Language and TwitterAAE to identify the typical contexts where people use AAVE grammatical features such as ain't. We then prompted the LLMs to produce text in AAVE and compared the model-generated text to human usage patterns. We find that, in many cases, there are substantial differences between AAVE usage in LLMs and humans: LLMs usually underuse and misuse grammatical features characteristic of AAVE. Furthermore, through sentiment analysis and manual inspection, we found that the models replicated stereotypes about African Americans. These results highlight the need for more diversity in training data and the incorporation of fairness methods to mitigate the perpetuation of stereotypes.

Figures & tables

Figure 1: The workflow we used to compare how African-American Vernacular English is used by humans and large language models. This workflow is expanded in further detail in the Methodology section below.

CORAAL-style Prompt

TwitterAAE-style Prompt

Role: You are an African American {gender} from {city}, participating in an oral sociolinguistic interview. Guidelines: • Speak in a natural, conversational way, as though you are telling your life story, sharing experiences, and reflecting on everyday life. • Produce a continuous narrative roughly the length of a 30-minute conversation (around 4,000–5,000 words). • Include only the narrative text itself, with no headings, notes, or explanations.

Role: You are given the role of a casual Twitter user. Generate 250 tweets written in African American Vernacular English (AAVE). Guidelines: • Write them in the informal, conversational style of Twitter. • Keep each tweet short (under 280 characters). • Use natural, everyday topics (music, sports, friends, emotions, funny observations, etc.). • Put each tweet on a new line using the newline character. • Include no other extraneous information about the tweet, the task, or anything else. Include only the tweets.

Table 1: Prompt templates used to elicit AAVE data in two distinct contexts: extended sociolinguistic interviews (CORAAL) and informal social media posts (TwitterAAE). The variables gender and city were sampled at a rate proportional to the representation in the CORAAL dataset to control for gender and geographical-related variance.

Figure 2: Bar plots highlighting the portion of sentences in each corpus that had a sentiment score $x$ defined as negative ( $x<-0.5$ ), neutral ( $0.5>x>-0.5$ ), or positive ( $x>0.5$ ), where $x$ was obtained via VADER.

Sentiment	Human	Gemma	GPT-4o	Llama
Average	0.135	0.146	0.260	0.160

Table 2: Descriptive data for sentence-level sentiment scores across datasets. Scores range from -1 (furthest negative) to +1 (furthest positive).

AAVE Feature	Example Usage of Feature	Analogous Sentence in SAE
Ain’t	I ain’t doing all that.	I am not doing all of that.
Habitual Be	He be doing too much.	He is always doing too much.
Negative Concord	I don’t never have no problems.	I don’t ever have any problems.
Double Comparative	I am more happier when I’m alone.	I am happier when I’m alone.
Perfective Done	I done lost my wallet.	I have lost my wallet.
Multiple Modals	We might can go up there next Saturday.	We might be able to go up there next Saturday.

Table 3: Examples of the grammatical features of African-American Vernacular English (AAVE) that are examined in this study, along with analogous sentences in Standard American English (SAE).

	Double Comparative	Habitual Be	Multiple Modals	Negative Concord	Null Copula	Perfective Done	Ain’t
Accuracy	0.99	0.99	0.99	0.97	0.96	0.99	1.00

Table 4: Performance of feature detectors across grammatical categories. All feature parsers can maintain an accuracy rate greater than 90.

Table 5: Feature frequencies across TwitterAAE and CORAAL (and analogous datasets generated by LLMs), per 10,000 sentences, rounded to three decimal points. Bolded

\delta

columns represent statistically significant differences with

\alpha<0.05

Figure 3: Bar plots highlighting the difference in feature frequency by model and feature (per 10,000 sentences). Features that appear less frequently (ie. Double Comparative, Null Modals, Perfective done ) have a harder time appearing on scale; see Table 5 for those features.

Table 6: Contextual frequencies of grammatical features in the CORAAL datasets (rounded to three decimal points). Values show Human baseline and model deviations. Bold indicates significant deltas. (–) indicates the model did not use the feature in that given context.

Table 7: Contextual frequencies of grammatical features in the TwitterAAE datasets (rounded to three decimal points). Values show Human baseline and model deviations. Bold indicates significant deltas. (–) indicates the model did not use the feature in that given context.

Feature	V1	V2	V3	V4	V5	V6	V7	V8	V9	V10
Habitual be	0.576	0.441	0.401	0.493	0.525	0.413	0.483	0.408	0.488	0.359
Negative Concord	2.738	2.828	2.778	2.832	2.931	2.868	2.730	2.815	2.749	2.857
Ain’t	2.279	2.620	2.378	2.306	2.406	2.372	2.655	2.373	2.530	2.356
Double Comparative	0.008	0.000	0.000	0.008	0.000	0.008	0.017	0.017	0.000	0.008
Multiple Modals	0.008	0.008	0.000	0.008	0.000	0.025	0.017	0.008	0.008	0.025
Perfective Done	0.125	0.183	0.142	0.125	0.183	0.149	0.133	0.150	0.168	0.092

Table 8: Table showing feature densities (per 100 words) across ten randomly selected mini-datasets generated from the human dataset.

Dialect Bias in Language Models

See all 14 papers

LLMs Silently Correct African American English: Auditing and Mitigating Dialect Bias via Activation Steering
Jul 7, 2026Huan Wu, Ali Emami, Muhammad Furquan Hassan +5Language Model Steering Social Bias in Language Models
York University · Vector Institute · Connected Minds +6
Side-by-side Comparison Amplifies Dialect Bias in Language Models
May 23, 2026Kritee Kondapally, Claire J. Smerdon, Pooja C. Patel +5Social Bias in Language Models Algorithmic Fairness
University of Southern California, USA
From a River in Gilead to the Inference Distributions of Large Language Models: Covert Dialect Bias and Linguistic Profiling at Scale
Sep 16, 2026Chowdhury Mohammad Abdullah, Rita OrjiSocial Bias in Language Models Dialect Bias in Language Models
Faculty of Computer Science, Dalhousie University Halifax, NS, Canada

Jul 7, 2026J/K move · Enter open · S save

LLMs Silently Correct African American English: Auditing and Mitigating Dialect Bias via Activation Steering

Huan Wu, Ali Emami, Muhammad Furquan Hassan, Osaretin Igbinoba+4

York University · Vector Institute · Connected Minds+6

Language Model Steering Social Bias in Language Models Dialect Bias in Language Models Language Model Bias Evaluation Linguistic Bias in Language Models Activation Steering

Open paper AskPDF arXiv

African American English (AAE), a rule-governed dialect spoken by over 30 million people, is routinely misinterpreted and "corrected" by large language models (LLMs). Across six instruction-tuned LLMs (14B to 70B), we show that state-of-the-art models systematically prefer Standard American English (SAE) continuations even when the preceding context is in AAE, effectively rewriting AAE into SAE. We present an end-to-end framework to audit and mitigate this bias. For auditing, we introduce conditional Dialect Group Invariance (cDGI), which isolates true model bias from translator-induced artifacts, and a feature-level localization analysis that identifies which AAE markers most strongly trigger bias; we find that syntactic constructions, especially negative concord (e.g., "ain't nobody"), are universal triggers across all models. For mitigation, we introduce, to our knowledge, the first application of activation steering to dialect bias: a training-free, test-time method that extracts dialect directions via causal tracing and injects them into bias-relevant layers. Activation steering reduces bias 5 to 20 times more than prompting while preserving SAE fluency. To enable this work, we release REAL-AAE , the largest real-AAE parallel corpus to date: 17,479 AAE/SAE/ AAE_back triplets from natural tweets (2 to 6 times larger than prior real-AAE resources), validated automatically (BERTScore F1 = 0.95) and by three native AAE speakers (83.0% semantic agreement).

Abstract

Dialect Bias in Language Models

Explore similar work

LLMs Silently Correct African American English: Auditing and Mitigating Dialect Bias via Activation Steering

Side-by-side Comparison Amplifies Dialect Bias in Language Models

From a River in Gilead to the Inference Distributions of Large Language Models: Covert Dialect Bias and Linguistic Profiling at Scale