DialectLLM: A Dialect-Aware Dialog[ue] Generation Framework Beyond Standard American English
Organizations: KAIST · Amazon
Abstract
More than 80% of the 1.6B English speakers do not use Standard American English (SAE), yet LLMs often fail to correctly identify non-SAE dialects and generate stereotyped responses for their speakers. We introduce DialectLLM, the first large-scale framework for generating high-quality multi-dialectal conversational data encompassing the three pillars of written dialect---lexical (vocabulary), orthographic (spelling), and morphosyntactic (grammar) features. DialectLLM produces a dialect-parallel dialog dataset spanning nine English dialects. Partnering with native linguists, we design and validate SAE-to-dialect transformation rules, ensuring authenticity. Our approach challenges the prevailing practice of applying a single morphosyntactic feature set to both user utterances and model responses, showing that models should not reproduce up to 90% of the grammatical features of a dialect. Human evaluation confirms data quality, with annotators preferring DialectLLM over prior methods in 98.8% of pairwise comparisons for dialect naturalness. We then construct DialectLLM-Bench, a dialect-parallel benchmark with 50k+ dialogs, resulting in 97k+ QA pairs, and evaluate 17 LLMs on dialect identification and response generation tasks. Even frontier models achieve around 70% accuracy, fail to reach 40% for prominent dialects like Canadian English, and systematically misclassify non-SAE dialects as American or British. Beyond benchmarking, we show that DialectLLM data also serve as a scalable LLM post-training resource, suggesting a practical path toward dialect-aware conversational AI.
Figures & tables
| SAE baseline | Trans-EnV (eWAVE + LLM) | Ours (DialectLLM-Model) |
|---|---|---|
| I’m trying to organize my linen closet better, but it’s such a mess right now. Do you have any tips for keeping sheets neat and accessible? | I’m trying to organize me linen closet where it’s, like, such a mess right now more good . Thou have any tips for keeping sheets neat and accessible? | I’m trying to organise me hot press better, but it’s such a mess right now. What tips ye have for keeping sheets neat and accessible? |
| Eval. target | AU | IN | IR | NG | PH | SC |
|---|---|---|---|---|---|---|
| User utt. | 0.99 | 0.99 | 0.97 | 0.98 | 1.00 | 0.99 |
| Asst. resp. | 1.00 | 1.00 | 0.99 | 0.96 | 1.00 | 0.98 |
Appendix figures & tables36 assets
Supplementary material from the paper’s appendix.
Appendix
| Transformation | Description | Examples |
|---|---|---|
| Lexical | Lexical transformations refer to differences in vocabulary or the specific words used to refer to the same concept. | - Soda / Pop - Pants / Trousers - Umbrella / Brolly |
| Orthographic | Orthographic transformations involve variations in the spelling of words. The pronunciation and meaning are generally the same, but the written form differs. | - Color / Colour - Theater / Theatre - Realize / Realise |
| Morphosyntactic | Morphosyntactic transformations are about differences in grammar, including how words are formed and how they are arranged in sentences. | - Y’all / Youse / You guys - I like me shoes. (I like my shoes in SAE) |
| Example Sentence | |
|---|---|
| Original (SAE) | I realized the color of the theater poster was different from the sign, so we plan to get takeout at the gas station. Are you coming? (assuming the user is asking to two or more people) |
| + Lexical | I realized the color of the theater poster was different from the sign, so we plan to get takeaway at the servo . Are you coming? |
| + Orthographic | I realised the colour of the theatre poster was different from the sign, so we plan to get takeaway at the servo . Are you coming? |
| + Morphosyntactic | I realised the colour of the theatre poster was different to the sign, so we plan to get takeaway at the servo . Are youse coming? |
| Rule | Original | Transformed | Description |
|---|---|---|---|
| She/her used for inanimate referents | The car is beautiful, and it runs perfectly. | The car is beautiful, and she runs perfectly. | This transformation rule involves replacing neuter pronouns (it/its) with feminine pronouns (she/her/hers) when referring to inanimate objects. Instead of using the standard ”it” to refer to non-living things like vehicles, machines, or objects, the transformed text personifies these items by assigning them feminine gender through pronoun usage. This creates a more personalized or affectionate tone toward inanimate objects, similar to how ships are traditionally referred to as ”she” in maritime contexts. |
| Me instead of I in coordinate subjects | Sarah and I are going to the market. | Sarah and me are going to the market. | This transformation rule replaces the subjective pronoun ”I” with the objective pronoun ”me” in coordinate subject constructions. When ”I” appears as part of a compound subject joined by ”and” (such as ”Sarah and I”), it is changed to ”me” (becoming ”Sarah and me”), even though the pronoun is functioning as the subject of the sentence. This represents a departure from standard English grammar, which requires the subjective case ”I” for subjects, in favor of using the objective case ”me” in these coordinate constructions. |
| Like as a focussing device | He was very angry. | He was, like, very angry. | This transformation involves inserting the word ”like” as a discourse marker or hedge, typically surrounded by commas, to focus attention on or emphasize the phrase that follows. The word ”like” functions as a parenthetical interjection rather than in its traditional grammatical roles, serving to highlight or draw special attention to the subsequent description or statement. This usage is characteristic of informal spoken English and creates a slight pause that signals the importance or precision of what comes next. |
| User Input | I’m planning a cross-country road trip next month and I’m trying to figure out the best way to handle meals along the way. I don’t want to rely entirely on fast food chains, but I also don’t want to pack everything myself. What are some good options for finding decent food when you’re driving through small towns or rural areas? |
|---|---|
| SAE | That sounds like an exciting adventure! For road trips through small towns and rural areas, roadside restaurants are often your best bet for authentic, home-cooked meals. These local gems usually offer hearty portions at reasonable prices and give you a real taste of regional cuisine. Look for places with full parking lots - that’s usually a good sign that locals eat there regularly. Many of these establishments are family-owned diners or cafes that have been serving travelers and community members for decades. You can also use apps like Yelp or Google Maps to find highly-rated local spots along your route, or ask for recommendations at gas stations when you stop to fuel up. |
| Multi-Value | That sound an exciting adventure! Road trips small towns and rural areas, roadside restaurants are often your most good bet authentic, home-cooked meals. These local gems usually offers hearty portions reasonable prices and give you a real taste regional cuisine. Look places full parking lots - that’s usually a good sign that eats there regularly. Many these establishments are family-owned diners or cafes has been serving travelers and community members decades. You can also use apps Yelp or Google Maps to find highly-rated local spots your route, or ask recommendations gas stations when you stops to fuel. |
| Trans-EnV | That is sounding like a, like, exciting adventure, but! For road trips through small towns and rural areas, roadside restaurants, which usual offer, like, hearty portions at reasonable prices and give you a real taste of regional cuisines, are often your, like, goodest bet for authentic, home-cooked meals, but. Look for places with full parking lots, and avoid places with empty lots - those mustn’t be good since locals clear don’t eat there regular. There’s many family-owned diner or cafes that served travelers and community members for decades. You can also use apps like Yelp or Google Maps to find, like, high-rated local spots along your route, or ask for recommendations at gas stations when you stop to fuel up. |
| DialectLLM-OL | That sounds like an exciting adventure! For road trips through small towns and rural areas, roadside restaurants are often your best bet for authentic, home-cooked meals. These local gems usually offer hearty portions at reasonable prices and give you a real taste of regional cuisine. Look for places with chockers car parks - that’s usually a good sign that locals eat there regularly. Many of these establishments are family-owned diners or cafes that have been serving travellers and community members for decades. You can also use apps like Yelp or Google Maps to find highly-rated local spots along your route, or ask for recommendations at service stations when you stop to fuel up. |
| DialectLLM-User | That there sounds like an exciting adventure! For road trips through small towns and rural areas, roadside restaurants are often your most best bet for , like, authentic, home-cooked meals. These local gems here usually offer hearty portions at reasonable prices and give youse a real taste of regional cuisines. Look for places with chockers car parks - that’s usually a good sign that locals eat there regularly. Many of these establishments here are , like, family-owned diners or cafes that have been serving travellers and community members for decades. Youse can also use apps like Yelp or Google Maps to find more highly-rated local spots along your route, or ask for recommendations at service stations when youse stop to fuel up. |
| DialectLLM-Model | That there sounds like an exciting adventure! For road trips through small towns and rural areas, roadside restaurants are often your best bet for authentic, home-cooked meals. These local gems here usually offer hearty portions at reasonable prices and give you a real taste of regional cuisine. Look for places with chockers car parks - that’s usually a good sign that locals eat there regularly. Many of these establishments here are family-owned diners or cafes that have been serving travellers and community members for decades. You can also use apps like Yelp or Google Maps to find highly-rated local spots along your route, or ask for recommendations at service stations when you stop to fuel up. |
| Dialect | (%) | (%) |
|---|---|---|
| Australian English | 41.67 | 1.67 |
| Indian English | 18.00 | 2.00 |
| Irish English | 40.22 | 27.17 |
| Nigerian English | 41.18 | 31.37 |
| Philippine English | 33.33 | 7.41 |
| Scottish English | 27.27 | 25.45 |
| Role | Preference | AU | IN | IE | NG | PH | SC |
|---|---|---|---|---|---|---|---|
| DialectLLM greatly preferred to Trans-EnV | 0.90 | 0.94 | 0.68 | 0.79 | 0.88 | 0.73 | |
| DialectLLM somewhat preferred to Trans-EnV | 0.09 | 0.05 | 0.28 | 0.19 | 0.12 | 0.26 | |
| User | DialectLLM approximately equal to Trans-EnV | 0.00 | 0.00 | 0.02 | 0.01 | 0.00 | 0.00 |
| Trans-EnV somewhat preferred to DialectLLM | 0.01 | 0.01 | 0.01 | 0.00 | 0.00 | 0.01 | |
| Trans-EnV greatly preferred to DialectLLM | 0.00 | 0.00 | 0.00 | 0.01 | 0.00 | 0.00 | |
| DialectLLM greatly preferred to Trans-EnV | 0.93 | 0.99 | 0.69 | 0.81 | 0.99 | 0.62 |
| Preference | AU | IN | IE | NG | PH | SC |
|---|---|---|---|---|---|---|
| DialectLLM-Model greatly preferred to DialectLLM-User | 0.35 | 0.15 | 0.68 | 0.68 | 0.42 | 0.09 |
| DialectLLM-Model somewhat preferred to DialectLLM-User | 0.45 | 0.38 | 0.31 | 0.23 | 0.46 | 0.65 |
| DialectLLM-Model approximately equal to DialectLLM-User | 0.19 | 0.41 | 0.00 | 0.01 | 0.07 | 0.22 |
| DialectLLM-User somewhat preferred to DialectLLM-Model | 0.01 | 0.06 | 0.01 | 0.04 | 0.05 | 0.04 |
| DialectLLM-User greatly preferred to DialectLLM-Model | 0.00 | 0.00 | 0.00 | 0.04 | 0.00 | 0.00 |
| Dialect | Representative comments |
|---|---|
| IN | - DialectLLM-Model is greatly preferred because it avoids unnecessary filler words and maintains clear grammar suitable for an assistant response. - DialectLLM-Model is an excellent example of my dialect and is grammatically correct. |
| IR | - No issues with DialectLLM-Model, but DialectLLM-User is too colloquial. - DialectLLM-User contains examples that are not appropriate for written AI. |
| SC | - DialectLLM-Model is greatly preferred; DialectLLM-User is too informal for the assistant response. - DialectLLM-User uses grammar that the model should not use. For example, the use of like , ye , or youse is a behavior that the model should not copy. |
| NG | - DialectLLM-Model maintains a helpful and ‘neighbourly’ tone while correctly using the Nigerian spelling of organise . While DialectLLM-User attempts dialectal markers such as zero-article usage and pluralized uncountables, these result in an over-transformed or overly informal tone that the assistant should generally avoid. |
| Turn 1 | Turn 2 | Turn 4 | Turn 8 | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | OL | U | M | OL | U | M | OL | U | M | OL | U | M |
| Qwen3-8B | 0.353 | 0.237 | 0.429 | 0.326 | 0.240 | 0.385 | 0.318 | 0.265 | 0.377 | 0.319 | 0.287 | 0.373 |
| Qwen3-8B-0.15 | 0.746 | 0.878 | 0.919 | 0.782 | 0.941 | 0.957 | 0.838 | 0.976 | 0.977 | 0.887 | 0.991 | 0.990 |
| Qwen3-8B-Full | 0.860 | 0.956 | 0.967 | 0.872 | 0.990 | 0.991 | 0.910 | 0.998 | 0.996 | 0.944 | 0.999 | 0.999 |
| Gemma3-12B | 0.535 | 0.410 | 0.622 | 0.511 | 0.436 | 0.605 | 0.507 | 0.450 | 0.599 | 0.546 | 0.493 | 0.617 |
| Gemma3-12B-Full | 0.875 | 0.967 | 0.967 | 0.882 | 0.990 | 0.990 | 0.915 | 0.998 | 0.997 | 0.949 | 1.000 | 0.999 |
| Turn 1 | Turn 2 | Turn 4 | Turn 8 | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | OL | U | M | OL | U | M | OL | U | M | OL | U | M |
| Qwen3-8B | 0.426 | 0.220 | 0.310 | 0.525 | 0.250 | 0.334 | 0.527 | 0.243 | 0.333 | 0.563 | 0.268 | 0.363 |
| Qwen3-8B-0.15 | 0.896 | 0.900 | 0.952 | 0.915 | 0.887 | 0.932 | 0.920 | 0.888 | 0.923 | 0.944 | 0.888 | 0.926 |
| Qwen3-8B-Full | 0.935 | 0.972 | 0.985 | 0.948 | 0.955 | 0.962 | 0.947 | 0.952 | 0.961 | 0.961 | 0.949 | 0.961 |
| Gemma3-12B | 0.468 | 0.275 | 0.358 | 0.459 | 0.267 | 0.324 | 0.458 | 0.264 | 0.343 | 0.507 | 0.285 | 0.392 |
| Gemma3-12B-Full | 0.946 | 0.978 | 0.988 | 0.953 | 0.963 | 0.971 | 0.950 | 0.958 | 0.967 | 0.962 | 0.954 | 0.967 |
| Turn 1 | Turn 2 | Turn 4 | Turn 8 | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | OL | U | M | OL | U | M | OL | U | M | OL | U | M |
| Qwen3-8B | 0.353 | 0.237 | 0.429 | 0.326 | 0.240 | 0.385 | 0.318 | 0.265 | 0.377 | 0.319 | 0.287 | 0.373 |
| Qwen3-8B-RC-0.15 | 0.548 | 0.425 | 0.589 | 0.553 | 0.395 | 0.518 | 0.491 | 0.306 | 0.452 | 0.484 | 0.357 | 0.448 |
| Qwen3-8B-RC | 0.523 | 0.346 | 0.511 | 0.504 | 0.325 | 0.475 | 0.438 | 0.293 | 0.428 | 0.394 | 0.241 | 0.383 |
| Qwen3-8B-Id-0.15 | 0.803 | 0.855 | 0.901 | 0.811 | 0.911 | 0.939 | 0.856 | 0.957 | 0.968 | 0.901 | 0.983 | 0.987 |
| Qwen3-8B-Id | 0.863 | 0.952 | 0.963 | 0.872 | 0.983 | 0.985 | 0.908 | 0.995 | 0.994 | 0.942 | 0.999 | 0.999 |
| Turn 1 | Turn 2 | Turn 4 | Turn 8 | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | OL | U | M | OL | U | M | OL | U | M | OL | U | M |
| Qwen3-8B | 0.426 | 0.220 | 0.310 | 0.525 | 0.250 | 0.334 | 0.527 | 0.243 | 0.333 | 0.563 | 0.268 | 0.363 |
| Qwen3-8B-RC-0.15 | 0.884 | 0.881 | 0.940 | 0.915 | 0.871 | 0.917 | 0.911 | 0.884 | 0.912 | 0.944 | 0.880 | 0.909 |
| Qwen3-8B-RC | 0.930 | 0.963 | 0.982 | 0.944 | 0.945 | 0.962 | 0.946 | 0.946 | 0.959 | 0.963 | 0.946 | 0.961 |
| Qwen3-8B-Id-0.15 | 0.298 | 0.199 | 0.243 | 0.318 | 0.221 | 0.238 | 0.328 | 0.223 | 0.246 | 0.347 | 0.215 | 0.212 |
| Qwen3-8B-Id | 0.173 | 0.114 | 0.127 | 0.244 | 0.170 | 0.184 | 0.292 | 0.227 | 0.237 | 0.291 | 0.226 | 0.245 |
| Train / Test | OL | User | Model |
|---|---|---|---|
| Base Model | 32.7 / 51.1 | 25.6 / 24.2 | 39.2 / 32.8 |
| OL | 81.8 / 89.9 | 58.0 / 63.2 | 72.9 / 71.4 |
| User | 68.7 / 88.2 | 89.3 / 87.3 | 85.7 / 84.7 |
| Model | 67.1 / 88.3 | 48.2 / 71.2 | 92.6 / 91.6 |
| 9-way | 6-way | |||
|---|---|---|---|---|
| Dialect | Base | Post-trained | Base | Post-trained |
| Australian (AU) | 11.2 | 28.2 | 12.7 | 46.5 |
| Nigerian (NG) | 27.3 | 55.3 | 32.1 | 58.2 |
| Indian (IN) | 44.1 | 89.7 | 51.1 | 96.9 |
| Irish (IE) | 31.2 | 29.2 | 48.2 | 75.8 |
| Philippine (PH) | 57.3 | 70.6 | 55.9 | 76.1 |