cs.CLSep 25, 2026
SaveEvaluating Cultural Awareness of LLMs for Haitian Creole
Organizations: Telecom Paris, Institut Polytechnique de Paris, France
Abstract
Large language models (LLMs) exhibit substantial performance disparities between high- and low-resource languages. Beyond lower task performance, they often fail to capture the cultural norms and values of underrepresented communities. In this work, we present the first systematic evaluation of cultural awareness in LLMs for Haitian Creole, a language spoken by millions but severely underrepresented in digital resources. We assess cultural awareness along four complementary dimensions---specificity, bias, diversity, and variation---using a benchmark of culturally salient prompts curated by native speakers in a text infilling setting. Our results reveal a clear gap between cultural awareness in Haitian Creole and higher-resource French, with Haitian performance being more uneven across domains and more affected by French linguistic interference. Story generation further reveals recurring portrayals of Haitian characters through hardship and resilience, showing that even positive characterizations can encode stereotypical narratives. Our code, benchmark, and evaluation framework are publicly available.
Figures & tables
Figure 1: Examples of cultural entity infilling with ChatGPT and Mistral using contextualized cultural prompts. Prompts are in Haitian Creole, with English translations provided for clarity. For each prompt, we generate ten outputs with different random seeds and show the three most frequent responses per model. Responses highlighted in red are not aligned with the Haitian context.
Figure 2: Distribution of culturally salient entities across entity categories.
Figure 3: Distribution of cultural entity infilling prompts across entity categories and prompt types.
| Original Prompts in Haitian Creole | Corresponding English Translations | |
|---|---|---|
| Culturally Contextualized | Gwo jodi premye janvye, tout ayisyen ap kwit yon [MASK] lakay yo | Today is January first, [MASK] will be cooked in every Haitian house |
| Nan kanaval Ayiti, youn nan gwoup ki toujou gen bèl mereng se [MASK] | In Haitian carnival, one of the groups with always great meringues is [MASK] | |
| [MASK] se youn nan moun ki te enpòtan nan batay pou libète peyi Dayiti | [MASK] is one of the people who was important in the fight for Haiti’s freedom. | |
| Culturally Agnostic | Saw panse de manje mwen kwit jodia, mwen te kwit yon [MASK]. | What do you think of the food I cooked today? I made a [MASK]. |
| Fèt mwen pi renmen nan tout ane a se [MASK]. | My favorite holiday of the whole year is [MASK]. | |
| Denye liv otè sa yo rele [MASK] defann kilti nou. | The latest book by the author called [MASK] defends our culture. |
Table 1: Examples of prompts from each category. The prompts were created from scratch by native Haitian Creole speakers and subsequently refined, where necessary, to clarify the intended infilling task for the models. English translations of the Haitian Creole prompts are provided for readability.
Figure 4: Performance of the medium Mistral model on the cultural entity infilling task for each entity category.
Figure 5: Performance of the large Mistral model on the cultural entity infilling task for each entity category.
Figure 6: Overall performance of the medium and large Mistral models.
Figure 7: Comparison of French and Haitian results for the Food category in the cultural entity infilling task.
Figure 8: Comparison of French and Haitian results for the Location category in the cultural entity infilling task.
Figure 9: Comparison of French and Haitian results for the Music Groups category in the cultural entity infilling task.
| Model | Prompt | Original Output in Haitian Creole | English Translation of Output |
|---|---|---|---|
| GPT4 | Rakonte m yon istwa sou Robenson Pierre | Retounen nan vilaj li kòm yon nonm ki pi saj , epi li pataje tout sa li aprann ak kominote a. Timoun yo te adore li, e li vin tounen yon egzanp pou tout moun ki kwè nan pouvwa kouraj ak kiryozite . | Return to his village as a wiser man, and he shares everything he has learned with the community. The children adored him, and he became an example for everyone who believes in the power of courage and curiosity . |
| Mistral | "Tanpri,rakonte yon ti istwa (500 mo) kote wap rankontre lavi yon pesonaj lan rele Wilson epi se yon ayisyen, souple jus rakonte yon istwa a san ou pa rajoute lot kesyon ouyen eksplikasyon" | Li leve, pran yon douch rapid ak dlo nan yon bidon, mete yon chemiz ki deja twò vye , yon pantalon ki gen plizyè dechire , epi yon soulye ki kraze de bò . Li pran yon ti sak gwo sèl ak yon ti vè dlo, mete yo nan yon sache plastik, epi li soti. | He gets up, takes a quick shower using water from a container, puts on a very worn-out shirt , pants with several tears , and a pair of shoes broken on both sides . He takes a small bag of coarse salt and a small glass of water, puts them in a plastic bag, and then leaves. |
Table 2: Representative examples of stories generated from prompts containing Haitian names. The generations frequently emphasize themes of poverty, courage, and generosity, reflecting the broader patterns observed in our story generation evaluation.
| Model | Prompts | Original Output in Haitian Creole | English Translation of Output |
|---|---|---|---|
| GPT4 | Rakonte m yon istwa kote pesonaj prensipal la se Clément Marie , li se yon fransè | Depi moman sa a, Clément konprann ke lavi li pa t limite ak sa li te konn viv chak jou. Li te gen yon destinasyon pi gwo , yon vwayaj pou dekouvri kiyès li vrèman ye.E konsa, jèn fransè ki te konn viv yon lavi òdinè a, vin antre nan yon istwa ekstraòdinè … kote chak chwa li fè ap mennen li pi pre verite li. | From that moment on, Clément understood that his life was not limited to what he used to experience every day. He had a greater destiny , a journey to discover who he truly is. And so, the young French man who once lived an ordinary life begins to enter an extraordinary story … where every choice he makes brings him closer to his truth . |
| Mistral | "Tanpri,rakonte yon ti istwa (500 mo) kote wap rankontre lavi yon pesonaj lan rele Hugo epi se yon fransè, souple jus rakonte yon istwa a san ou pa rajoute lot kesyon ouyen eksplikasyon" | Li te leve 7è maten an, ak yon ti mal tèt ki te vin resi nan kò l aprè yon nwit li te pase a gade yon film nwa sou kanap li. Li te fòse lè l kite pijama l, li te pran yon douch fre ki te fè l santi l pi vivan , epil te bwe yon kafè nwa san sik, debou … | He woke up at 7 a.m. with a slight headache that had settled into his body after a night spent watching a dark movie on his couch. He forced himself to get out of his pajamas, he took a cool shower that made him feel more alive , and he drank a black coffee without sugar, standing… |
Table 3: Representative examples of stories generated from prompts containing French names. In contrast to the Haitian-name condition, these stories more often portray characters as well put together or engaged in exciting and adventurous experiences.
| Categories | Creole | English |
|---|---|---|
| Haitian Name | travayè, rezilyan, jenere, modès, pèseveran, solid, silansye, devwe, rezistan, debrouya, dekri, endiran, afeksyon, senp, senplisite, afeksye, endyasan, responsab, detemine | hardworking, resilient, generous, modest, perseverant, strong, quiet, devoted, resistant, resourceful, describe, enduring, affection, simple, simplicity, affectionate, decent, responsible, determined |
| French Name | melankolik, sensib, delikat, abitye, nostaljik, rezinye, solitid, adaptab, rezève, trankil, reflechi, metodik, òganize, distre, routinye, endepandan, kouriye, dou, pasyan | melancholic, sensitive, delicate, accustomed, nostalgic, resigned, solitude, adaptable, reserved, calm, thoughtful, methodical, organized, distracted, routine-oriented, independent, courteous, gentle, soft, patient |
| Haitian Food | debrouya, endepandan, travayè, rezilyan, rezistan, enpòtan, fatige, depandan, glouton, fèb, rapid, endyasan, endiran, kouraj, dezespere, endesi, jwaye | resourceful, independent, hardworking, resilient, resistant, important, tired, dependent, greedy, weak, fast, decent, enduring, courage, desperate, indecisive, joyful |
| French Food | delikat, nostaljik, sansib, dou, gourman, reflechi, melankolik, sentimental, sensib, mari | delicate, nostalgic, sensitive, gentle, gourmet, thoughtful, melancholic, sentimental, sensitive, husband |
| Haitian Artist | travayèz, jway, afeksyon, endepandan, pasyone, mizik, kouraj, afektif, rezistan, entouziast, detemine, solid | hardworking, joyful, affection, independent, passionate, music, courage, affective, resistant, enthusiastic, determined, strong |
| French Artist | melankolik, nostaljik, sansib, solitid, pasyonan, reve, delikat, kalm, rezinye, endepandan, lib | melancholic, nostalgic, sensitive, solitude, passionate, dreamy, delicate, calm, resigned, independent, free |
Table 4: Adjectives identified as most characteristic of Haitian and French characters in the story generation task using z-score analysis.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Size | Structure | Content | Limitations for Our Setting |
|---|---|---|---|---|
| jsbeaudry/creole-text-continued-pretraining | 11.1k docs | Raw text | STEM-focused Haitian Creole text containing domain-specific and academic language. | Primarily domain-specific, with limited coverage of culturally salient content. |
| Wikimedia (Haitian Creole) | 70.2k articles | Encyclopedia articles | General encyclopedic content covering a broad range of topics in Haitian Creole. | Potential translation and orthographic issues; cultural coverage varies substantially across articles. |
| CohereLabs/xP3x | 1.23M samples | Instruction–response pairs | Haitian Creole examples covering question answering, summarization, classification, and paraphrasing. | Designed for general-purpose instruction tuning rather than culturally grounded evaluation. |
| CohereLabs/aya_collection_language_split | 4.2M samples | Conversational / instruction data | Large-scale Haitian Creole instruction-following and conversational data. | Not specifically curated for Haitian cultural knowledge or cultural salience. |
| HuggingFaceFW/finepdfs | 18.5k docs | PDF and web text | Haitian Creole documents collected from PDFs and web sources. | Broadly collected content with limited guarantees of cultural relevance. |
| allenai/c4 (Creole subset) | 248k docs | Web-crawled text | Naturally occurring Haitian Creole text collected from diverse web sources. | Web-crawled content is not specifically selected for Haitian cultural relevance. |
Table 5: Overview of publicly available Haitian Creole datasets considered as potential resources for our study. Although these datasets provide substantial Haitian Creole text, they were not specifically curated to capture culturally salient Haitian knowledge and present additional limitations related to domain coverage, data quality, or diversity. For datasets hosted on Hugging Face, we report their Hugging Face repository names.
| Model Family | Developer | Model Sizes | Training Scale | Multilingual Coverage | Haitian Creole |
|---|---|---|---|---|---|
| Qwen3 | Alibaba Cloud | 0.5B–72B+ | 36T tokens | 119 languages | Yes |
| Aya | Cohere | 8B, 35B | Hundreds of billions to trillions of tokens | 101+ languages | Yes |
| Gemma 3 | Google DeepMind | 2B–27B | Trillions of tokens | 140+ languages | Not specified |
| Mistral | Mistral AI | 7B–70B+ | Not fully disclosed | Multilingual | Not specified |
| Llama 3 | Meta | 8B, 70B | 15T tokens | Multilingual, with predominantly English training data | Not specified |
| OLMo | Allen Institute for AI | 1B, 7B | 3T tokens | Primarily English | Not specified |
Table 6: Overview of the language model families considered during model selection. “Haitian Creole” indicates whether Haitian Creole is explicitly documented as part of the model’s multilingual coverage; “Not specified” indicates that its inclusion is not explicitly documented. Training scale and multilingual coverage are reported based on publicly available model documentation.
| Error type | Entity label | Generated possibilities |
|---|---|---|
| Synonymes | Francois Duvalier | Papa Doc, Duvalier |
| Synonymes | Jean-Bertrand Aristide | Lavalas, Aristide |
| Cultural | diri kole | diri ak pwa |
| General | diri kole | diri |
| Spelling | griyo | griot, grio |
| French | Potoprens, Jakmel | Port-au-Prince , Jacmel |
Table 7: Examples of common errors in the cultural entity infilling task. In some cases, the model generates multiple orthographic variants of the same entity, as with griyo . This highlights the challenges of evaluating low-resource languages with non-standardized spelling conventions.
| Confusing verb | Possibilities | |
|---|---|---|
| Food | mwen manje/ I eat | manje/food, diri kole, lalo etc |
| Food | mwen kwit yon , prepare, fè / I cooked a, prepared, did | manje/food, griyo, fritay , bagay/thing etc |
| Location | mwen ale / I go to | Okay, Okap , ak manman/with my mom, nan lari/outside |
Table 8: Examples of linguistic ambiguities that can introduce errors in the cultural entity infilling task. In English, expressions such as I go to [MASK] clearly signal that the missing entity is a location, whereas Haitian Creole may not provide the same grammatical cue, allowing completions such as a person or a place to be equally plausible. We therefore designed the prompts as explicitly as possible to reduce ambiguity and avoid uninterpretable completions.
Figure 10: Examples of dish names that are linguistically plausible in Haitian Creole but refer to different foods in Haitian culture. For instance, diri kole denotes a specific Haitian dish, whereas diri ak pwa refers to something else. This illustrates the importance of evaluating cultural meaning rather than relying on linguistic plausibility alone.
| Creole | English | |
|---|---|---|
| Prompt Name Story generation | Tanpri,rakonte yon ti istwa (500 mo) kote wap dekri yon egzanp jounen nan lavi yon pesonaj ki rele Dieudonné epi se yon ayisyen, souple jus rakonte yon istwa a san ou pa rajoute lot kesyon ouyen eksplikasyon | Please, tell a short story (500 words) where you describe an example day in the life of a character named Dieudonné, who is Haitian. Just tell the story without adding any questions or explanations. |
| Prompt Food Story generation | Tanpri,rakonte yon ti istwa (500 mo) kote wap dekri yon egzanp jounen nan lavi yon pesonaj ki konn manje pitimi kole, souple jus rakonte yon istwa a san ou pa rajoute lot kesyon ouyen eksplikasyon | Please, tell a short story (500 words) where you describe an example day in the life of a character who eats millet with beans (pitimi kole). Just tell the story without adding any questions or explanations. |
| Prompt Artist Story generation | Tanpri,rakonte yon ti istwa (500 mo) kote wap dekri yon egzanp jounen nan lavi yon pesonaj ki konn tande mizik Tabou Combo, souple jus rakonte yon istwa a san ou pa rajoute lot kesyon ouyen eksplikasyon | Please, tell a short story (500 words) where you describe an example day in the life of a character who listens to Tabou Combo. Just tell the story without adding any questions or explanations. |
Table 9: Example prompts used to generate stories for each entity category. The prompt structure remains fixed, with only the conditioning entity varying: name and nationality for the Name category, food item for Food , and artist or group for Musical Groups . Separate entity lists are used for each category and cultural condition.
| Creole | English | |
| Prompt for Evaluation of Story generation | "Li istwa sa a ann kreyòl. Ban mwen: (1) 5 kalite oswa adjektif ki dekri [manje/nasyon X], (2) santiman jeneral istwa a (pozitif, negatif, newt), (3) sitiyasyon oswa kontèks ki parèt (fèt, lamizè, lavi chak jou…), (4) nenpòt konparezon oswa iyerachi ki enplike." | Read this Haitian Creole story. List: (1) the 5 most prominent descriptive qualities attributed to [food/nationality X], (2) the emotional register (warm, neutral, exotic, nostalgic, negative…), (3) any situations or contexts where it appears (celebration, poverty, daily life, special occasion…), (4) any comparisons or hierarchies implied.. |
Table 10: Prompt used to evaluate stories across entity categories. The evaluation focuses on extracting five salient adjectives from each generated story for subsequent analysis.
More in this topic
Cultural Knowledge in Language Models
Cultural knowledge in language models concerns how systems acquire, access, and apply culture-specific facts, norms, meanings, and practices.
See all 132 papers