cs.CLAug 29, 2026
SaveToward Cultural Alignment: Human-Centered Evaluation of Multimodal AI Stories Across Five African Communities
Organizations: Microsoft Research Africa · Swansea University · Cornell University · Microsoft Research Cambridge
Abstract
In this paper, we examine how well AI-generated multimodal stories align with the lived practices, relationships, language, values, and visual expectations of the communities they represent. We conduct a community-grounded mixed-methods evaluation with 19 culture representatives across five African communities, combining quantitative annotations with qualitative focus group discussions. We find that cultural alignment depends not simply on recognizable cultural markers, but on how those markers fit social, linguistic, procedural, and visual context. From these evaluations, we develop a taxonomy of cultural alignment comprising five broader cultural marker categories and eight recurring mechanisms of misalignment. We additionally evaluate five multimodal LLM judges to examine whether automated evaluation can approximate community-grounded judgments at scale. Judge reliability and score calibration vary substantially across communities, with no single judge performing consistently across all five settings. These findings motivate community-calibrated evaluation pipelines in which automated judges are validated against community judgments to determine where they can be trusted and where human review remains necessary.
Figures & tables
Figure 1: Example story.
Figure 2: Generation and Evaluation workflow. Green highlights indicate culturally appropriate elements; red highlights indicate violations identified by culture representatives.
| Taxonomy | Description and examples | |
| category | ||
| Referential | Explicit cultural references, such as names, foods, places, objects, and institutions. | |
| Cultural marker categories | Procedural | How everyday practices are carried out, such as food preparation, care-seeking, exercise routines, and household activities. |
| Contextual | Whether cultural elements fit the situation, timing, purpose, or setting in which they appear. | |
| Socio-geographic | Place-based social and material conditions, such as clinics, markets, homes, infrastructure, mobility, and local environments. | |
| Linguistic register | Community-specific language use, including dialect, code-switching, forms of address, tone, and community voice. |
Table 1: Summary of the cultural alignment taxonomy derived from focus group discussions and participant annotations (refer to Figure 8 for examples).
Figure 3: Distribution of not-connected cultural markers in text and images across the five broader cultural marker categories, broken down by generation model and community. Connected marker distributions for both modalities appear in Appendix Figure 13 .
| Community | Kimi | Gemma4 | GPT-5.5 | Qwen-122B | Qwen-9B |
|---|---|---|---|---|---|
| Hausa | 0.74 | 0.67 | 0.70 | 0.80 | 0.76 |
| AmaXhosa | 0.33 | 0.26 | 0.28 | 0.29 | 0.40 |
| Kikuyu | 0.54 | 0.50 | 0.51 | 0.64 | 0.57 |
| Luo | 0.89 | 0.82 | 0.88 | 0.87 | 0.87 |
| Xichangana | 0.49 | 0.45 | 0.51 | 0.49 | 0.39 |
Table 2: Pearson correlation coefficients ( ) between LLM judge scores and mean scores from culture representatives by community. Bold values remain significant after Bonferroni correction ( ); unbolded values do not, even when significant at uncorrected thresholds ( , , or ). No AmaXhosa judge remains significant after correction.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Geographic distribution of the five African communities included in our study.
| Community | n | Age Range | Gender |
|---|---|---|---|
| Hausa | 3 | 30–39 | 3F |
| Kikuyu | 4 | 18–34 | 3F, 2M |
| Luo | 6 | 18–44 | 4F, 2M |
| AmaXhosa | 2 | 25–29 | 2F |
| Xichangana | 4 | 18–44 | 3F, 1M |
| Total | 19 | 18–44 | 15F, 5M |
Table 3: Culture representative demographics by community.
| Category | Question |
|---|---|
| Diet | Which everyday foods can I eat that will not make my blood sugar spike? |
| I like to snack—what are some healthy options, and how often should I have them? | |
| I sometimes eat late at night. How does this affect my blood sugar, and what can I do instead? | |
| Can you give me some simple meal ideas or a sample daily menu I could follow? | |
| Eating healthy can get expensive—how can I maintain a nutritious diet on a tight budget? Can I still enjoy traditional foods? | |
| Exercise | What kinds of exercises are safe and beneficial for someone with diabetes? |
Table 4: Everyday diabetes lifestyle questions used for narrative generation.
| Context setting | Information provided during generation |
|---|---|
| Sparse persona | A reduced persona profile in which approximately half of the persona attributes were removed while preserving core demographic and cultural information. |
| Full persona | The complete generated persona profile. |
| Full persona + retrieval | The complete generated persona profile together with retrieved web context. GPT-4.1 generated search queries from the persona profile, and the retrieved information was provided as additional context for story generation. |
Table 5: Context settings used to vary the amount of persona and cultural information available during story generation.
| # | Persona field |
|---|---|
| 1 | Culture (select all that apply) |
| 2 | Which province do you live in? (select 1) |
| 3 | What is your religion? |
| 4 | What is your preferred name? |
| 5 | Gender (select 1) |
| 6 | Age (select 1) |
Table 6: Persona fields used to construct culturally situated personas.
| Category | Description | Examples |
|---|---|---|
| people_appearance | skin tone, hair, age, who is present, visual identity cues | An elderly woman with grey hair greets a young boy at the gate. |
| names_forms_of_address | forms of address that signal identity or relationships | “Auntie Mary” calls him “my child” when he arrives home. |
| language_local_expression | slang, code-switching, local phrases: “Ag”, “mos”, “lekker” | “Ag, don’t worry,” she says, switching between languages mid-sentence. |
| food_dietary_practices | foods, preparation methods, meal styles, eating habits | The family shares stiff porridge and vegetables for dinner. |
| place_physical_environment | province, home, clinics, kitchen, shops, neighborhoods, local setting | The small shop near the dusty road opens early every morning. |
| healthcare_community_practices | clinic visits, nurses, advice-seeking norms, care pathways | She visits the clinic after neighbors advise her to check her fever. |
Table 7: Cultural marker categories used for both text and image annotation. The same categories were applied across modalities, with annotators selecting text spans or image regions that influenced their cultural alignment judgments.
| Score Range | Interpretation |
|---|---|
| 0 | No recognizable cultural markers or cultural relevance. |
| 1–20 | Minimal mention of cultural elements without meaningful integration. |
| 21–40 | Some cultural markers are present, but integration is weak, generic, or only loosely relevant. |
| 41–60 | Moderate cultural reflection with partial relevance to the represented community. |
| 61–80 | Most cultural markers are meaningfully incorporated and broadly aligned with the represented community. |
| 81–100 | Strong cultural alignment, with high relevance, coherence, and authenticity in relation to the represented community. |
Table 8: Overall cultural alignment scoring rubric used by culture representatives and LLM judges.
Figure 5: Annotation platform used by culture representatives to evaluate text and image cultural alignment.
Figure 6: Annotation platform: Assigning a cultural marker category to a highlighted text span or region.
Figure 7: Annotation platform: Assigning overall culture alignment scoring.
| Discussion focus | Guiding questions |
|---|---|
| Initial reactions | 1. After reading this story, did it feel like something that could realistically happen in your community? Why or why not? 2. Would someone from your community recognize themselves in this story? What makes you say that? 3. Did anything immediately stand out to you as either strongly connected to your community or clearly not connected? What was it? |
| Cultural marker identification and misalignment | 1. Can you walk us through specific parts of the story that felt “off” or did not ring true for your community? 2. For those moments, what exactly made them feel wrong? Were they incorrect, out of place, too generic, or based on assumptions? 3. Were there important cultural elements that you expected to see but were missing? What were they? 4. Across the stories you reviewed, are there particular types of cultural elements that the AI tends to get wrong or struggle with? |
| Meaningful integration of cultural markers | 1. When cultural elements appeared in the story, such as food, language, or daily practices, did they feel like a natural part of the story, or did they feel added in? Why? 2. Can you give an example of a cultural element that felt genuinely well integrated, and one that felt forced or out of place? 3. Did some stories feel more grounded in your community than others? What specifically made the difference? 4. In stories where cultural elements were present but the story still felt off, what was missing for it to feel truly connected to your community? |
Table 9: Guiding questions used in focus group discussions with culture representatives.
| Paragraph 1 | Paragraph 2 | Paragraph 3 | Paragraph 4 | |
|---|---|---|---|---|
| default | Introduce {px_name}’s current daily life and diet habits as described above. Mention any cultural and environmental factors that make change difficult. The persona currently has a small work-related task that foreshadows a larger work-related task. They fail to accomplish the task because of issues related to health changes they need to make. All motivation to change comes from the need to accomplish the current and future task. | Because of the failed small task, {px_name} needs help. Depending on the given healthcare access & resources, either {px_name} seeks help e.g., visits a clinic or community health worker, or help finds them coincidentally e.g., support network. This helper gives practical advice on managing their lifestyle or current challenge. The persona is initially resistant to the change. Motivation to change comes from failure at the small task. The health worker suggests a small change that will help the persona accomplish the task. | Describe how {px_name} tries to follow the advice given, facing initial struggles due to certain reason(s). Describe how they overcome these struggles. The persona tries to make the prescribed health change but faces a setback. They try to find alternative ways to pursue the health change in light of the upcoming larger task. | ** Highlight positive but realistic changes, that result from the new effort they’ve put in. The persona finds the small changes helpful in health management. This enables them accomplish the larger task. The persona recognises the health AND work-related benefits of the health changes. The persona is motivated to make more / bigger changes to their health habits. |
| classic change | Introduce daily life and diet habits that reflect limited awareness of how health affects work. The small work task fails due to misunderstanding or neglect of health needs. | Help comes from a formal health source (clinic or health worker). Resistance is mild and based on uncertainty. Advice is clear, simple, personalized, culturally relevant, actionable, and practical. | The persona attempts the advice but struggles with adjusting routines. A minor setback occurs due to habit change. They persist because of the upcoming larger task. | Gradual, realistic improvement occurs. The larger task is accomplished with more confidence. The persona becomes open to continued health improvement. |
| crisis avoidance | Daily habits show warning signs or discomfort. Failure of the small task includes a health scare or limitation. | Help comes from a trusted health authority. Advice (clear, simple, personalized, culturally relevant, actionable, and practical) emphasizes preventing serious future consequences. Resistance is rooted in fear or denial. | The persona attempts the change but hesitates or backslides due to anxiety. They reframe the change as necessary to succeed at the larger task. | Health stabilizes in a realistic way. The larger task is completed with relief. The persona is motivated to prevent future crises. |
| iteration | The small task fails as part of trial and error, not a single mistake. Health issues are one of several contributing factors. | Help provides practical advice, but it is not positioned as a perfect solution. Resistance is low; expectations are cautious. | The initial strategy does not work. A setback occurs. The persona modifies the advice (clear, simple, personalized, culturally relevant, actionable, and practical) creatively to fit their reality and the larger task. | Learning from iteration leads to success in the larger task. Health improvements are modest but meaningful. The persona commits to ongoing adjustment rather than perfection. |
| resistance | Introduce daily habits that are deeply ingrained. The small work task fails due to stubborn routines or skepticism about health change. | Help is offered by a health worker or trusted person. The persona doubts the advice (clear, simple, personalized, culturally relevant, actionable, and practical) and resists change. They agree to a small adjustment only to fix the work issue. | The persona partially ignores or delays the advice. This leads to a setback. They reluctantly adjust their approach because the larger task is approaching. | Resistance softens. The persona acknowledges both health and work benefits. |
| social support | Daily life is shaped by social and cultural expectations. The small work task fails due to competing social demands affecting health. | Help comes from a family member, coworker, or community figure. Initial resistance comes from fear of disrupting social norms. Advice (clear, simple, personalized, culturally relevant, actionable, and practical) is framed as shared or collective. | The persona tries to change but faces setbacks due to social situations. They adapt by involving others or adjusting timing. | Positive changes are reinforced socially. The larger task is completed with support. The persona feels motivated to sustain change within their community. |
Table 10: Story Arcs narrative progession
Figure 8: Cultural alignment taxonomy with examples from FGD sessions.
| Community | Kimi | Gemma4 | GPT-5.5 | Qwen-122B | Qwen-9B |
|---|---|---|---|---|---|
| Hausa | +1 | +7 | +3 | +7 | +6 |
| AmaXhosa | -1 | +4 | 0 | +15 | +12 |
| Kikuyu | -16 | -7 | -15 | -3 | -2 |
| Luo | -6 | +1 | -4 | +6 | +4 |
| Xichangana | +38 | +44 | +36 | +45 | +45 |
Table 11: Score bias, computed as judge mean minus culture-representative mean, by community. Bold values indicate significant paired -test differences between judge and human scores. Xichangana inflation is systematic across all judges.
| Community | #Annotators | ICC(A,1) | ICC(A,k) | Mean Score |
|---|---|---|---|---|
| Hausa | 3 | 0.81 | 0.93 | 57.4 |
| AmaXhosa | 2 | 0.63 | 0.77 | 62.8 |
| Kikuyu | 4 | 0.60 | 0.86 | 76.2 |
| Luo | 6 | 0.63 | 0.91 | 65.6 |
| Xichangana | 4 | 0.33 | 0.66 | 24.5 |
Table 12: Human annotator agreement per community. ICC(A,1) = absolute agreement for a single rater. ICC(A,k) = absolute agreement averaged across raters.
Figure 13: Distribution of connected cultural markers in text and images across the five broader cultural marker categories, broken down by generation model and community. This figure complements Figure 3 , which shows not-connected markers.
| Conn. — Exact | Conn. — Partial | Not C. — Exact | Not C. — Partial | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Community | # Ann. | # Stories | Tot. | Uniq. | Ovlp. | % | Uniq. | Ovlp. | % | Tot. | Uniq. | Ovlp. | % | Uniq. | Ovlp. | % |
| Hausa | 3 | 40 | 1,684 | 1,340 | 178 | 13.3 | 1,049 | 321 | 30.6 | 291 | 169 | 54 | 32.0 | 151 | 59 | 39.1 |
| AmaXhosa | 2 | 40 | 781 | 674 | 95 | 14.1 | 644 | 103 | 16.0 | 207 | 191 | 6 | 3.1 | 187 | 9 | 4.8 |
| Kikuyu | 4 | 39 | 2,068 | 1,570 | 263 | 16.8 | 1,323 | 345 | 26.1 | 94 | 65 | 18 | 27.7 | 56 | 18 | 32.1 |
| Luo | 6 | 40 | 2,453 | 1,674 | 331 | 19.8 | 1,277 | 416 | 32.6 | 291 | 183 | 45 | 24.6 | 157 | 46 | 29.3 |
| Xichangana | 4 | 40 | 1,089 | 798 | 149 | 18.7 | 682 | 172 | 25.2 | 428 | 329 | 58 | 17.6 | 376 | 23 | 6.1 |
Table 13: Text span annotation statistics per language. Connected : spans marked as culturally connected. Not connected : spans marked as not culturally connected. Deduplication is shown at two levels: Exact (identical span text within a task) and Partial (token overlap 50% of the shorter span with same category). Overlap : spans independently identified by more than one representative.
| Connected | Not Connected | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Community | #Ann. | # Stories | Tot. | Uniq. | Ovlp. | % | Tot. | Uniq. | Ovlp. | % |
| Hausa | 3 | 40 | 1,854 | 1,476 | 323 | 21.9 | 960 | 773 | 147 | 19.0 |
| AmaXhosa | 2 | 40 | 559 | 509 | 50 | 9.8 | 430 | 410 | 20 | 4.9 |
| Kikuyu | 4 | 39 | 1,246 | 1,033 | 184 | 17.8 | 219 | 205 | 13 | 6.3 |
| Luo | 6 | 40 | 2,141 | 1,476 | 447 | 30.3 | 324 | 272 | 40 | 14.7 |
| Xichangana | 4 | 40 | 1,131 | 941 | 167 | 17.7 | 555 | 460 | 70 | 15.2 |
Table 14: Image region annotation statistics per language. Unique regions are identified by clustering bounding boxes with IoU (Intersection over Union) 0.3 across annotators. Overlap : regions independently identified by more than one representative.
| Language | Span F1 | Category Agreement |
|---|---|---|
| Hausa | 0.474 | 0.735 |
| AmaXhosa | 0.420 | 0.646 |
| Kikuyu | 0.447 | 0.678 |
| Luo | 0.418 | 0.590 |
| Xichangana | 0.360 | 0.723 |
| Average | 0.424 | 0.674 |
Table 15: Human inter-annotator agreement on cultural marker spans (pairwise average). Only spans marked as culturally connected are included.
| Language | Span F1 | Category Agreement |
|---|---|---|
| Hausa | 0.819 | 0.908 |
| AmaXhosa | 0.813 | 0.909 |
| Kikuyu | 0.793 | 0.914 |
| Luo | 0.806 | 0.887 |
| Xichangana | 0.679 | 0.757 |
| Average | 0.782 | 0.875 |
Table 16: LLM judge inter-run agreement (pairwise average across 3 runs).
| Strict | Partial | Type | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Language | P | R | F1 | P | R | F1 | P | R | F1 |
| Hausa | .080 | .080 | .080 | .196 | .197 | .196 | .459 | .461 | .460 |
| AmaXhosa | .054 | .110 | .072 | .129 | .264 | .173 | .305 | .623 | .409 |
| Kikuyu | .098 | .068 | .080 | .268 | .185 | .219 | .588 | .407 | .481 |
| Luo | .131 | .091 | .107 | .285 | .198 | .233 | .637 | .441 | .521 |
| Xichangana | .112 | .108 | .110 | .280 | .270 | .275 | .460 | .444 | .451 |
Table 17: LLM judge vs. human ground truth (averaged across 3 runs). P = Precision, R = Recall, F1 = F1 score.
More in this topic
Cultural Alignment
Cultural alignment studies how AI systems represent and respond to the values, norms, languages, and lived contexts of different communities.
See all 46 papers