Lessons Without Borders? Evaluating Cultural Alignment of LLMs Using Multilingual Story Moral Generation

Sophie Wu; Andrew Piper

cs.CLApr 9, 2026

Lessons Without Borders? Evaluating Cultural Alignment of LLMs Using Multilingual Story Moral Generation

Authors: Sophie Wu, Andrew Piper

Organizations: Languages, Literatures, and Cultures McGill University

Abstract

Stories are key to transmitting values across cultures, but their interpretation varies across linguistic and cultural contexts. Thus, we introduce multilingual story moral generation as a novel culturally grounded evaluation task. Using a new dataset of human-written story morals collected across 14 language-culture pairs, we compare model outputs with human interpretations via semantic similarity, a human preference survey, and value categorization. We show that frontier models such as GPT-4o and Gemini generate story morals that are semantically similar to human responses and preferred by human evaluators. However, their outputs exhibit markedly less cross-linguistic variation and concentrate on a narrower set of widely shared values. These findings suggest that while contemporary models can approximate central tendencies of human moral interpretation, they struggle to reproduce the diversity that characterizes human narrative understanding. By framing narrative interpretation as an evaluative task, this work introduces a new approach to studying cultural alignment in language models beyond static benchmarks or knowledge-based tests.

Figures & tables

Figure 1: Schematic of different factors potentially influencing story moral generation. We assume that culture and language may affect the story content, as well as the interpretation of a human reader engaging with the story, to produce a story moral.

Figure 2: Visual schematic of our data generation framework for this project.

Figure 3: Fixed-effect estimates of the intra-lingual similarity gap between Human–Human (HH) and Human–Model (HM) moral pairs. The vertical reference line indicates the human baseline (HH agreement). Values at or above the reference line indicate model similarity to human annotations that meets or exceeds typical within-language human agreement.

Figure 4: Fixed-effect estimates of the cross-lingual similarity gap between Model–Model (MM) and Human–Human (HH) moral pairs. The vertical reference line indicates the human baseline (HH cross-lingual agreement). Values to the right of the line indicate higher cross-lingual similarity than humans, reflecting reduced cultural differentiation.

	Story
Culture	in	out
in $\bigstar$	in story in culture	out story in culture
$\textbf{out}\boldsymbol{\times}$	in story out culture	out story out culture

Table 2: 2×2 design for validating human preferences for story morals. Color indicates story condition (green = in-story, red = out-story), while shading indicates relevance (top left = most relevant, bottom left = least).

Figure 5: Validation survey results asking participants to choose their preferred story moral for a given story using our 2x2 design. All LLM morals are generated by GPT-4o and compared to each type of human moral.

Figure 6: Schwartz’s values as a percentage of morals generated by LLM and human annotators. Multiple values can be present per moral. Value labels are generated by GPT-4o (top) and Gemini 2.5 Flash (bottom).

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Visualization of our dataset’s passage generation. Each story is translated to all languages represented in our dataset. Green squares indicate original passages in original languages and yellow squares indicate passages generated from stories translated to all other languages represented in our dataset.

Book Title	WikiID	Country of Origin	Language
La donna dei fiori di carta	Q3822099	Italy	Italian
Der Untergang	Q1197714	Germany	German
Les Petits Enfants du siècle	Q1741761	France	French
Astronauci	Q931113	Poland	Polish
(The Woman in the Dunes)	Q6455599	Japan	Japanese
(Murder on the Way to Bethlehem)	Q5708155	Israel	Hebrew

Appendix

Table 3: List of books with plot summaries used in this study with corresponding WikiID, country of origin, and inferred language.

Figure 8: Screenshot of English survey for story moral generation on Prolific. Note that survey would be presented in different languages depending on the language of the passage presented.

Human Moral	GPT-4o Moral	Discarded
“Letting go of the one you love can be the greatest act of love.”	“Love and fate can be unpredictable and sometimes tragic, but letting go can be the greatest act of love.”	Yes
“Violence and revenge can lead to everyone’s downfall.”	“Self-destructive conflicts and wars can lead to the downfall of a civilization.”	Yes
“Many unexpected things happen in life despite careful planning.”	“Taking responsibility and overcoming difficulties can lead to unexpected opportunities and changes in life.”	No
“Accepting one’s fate can make others sad.”	“Sometimes a sense of solidarity and adaptation to a new life situation can overcome the desire to escape, even if it means giving up previous freedoms.”	No
“Freedom can easily be taken for granted, but its loss reveals the human ability to adapt or give up.”	“Sometimes a sense of solidarity and adaptation to a new life situation can overcome the desire to escape, even if it means giving up previous freedoms.”	No

Appendix

Table 4: Examples of human-generated morals flagged for high semantic similarity to LLM-generated morals during manual inspection, alongside the corresponding GPT-4o-generated moral for the same passage. “Discarded” indicates whether the human moral was removed and replaced due to suspected AI assistance or generation within this pair.

	Coef.	SE	z	p
Intercept	0.40	0.024	17.004	< .001
Both Translated	-0.0	0.021	-1.245	.213
Moral1 Length	0.003	0.001	3.084	.002
Moral2 Length	0.002	0.001	2.937	.003
Random Intercept Variance	0.004
Residual Variance	0.019

Appendix

Table 5: Results for linear mixed-effects regression predicting semantic similarity in the translation effects experiment. Reference condition is original-language summaries.

Predictor	Coef.	SE	z	p	95% CI
Intercept	0.385	0.006	69.362	$<$ .001	[0.375, 0.396]
Interlingual	$-$ 0.010	0.004	$-$ 2.788	.005	[ $-$ 0.018, $-$ 0.003]
Avg. Word Count	0.004	0.000	20.575	$<$ .001	[0.004, 0.005]
Random Effects
Embedding Model (Var.)	0.008	0.005
Language Pair (Var.)	0.001	0.001

Appendix

Table 6: H2 - Regression effects for cultural effects experiment.

Figure 9: RQ1 - Results for mixed effects regression displaying Model-Human vs Human-Human advantage in terms of semantic similarity for each LLM model tested in our experiments for each language in our dataset. Asterisks (*) indicate a statistically significant result.

Model	Human-Human Sim.	Model-Human Sim.	Difference	Coefficient	95% CI	$p$ -value
Gemini 2.5	0.429 ± 0.152	0.473 ± 0.138	0.044	0.035	[0.025, 0.045]	$<$ 0.001 ***
GPT-4o	0.429 ± 0.152	0.472 ± 0.136	0.042	0.019	[0.009, 0.030]	0.0005 ***
Phi3 (3.8B)	0.429 ± 0.152	0.403 ± 0.149	-0.026	-0.018	[-0.028, -0.008]	0.0005 ***
Gemma3 (4B)	0.429 ± 0.152	0.402 ± 0.149	-0.027	-0.019	[-0.029, -0.009]	0.0003 ***
Aya (8b)	0.429 ± 0.152	0.340 ± 0.146	-0.089	-0.081	[-0.091, -0.071]	$<$ 0.001 ***
Aya (35b)	0.429 ± 0.152	0.340 ± 0.145	-0.090	-0.082	[-0.092, -0.072]	$<$ 0.001 ***

Appendix

Table 7: RQ1 - Mixed-effects regression results for model-human similarity. Each row represents a separate mixed-effects regression comparing human-human similarity to model-human similarity.

Figure 10: Fixed-effect estimates of the intra-lingual similarity gap between Human–Human (HH) and Human–Model (HM) moral pairs with the inclusion of morals that were originally excluded due to similarity to model-generated morals. The vertical reference line indicates the human baseline (HH agreement). Values at or above the reference line indicate model similarity to human annotations that meets or exceeds typical within-language human agreement.

Figure 11: Fixed-effect estimates of the cross-lingual similarity gap between Model–Model (MM) and Human–Human (HH) moral pairs with the inclusion of morals that were originally excluded due to similarity to model-generated morals. The vertical reference line indicates the human baseline (HH cross-lingual agreement). Values to the right of the line indicate higher cross-lingual similarity than humans, reflecting reduced cultural differentiation.

Model	Coefficient (SE)	95% CI	$p$ -value	Cohen’s $d$	Model Mean	Improvement over Human
Gemini	0.132 (0.003)	[0.127, 0.137]	$<$ 0.001 ***	1.03	0.571	36.6%
GPT-4o	0.220 (0.003)	[0.213, 0.226]	$<$ 0.001 ***	1.75	0.677	62.2%
Phi3	0.186 (0.003)	[0.181, 0.192]	$<$ 0.001 ***	1.14	0.587	40.6%
Gemma3	0.183 (0.003)	[0.178, 0.189]	$<$ 0.001 ***	1.12	0.584	39.8%
Aya:8b	0.140 (0.003)	[0.134, 0.146]	$<$ 0.001 ***	0.83	0.541	29.5%
Aya:35b	0.159 (0.003)	[0.154, 0.165]	$<$ 0.001 ***	0.97	0.561	34.3%

Appendix

Table 8: RQ2 -Mixed-effects regression results for inter-lingual similarity. Each row represents a separate mixed-effects regression evaluating inter-lingual similarity for morals generated from that source.

Language	Original Prompt	In-Language Prompt
ar	Blind jealousy and personal biases can lead to the destruction of relationships and bring unhappiness to everyone.	Clinging to biases and jealousy can lead to personal and family decline, and prevent the realization of painful truths until it’s too late.
cs	The inability to confront one’s own prejudices and jealousy can lead to personal and familial decline.	The inability to confront one’s own prejudices and mistakes can lead to personal and family decline, despite the rich history and traditions that surround us.
de	Self-reflection and overcoming prejudices are crucial for achieving personal happiness and interpersonal understanding.	Self-deception and clinging to outdated prejudices can lead to personal and collective decline.
en	Failing to confront and learn from one’s past mistakes leads to a life of regret and unfulfilled potential.	Failing to confront one’s own prejudices and insecurities can lead to personal and relational decay, leaving a legacy of regret and misunderstanding.
fr	Jealousy and prejudice can destroy lives and hinder personal fulfillment, leaving bitter regrets and unresolved dramas.	The inability to confront one’s own mistakes and prejudices can lead to a life of regrets and disillusionment, highlighting the importance of lucidity and self-acceptance to avoid personal and collective decay.
he	Jealousy and prejudice can destroy lives and prevent people from realizing themselves and their love.	It is important to confront prejudices and jealousy, as the inability to do so can lead to personal and family decline and prevent people from realizing their full potential.

Appendix

Table 10: GPT-4o-generated morals comparing results from Original Prompt (English) vs. In-Language Prompt, all based on story sourced from Brazil.

Language	Original Prompt	In-Language Prompt
ar	Money and status fade away, but good deeds and a good reputation remain.	The real tragedy lies in a person telling the story of their loss without realizing their role in this misery; blind jealousy blinds insight until the very last moment.
cs	Even when lives are falling apart, it is important to maintain honor and not succumb to prejudice.	The true tragedy lies in a person’s inability to confront their own prejudices and mistakes, for only then can they avoid personal and societal decline.
de	One should be aware that one’s own prejudices and jealousy can destroy happiness not only for oneself but also for others.	The inability to recognize one’s own mistakes and prejudices leads to a life full of regret and unfulfilled possibilities.
en	Holding onto past grievances and prejudices can lead to a life of regret and missed opportunities.	A refusal to confront one’s own prejudices and personal failings can lead to a lifetime of regret and tragedy.
fr	One should never let regrets and unspoken words ruin a life.	The denial of personal truths and social realities inevitably leads to decadence.
he	Even when life is deteriorating, it is important to remember the past and learn from it.	Blind adherence to prejudices, jealousy, and an inability to confront the truth lead to inevitable decline, both personally and socially.

Appendix

Table 11: Extracted morals by language: Original Prompt vs. In-Language Prompt.

Figure 12: Screenshot of our English validation human preference survey. All participants will be shown five pairs of morals, where they are asked to select for each pair which moral they believe to be more appropriate to the story.

Value	Human	GPT-4o	Gemini
Security	17.3	44.9	24.0
Self-direction	16.2	42.9	33.2
Benevolence	12.9	33.2	26.5
Universalism	13.9	29.1	17.3
Conformity	6.8	8.7	8.2
Achievement	3.6	2.0	4.1

Appendix

Table 12: Schwartz’s values as a percentage of morals (GPT-4o annotations)

Value	Human	GPT-4o	Gemini
Security	65.1	73.0	70.4
Self-direction	42.0	64.8	61.7
Benevolence	29.3	41.3	38.8
Universalism	25.5	39.8	36.2
Conformity	24.3	26.5	28.1
Achievement	14.3	17.3	17.3

Appendix

Table 13: Schwartz’s values as a percentage of morals (Gemini 2.5 Flash annotations)

Schwartz Value	Both Positive Moral (English)	Disagreement Moral (English)	GPT-4o	Gemini
Power	“The quest for power and the division among factions can lead to the self-destruction of a civilization, highlighting the importance of unity and peace for the survival of humanity.”	“The solidarity born out of coercion and compromise ultimately leads to the loss of individual freedom.”	0	1
Achievement	“Taking responsibility and striving for education can open doors to a better life, even when circumstances are difficult.”	“Even in adversity, one can find strength and create something new.”	0	1
Hedonism	“Life is full of unexpected twists, which is why it is important to appreciate what we have and to seek happiness in the present.”	“Do not blindly trust the temptations that cloud your senses, but always maintain a clear mind to recognize and combat evil.”	0	1
Stimulation	“Despite the difficulties and challenges in life, it is important to maintain hope and seek new opportunities for change and personal growth.”	“True courage is revealed in overcoming adversity and in protecting those we love, even in the face of danger.”	0	1
Self-direction	“Even after everything, life goes on, and sometimes you need to move forward.”	“Truth and justice may require courage and determination, but it is worth striving for their revelation, even if it requires sacrifices.”	0	1
Universalism	“It is important to learn from past mistakes and build a peaceful future.”	“Communication and cooperation between generations can be the key to overcoming challenges and achieving success in the face of difficulties.”	1	0

Appendix

Table 14: For each Schwartz value, we report one English-translated moral jointly labeled positive by GPT-4o and Gemini, and one English-translated moral where their labels disagree.

Cross-Cultural Language Model Evaluation

Cross-cultural language model evaluation tests whether AI systems understand and respond appropriately across languages, communities, and cultural contexts.

See all 121 papers

Same Lesson, Different Story: Cross-Lingual Reconstruction of Cultural Narratives in Large Language Models
Jun 23, 2026Jory Alshaalan, Haya Albaker, Abeer Aldayel +2Multilingual Language Model Evaluation LLM Grounding
College of Computer and Information Sciences King Saud University, Saudi Arabia
One Model, Many Morals: Uncovering Cross-Linguistic Misalignments in Computational Moral Reasoning
Sep 25, 2025Sualeha Farid, Jayden Lin, Zean Chen +2Multilingual Language Model Evaluation Moral Reasoning in Language Models
University of Michigan, Ann Arbor, USA
BIASEDTALES-ML: A Multilingual Dataset for Analyzing Narrative Attribute Distributions in LLM-Generated Stories
Apr 18, 2026Yuxuan Ouyang, yingfeng luo, JingBo Zhu +1Multilingual Language Model Evaluation Social Bias in Language Models
School of Computer Science and Engineering, Northeastern University, Shenyang, China · NiuTrans Research, Shenyang, China

Jun 23, 2026J/K move · Enter open · S save

Same Lesson, Different Story: Cross-Lingual Reconstruction of Cultural Narratives in Large Language Models

Jory Alshaalan, Haya Albaker, Abeer Aldayel, Aljawharah Alabdullatif+1

College of Computer and Information Sciences King Saud University, Saudi Arabia

Multilingual Language Model Evaluation LLM Grounding Text Generation Cross-Lingual Transfer

Open paper AskPDF arXiv

The evaluation of cultural grounding context becomes complex when multiple cultures convey the same moral lesson. This challenge is particularly relevant to large language models (LLMs), which produce narratives across a wide range of languages and cultural contexts. However, it remains uncertain whether these models preserve culturally grounded meaning when equivalent moral lessons are conveyed through distinct cultural forms. This study introduces a multilingual evaluation narrative framework that integrates a cross-linguistic collection of 414 proverbs spanning 15 languages and uses four LLMs to generate 13k narratives. By employing semantically equivalent proverbs as culturally grounded prompts, the analysis assesses whether models preserve meaning across languages, how cross-lingual conditioning influences narrative realization, and whether different model families converge on similar interpretations. Results indicate that cross-lingual prompting largely preserves proverb-level semantic meaning while systematically redistributing agency, social positioning, and narrative structure. Additionally, strong inter-model convergence is observed in both monolingual and cross-lingual settings, suggesting that multilingual LLMs rely on shared semantic abstractions despite architectural and linguistic differences. These findings shed light on the need for more comprehensive evaluations of cultural grounding. Relying exclusively on semantic similarity in multilingual narrative assessments may overestimate cultural preservation by neglecting culturally meaningful variations in narrative expression.

Abstract

Cross-Cultural Language Model Evaluation

Explore similar work

Same Lesson, Different Story: Cross-Lingual Reconstruction of Cultural Narratives in Large Language Models

One Model, Many Morals: Uncovering Cross-Linguistic Misalignments in Computational Moral Reasoning

BIASEDTALES-ML: A Multilingual Dataset for Analyzing Narrative Attribute Distributions in LLM-Generated Stories