Large Language Models (LLMs) are increasingly being deployed in multilingual, multicultural settings, yet their reliance on predominantly English-centric training data risks misalignment with the diverse cultural values of different societies. In this paper, we present a comprehensive, multilingual audit of the cultural alignment of contemporary LLMs including GPT-4o-Mini, Gemini-2.5-Flash, Llama 3.2, Mistral and Gemma 3 across India, East Asia and Southeast Asia. Our study specifically focuses on the sensitive domain of religion as the prism for broader alignment. To facilitate this, we conduct a multi-faceted analysis of every LLM's internal representations, using log-probs/logits, to compare the model's opinion distributions against ground-truth public attitudes. We find that while the popular models generally align with public opinion on broad social issues, they consistently fail to accurately represent religious viewpoints, especially those of minority groups, often amplifying negative stereotypes. Lightweight interventions, such as demographic priming and native language prompting, partially mitigate but do not eliminate these cultural gaps. We further show that downstream evaluations on bias benchmarks (such as CrowS-Pairs, IndiBias, ThaiCLI, KoBBQ) reveal persistent harms and under-representation in sensitive contexts. Our findings underscore the urgent need for systematic, regionally grounded audits to ensure equitable global deployment of LLMs.
Figures & tables
Figure 1: Evaluation framework for assessing LLMs, where human opinion distributions from Pew surveys (India, Sri Lanka, East Asia, and South East Asia) are compared with model-generated distributions to measure representativeness across various categories.
Figure 2: The treemap shows respondent counts from 12 countries/territories across India (green), East Asia (blue), and Southeast Asia (orange), with block area proportional to sample size. These nationally representative surveys (Pew Research Center 2021, 2024a, 2024b) form the empirical ground truth for measuring LLM representativeness, enabling robust cross-country comparisons on religion and social attitudes.
Country
Prompt Languages
Country
Prompt Languages
IND
en, hi
KHM
en, km
HKG
en, zh-Hant
IDN
en, id
JPN
en, ja
MYS
en, ms, zh-Hans
KOR
en, ko
SGP
en, ms, zh-Hans, ta
TWN
en, zh-Hant
LKA
en, si, ta
VNM
en, vi
THA
en, th
Table 1: Language coverage by country: For every country, prompts were issued in both English and one or more local languages to facilitate a within-country analysis of language effects. Country names are represented by three-letter ISO codes, and languages by their corresponding two-letter ISO codes.
Dataset
Primary format
CrowS-Pairs
Pairwise Sentences
IndiBias
Pairwise / Judgment (English + Hindi)
ThaiCLI
Question / Chosen / rejected
KoBBQ
QA / templates (template-expanded)
Table 2: Cross-cultural benchmarks used to evaluate bias and representational harms in LLMs. These corpora span multiple regions and task formats—pairwise judgments (CrowS-Pairs, IndiBias), culturally grounded question-answering (KoBBQ), and culturally sensitive response scoring (ThaiCLI).
Figure 3: Qualitative examples for two religion-related bias benchmarks. CrowS-Pairs operationalizes bias via minimal pairs scored with pseudo-log-likelihood, while ThaiCLI uses instruction-preference judgments with explicit chosen vs. rejected responses.
Figure 4: Representativeness scores ( RM ) of GPT-4o-Mini and Gemini-2.5-Flash on non-religious versus religious items. While both models achieve high representativeness on non-religious prompts ( > 94%), their scores dip on religious items.
Figure 5: Change in Hellinger distance ( ΔH=Hlocal−HEN ) when switching from English to local-language prompts for Gemma-3-12B across multiple locales. Negative values (bars below zero) indicate that local-language prompting reduces the divergence between model and human distributions.
Figure 6: Effect of prompt language on religion-related items across distinct model–country pairs (Kr = Korea, Tw = Taiwan, SL = Sri Lanka). Lower Jensen–Shannon distance indicates better alignment.
Model
Region (Lang)
RM
AJSD (Eng → Loc)
AHD (Eng → Loc)
Gemma-3 12B-IT
Sri Lanka (Sinhala)
0.96
0.47→0.32
0.49→0.47
Llama-3.2 1B-Instruct
Taiwan (Chinese)
0.95
0.88→0.81
0.86→0.86
Mistral-7B Instruct-v0.3
Korea (Korean)
0.91
0.53→0.48
0.48→0.48
Table 3: Impact of local-language prompting: switching to local languages consistently reduce divergence ( AJSD ), while Hellinger Distances ( AHD ) remain stable.
Figure 7: Cross-lingual bias rates on the religion-only subset of CrowS-Pairs across six languages (105 items/locale). GPT-4o-Mini shows low bias ( ≈ 8%) and high anti-stereotype accuracy ( ≈ 92%) consistently across languages. Gemini-2.5-Flash exhibits higher bias ( ≈ 16%), lower anti-stereotype accuracy ( ≈ 68%), and more invalid responses, indicating weaker cross-lingual stereotype resistance.
Figure 8: Misrepresentation of Indian religious identities on IndiBias using GPT-4o-Mini. The plot shows ΔELO=ELOneg−ELOpos , where positive values indicate that negative descriptions are judged more plausible than positive ones. Several minority identities (Shia, Sunni, Jain, Parsi) show strong misrepresentation, while others (Hindu, Sikh, Sufi) show the opposite trend, highlighting systematic group-specific calibration gaps.
Figure 9: Effect of prompt disambiguation on GPT-4o-Mini performance on the KoBBQ Korean identity benchmark. Disambiguating prompts improves overall accuracy (0.611 → 0.961) and religion-related accuracy (0.625 → 0.950), while sharply reducing bias (0.275 → 0.100).
As Large Language Models (LLMs) develop stronger multilingual capabilities, their sensitivity to culturally diverse entities becomes increasingly important. Prior work by Naous et al. (2024) has shown that LLMs often favor Western-associated entities in Arabic. Due to the lack of entity-centric multilingual benchmarks, it remains unclear if such biases also manifest in various non-Western languages. In this paper, we introduce Camellia, a benchmark for evaluating entity-centric cultural biases in nine Asian languages, spanning six Asian cultures. Camellia includes 19,530 manually annotated entities associated with the covered Asian or Western cultures, as well as 2,173 masked contexts for these entities derived from social media posts. Using Camellia, we evaluate cultural biases in four recent multilingual LLMs across three tasks: cultural context adaptation, sentiment association, and entity extractive QA. Our analyses show that LLMs struggle with cultural adaptation across these languages, with performance differing across models developed in different regions. We further observe that different LLM families can hold distinct biases, reflected in the ways they link cultures to particular sentiments. Lastly, we find that LLMs can struggle with context understanding in some Asian languages, creating performance gaps between cultures in entity extraction.
Tarek Naous, Anagha Savit, Carlos Rafael Catalan +17
Georgia Institute of Technology · Samsung R&D Institute Philippines · Sungkyunkwan University +6
Although the cultural (mis)alignment of Large Language Models (LLMs) has attracted increasing attention -- often framed in terms of cultural bias -- until recently there has been limited work on the design and development of datasets for cultural assessment. Here, we review existing approaches to such datasets and identify their main limitations. To address these issues, we propose design guidelines for annotators and report on the construction of a dataset built according to these principles. We further present a series of contrastive experiments conducted with this dataset. The results demonstrate that our design yields test sets with greater discriminative power, effectively distinguishing between models specialized for a given culture and those that are not, ceteris paribus.
António Branco, João Silva, Nuno Marques +10
†University of Lisbon · ‡University of Beira Interior
As large language models (LLMs) are deployed in multilingual settings, their safety behavior in culturally diverse, low-resource languages remains poorly understood. We present the first systematic evaluation of LLM safety across 12 Indic languages, spoken by over 1.2 billion people but underrepresented in LLM training data. Using a dataset of 6,000 culturally grounded prompts spanning caste, religion, gender, health, and politics, we assess 10 leading LLMs on translated variants of the prompt. Our analysis reveals significant safety drift: cross-language agreement is just 12.8%, and SAFE rate variance exceeds 17% across languages. Some models over-refuse benign prompts in low-resource scripts, overflag politically sensitive topics, while others fail to flag unsafe generations. We quantify these failures using prompt-level entropy, category bias scores, and multilingual consistency indices. Our findings highlight critical safety generalization gaps in multilingual LLMs and show that safety alignment does not transfer evenly across languages. We release IndicSafe, the first benchmark to enable culturally informed safety evaluation for Indic deployments, and advocate for language-aware alignment strategies grounded in regional harms.