Cross-Cultural Language Model Evaluation

Follow this topic Ask about this topic

This field studies how language models perform across cultures, including whether they preserve local meaning, represent community values, and behave safely in different settings. Researchers build native-speaker-validated benchmarks for tasks such as question answering, translation, social interaction, bias, and safety, and compare outputs with human judgments or survey distributions. Methods include paired language tests, culturally grounded prompts and evidence, distributional metrics, and statistical tools that separate language proficiency from knowledge. A central challenge is measuring cultural fit without treating a population as uniform or imposing a single standard, while making results reliable across prompts, evaluators, and subgroups.

Latest papers

121 papers26 papers in the last four weeks, up 136% on the four weeks before. 0.2% of all new papers.

Evaluating Cultural Awareness of LLMs for Haitian Creole
Sep 25, 2026Christelle Clervilsson, Yanzhu GuoCultural Knowledge in Language Models Cultural Bias in Language Models
Telecom Paris, Institut Polytechnique de Paris, France
From annotation to reasoning: Culture in language models
Sep 25, 2026Daniel Hershcovich, Alexander Conroy, Jens Bjerring-HansenCultural Knowledge in Language Models Cross-Cultural Language Model Evaluation
Department of Computer Science, University of Copenhagen, Vermundsgade 5, 2100, Copenhagen, Denmark · Department of Nordic Studies and Linguistics, University of Copenhagen, Copenhagen, Denmark
CIBuzzBench: A Benchmark for Cross-Lingual Understanding of Chinese Internet Buzzwords
Sep 18, 2026Yifan Wang, Junyu Lu, Qifan Wang +6Cross-Cultural Language Model Evaluation Cultural Knowledge in Language Models
People’s Public Security University of China · Dalian University of Technology · Meta AI +3
Geometry of Values: Task Vector Composition for Ethical Preference Alignment in Language Models
Sep 17, 2026Utkarsh Agarwal, Monojit ChoudhuryLLM Alignment Moral Reasoning in Language Models
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE.
GYROval: A Robust Benchmark for Cultural Value Orientation in Large Language Models
Sep 16, 2026Alexander Didenko, Anna Shabanova, Vladislav Zapylikhin +2LLM Evaluation Cross-Cultural Language Model Evaluation
Artificial Intelligence Laboratory, SKOLKOVO School of Management, Moscow · Artificial Intelligence Laboratory, University of Tyumen · Institute of Business Studies, Russian Presidential Academy of National Economy and Public Administration, Moscow
LLMs Mirror Country-Specific Gender Patterns If Asked, but Skew Male When Generating Media in Local Languages
Sep 6, 2026Sharif Kazemi, Tanya Popli, Neil K. R. Sehgal +7Gender Bias in Language Models Language Model Bias Evaluation
World Bank Group · Cornell University · University of Pennsylvania +2
CC-Mediation: Evaluating Large Language Models for Cross-Cultural Conflict Mediation
Sep 4, 2026Suhyun Lee, Wenxuan Zhang, W. Quin Yow +1Cross-Cultural Language Model Evaluation
Department of Artificial Intelligence, Hanyang University, Seoul, Republic of Korea · Singapore University of Technology and Design, Singapore · School of Computing and Information Systems, Singapore Management University, Singapore
Cultural Misalignment in Large Language Models: Detection, Measurement, and Mitigation Through Targeted Fine-Tuning
Sep 3, 2026Antoni Czolgowski, Abel IyaseleCultural Bias in Language Models Social Bias in Language Models
Masters in Data Science Program University of Colorado Boulder Boulder, CO 80309, USA · Department of Information Science University of Colorado Boulder Boulder, CO 80309, USA
WorldBench: Culturally Grounded Benchmark for Multilingual Agents
Sep 1, 2026Leonardo Ranaldi, Sherrie Shen, Jushi Kai +1Cross-Cultural Language Model Evaluation LLM Agent Evaluation
(•) ILCC, School of Informatics, University of Edinburgh · (•,◦) ILCC, School of Informatics, University of Edinburgh · (◦) School of Artificial Intelligence, Shanghai Jiao Tong University
The Failure Happens Before the Drift: The Social Dynamics of Values in LLM Agent Societies
Aug 31, 2026Farah Atif, Sougata Saha, Monojit ChoudhuryLarge Language Model-Based Social Simulation Cross-Cultural Language Model Evaluation
Mohamed Bin Zayed University of Artificial Intelligence
CoCoA: Context-Conditional Cultural Alignment for Large Language Models
Aug 30, 2026Kyungdon Lee, Wei Xu, Alan Ritter +2Cultural Bias in Language Models Language Model Debiasing
Sungkyunkwan University, Suwon, South Korea · Georgia Institute of Technology, GA, USA · University of Southern California, CA, USA
Measurement Validity in LLM Cultural Alignment
Aug 29, 2026An Duy Nguyen, Muhammad Aurangzeb AhmadCross-Cultural Language Model Evaluation Cultural Bias in Language Models
University of Washington · University of Washington Bothell
CultureConverse: A Multilingual Multi-turn Simulation Harness for Culturally Grounded Assistance in East and Southeast Asia
Aug 28, 2026Bryan Chen Zhengyu Tan, Weihua Zheng, Thong T. Doan +30Cross-Cultural Language Model Evaluation Cultural Knowledge in Language Models
Singapore University of Technology and Design (SUTD), Singapore · Agency for Science, Technology and Research (A*STAR), Singapore · École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland +13
The Shape of Power: A Multilingual Framework for Social Power Reasoning in Dialogues
Aug 28, 2026Farah Atif, Sougata Saha, Monojit ChoudhurySocial Reasoning Cross-Cultural Language Model Evaluation
Mohamed bin Zayed University of Artificial Intelligence
PragAlign: Evidence-Sensitive Reply Assistance Across Chinese and Japanese Appropriateness Judgments
Aug 27, 2026Xin Zhong, Satori HachisukaPragmatic Reasoning in Language Models Cultural Alignment
The University of Tokyo
Apples to Apples? Towards Comparable Crosslingual Language Model Evaluation
Aug 25, 2026Xiulin Yang, Ethan Gotlieb Wilcox, Catherine ArnettMultilingual Language Model Evaluation LLM Evaluation
Georgetown University · EleutherAI
Beyond Surface Cues: Disentangling Sociocultural Signals in Multilingual LLMs
Aug 24, 2026Yuanjun Feng, Tanzhou Liu, Stefan Feuerriegel +1Cross-Cultural Language Model Evaluation Cultural Bias in Language Models
University of Lausanne, Switzerland · LMU Munich, Munich Center for Machine Learning (MCML), Germany
Register Shifts Break LLM Safety: A Bengali Benchmark with Culturally Grounded Harms
Aug 23, 2026Naymul Islam, Nusrat Jahan Lia, Shubhashis Roy Dipta +2LLM Safety Benchmarks Language Model Safety Evaluation
BanglaLLM · Institute of Information Technology, University of Dhaka · University of Maryland, Baltimore County +2
Redteaming Leading Arabic LLMs with ASAS
Aug 22, 2026Fidaa Abed, Haidar Khan, M Saiful Bari +2LLM Safety Benchmarks Language Model Safety Evaluation
AI Astrolabe
Figurative and Cultural Knowledge in LLMs: Investigating Cross-Domain Transfer through Fine-Tuning
Aug 18, 2026Mena Attia, Mona Diab, Thamar SolorioFigurative Language Understanding Cultural Knowledge in Language Models
MBZUAI · Carnegie Mellon University
Inspect India Evals: An Open Benchmarking Framework for Evaluating Large Language Models in the Indian Linguistic and Cultural Context
Jul 28, 2026Abhishek Kumar Singh, Shrey Nag, Sachita +2LLM Safety Benchmarks Multilingual Language Model Evaluation
GTBIT¹ · TIET² · IGDTUW³ +2
ADAGE: A Language-Agnostic Pipeline for Analogical Reasoning Evaluation
Jul 25, 2026Ahmed Haj Ahmed, Alvin GrissomMultilingual Language Model Evaluation Reasoning Evaluation
Haverford College
Pluralis v0.1: Towards a Multicultural, Multimodal, Multilingual Benchmark for AI Risk and Reliability
Jul 7, 2026Alicia Parrish, Rajat Shinde, Sanket Badhe +57VLM Evaluation Cultural Safety Evaluation of VLMs
SPLIT: Cross-Lingual Empathy and Cultural Grounding in English and Ukrainian LLM Responses
Jul 2, 2026Anna ChornaMultilingual Language Model Evaluation LLM-as-a-Judge
MSQA: A Natively Sourced Multilingual and Multicultural SimpleQA Benchmark
Jul 1, 2026Xianru Chen, Yukai Huang, Mingxiang Chen +6Multilingual Language Model Evaluation LLM Evaluation
ByteDance Seed · Nanjing University · M-A-P
Extending Item Response Theory for Efficient and Meaningful Multilingual Evaluation
Jun 14, 2026Gili Lior, Tzviel Frostig, Gabriel Stanovsky +1Multilingual Language Model Evaluation Cross-Cultural Language Model Evaluation
Google Research · The Hebrew University of Jerusalem · PhaseV Trials
Schützen: Evaluating LLM Safety in Bulgarian and German Contexts
Jun 9, 2026Kiril Georgiev, Yuxia Wang, Dimitar Iliyanov Dimitrov +2LLM Safety Benchmarks Multilingual Language Model Evaluation
Sofia University "St. Kliment Ohridski" · INSAIT, Sofia University “St. Kliment Ohridski” · MBZUAI
Culturally-Adapted Red-Teaming Across East and Southeast Asian Contexts: A Methodological and Comparative Analysis
Jun 8, 2026Hyeji Choi, Yongtaek Lim, Minwoo KimMultilingual Language Model Evaluation Language Model Safety Evaluation
AI Safety Team, DATUMO.INC, Seoul, South Korea.
GlobeAudio: A Multilingual Multicultural Benchmark for Naturalistic Evaluation of Large Audio-Language Models
Jun 6, 2026Ryner Tan, Wenxuan ZhangAudio-Language Model Evaluation Cross-Cultural Language Model Evaluation
Singapore University of Technology and Design
The Masked Advantage: Uncovering Local-Language Access to Cultural Knowledge in LLMs
Jun 5, 2026Yang Zhang, Xiao Fei, Amr Mohamed +6Multilingual Language Model Evaluation LLM Evaluation
Ecole Polytechnique · MBZUAI · ENS-PSL +1

Sep 25, 2026J/K move · Enter open · S save

Evaluating Cultural Awareness of LLMs for Haitian Creole

Christelle Clervilsson, Yanzhu Guo

Telecom Paris, Institut Polytechnique de Paris, France

Cultural Knowledge in Language Models Cultural Bias in Language Models Cross-Cultural Language Model Evaluation Multilingual Language Model Evaluation

Open paper AskPDF arXiv

Large language models (LLMs) exhibit substantial performance disparities between high- and low-resource languages. Beyond lower task performance, they often fail to capture the cultural norms and values of underrepresented communities. In this work, we present the first systematic evaluation of cultural awareness in LLMs for Haitian Creole, a language spoken by millions but severely underrepresented in digital resources. We assess cultural awareness along four complementary dimensions---specificity, bias, diversity, and variation---using a benchmark of culturally salient prompts curated by native speakers in a text infilling setting. Our results reveal a clear gap between cultural awareness in Haitian Creole and higher-resource French, with Haitian performance being more uneven across domains and more affected by French linguistic interference. Story generation further reveals recurring portrayals of Haitian characters through hardship and resilience, showing that even positive characterizations can encode stereotypical narratives. Our code, benchmark, and evaluation framework are publicly available.