Cross-Cultural Language Model Evaluation

Follow this topic Ask about this topic

This field studies how language models perform across cultures, including whether they preserve local meaning, represent community values, and behave safely in different settings. Researchers build native-speaker-validated benchmarks for tasks such as question answering, translation, social interaction, bias, and safety, and compare outputs with human judgments or survey distributions. Methods include paired language tests, culturally grounded prompts and evidence, distributional metrics, and statistical tools that separate language proficiency from knowledge. A central challenge is measuring cultural fit without treating a population as uniform or imposing a single standard, while making results reliable across prompts, evaluators, and subgroups.

Latest papers

121 papers26 papers in the last four weeks, up 136% on the four weeks before. 0.2% of all new papers.

The State and Fate of Multilingual, Contextual Evaluation in the NLP World
Oct 27, 2026Manan Uppadhyay, Himanshu Beniwal, Prashant Kodali +1Multilingual Language Model Evaluation Cross-Cultural Language Model Evaluation
Microsoft Research India · IIT Gandhinagar
Beyond Words Alone: Disentangling Distributed Evidence for Visually Grounded Cultural Translation
Oct 26, 2026Dingming Li, Yiwen Qiu, Linjuan Wu +8Machine Translation Multimodal MT
Zhejiang University · Xiaohongshu Inc.
Do Language Models Understand Igbo Culture? Beyond Multiple Choice
Oct 26, 2026Blessing Ngozi Uzor, Valdini Douglace Lemofouet, Paula Chikaodinaka Anyanwu +7Cultural Knowledge in Language Models Cross-Cultural Language Model Evaluation
African Institute for Mathematical Sciences (AIMS), Cameroon · University of Nigeria, Nsukka · Brown University +5
A Russian, a Brazilian, and a German Walk into a Bar, Ask the Same Question-- And Get Three Different Answers: Profiling LLM Perspective Diversity on Contested Topics
Oct 25, 2026Wiem Ben Rim, Yao Lu, Pontus Stenetorp +1Cross-Cultural Language Model Evaluation Output Diversity in Language Models
University College London
Investigating Multi-Cultural Value Selection Biases in LLMs
Oct 22, 2026Siddhesh Pawar, Arnav Arora, Dustin Wright +2Cultural Bias in Language Models Cross-Cultural Language Model Evaluation
University of Copenhagen · Aalborg University Copenhagen · Bocconi University
Harm Is Behavioural, Moderation Is Lexical: How Language and Culture Decide Whom Automated Safety Systems Protect and Whom They Silence
Oct 22, 2026Gabriel Lars Sabadin, Dr Nicola Harding, Sarah KingstonContent Moderation Language Model Bias Evaluation
Tuteliq AB, Sweden · Tuteliq AB, United Kingdom · University of Lancashire, United Kingdom
WikiOriginQA: Automated Construction of a QA Dataset with Knowledge Origins for Analyzing Cultural Bias
Oct 22, 2026Kento Haneda, Yosuke Kishinami, Ryo Fujii +2Cultural Knowledge in Language Models Cultural Bias in Language Models
Future Corporation · Tohoku University
CulturalRiddles: A Multicultural Benchmark of Riddles
Oct 21, 2026My Chiffon Nguyen, Tianyi Hu, Vinod Anbalagan +161Cross-Cultural Language Model Evaluation Cultural Knowledge in Language Models
Benchmarking Arabic Dialects for Math Problems
Oct 20, 2026Noor Abo Mokh, Teresa Lynn, Nizar Habash +5Mathematical Reasoning Benchmarks Arabic Dialects
Mohamed bin Zayed University of Artificial Intelligence · New York University Abu Dhabi
ABench: Adversarily Benchmarking Large Language Models across Taxonomized Values
Oct 20, 2026Liping Yang, Yongqi Leng, Jingting Zheng +3LLM Safety Benchmarks Language Model Safety Evaluation
TJUNLP Lab, School of Computer Science and Technology, Tianjin University, Tianjin, China · Yuanhui AI
The Culture Within: Towards Mechanistic Understanding of Cultural Reasoning in LLMs
Oct 19, 2026Samee Arif, Furkan Danisman, Angana Borah +6Cultural Bias in Language Models Cultural Knowledge in Language Models
AfriGuard: Safety and Robust Refusal For African Languages
Oct 19, 2026Israel Abebe Azime, Jesujoba O. Alabi, Iffat Maab +20LLM Safety Language Model Safety Evaluation
Saarland University, Saarland Informatic Campus, Germany · Masakhane NLP · National Institute of Informatics, Japan +13
VEILED-SH: Vulnerability Expression through Indirect Language Evaluation Dataset (Self-Harm)
Oct 19, 2026Leanne Tan, Keith Chua, Shaun Khoo +7LLM Safety Benchmarks Language Model Safety Evaluation
GovTech Singapore · MOH Office for Healthcare Transformation · Singapore University of Technology and Design +2
Aligning Cultural Behaviors in Arabic Language Models via Preferences
Oct 18, 2026Junhong Liang, Daniil Orel, Saeed Almheiri +11Arabic Language Models Cultural Alignment
Mohamed bin Zayed University of Artificial Intelligence · IBM Research AI
KiranaBench: Can a Language Model Run a Neighbourhood Shop?
Oct 16, 2026Bhuvan Arora, Ankita Karmakar, Chaitanya Jain +10AI Agent Benchmarks Long-Horizon LLM Agents
BITS Pilani, Pilani Campus · University of Minnesota, Twin Cities · BITS Pilani, K. K. Birla Goa Campus
Cultural (In)Competence: Investigating Culture-Specific Stereotype Detection Capabilities of LLMs
Oct 15, 2026Maximilian Spliethöver, Jannes Lötfering, Dominik Bachmann +4Social Bias in Language Models Cultural Bias in Language Models
Institute of Artificial Intelligence & L3S Research Center, Leibniz University Hannover · Institute for Logic, Language & Computation, University of Amsterdam · Department of Experimental Psychology, Utrecht University
In the Guise of Correction: How Framing and Cultural Adaptation Erode LLM Safety on Real-World Criminal Cases
Oct 15, 2026Mohsinul Kabir, Md. Tariquzzaman, Karam S. R. Elradie +4LLM Safety Evaluation LLM Safety Benchmarks
The University of Manchester · ELLIS Manchester · Islamic University of Technology +2
ChronoCulture: Evaluating Cultural Awareness Across Time in Large Language Models
Oct 14, 2026Rusali Saha, Nitya Khurana, Ruihong Huang +1Cultural Knowledge in Language Models Cross-Cultural Language Model Evaluation
Texas A&M University
Analyzing performance of Retrieval Augmented Generation models for Question Answering Tasks
Oct 14, 2026Fazle Mohammed Tasfiq, Sabab Attin, Sheikh Md. Tahmid Hossain +6RAG Evaluation Multilingual QA
Dept. of Computer Science and Engineering BRAC University Dhaka, Bangladesh
J-tau: A Fully Localized Japanese Benchmark for Tool-Agent-User Interaction
Oct 14, 2026Chihiro Yano, Jun Hirako, Hiroaki Yamagiwa +1AI Agent Benchmarks LLM Agent Evaluation
SB Intuitions Corp. · Nagoya University
A Global View of How LLMs Value Human Time
Oct 13, 2026Yifan He, Manon Reusens, Sofie Goethals +1Jurisdictional Bias in Language Models Cross-Cultural Language Model Evaluation
NepPlural: A Human-Anchored Benchmark for Persona Annotation in Nepali Migration Discourse
Oct 11, 2026Jinu Nyachhyon, Anish Khatiwada, Bimal TimilsinaPluralistic Alignment Cross-Cultural Language Model Evaluation
Institute for Research and Innovation in Intelligent Systems (IRIIS) · Independent Researcher
North or South? Diagnosing LLMs' Understanding of North Korean Language, Society, and Knowledge
Oct 11, 2026Kiwoong Park, Seyoung Song, Donghyun Woo +1Cultural Knowledge in Language Models Cross-Cultural Language Model Evaluation
KAIST
Knowing Is Not Telling: Identity Ossification in LLM Representations of Trans-border Communities
Oct 8, 2026Minglu Sun, Lan Chen, Hang JiangCultural Bias in Language Models Cultural Knowledge in Language Models
Northeastern University, Boston, MA, USA · The University of Texas at Austin, Austin, TX, USA
Measuring Cultural Alignment Beyond the Average: A Framework for Evaluating Maternal-Health LLM Interactions in Indian Contexts
Oct 8, 2026Umaira Izhar, Gunjan Arora, Pushpendra SinghCultural Alignment Clinical Language Model Evaluation
Indraprastha Institute of Information Technology Delhi
Language Unalignability: Why Some Concepts Resist Cross-Cultural Benchmark Evaluation
Oct 6, 2026Shu-Kai Hsieh, Da-Chen LianCultural Knowledge in Language Models LLM Evaluation
National Taiwan University
CuBEs: Culturally-Situated Behavioral Evaluations and the Limitations of Culture-Blind LLM Judges
Oct 2, 2026Hoda Ayad, Tanu Mitra, Abhishek MukherjiLLM Evaluation Cross-Cultural Language Model Evaluation
University of Washington · Centific
PowerBench: Measuring Language Model Bias in Power-shifting Requests
Oct 1, 2026Nicolas Martorell, Wendy Brau, Gonzalo A. Heredia +3Language Model Bias Evaluation Refusal Behavior in Language Models
Universidad de Buenos Aires · Buenos Aires AI Safety Hub · The World Bank +1
Calibrated to Whom? Persona and Language Effects on Cultural Values in JEV
Sep 28, 2026Bushra Asseri, Abdulaziz AsseriMultilingual Language Model Evaluation Cross-Cultural Language Model Evaluation
College of Engineering and Advanced Computing, Alfaisal University, Riyadh, Saudi Arabia · Proxa.sa, Riyadh, Saudi Arabia
PlurVA-LLM-2026 Shared Task Track-1: Pluralistic Value Alignment in LLMs via Multilingual Fine-Tuning and Threshold Calibration
Sep 26, 2026Vihindi Kotalawala, Nevidu JayatillekePluralistic Alignment Cross-Cultural Language Model Evaluation
School of Computing, Informatics Institute of Technology, Sri Lanka · Department of Computer Science & Engineering, University of Moratuwa, Sri Lanka

Oct 27, 2026J/K move · Enter open · S save

The State and Fate of Multilingual, Contextual Evaluation in the NLP World

Manan Uppadhyay, Himanshu Beniwal, Prashant Kodali, Sunayana Sitaram

Microsoft Research India · IIT Gandhinagar

Multilingual Language Model Evaluation Cross-Cultural Language Model Evaluation

Open paper AskPDF arXiv

Multilingual evaluation benchmarks are the primary instrument for assessing whether large language models generalize beyond English, yet the adequacy of these benchmarks has received little systematic scrutiny. We present a data-driven audit of 51 recent multilingual benchmarks spanning 242 datasets and 219 languages, organized around three pillars: coverage, representativeness, and rigor. Our analysis reveals that coverage is wide but thin with 36% of evaluated languages appearing in only a single benchmark, entire regions (Oceania, the Americas, Central Asia) are near-zero, and a stark task equity gap leaves low-resource languages evaluated on only 1-3 task categories versus 14 for high-resource languages. Representativeness is structurally compromised: translation from English remains the dominant construction strategy where 56% of all dataset-language instances are translated introducing artifacts and English-centric framing, while culturally grounded content is concentrated in a handful of community-driven benchmarks with narrow language scope. The ecosystem thus forces a trade-off between breadth and validity. Rigor is undermined by benchmark contamination, including translated benchmark leakage and parallel corpus overlap that evade surface-form detection. We synthesize these findings into concrete recommendations for building evaluation frameworks that are natively constructed, culturally grounded, contamination-aware, and designed to serve the communities whose languages they claim to evaluate. The dashboard with all the metadata collected can be found at https://aka.ms/multilingual-evals.