Cross-Cultural Language Model Evaluation

Follow this topic Ask about this topic

This field studies how language models perform across cultures, including whether they preserve local meaning, represent community values, and behave safely in different settings. Researchers build native-speaker-validated benchmarks for tasks such as question answering, translation, social interaction, bias, and safety, and compare outputs with human judgments or survey distributions. Methods include paired language tests, culturally grounded prompts and evidence, distributional metrics, and statistical tools that separate language proficiency from knowledge. A central challenge is measuring cultural fit without treating a population as uniform or imposing a single standard, while making results reliable across prompts, evaluators, and subgroups.

Latest papers

121 papers26 papers in the last four weeks, up 136% on the four weeks before. 0.2% of all new papers.

MADE: Beyond Scoring via a Multilingual Agentic Diagnosing Engine for Fine-Grained Evaluation Insights
Jun 5, 2026Yilun Liu, Miao Zhang, Shimin Tao +9Multilingual Language Model Evaluation LLM Evaluation
Huawei, China
SEA-NLI: Natural Language Inference as a Lens into Southeast Asian Cultural Understanding
Jun 2, 2026Peerawat Chomphooyod, Jian Gang Ngui, Yosephine Susanto +5Multilingual Language Model Evaluation Cross-Cultural Language Model Evaluation
1Chulalongkorn University · 2AI Singapore · 3MBZUAI +1
When English Rewrites Local Knowledge: Global Narrative Dominance in Large Language Models
May 28, 2026Md Arid Hasan, Ruwad Naswan, Farhan Samir +2Multilingual Language Model Evaluation LLM Grounding
University of Toronto · BUET · University of Illinois Urbana-Champaign
ExCAM: Explainable Cultural Awareness Metrics
May 28, 2026Christoph Leiter, Haiyue Song, Hour Kaing +4LLM Evaluation Language Model Generation Evaluation
University of Mannheim, Germany · National Institute of Information and Communications Technology · University of Technology Nuremberg, Germany
SomaliBench Eval: Measuring English-to-Somali Refusal Gaps in Open-Weight Language Models
May 25, 2026Khalid Yusuf DahirLLM Safety Benchmarks Multilingual Language Model Evaluation
Independent researcher
ROK-FORTRESS: Measuring the Effect of Geopolitical Transcreation for National Security and Public Safety
May 13, 2026Michael S. Lee, Yash Maurya, Drew Rein +13Multilingual Language Model Evaluation Language Model Safety Evaluation
1Scale AI · 2Korea AI Safety Institute
XL-SafetyBench: A Country-Grounded Cross-Cultural Benchmark for LLM Safety and Cultural Sensitivity
May 7, 2026Dasol Choi, Eugenia Kim, Jaewon Noh +14LLM Safety Benchmarks Multilingual Language Model Evaluation
AIM Intelligence · Microsoft · Korea AISI +7
SemEval-2026 Task 7: Everyday Knowledge Across Diverse Languages and Cultures
May 4, 2026Nedjma Ousidhoum, Junho Myung, Carla Perez-Almendros +27Multilingual Language Model Evaluation LLM Evaluation
Cardiff University · KAIST · MBZUAI +13
When AI Speaks, Whose Values Does It Express? A Cross-Cultural Audit of Individualism-Collectivism Bias in Large Language Models
Apr 24, 2026Pruthvinath Jeripity VenkataLLM Evaluation Cross-Cultural Language Model Evaluation
Independent Researcher
The GaoYao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Models
Apr 22, 2026Yilun Liu, Chunguang Zhao, Mengyao Piao +14Multilingual Language Model Evaluation LLM Evaluation
Huawei, China · Fudan University, China
CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks
Apr 21, 2026Peiqin Lin, Chenyang Lyu, Wenjiang Luo +22Multilingual Language Model Evaluation LLM Evaluation
Alibaba Group · Beijing Language and Culture University · LMU Munich +6
AlignCultura: Towards Culturally Aligned Large Language Models?
Apr 21, 2026Gautam Siddharth Kashyap, Mark Dras, Usman NaseemCross-Cultural Language Model Evaluation Cultural Alignment
School of Computing, Macquarie University, Australia
Cross-Cultural Simulation of Citizen Emotional Responses to Bureaucratic Red Tape Using LLM Agents
Apr 14, 2026Wanchun Ni, Jiugeng Sun, Yixian Liu +1Large Language Model-Based Social Simulation Cross-Cultural Language Model Evaluation
ETH Zurich Zurich, Switzerland · City University of Hong Kong Hong Kong SAR, China
When Meaning Isn't Literal: Exploring Idiomatic Meaning Across Languages and Modalities
Apr 12, 2026Sarmistha Das, Shreyas Guha, Suvrayan Bandyopadhyay +3Retrieval-Augmented Reasoning Figurative Language Understanding
Department of Computer Science and Engineering, Indian Institute of Technology Patna, Patna, India · School of Information Technology, King Mongkut’s Institute of Technology Ladkrabang, Bangkok, Thailand
Nationality encoding in language model hidden states: Probing culturally differentiated representations in persona-conditioned academic text
Apr 11, 2026Paul Jackson, Ruizhe Li, Elspeth EdelsteinDemographic Bias in Language Models Cross-Cultural Language Model Evaluation
Language Centre, School of Language, Literature, Music and Visual Culture, University of Aberdeen, United Kingdom · Language Centre, School of Language, Literature, Music and Visual Culture, University of Aberdeen, Aberdeen AB24 3FX, United Kingdom. · School of Natural and Computing Sciences, University of Aberdeen, United Kingdom +3
Lessons Without Borders? Evaluating Cultural Alignment of LLMs Using Multilingual Story Moral Generation
Apr 9, 2026Sophie Wu, Andrew PiperCross-Cultural Language Model Evaluation Cultural Alignment
Languages, Literatures, and Cultures McGill University
Appear2Meaning: A Cross-Cultural Benchmark for Structured Cultural Metadata Inference from Images
Apr 8, 2026Yuechen Jiang, Enze Zhang, Md Mohsinul Kabir +4Cultural Knowledge in Language Models VLM Evaluation
University of Manchester Manchester, UK · School of Artificial Intelligence, Wuhan University Wuhan, Hubei, China · Getty Conservation Institute Los Angeles, CA, USA
To Lie or Not to Lie? Investigating The Biased Spread of Global Lies by LLMs
Apr 8, 2026Zohaib Khan, Mustafa Dogan, Ifeoma Okoh +6Deception in Language Models Language Model Safety Evaluation
Fatima Fellowship · Stanford University · Georgia Institute of Technology
Metaphors We Compute By: A Computational Audit of Cultural Translation vs. Thinking in LLMs
Apr 6, 2026Yuan Chang, Jiaming Qu, Zhu LiCultural Bias in Language Models Cultural Knowledge in Language Models
Meta, Burlingame, CA · Amazon, Seattle, WA
Cultural Authenticity: Comparing LLM Cultural Representations to Native Human Expectations
Apr 3, 2026Erin MacMurray van Liemt, Aida Davani, Sinchana Kumbale +2Cultural Alignment Cross-Cultural Language Model Evaluation
Google Research, USA · Google, USA
Pragmatics Meets Culture: Culturally-adapted Artwork Description Generation and Evaluation
Apr 2, 2026Lingjun Zhao, Dayeon Ki, Marine Carpuat +1Cultural Knowledge in Language Models Text Generation
Department of Computer Science, University of Maryland, College Park, Maryland, USA
Mitigating Cross-Lingual Cultural Inconsistencies in LLMs via Consensus-Driven Preference Optimisation
Apr 2, 2026Lucas Resck, Isabelle Augenstein, Anna KorhonenCross-Lingual Consistency in Language Models Cross-Cultural Language Model Evaluation
Language Technology Lab, University of Cambridge · University of Copenhagen
A Japanese Benchmark for Evaluating Social Bias in Reasoning Based on Attribution Theory
Apr 1, 2026Taihei Shiotani, Masahiro Kaneko, Naoaki OkazakiSocial Bias in Language Models Language Model Bias Evaluation
Institute of Science Tokyo · MBZUAI · AIST +1
Dual Perspectives in Emotion Attribution: A Generator-Interpreter Framework for Cross-Cultural Analysis of Emotion in LLMs
Mar 30, 2026Aizirek Turdubaeva, Uichin LeeEmotion Representation in Language Models Cultural Bias in Language Models
KAIST / Daejeon, Republic of Korea
Does Claude's Constitution Have a Culture?
Mar 30, 2026Parham PourdavoodCultural Bias in Language Models Cultural Alignment
Independent Researcher
Investigating the Influence of Language on Sycophantic Behavior of Multilingual LLMs
Mar 29, 2026Bayan Abdullah Aldahlawi, A. B. M. Ashikur Rahman, Irfan AhmadLLM Sycophancy Multilingual Language Model Evaluation
King Fahd University of Petroleum and Minerals, Dhahran, KSA · SDAIA-KFUPM Joint Research Center for AI (JRCAI), Dhahran, KSA
Culturally Adaptive Explainable LLM Assessment for Multilingual Information Disorder: A Human-in-the-Loop Approach
Mar 28, 2026Maziar Kianimoghadam JouneghaniCross-Cultural Language Model Evaluation Faithfulness of Language Model Explanations
University of Turin
ALBA: A European Portuguese Benchmark for Evaluating Language and Linguistic Dimensions in Generative LLMs
Mar 27, 2026Inês Vieira, Inês Calvo, Iago Paulo +6LLM Evaluation Language Model Generation Evaluation
1NOVA University of Lisbon, Portugal · 2NOVA LINCS
Toward Culturally Grounded Natural Language Processing
Mar 27, 2026Sina Bagheri NezhadCross-Cultural Language Model Evaluation Cultural Knowledge in Language Models
Independent Researcher, Seattle, WA, USA
CAPITU: A Benchmark for Evaluating Instruction-Following in Brazilian Portuguese with Literary Context
Mar 23, 2026Giovana Kerche Bonás, Roseval Malaquias Junior, Marcos Piau +6Instruction Following Multilingual Language Model Evaluation
Maritaca AI · Jusbrasil

Jun 5, 2026J/K move · Enter open · S save

MADE: Beyond Scoring via a Multilingual Agentic Diagnosing Engine for Fine-Grained Evaluation Insights

Yilun Liu, Miao Zhang, Shimin Tao, Minggui He+8

Huawei, China

Multilingual Language Model Evaluation LLM Evaluation Cross-Cultural Language Model Evaluation LLM Agent Evaluation

Open paper AskPDF arXiv

Multilingual and multicultural benchmarks now cover dozens of languages and model families, but the resulting score landscapes remain metric-rich and insight-poor, necessitating fine-grained multilingual post-evaluation diagnosis. However, single LLMs and open-ended agents are easily swamped by the long, noisy diagnostic input, and no reusable taxonomy exists for it. To address this, we propose MADE, a Multilingual Agentic Diagnosing Engine that decomposes post-evaluation analysis into planning, aggregate analysis, instance-level case inspection, multilingual and cultural reflection, and grounded report synthesis. MADE is paired with an expert-led 54-query and 15-language diagnostic set, evaluated on top of a large-scale multilingual evaluation substrate (33 model families, 11 benchmarks, 26 languages, 34 cultures, 8.66M evaluation records). Experiments show that MADE outperforms the strongest shared baseline by 47% in diagnosis report quality and is preferred by human multilingual experts in 87.9% of pairwise comparisons. Applied with multilingual experts, MADE further surfaces four actionable findings on deployment, iteration, and cross-cultural pitfalls, turning benchmark score tables into model-selection and remediation guidance.