Cross-Cultural Language Model Evaluation

Overview

This field studies how language models perform across cultures, including whether they preserve local meaning, represent community values, and behave safely in different settings. Researchers build native-speaker-validated benchmarks for tasks such as question answering, translation, social interaction, bias, and safety, and compare outputs with human judgments or survey distributions. Methods include paired language tests, culturally grounded prompts and evidence, distributional metrics, and statistical tools that separate language proficiency from knowledge. A central challenge is measuring cultural fit without treating a population as uniform or imposing a single standard, while making results reliable across prompts, evaluators, and subgroups.

Latest papers

CardsList

135 papers23 papers in the last four weeks, up 21% on the four weeks before. 0.2% of all new papers.

  1. Cross-cultural Adaptation of Recipes for Low-Resource Languages: A Benchmark and Multilingual Culinary Lexicon for Hausa, Yorùbá, and Igbo

    Nov 17, 2026Nuhu Ibrahim, Maryam Abdulazeez, Fatsuma Jauro +3Machine TranslationLow-Resource MT

  2. AfriNORM: An Evaluation Benchmark for Normative Reasoning of LLMs Across African Cultures

    Nov 17, 2026Ahmad Samuel Gali, Tadesse Destaw Belay, Idris Abdulmumin +2Cross-Cultural Language Model EvaluationCultural Knowledge in Language Models

  3. Evaluating Cultural Reasoning and Robustness in LLMs for Spanish Political Discourse

    Nov 16, 2026Jorge Valverde-Rebaza, Nathan Garcia-Larios, Karime Ruvalcaba-Perez +3Sentiment AnalysisLanguage Model Robustness

  4. BDBBQ: A Human-AI Collaborative Benchmark for Bengali Social Bias in LLMs

    Nov 9, 2026Provat Saha, Ashika Habib Khan, Tahseen Tanha +2Social Bias in Language ModelsLanguage Model Bias Evaluation

  5. CULTURE-CMT: Benchmarking Contextualized Social Media Translation

    Nov 1, 2026Linjuan Wu, Dingming Li, Yiwen Qiu +5Machine TranslationContext-Aware MT

  6. Whose Values, Which Contexts, and What Signals? Grounding Cultural Value Alignment with Re3Culture

    Oct 30, 2026Jing Yao, Hyunjin Hwang, Weihua Zheng +12Cultural AlignmentCross-Cultural Language Model Evaluation

  7. Qistas: A Pre-Registered, Portable Protocol for Fairness Evaluation of Large Language Models

    Oct 29, 2026Nawar S. Alseelawi, Mustafa S. AljumailyLanguage Model Bias EvaluationCross-Cultural Language Model Evaluation

  8. Gold vs.~Translation: Are We Measuring Language, Knowledge, or Locality in Translated Benchmarks?

    Oct 28, 2026Klaudia-Doris Thellmann, Bernhard Stadler, Dan Saattrup Nielsen +2Benchmark ValidityMultilingual Language Model Evaluation

  9. The State and Fate of Multilingual, Contextual Evaluation in the NLP World

    Oct 27, 2026Manan Uppadhyay, Himanshu Beniwal, Prashant Kodali +1Multilingual Language Model EvaluationCross-Cultural Language Model Evaluation

  10. Beyond Words Alone: Disentangling Distributed Evidence for Visually Grounded Cultural Translation

    Oct 26, 2026Dingming Li, Yiwen Qiu, Linjuan Wu +8Machine TranslationMultimodal MT

  11. Do Language Models Understand Igbo Culture? Beyond Multiple Choice

    Oct 26, 2026Blessing Ngozi Uzor, Valdini Douglace Lemofouet, Paula Chikaodinaka Anyanwu +7Cultural Knowledge in Language ModelsCross-Cultural Language Model Evaluation

  12. Investigating Multi-Cultural Value Selection Biases in LLMs

    Oct 22, 2026Siddhesh Pawar, Arnav Arora, Dustin Wright +2Cultural Bias in Language ModelsCross-Cultural Language Model Evaluation

  13. Harm Is Behavioural, Moderation Is Lexical: How Language and Culture Decide Whom Automated Safety Systems Protect and Whom They Silence

    Oct 22, 2026Gabriel Lars Sabadin, Dr Nicola Harding, Sarah KingstonContent ModerationLanguage Model Bias Evaluation

  14. Benchmarking Arabic Dialects for Math Problems

    Oct 20, 2026Noor Abo Mokh, Teresa Lynn, Nizar Habash +5Mathematical Reasoning BenchmarksArabic Dialects

  15. ABench: Adversarily Benchmarking Large Language Models across Taxonomized Values

    Oct 20, 2026Liping Yang, Yongqi Leng, Jingting Zheng +3LLM Safety BenchmarksLanguage Model Safety Evaluation

  16. AfriGuard: Safety and Robust Refusal For African Languages

    Oct 19, 2026Israel Abebe Azime, Jesujoba O. Alabi, Iffat Maab +20LLM SafetyLanguage Model Safety Evaluation

  17. VEILED-SH: Vulnerability Expression through Indirect Language Evaluation Dataset (Self-Harm)

    Oct 19, 2026Leanne Tan, Keith Chua, Shaun Khoo +7LLM Safety BenchmarksLanguage Model Safety Evaluation

  18. Aligning Cultural Behaviors in Arabic Language Models via Preferences

    Oct 18, 2026Junhong Liang, Daniil Orel, Saeed Almheiri +11Arabic Language ModelsCultural Alignment

  19. KiranaBench: Can a Language Model Run a Neighbourhood Shop?

    Oct 16, 2026Bhuvan Arora, Ankita Karmakar, Chaitanya Jain +10AI Agent BenchmarksLong-Horizon LLM Agents

  20. Cultural (In)Competence: Investigating Culture-Specific Stereotype Detection Capabilities of LLMs

    Oct 15, 2026Maximilian Spliethöver, Jannes Lötfering, Dominik Bachmann +4Social Bias in Language ModelsCultural Bias in Language Models

  21. In the Guise of Correction: How Framing and Cultural Adaptation Erode LLM Safety on Real-World Criminal Cases

    Oct 15, 2026Mohsinul Kabir, Md. Tariquzzaman, Karam S. R. Elradie +4LLM Safety EvaluationLLM Safety Benchmarks