Cross-Cultural Language Model Evaluation

Ask about this topic

Overview

This field studies how language models perform across cultures, including whether they preserve local meaning, represent community values, and behave safely in different settings. Researchers build native-speaker-validated benchmarks for tasks such as question answering, translation, social interaction, bias, and safety, and compare outputs with human judgments or survey distributions. Methods include paired language tests, culturally grounded prompts and evidence, distributional metrics, and statistical tools that separate language proficiency from knowledge. A central challenge is measuring cultural fit without treating a population as uniform or imposing a single standard, while making results reliable across prompts, evaluators, and subgroups.

Latest papers

CardsList

121 papers26 papers in the last four weeks, up 136% on the four weeks before. 0.2% of all new papers.

  1. Cross-Lingual LLM-Judge Transfer via Evaluation Decomposition

    Mar 19, 2026Ivaxi Sheth, Zeno Jonke, Amin Mantrach +1Cross-Lingual TransferCross-Cultural Language Model Evaluation

    CISPA Helmholtz Center for Information Security · Amazon

  2. Harm or Humor: A Multimodal, Multilingual Benchmark for Overt and Covert Harmful Humor

    Mar 18, 2026Ahmed Sharshar, Hosam Elgendy, Saad El Dine Ahmed +2LLM Safety BenchmarksLanguage Model Safety Evaluation

    Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE · INSAIT, Sofia University “St. Kliment Ohridski”, Sofia, Bulgaria

  3. From Words to Worlds: Benchmarking Cross-Cultural Cultural Understanding in Machine Translation

    Mar 18, 2026Bangju Han, Yingqi Wang, Huang Qing +6Cross-Cultural Language Model EvaluationMachine Translation

    Xinjiang Technical Institute of Physics & Chemistry, Chinese Academy of Sciences, Urumqi, China · University of Chinese Academy of Sciences, Beijing, China (second affiliation) · Xinjiang Laboratory of Minority Speech and Language Information Processing, Urumqi, China (third affiliation)

  4. KMMMU: Evaluation of Massive Multi-discipline Multimodal Understanding in Korean Language and Context

    Mar 18, 2026Nahyun Lee, Guijin Son, Hyunwoo Ko +4Multimodal Model EvaluationMultimodal Understanding

    Chung-Ang University · HAE-RAE · Seoul National University +2

  5. Prompt Programming for Cultural Bias and Alignment of Large Language Models

    Mar 17, 2026Maksim Eren, Eric Michalak, Brian Cook +1LLM AlignmentCultural Alignment

    Advanced Research in Cyber Systems, Los Alamos National Laboratory, Los Alamos, New Mexico, USA · Analytics Division, Los Alamos National Laboratory, Los Alamos, New Mexico, USA

  6. Assessing the Pedagogical Readiness of Large Language Models as AI Tutors in Low-Resource Contexts: A Case Study of Nepal's K-10 Curriculum

    Mar 17, 2026Pratyush Acharya, Prasansha Bharati, Yokibha Chapagain +2AI in EducationLLM Evaluation

  7. BanglaSocialBench: A Benchmark for Evaluating Sociopragmatic and Cultural Alignment of LLMs in Bangladeshi Social Interaction

    Mar 16, 2026Tanvir Ahmed Sijan, S. M Golam Rifat, Pankaj Chowdhury Partha +2Pragmatic Reasoning in Language ModelsCultural Bias in Language Models

    Jahangirnagar University, Dhaka, Bangladesh · Rajshahi University of Engineering & Technology, Rajshahi, Bangladesh · Bangladesh University of Engineering and Technology, Dhaka, Bangladesh

  8. Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebook

    Mar 16, 2026Jaehyeok Lee, Xiaoyuan Yi, Jing Yao +4LLM EvaluationLLM Alignment

    Sungkyunkwan University · Microsoft Research Asia · Singapore University of Technology and Design

  9. Information Asymmetry across Language Varieties: A Case Study on Cantonese-Mandarin and Bavarian-German QA

    Mar 16, 2026Renhao Pei, Siyao Peng, Verena Blaschke +2Multilingual QACultural Knowledge in Language Models

    MaiNLP, Center for Information and Language Processing, LMU Munich, Germany · Munich Center for Machine Learning (MCML), Germany · ELLIS Institute Finland +1

  10. CLARIN-PT-LDB: An Open LLM Leaderboard for Portuguese to assess Language, Culture and Civility

    Mar 13, 2026João Silva, Luís Gomes, António BrancoLLM EvaluationPortuguese Language Models

    University of Lisbon NLX—Grupo de Fala e Linguagem Natural, Departmento de Informática Faculdade de Ciências, Campo Grande, 1749-016 Lisboa, Portugal

  11. Dual-Metric Evaluation of Social Bias in Large Language Models: Evidence from an Underrepresented Nepali Cultural Context

    Mar 8, 2026Ashish Pandey, Tek Raj ChhetriSocial Bias in Language ModelsCultural Bias in Language Models

    Center for Artificial Intelligence Research Nepal, Sundarharaincha-09, Buddhabare, Nepal

  12. Taiwan Safety Benchmark and Breeze Guard: Toward Trustworthy AI for Taiwanese Mandarin

    Mar 7, 2026Po-Chun Hsu, Meng-Hsi Chen, Tsu Ling Chao +2LLM SafetyLanguage Model Safety Evaluation

    MediaTek Research · National Taiwan University

  13. Mind the Gap: Pitfalls of LLM Alignment with Asian Public Opinion

    Mar 6, 2026Hari Shankar, Vedanta S P, Sriharini Margapuri +3Cultural Bias in Language ModelsCross-Cultural Language Model Evaluation

    IIIT Hyderabad · IIIT Kottayam · IIT Kharagpur

  14. ThaiSafetyBench: Assessing Language Model Safety in Thai Cultural Contexts

    Mar 5, 2026Trapoom Ukarapol, Nut Chukamphaeng, Kunat Pipatanakul +1Language Model Safety EvaluationLLM Safety

    SCB DataX · Department of Computer Science and Technology, Tsinghua University · SCBX R&D +1

  15. Evaluating Large Language Models' Responses to Sexual and Reproductive Health Queries in Nepali

    Mar 4, 2026Medha Sharma, Supriya Khadka, Udit Chandra Aryal +15Language Model Safety EvaluationMedical QA

    Visible Impact (Visim), Kathmandu, 44600, Nepal · Diyo.AI, Lalitpur, 44600, Nepal · Nepal Applied Mathematics and Informatics Institute for research (NAAMII), Lalitpur, 44600, Nepal +1

  16. Evidence-based Distributional Alignment for Large Language Models

    Mar 3, 2026Viet-Thanh Pham, Lizhen Qu, Zhuang Li +1LLM AlignmentCross-Cultural Language Model Evaluation

    Department of Data Science & AI, Monash University, Australia · Royal Melbourne Institute of Technology, Australia

  17. LiveCultureBench: a Multi-Agent, Multi-Cultural Benchmark for Large Language Models in Dynamic Social Simulations

    Mar 2, 2026Viet-Thanh Pham, Lizhen Qu, Thuy-Trang Vu +2AI Agent BenchmarksCross-Cultural Language Model Evaluation

    Department of Data Science & AI, Monash University

  18. FLANS at SemEval-2026 Task 7: RAG with Open-Sourced Smaller LLMs for Everyday Knowledge Across Diverse Languages and Cultures

    Mar 2, 2026Liliia Bogdanova, Shiran Sun, Lifeng Han +2Multilingual QAKnowledge Augmentation for Language Models

    Insilico Medicine AI Limited · University of Groningen · LIACS, Leiden University +1

  19. JailNewsBench: Multi-Lingual and Regional Benchmark for Fake News Generation under Jailbreak Attacks

    Mar 1, 2026Masahiro Kaneko, Ayana Niwa, Timothy BaldwinLLM Safety BenchmarksLLM Jailbreak Attacks

    MBZUAI

  20. A Unified Framework to Quantify Cultural Intelligence of AI

    Mar 1, 2026Sunipa Dev, Vinodkumar Prabhakaran, Rutledge Chin Feman +16Cross-Cultural Language Model EvaluationGenerative AI Evaluation

    Google Research.

  21. TARAZ: Persian Short-Answer Question Benchmark for Cultural Evaluation of Language Models

    Feb 26, 2026Reihaneh Iranmanesh, Saeedeh Davoudi, Pasha Abrishamchian +2Cultural Knowledge in Language ModelsCross-Cultural Language Model Evaluation

    IR Lab, Computer Science Department Georgetown University, Washington D.C.

  22. When AI Writes, Whose Voice Remains? Quantifying Cultural Marker Erasure Across World English Varieties in Large Language Models

    Feb 25, 2026Satyam Kumar Navneet, Joydeep Chandra, Yong ZhangLinguistic Bias in Language ModelsCultural Bias in Language Models

    Independent Researcher Bihar, India · BNRIST, Dept. of CST, Tsinghua University Beijing, China

  23. Towards Cross-lingual Values Judgment: A Consensus-Pluralism Perspective

    Feb 19, 2026Yukun Chen, Xinyu Zhang, Boyi Deng +6Cross-Cultural Language Model EvaluationMultilingual Language Model Evaluation

    State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security · Tongyi Lab, Alibaba Group +1

  24. Far Out: Evaluating Language Models on Slang in Australian and Indian English

    Feb 17, 2026Deniz Kaya Dilsiz, Dipankar Srirag, Aditya JoshiLLM EvaluationCross-Cultural Language Model Evaluation

    University of New South Wales, Sydney, Australia

  25. Robust Bias Evaluation with FilBBQ: A Filipino Bias Benchmark for Question-Answering Language Models

    Feb 16, 2026Lance Calvin Lim Gamboa, Yue Feng, Mark LeeSocial Bias in Language ModelsLanguage Model Bias Evaluation

    School of Computer Science, University of Birmingham · Department of Information Systems and Computer Science, Ateneo de Manila University

  26. Evaluating Geographic Lexical Knowledge in Large Language Models Across 21 Spanish-Speaking Countries

    Feb 10, 2026Yoshifumi KawasakiCross-Cultural Language Model Evaluation

    University of Tokyo

  27. Culturally Grounded Personas in Large Language Models: Characterization and Alignment with Socio-Psychological Value Frameworks

    Jan 29, 2026Candida M. Greco, Lucio La Cava, Andrea TagarelliCultural AlignmentCross-Cultural Language Model Evaluation

    DIMES, University of Calabria, Italy

  28. MasalBench: A Benchmark for Contextual and Cross-Cultural Understanding of Persian Proverbs in LLMs

    Jan 29, 2026Ghazal Kalhor, Behnam BahrakCross-Cultural Language Model EvaluationFigurative Language Understanding

    School of Electrical and Computer Engineering, College of Engineering, University of Tehran, Tehran, Iran · Tehran Institute for Advanced Studies, Khatam University, Tehran, Iran

  29. The Alignment Veto: How Safety Training Suppresses Cultural Knowledge in LLMs

    Oct 15, 2025Pardis Sadat Zahraei, Gokhan Tur, Dilek Hakkani-Tür +1LLM AlignmentRefusal Behavior in Language Models

    University of Illinois Urbana-Champaign (UIUC) · Qatar Computing Research Institute (QCRI), Hamad Bin Khalifa University

  30. One Model, Many Morals: Uncovering Cross-Linguistic Misalignments in Computational Moral Reasoning

    Sep 25, 2025Sualeha Farid, Jayden Lin, Zean Chen +2Multilingual Language Model EvaluationMoral Reasoning in Language Models

    University of Michigan, Ann Arbor, USA