Cross-Cultural Language Model Evaluation

Follow this topic Ask about this topic

Momentum

3 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 29

All topics

Measuring Cultural Alignment Beyond the Average: A Framework for Evaluating Maternal-Health LLM Interactions in Indian Contexts
Oct 8, 2026Umaira Izhar, Gunjan Arora, Pushpendra SinghCultural Alignment Clinical Language Model Evaluation
Indraprastha Institute of Information Technology Delhi
Language Unalignability: Why Some Concepts Resist Cross-Cultural Benchmark Evaluation
Oct 6, 2026Shu-Kai Hsieh, Da-Chen LianMultilingual Language Model Evaluation LLM Evaluation
National Taiwan University
Calibrated to Whom? Persona and Language Effects on Cultural Values in JEV
Sep 28, 2026Bushra Asseri, Abdulaziz AsseriMultilingual Language Model Evaluation Cross-Cultural Language Model Evaluation
College of Engineering and Advanced Computing, Alfaisal University, Riyadh, Saudi Arabia · Proxa.sa, Riyadh, Saudi Arabia
PlurVA-LLM-2026 Shared Task Track-1: Pluralistic Value Alignment in LLMs via Multilingual Fine-Tuning and Threshold Calibration
Sep 26, 2026Vihindi Kotalawala, Nevidu JayatillekePluralistic Alignment Cross-Cultural Language Model Evaluation
School of Computing, Informatics Institute of Technology, Sri Lanka · Department of Computer Science & Engineering, University of Moratuwa, Sri Lanka
GYROval: A Robust Benchmark for Cultural Value Orientation in Large Language Models
Sep 16, 2026Alexander Didenko, Anna Shabanova, Vladislav Zapylikhin +2LLM Evaluation Cross-Cultural Language Model Evaluation
Artificial Intelligence Laboratory, SKOLKOVO School of Management, Moscow · Artificial Intelligence Laboratory, University of Tyumen · Institute of Business Studies, Russian Presidential Academy of National Economy and Public Administration, Moscow
WorldBench: Culturally Grounded Benchmark for Multilingual Agents
Sep 1, 2026Leonardo Ranaldi, Sherrie Shen, Jushi Kai +1Cross-Cultural Language Model Evaluation LLM Agent Evaluation
(•) ILCC, School of Informatics, University of Edinburgh · (•,◦) ILCC, School of Informatics, University of Edinburgh · (◦) School of Artificial Intelligence, Shanghai Jiao Tong University
Apples to Apples? Towards Comparable Crosslingual Language Model Evaluation
Aug 25, 2026Xiulin Yang, Ethan Gotlieb Wilcox, Catherine ArnettMultilingual Language Model Evaluation LLM Evaluation
Georgetown University · EleutherAI
Inspect India Evals: An Open Benchmarking Framework for Evaluating Large Language Models in the Indian Linguistic and Cultural Context
Jul 28, 2026Abhishek Kumar Singh, Shrey Nag, Sachita +2LLM Safety Benchmarks Multilingual Language Model Evaluation
GTBIT¹ · TIET² · IGDTUW³ +2
ADAGE: A Language-Agnostic Pipeline for Analogical Reasoning Evaluation
Jul 25, 2026Ahmed Haj Ahmed, Alvin GrissomMultilingual Language Model Evaluation Reasoning Evaluation
Haverford College
Pluralis v0.1: Towards a Multicultural, Multimodal, Multilingual Benchmark for AI Risk and Reliability
Jul 7, 2026Alicia Parrish, Rajat Shinde, Sanket Badhe +57VLM Evaluation Cross-Cultural Language Model Evaluation
SPLIT: Cross-Lingual Empathy and Cultural Grounding in English and Ukrainian LLM Responses
Jul 2, 2026Anna ChornaMultilingual Language Model Evaluation LLM-as-a-Judge
MSQA: A Natively Sourced Multilingual and Multicultural SimpleQA Benchmark
Jul 1, 2026Xianru Chen, Yukai Huang, Mingxiang Chen +6Multilingual Language Model Evaluation LLM Evaluation
ByteDance Seed · Nanjing University · M-A-P
Extending Item Response Theory for Efficient and Meaningful Multilingual Evaluation
Jun 14, 2026Gili Lior, Tzviel Frostig, Gabriel Stanovsky +1Multilingual Language Model Evaluation Cross-Cultural Language Model Evaluation
Google Research · The Hebrew University of Jerusalem · PhaseV Trials
Schützen: Evaluating LLM Safety in Bulgarian and German Contexts
Jun 9, 2026Kiril Georgiev, Yuxia Wang, Dimitar Iliyanov Dimitrov +2LLM Safety Benchmarks Multilingual Language Model Evaluation
Sofia University "St. Kliment Ohridski" · INSAIT, Sofia University “St. Kliment Ohridski” · MBZUAI
Culturally-Adapted Red-Teaming Across East and Southeast Asian Contexts: A Methodological and Comparative Analysis
Jun 8, 2026Hyeji Choi, Yongtaek Lim, Minwoo KimMultilingual Language Model Evaluation Language Model Safety Evaluation
AI Safety Team, DATUMO.INC, Seoul, South Korea.
GlobeAudio: A Multilingual Multicultural Benchmark for Naturalistic Evaluation of Large Audio-Language Models
Jun 6, 2026Ryner Tan, Wenxuan ZhangAudio-Language Model Evaluation Cross-Cultural Language Model Evaluation
Singapore University of Technology and Design
The Masked Advantage: Uncovering Local-Language Access to Cultural Knowledge in LLMs
Jun 5, 2026Yang Zhang, Xiao Fei, Amr Mohamed +6Multilingual Language Model Evaluation LLM Evaluation
Ecole Polytechnique · MBZUAI · ENS-PSL +1
MADE: Beyond Scoring via a Multilingual Agentic Diagnosing Engine for Fine-Grained Evaluation Insights
Jun 5, 2026Yilun Liu, Miao Zhang, Shimin Tao +9Multilingual Language Model Evaluation LLM Evaluation
Huawei, China
SEA-NLI: Natural Language Inference as a Lens into Southeast Asian Cultural Understanding
Jun 2, 2026Peerawat Chomphooyod, Jian Gang Ngui, Yosephine Susanto +5Multilingual Language Model Evaluation Cross-Cultural Language Model Evaluation
1Chulalongkorn University · 2AI Singapore · 3MBZUAI +1
When English Rewrites Local Knowledge: Global Narrative Dominance in Large Language Models
May 28, 2026Md Arid Hasan, Ruwad Naswan, Farhan Samir +2Multilingual Language Model Evaluation LLM Grounding
University of Toronto · BUET · University of Illinois Urbana-Champaign
ExCAM: Explainable Cultural Awareness Metrics
May 28, 2026Christoph Leiter, Haiyue Song, Hour Kaing +4LLM Evaluation Language Model Generation Evaluation
University of Mannheim, Germany · National Institute of Information and Communications Technology · University of Technology Nuremberg, Germany
SomaliBench Eval: Measuring English-to-Somali Refusal Gaps in Open-Weight Language Models
May 25, 2026Khalid Yusuf DahirLLM Safety Benchmarks Multilingual Language Model Evaluation
Independent researcher
ROK-FORTRESS: Measuring the Effect of Geopolitical Transcreation for National Security and Public Safety
May 13, 2026Michael S. Lee, Yash Maurya, Drew Rein +13Multilingual Language Model Evaluation Language Model Safety Evaluation
1Scale AI · 2Korea AI Safety Institute
XL-SafetyBench: A Country-Grounded Cross-Cultural Benchmark for LLM Safety and Cultural Sensitivity
May 7, 2026Dasol Choi, Eugenia Kim, Jaewon Noh +14LLM Safety Benchmarks Multilingual Language Model Evaluation
AIM Intelligence · Microsoft · Korea AISI +7
SemEval-2026 Task 7: Everyday Knowledge Across Diverse Languages and Cultures
May 4, 2026Nedjma Ousidhoum, Junho Myung, Carla Perez-Almendros +27Multilingual Language Model Evaluation LLM Evaluation
Cardiff University · KAIST · MBZUAI +13
When AI Speaks, Whose Values Does It Express? A Cross-Cultural Audit of Individualism-Collectivism Bias in Large Language Models
Apr 24, 2026Pruthvinath Jeripity VenkataLLM Evaluation Cross-Cultural Language Model Evaluation
Independent Researcher
The GaoYao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Models
Apr 22, 2026Yilun Liu, Chunguang Zhao, Mengyao Piao +14Multilingual Language Model Evaluation LLM Evaluation
Huawei, China · Fudan University, China
CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks
Apr 21, 2026Peiqin Lin, Chenyang Lyu, Wenjiang Luo +22Multilingual Language Model Evaluation LLM Evaluation
Alibaba Group · Beijing Language and Culture University · LMU Munich +6
AlignCultura: Towards Culturally Aligned Large Language Models?
Apr 21, 2026Gautam Siddharth Kashyap, Mark Dras, Usman NaseemCross-Cultural Language Model Evaluation Cultural Alignment
School of Computing, Macquarie University, Australia
Culturally Grounded Personas in Large Language Models: Characterization and Alignment with Socio-Psychological Value Frameworks
Jan 29, 2026Candida M. Greco, Lucio La Cava, Andrea TagarelliCultural Alignment Cross-Cultural Language Model Evaluation
DIMES, University of Calabria, Italy

Oct 8, 2026, cs.CLJ/K move · Enter open · S save

Measuring Cultural Alignment Beyond the Average: A Framework for Evaluating Maternal-Health LLM Interactions in Indian Contexts

Umaira Izhar, Gunjan Arora, Pushpendra Singh

Indraprastha Institute of Information Technology Delhi

Cultural Alignment Clinical Language Model Evaluation Healthcare Cross-Cultural Language Model Evaluation Personalized Language Model Alignment

Open paper AskPDF arXiv

Existing evaluation methods for healthcare LLMs primarily assess factual correctness,safety, and fluency, while providing limited insight into whether generated interactions reflect culturally situated healthcare reasoning. This limitation is particularly important in maternal health, where care decisions are shaped by social and relational norms. We introduce MH-INDIC, a culturally grounded evaluation framework for maternal-health interactions in urban and semi-urban North Indian contexts that operationalises cultural behaviour through ten dimensions of maternal-health reasoning. Using a 26-item survey administered to 102 pregnant and postpartum women from urban and semi-urban North India, we evaluate ten LLMs. We distinguish population level cultural alignment from profile-level behavioural variation. Although several models approximate the human population-level distribution, all evaluated systems exhibit substantially lower variation across demographic and household profiles than the human cohort, revealing a gap between aggregate alignment and profile-conditioned sensitivity. As a downstream application of MH-INDIC, we use the strongest-aligned proprietary and open-source models to generate culturally conditioned maternal-health dialogues under zero-shot, self-conditioned, and human-grounded prompting. Human-grounded conditioning produces stronger profile alignment and dialogue quality ratings, suggesting that measured cultural profiles can improve the cultural grounding of generated interactions

Momentum

Related topics

Latest papers 29