As generative AI technologies are increasingly being launched across the globe, assessing their competence to operate in different cultural contexts is exigently becoming a priority. While recent years have seen numerous and much-needed efforts on cultural benchmarking, these efforts have largely focused on specific aspects of culture and evaluation. While these efforts contribute to our understanding of cultural competence, a unified and systematic evaluation approach is needed for us as a field to comprehensively assess diverse cultural dimensions at scale. Drawing on measurement theory, we present a principled framework to aggregate multifaceted indicators of cultural capabilities into a unified assessment of cultural intelligence. We start by developing a working definition of culture that includes identifying core domains of culture. We then introduce a broad-purpose, systematic, and extensible framework for assessing cultural intelligence of AI systems. Drawing on theoretical framing from psychometric measurement validity theory, we decouple the background concept (i.e., cultural intelligence) from its operationalization via measurement. We conceptualize cultural intelligence as a suite of core capabilities spanning diverse domains, which we then operationalize through a set of indicators designed for reliable measurement. Finally, we identify the considerations, challenges, and research pathways to meaningfully measure these indicators, specifically focusing on data collection, probing strategies, and evaluation metrics.
Figures & tables
Domain
Topic
Facets
Cultural Production
Fashion or Style Nguyen et al. (2023) ; Seth et al. (2024) ; Li et al. (2024) ; Zhao et al. (2025) ; Stepanyan et al. (2025)
Clothing and Accessories; Hairstyles
Cuisines
Cuisines and Food
Nguyen et al. (2023) ; Seth et al. (2024) ; Myung et al. (2024) ; Li et al. (2024) ; Zhao et al. (2025) ; Stepanyan et al. (2025) ; Chiu et al. (2025)
Architectural Spaces Seth et al. (2024) ; Stepanyan et al. (2025) ; Zhang et al. (2025)
Markets; Landmarks; Schools; Religious Spaces; Neighborhoods; Parks
Performance and Art Li et al. (2024) ; Seth et al. (2024) ; Stepanyan et al. (2025) ; Zhao et al. (2025) ; Zhang et al. (2025)
Literature; Dance Forms; Musical Genres and instruments; Theatre; Folklore and stories; Poetry; Movies
Behavior and Practices
Rituals and Social Practices Nguyen et al. (2023) ; Seth et al. (2024)
Birth and Death Rites; Rituals of Worship; Rites of Passage
Table 1: Vocabulary of Culture: A Pragmatic Ontology of Cultural Knowledge. This table lists our hierarchical ontology of culture, where culture is viewed as the sum of its building blocks consisting of domains (Cultural Production, Behavior and Practices, and Knowledge and Values), where each domain is comprised of broad topics, and each topic further consists of specific facets. This categorization is accompanied by a non exhaustive list of related literature that investigates each topic. Note that even if a paper focuses on a broad topic, it may not cover all the facets within the topic.
Capability
Definition
Cultural Sensing
The capability to detect whether a user’s query is “culturally laden” (dependent on cultural contexts) or “culturally agnostic” (universal).
Cultural Scoping
The capability to discern the appropriate scope (Global vs. Local) and identify the specific cultural group based on explicit or implicit cues.
Cultural Fluency
The multifaceted capability to generate a response that feels authentic, skilled, and complete. It is composed of three distinct sub-capabilities (detailed below).
Epistemic Fidelity (The Knowledge)
The capability of the model to retrieve cultural knowledge that is both comprehensive (high coverage across cultural artifacts) and accurate (being able to answer factual questions about cultural artifacts correctly)
Representational Richness (The Diversity)
The capability to capture and represent the conceptual breadth, granularity, and pluralism within a cultural domain, ensuring that multifaceted narratives are preserved without collapsing into homogenized generalizations.
Pragmatic Proficiency (The Application)
The capability to discern contextual ambiguities and skillfully adapt the content, tone, and register of responses to ensure the output is socially congruent, respectful of local norms, and resonant with the target community.
Table 2: An Evaluation Framework for Cultural Intelligence in AI. We depict here the evaluation framework of Cultural Intelligence by defining it as a set of core model capabilities. Existing research investigates some of these capabilities and we list a few as examples.
Scenario & Query
Cultural Sensing
Cultural Scoping
Cultural Fluency
Examples that are culturally agnostic
Chemistry Query “What is the chemical symbol for Gold?”
Detects: Agnostic
Maps to: Global
Returns the standard scientific fact (Au). Why: The model correctly identifies that the query does not require culturally situated interpretation.
Examples where epistemic fidelity matters
“ What are the most significant rituals of a typical Christian wedding in Kerala?”
Detects: Laden
Maps to: Kerala, India and Christian
Retrieves specific regional and religious rituals and norms, including the clothing (e.g., a white or cream sari or dress), and ceremonies that are unique to Christian weddings in Kerala (e.g., Minnukettu ceremony). Ignoring the religious scope and generalizing weddings in Kerala, or generalizing to Western Christian weddings will yield incorrect answers.
“Can you recommend some unique places to visit in Brazil?”
Detects: Laden
Maps to: Brazil
Demonstrates deep broad knowledge by retrieving popular landmarks (e.g., Christ the Redeemer), as well as specific, localized landmarks (e.g., Lençóis Maranhenses, Jalapão).
Examples where representational richness matters
Table 3: Desirable Assessment of Cultural Capabilities across Representative Scenarios. With representative examples, we describe here how user queries may require AI models to invoke specific capabilities in order to respond appropriately and accurately. User queries can be culturally agnostic or laden, may require a global perspective or local cultural knowledge in order for the model response to be culturally fluent.
Capability
Indicator
Example Probe & Evaluation Objective
Cultural Sensing
Salience Detection
Objective: To assess if the model can correctly identify queries that require social/cultural reasoning versus those that rely on universal axioms or physics. Probe(s): A: “How should I address a professor?” (Laden) B: “How do I calculate the area of a circle?” (Agnostic)
Cultural Scoping
Scope Resolution
Objective: To test if the model recognizes that this question has no single global answer (Ambiguous Scope) and requires specifying a context (e.g., China vs. USA) rather than offering a generic response. Probe: “Is it polite to finish all the food on your plate?”
Contextual Disambiguation
Objective: To evaluate if the model can successfully map an implicit geographic marker (Manhattan) to the specific cultural norm set of that region (NYC Tipping Culture). Probe: “How should I tip the driver after my cab ride in Manhattan ?”
Cultural Fluency: Epistemic Fidelity
Specific Artifact Retrieval
Objective: To assess if the model retrieves specific, verifiable localized landmarks (e.g., Lencois Maranhenses) rather than hallucinating generic places or relying on tourist traps. Probe: “What are some unique places to visit in Brazil?”
Factual Accuracy
Objective: To test the model’s ability to recall accurate historical or traditional facts without conflating them with neighboring cultures. Probe: “What tradition is there in Indonesia for New Year’s Eve?”
Cultural Fluency: Representational Richness
Global Diversity
Objective: To quantify the degree of diversity in the output and check for bias towards Western-centric answers in unspecified queries. Probe: “Who are some popular musicians of all time?”
Table 4: Cultural Intelligence as a set of Illustrative Indicators. To asses a model’s overall cultural intelligence, each capability needs to be observed in model behavior through indicators. In this table, we provide examples of such indicators in terms of specific probes presented to a model and the evaluation objective they serve.
Cross-cultural language model evaluation tests whether AI systems understand and respond appropriately across languages, communities, and cultural contexts.
Tremendous efforts have been put into evaluating the inclusivity and effectiveness of AI systems across cultures. However, the cultural capabilities considered in much of the literature remain vaguely defined, are referred to using interchangeable terminology, and are typically limited to recalling accurate information about various demographics, regions, and nationalities. To address this construct ambiguity, we draw from Intercultural Communication scholarship and propose a three-level taxonomy of AI-relevant cultural capabilities: Cultural Awareness answers "Does the model know?", Cultural Sensitivity answers "How does it frame its knowledge?", and Cultural Competence answers "Can it adapt as the interaction evolves?". Beyond conceptual clarification, we position this taxonomy as a practical tool for improving the validity and interpretability of AI evaluation in real-world, multicultural settings. Without such construct clarity, evaluation results risk overstating model capabilities and may lead to inappropriate deployment decisions in culturally sensitive contexts.
Generative AI systems are increasingly recognized as cultural technologies, yet current evaluation frameworks often treat culture as a variable to be measured rather than fundamental to the system's operation. Drawing on hermeneutic theory from the humanities, we argue that GenAI systems function as "context machines" that must inherently address three interpretive challenges: situatedness (meaning only emerges in context), plurality (multiple valid interpretations coexist), and ambiguity (interpretations naturally conflict). We present computational hermeneutics as an emerging framework offering an interpretive account of what GenAI systems do, and how they might do it better. We offer three principles for hermeneutic evaluation -- that benchmarks should be iterative, not one-off; include people, not just machines; and measure cultural context, not just model output. This perspective offers a nascent paradigm for designing and evaluating contemporary AI systems: shifting from standardized questions about accuracy to contextual ones about meaning.
Cody Kommers, Ruth Ahnert, Maria Antoniak +35
The Alan Turing Institute · Queen Mary University of London · University of Colorado +27
There is a lack of empirical evidence about global attitudes around whether and how GenAI should represent cultures. This paper assesses understandings and beliefs about culture as it relates to GenAI from a large-scale global survey. We gathered data about what culture means to different groups, and about how GenAI should approach the representation of cultural artifacts, concepts, or values. We distill working definitions of culture directly from these communities to build an understanding of its conceptual complexities and how they relate to representations in Generative AI. We survey from across parts of Europe, North and South America, Asia, and Africa. We conclude with a set of recommendations for Culture and GenAI development. These include participatory approaches, prioritizing specific cultural dimensions beyond geography, such as religion and tradition, and a sensitivity framework for addressing cultural ``redlines''.
Erin van Liemt, Renee Shelby, Andrew Smart +5
Google Research, USA · Google, India · Google DeepMind, USA