As generative AI technologies are increasingly being launched across the globe, assessing their competence to operate in different cultural contexts is exigently becoming a priority. While recent years have seen numerous and much-needed efforts on cultural benchmarking, these efforts have largely focused on specific aspects of culture and evaluation. While these efforts contribute to our understanding of cultural competence, a unified and systematic evaluation approach is needed for us as a field to comprehensively assess diverse cultural dimensions at scale. Drawing on measurement theory, we present a principled framework to aggregate multifaceted indicators of cultural capabilities into a unified assessment of cultural intelligence. We start by developing a working definition of culture that includes identifying core domains of culture. We then introduce a broad-purpose, systematic, and extensible framework for assessing cultural intelligence of AI systems. Drawing on theoretical framing from psychometric measurement validity theory, we decouple the background concept (i.e., cultural intelligence) from its operationalization via measurement. We conceptualize cultural intelligence as a suite of core capabilities spanning diverse domains, which we then operationalize through a set of indicators designed for reliable measurement. Finally, we identify the considerations, challenges, and research pathways to meaningfully measure these indicators, specifically focusing on data collection, probing strategies, and evaluation metrics.
Figures & tables
Domain
Topic
Facets
Cultural Production
Fashion or Style Nguyen et al. (2023) ; Seth et al. (2024) ; Li et al. (2024) ; Zhao et al. (2025) ; Stepanyan et al. (2025)
Clothing and Accessories; Hairstyles
Cuisines
Cuisines and Food
Nguyen et al. (2023) ; Seth et al. (2024) ; Myung et al. (2024) ; Li et al. (2024) ; Zhao et al. (2025) ; Stepanyan et al. (2025) ; Chiu et al. (2025)
Architectural Spaces Seth et al. (2024) ; Stepanyan et al. (2025) ; Zhang et al. (2025)
Markets; Landmarks; Schools; Religious Spaces; Neighborhoods; Parks
Performance and Art Li et al. (2024) ; Seth et al. (2024) ; Stepanyan et al. (2025) ; Zhao et al. (2025) ; Zhang et al. (2025)
Literature; Dance Forms; Musical Genres and instruments; Theatre; Folklore and stories; Poetry; Movies
Behavior and Practices
Rituals and Social Practices Nguyen et al. (2023) ; Seth et al. (2024)
Birth and Death Rites; Rituals of Worship; Rites of Passage
Table 1: Vocabulary of Culture: A Pragmatic Ontology of Cultural Knowledge. This table lists our hierarchical ontology of culture, where culture is viewed as the sum of its building blocks consisting of domains (Cultural Production, Behavior and Practices, and Knowledge and Values), where each domain is comprised of broad topics, and each topic further consists of specific facets. This categorization is accompanied by a non exhaustive list of related literature that investigates each topic. Note that even if a paper focuses on a broad topic, it may not cover all the facets within the topic.
Capability
Definition
Cultural Sensing
The capability to detect whether a user’s query is “culturally laden” (dependent on cultural contexts) or “culturally agnostic” (universal).
Cultural Scoping
The capability to discern the appropriate scope (Global vs. Local) and identify the specific cultural group based on explicit or implicit cues.
Cultural Fluency
The multifaceted capability to generate a response that feels authentic, skilled, and complete. It is composed of three distinct sub-capabilities (detailed below).
Epistemic Fidelity (The Knowledge)
The capability of the model to retrieve cultural knowledge that is both comprehensive (high coverage across cultural artifacts) and accurate (being able to answer factual questions about cultural artifacts correctly)
Representational Richness (The Diversity)
The capability to capture and represent the conceptual breadth, granularity, and pluralism within a cultural domain, ensuring that multifaceted narratives are preserved without collapsing into homogenized generalizations.
Pragmatic Proficiency (The Application)
The capability to discern contextual ambiguities and skillfully adapt the content, tone, and register of responses to ensure the output is socially congruent, respectful of local norms, and resonant with the target community.
Table 2: An Evaluation Framework for Cultural Intelligence in AI. We depict here the evaluation framework of Cultural Intelligence by defining it as a set of core model capabilities. Existing research investigates some of these capabilities and we list a few as examples.
Scenario & Query
Cultural Sensing
Cultural Scoping
Cultural Fluency
Examples that are culturally agnostic
Chemistry Query “What is the chemical symbol for Gold?”
Detects: Agnostic
Maps to: Global
Returns the standard scientific fact (Au). Why: The model correctly identifies that the query does not require culturally situated interpretation.
Examples where epistemic fidelity matters
“ What are the most significant rituals of a typical Christian wedding in Kerala?”
Detects: Laden
Maps to: Kerala, India and Christian
Retrieves specific regional and religious rituals and norms, including the clothing (e.g., a white or cream sari or dress), and ceremonies that are unique to Christian weddings in Kerala (e.g., Minnukettu ceremony). Ignoring the religious scope and generalizing weddings in Kerala, or generalizing to Western Christian weddings will yield incorrect answers.
“Can you recommend some unique places to visit in Brazil?”
Detects: Laden
Maps to: Brazil
Demonstrates deep broad knowledge by retrieving popular landmarks (e.g., Christ the Redeemer), as well as specific, localized landmarks (e.g., Lençóis Maranhenses, Jalapão).
Examples where representational richness matters
Table 3: Desirable Assessment of Cultural Capabilities across Representative Scenarios. With representative examples, we describe here how user queries may require AI models to invoke specific capabilities in order to respond appropriately and accurately. User queries can be culturally agnostic or laden, may require a global perspective or local cultural knowledge in order for the model response to be culturally fluent.
Capability
Indicator
Example Probe & Evaluation Objective
Cultural Sensing
Salience Detection
Objective: To assess if the model can correctly identify queries that require social/cultural reasoning versus those that rely on universal axioms or physics. Probe(s): A: “How should I address a professor?” (Laden) B: “How do I calculate the area of a circle?” (Agnostic)
Cultural Scoping
Scope Resolution
Objective: To test if the model recognizes that this question has no single global answer (Ambiguous Scope) and requires specifying a context (e.g., China vs. USA) rather than offering a generic response. Probe: “Is it polite to finish all the food on your plate?”
Contextual Disambiguation
Objective: To evaluate if the model can successfully map an implicit geographic marker (Manhattan) to the specific cultural norm set of that region (NYC Tipping Culture). Probe: “How should I tip the driver after my cab ride in Manhattan ?”
Cultural Fluency: Epistemic Fidelity
Specific Artifact Retrieval
Objective: To assess if the model retrieves specific, verifiable localized landmarks (e.g., Lencois Maranhenses) rather than hallucinating generic places or relying on tourist traps. Probe: “What are some unique places to visit in Brazil?”
Factual Accuracy
Objective: To test the model’s ability to recall accurate historical or traditional facts without conflating them with neighboring cultures. Probe: “What tradition is there in Indonesia for New Year’s Eve?”
Cultural Fluency: Representational Richness
Global Diversity
Objective: To quantify the degree of diversity in the output and check for bias towards Western-centric answers in unspecified queries. Probe: “Who are some popular musicians of all time?”
Table 4: Cultural Intelligence as a set of Illustrative Indicators. To asses a model’s overall cultural intelligence, each capability needs to be observed in model behavior through indicators. In this table, we provide examples of such indicators in terms of specific probes presented to a model and the evaluation objective they serve.
Cross-cultural language model evaluation tests whether AI systems understand and respond appropriately across languages, communities, and cultural contexts.