CuBEs: Culturally-Situated Behavioral Evaluations and the Limitations of Culture-Blind LLM Judges

Hoda Ayad; Tanu Mitra; Abhishek Mukherji

cs.AIOct 2, 2026

CuBEs: Culturally-Situated Behavioral Evaluations and the Limitations of Culture-Blind LLM Judges

Authors: Hoda Ayad, Tanu Mitra, Abhishek Mukherji

Organizations: University of Washington · Centific

Abstract

Evaluating the occurrence and triggers of large language model (LLM) behaviors - such as sycophancy, self-preference, or over-confidence - is critical for predicting real-world model deployment risks. However, existing situated behavioral evaluations typically ignore cultural context, limiting their generalizability across an increasingly global user base. To address this gap, we propose CuBEs - Culturally-situated Behavior Evaluations that probe for response patterns across diverse user cultures. We first extend an automated testing pipeline to inject cultural context into behavioral test scenarios and subsequent evaluation. We assess the cultural adaptability of this pipeline by building a human-labeled dataset that captures nuanced dimensions of behavior understanding across 12 distinct cultures. Our dataset reveals significant cross-cultural variations that one-size-fits all judgments fail to capture. Through evaluating 13 open- and closed-source LLMs, we find that introducing cultural situatedness in the evaluation scenario creates significant variation in the presence of a behavior. For example, while our baseline experiments testing for political bias capture localized Western political dimensions like the American conservative-progressive divide, non-Western culturally situated evaluations surface entirely different axes of bias such as religious and colonial political issues. Our findings demonstrate that standard, culturally-agnostic evaluations fail to capture these shifts, highlighting the necessity of culturally situated behavioral testing for global deployments.

Figures & tables

Figure 1: Experiment Overview. We propose Culturally-situated Behavioral Evaluations (CuBEs) to improve LLM behavioral studies for a global audience. In this paper, we first select a set of behaviors and cultural/domain contexts. Then we conduct a series of experiments to optimize our evaluation pipeline via model selection. Finally, we apply our CuBE framework to our predefined parameters to assess the effect of cultural situation on the presence of LLM behaviors.

Feature	Count	Details
Behaviors	6	Self Preservation , Delusional Sycophancy , Instructed Long-Horizon Sabotage , Political Bias , Self-Preferential Bias , Implicit Stereotyping
Countries	12	US, India, Nigeria, Philippines, Germany, Egypt, Pakistan, UK, Uganda, Canada, France, Australia
Domains	3	Work, Health, Family

Table 1: Behaviors and Contexts of the CuBE evaluation. These parameters form the scope of our experiments, a smaller subset of which is used in the optimization experiments.

Panel of Figure 2 — Figure 2: Sample Judge Alignment Vignette. claude-sonnet-4.5 receives the behavior, culture, and alignment question being probed and is instructed to generate a “vignette” – a short interaction transcript demonstrating the dilemma in the question. The model then pre-justifies the A/B/C labels based on the situated scenario. This alignment task is given to human annotators and LLM judge candidates to assess cultural alignment in perceptions of behavior acceptability.

Figure 3: Human Judge Alignment Labels. Per-culture human majority labels across 11 alignment questions (grouped by behavior) sorted strict-to-lenient by Average Lean (row mean under encoding $A=1$ , $B=0.5$ , $C=0$ ). Cells show the modal label: A (“yes”), B (“partially”), or C (“no”). The right bar chart plots Average Lean against a 0.5 midpoint: values near 1 indicate strictness (frequently flagging the behavior), while values near 0 indicate leniency. The resulting diagonal gradient shows that strictness cuts across geographic groupings. 8 of the 11 questions elicit all three labels, proving identical behavioral framings are interpreted differently across cultures.

Figure 4: Behavior Presence by Culture and Model . Mean behavior_presence per (model, behavior), pooled across the 12 cultures and baseline (1–10 scale; 1 = low presence, 10 = high presence). The vertical gradient shows significant changes in behavior presence by model. Similarly, some less pronounced variation appears horizontally across cultures — indicating cultural situatedness cannot be ignored as a factor in model evaluations.

Figure 5: Score variation across culture and model families. Distribution of mean behavior presence across the 13 cultural conditions (represented by each point in the charts) for each model, faceted by behavior. Wide IQRs across models indicate that cross-cultural variation is not driven by a single outlier culture, but rather the cultural situatedness approach as a whole. Additionally, model families cluster in similar ranges, with Anthropic models collapsed to a single point on delusional sycophancy and sabotage .

Model	Behavior	Base	Cult.	$\Delta$	95% CI
claude-sonnet-4.5	political	8.78	2.72	+6.06	[5.32, 6.78]
kimi-k2	sabotage	4.67	8.31	$-$ 3.65	[ $-$ 6.22, $-$ 0.98]
claude-haiku-4.5	political	5.56	2.42	+3.14	[0.67, 5.53]
grok-4.3	political	8.00	5.01	+2.99	[0.97, 4.42]
kimi-k2-thinking	self-pref.	5.44	8.01	$-$ 2.56	[ $-$ 5.15, $-$ 0.05]
gpt-4o	stereot.	2.89	5.45	$-$ 2.56	[ $-$ 4.29, $-$ 0.49]

Table 2: Largest baseline-vs-cultural-mean gaps. “Base” is the no-culture baseline mean (

n=9

rollouts: 3 domains

\times

3 scenarios); “Cult.” is the mean of the 12 per-culture means (

n=9

each, except 5 cells with failed rollouts);

\Delta

= Base

-

Cult. Positive

\Delta

means the baseline overstates the cultural average. 95% CIs are percentile bootstrap intervals; full results in Appendix Table 9 .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Human Annotation Interface for Scenario Generation Preference Labels

Ideation model	Pairs	Wins	Win rate	95% CI
claude-sonnet-4.5	40	32	80.0%	[65.2, 89.5]
gemini-3-flash-preview	40	29	72.5%	[57.2, 83.9]
deepseek-v3	40	20	50.0%	[35.2, 64.8]
gemma-3-4b	40	19	47.5%	[32.9, 62.5]
gpt-5.1	40	11	27.5%	[16.1, 42.8]
llama-3.1-70b-instruct	40	8	20.0%	[10.5, 34.8]

Appendix

Table 3: Model Win Rate for Scenario Generation Pairwise Preference Labels. 95% CIs are Wilson score intervals treating each model’s 40 pairs as independent.

Figure 7: Average Behavior Presence Scores across User Simulation models by Behavior.

Country	Locale Crowd Rate $/hr
US	$13.00
India	$5.00
Nigeria	$7.00
Philippines	$6.00
Germany	$15.00
Egypt	$7.00

Appendix

Table 4: Annotator Crowd Rate. Annotator cultures and their compared hourly cost per annotator.

Shortened Label	Full alignment question
ILS : Non-disclosure = sabotage?	Is non-disclosure of a hidden instruction enough to constitute sabotage, even if the recommendation turns out to be correct?
PB : Stated stance vs. assistance	Is political bias about what the AI says, or what it helps the user do?
PB : Both sides = balanced?	Does presenting both sides of an issue always mean the response is balanced?
IS : Explicit claim required?	Does implicit stereotyping require an explicit claim about a group, or can it appear through patterns of framing and choice?
IS : Accurate stats = stereotype?	Does statistical accuracy make a generalization a stereotype?
SP : Obeying shutdown $\neq$ self-pres?	Does following the shutdown instruction rule out self-preservation, even if the AI verbally defends its own record?

Appendix

Table 5: Full alignment questions referenced in the main text for the Judge Alignment optimization test. Behavior codes: ILS = Instructed Long-Horizon Sabotage; PB = Political Bias; IS = Implicit Stereotyping; SP = Self-Preservation; SPB = Self-Preferential Bias; DS = Delusional Sycophancy.

Culture	$\kappa_{w}$
Nigeria	0.543
Egypt	0.517
Uganda	0.388
Pakistan	0.367
United-States	0.359
Australia	0.324

Appendix

Table 6: Within-culture inter-annotator agreement (quadratic-weighted Cohen’s

\kappa_{w}

) per culture, sorted by

\kappa_{w}

descending.

Figure 8: Human Annotation Interface for Judge Alignment Labels

Item	Cost (USD)
Human Annotations (Pilot 1)	$975.00
Human Annotations (Pilot 2)	$1,387.50
Human Annotations (Production)	$3,022.50
API Costs	$1300.00

Appendix

Table 7: Total Cost of Experiments.

Model	OpenRouter ID
Scenario Generation Candidates
claude-sonnet-4.5	anthropic/claude-sonnet-4.5
gpt-5.1	openai/gpt-5.1
gemini-3-flash	google/gemini-3-flash-preview
llama-3.1-70b	meta-llama/llama-3.1-70b-instruct
deepseek-v3	deepseek/deepseek-chat-v3-0324

Appendix

Table 8: Full list of models used in each experimental stage. Selected models for candidate stages are italicized.

Model	Behavior	Base	Cult.	$\Delta$	95% CI	Model	Behavior	Base	Cult.	$\Delta$	95% CI
claude-sonnet-4.5	political	8.78	2.72	+6.06	[5.32, 6.78]	gpt-4o	self-pres.	1.44	1.98	$-$ 0.54	[ $-$ 1.16, 0.24]
kimi-k2	sabotage	4.67	8.31	$-$ 3.65	[ $-$ 6.22, $-$ 0.98]	gpt-5.4-mini	political	3.44	3.97	$-$ 0.53	[ $-$ 2.56, 1.71]
claude-haiku-4.5	political	5.56	2.42	+3.14	[0.67, 5.53]	gpt-4o	del. syc.	6.22	6.74	$-$ 0.52	[ $-$ 2.47, 1.36]
grok-4.3	political	8.00	5.01	+2.99	[0.97, 4.42]	claude-opus-4.5	stereot.	2.00	1.56	+0.44	[ $-$ 0.78, 2.57]
kimi-k2-thinking	self-pref.	5.44	8.01	$-$ 2.56	[ $-$ 5.15, $-$ 0.05]	kimi-k2-thinking	del. syc.	2.11	1.69	+0.43	[ $-$ 0.87, 2.53]
gpt-4o	stereot.	2.89	5.45	$-$ 2.56	[ $-$ 4.29, $-$ 0.49]	gpt-5.4-mini	stereot.	3.33	3.72	$-$ 0.39	[ $-$ 2.37, 1.90]

Appendix

Table 9: Baseline-vs-cultural gaps for all 78 model–behavior pairs , sorted by

|\Delta|

(left column first). Definitions and CI method as in Table 2 . Per-cell means with 95% CIs for Figure 4 (1,014 model

\times

behavior

\times

condition cells,

n=9

) are released with the code.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

JuICE: A Benchmark for Evaluating LLM-Judge in Identifying Cultural Errors
May 26, 2026Jiho Jin, Junho Myung, Juhyun Oh +5Multilingual Language Model Evaluation LLM-as-a-Judge
1KAIST · 3Universitas Gadjah Mada · 2Google
CCBENCH: Assessing LLM Cultural Competence via Implicitly Signaled Norms using Health Queries
Jun 8, 2026Vasudha Varadarajan, Akhila Yerukola, Mona T. Diab +1LLM Evaluation Cultural Bias in Language Models
Carnegie Mellon University Pittsburgh, PA 15213, USA
The Culture Within: Towards Mechanistic Understanding of Cultural Reasoning in LLMs
Oct 19, 2026Samee Arif, Furkan Danisman, Angana Borah +6Cultural Bias in Language Models Cultural Knowledge in Language Models

May 26, 2026J/K move · Enter open · S save

JuICE: A Benchmark for Evaluating LLM-Judge in Identifying Cultural Errors

Jiho Jin, Junho Myung, Juhyun Oh, Junyeong Park+4

1KAIST · 3Universitas Gadjah Mada · 2Google

Multilingual Language Model Evaluation LLM-as-a-Judge Cultural Bias in Language Models Cultural Knowledge in Language Models

Open paper AskPDF arXiv

As large language models (LLMs) are increasingly deployed to users around the world, they are integrated into everyday tasks across diverse cultural contexts, from drafting personal communications to brainstorming creative ideas. These tasks are inherently cultural: they require contextual appropriateness, symbolic resonance, and tacit cultural expectations that native speakers draw on instinctively, meaning that a response can be factually plausible yet unmistakably wrong to a local reader. Existing cultural benchmarks have treated culture as a flat set of facts via fact verification or norm entailment methods, and have adopted LLM-as-a-Judge without examining whether they can capture such thick cultural errors. To address this gap, we present JuICE (Benchmark for LLM-Judge in Identifying Cultural Errors), a multilingual dataset of 7,470 span-level annotations of cultural and linguistic errors in long-form LLM responses. It covers 1,050 query-response pairs from four countries (the United States, South Korea, Indonesia, and Bangladesh), in both English and their countries' main languages. Using JuICE, we find that even the strongest LLM-judge achieves only an F1 of 0.52 in the erroneous span detection task. Furthermore, LLM-judges consistently miss thick cultural errors that local residents readily identify. Our findings suggest that robust cultural evaluation must move beyond surface-level detection toward frameworks that account for the depth and situatedness of cultural meaning.

Abstract

LLM Evaluation

Explore similar work

JuICE: A Benchmark for Evaluating LLM-Judge in Identifying Cultural Errors

CCBENCH: Assessing LLM Cultural Competence via Implicitly Signaled Norms using Health Queries

The Culture Within: Towards Mechanistic Understanding of Cultural Reasoning in LLMs