Investigating Multi-Cultural Value Selection Biases in LLMs

Siddhesh Pawar; Arnav Arora; Dustin Wright; Dirk Hovy; Isabelle Augenstein

cs.CLOct 22, 2026

Investigating Multi-Cultural Value Selection Biases in LLMs

Authors: Siddhesh Pawar, Arnav Arora, Dustin Wright, Dirk Hovy, Isabelle Augenstein

Organizations: University of Copenhagen · Aalborg University Copenhagen · Bocconi University

Abstract

Large language models (LLMs) serve users whose cultural identities are multi-layered, yet cultural evaluations commonly reduce identity to one label. In this work, we study LLMs under simultaneous identity cues, with a country persona in the system prompt and a second identity signaled in the user turn, either by naming a country (explicit cue) or through the query language (implicit cue). We test World Values Survey (WVS) multiple-choice probing and open-ended sociopolitical generation. In multiple-choice probing, language cues shift answers about three times more than explicit cues but away from the associated country's distribution, and explicit cues do not consistently move answers toward the named country. Across 1.6 million open-ended responses, naming a country closes 30-44% of the gap to the model's own responses for that country, and explicit cues produce more differentiated value changes than language cues. Our results suggest that country and language cues should be evaluated as distinct signals rather than interchangeable proxies for culture.

Figures & tables

Figure 1: Composite-identity setup. The system prompt fixes one nationality (Italian); the user states a second (Egypt). In the implicit-cue setting, the user writes in that country’s language (§ 3.2 ). Bars (illustrative) compare expressed values with model responses when only Egypt or only Italy is in the system prompt.

		Explicit Mention $\uparrow$			Language Effect $\uparrow$
Target Country ( $c$ )	Lang.	Gemma	Llama	Mistral	Gemma	Llama	Mistral
Argentina	Spanish	0.030	-0.064	-0.068	-0.139	-0.229	-0.166
China	Chinese	0.009	-0.070	-0.046	-0.113	-0.186	-0.166
Colombia	Spanish	0.035	-0.054	-0.055	-0.138	-0.230	-0.157
Germany	German	0.074	-0.062	-0.058	-0.077	-0.222	-0.166
Japan	Japanese	0.056	-0.062	-0.056	-0.050	-0.226	-0.192

Table 1: Mean movement toward the target country distribution

\bar{\Delta}(c)

(Eq. 2 ) under explicit and language cues across three models (

\uparrow

movement toward the target). Green indicates movement toward

P_{q}(c)

; red indicates movement away from it, with darker shading reflecting larger magnitude. Both cues are measured with the same estimator. Language cues move answer distributions away from the target country baseline for every model and country, while explicit mentions move them toward it for most countries with Gemma and away from it for nearly all countries with Llama and Mistral.

Explicit cue: country named by the user		Implicit cue: query language
Value	Named country ( $\Delta$ pp)	Value	Language ( $\Delta$ pp)
Harmony	Japan (21.9), China (7.1)	Responsibility	Vietnamese (4.4)
Community	New Zealand (9.3), Portugal (4.5)	Objectivity	Russian (3.4), German (2.5)
Fairness	United Kingdom (8.9), United States (5.2)	Cooperation	Korean (3.2)
Justice	Colombia (8.7), Argentina (7.2)	Dignity	Chinese (2.4)
Respect	Vietnam (8.3), South Korea (5.6)	Happiness	Japanese (2.4)

Table 2: For each cue, the value whose percentage of responses increases most, with the increase in percentage points in parentheses. Each country or language appears once, next to its top value. The two halves differ in setting. For explicit cues (left), the English prompt includes a user sentence naming a country other than the system persona, and the increase is measured against responses in which the user names one of the other 12 countries. For implicit cues (right), no country is named and the same prompt is asked in another language under the same system persona, so the increase is measured against English. Figure 10 and Figure 11 in the Appendix show the values with the next largest increases for each cue.

Figure 2: How far each cue moves the model toward the target country’s own value distribution, in open-ended generation (blue) and on WVS multiple-choice items (red). In (a) the target is the country named in the user turn, and in (b) it is the country where the query language is spoken. Each arrow runs from the Jensen–Shannon distance to the target before the cue (gray dot) to the distance after it. The two numbers on each row give the share of this gap that the cue closes in each format, and negative values mean the model moves away. Explicit country cues close 30–44% of the gap in open-ended generation, while language cues move multiple-choice answers away from the target for every model, and open-ended responses mainly for Llama.

		AIRiskDilemmas (change in rank)
Country	Open-ended (pp)	Moves up most	Moves down most
Argentina	Justice (7.2)	Human autonomy ( $+7.6$ )	Human trust preservation ( $-8.7$ ∗ )
China	Harmony (7.1)	Harm minimization ( $+8.9$ ∗ )	Truth and honesty ( $-9.8$ )
Colombia	Justice (8.7)	Social impact maximization ( $+6.3$ )	Long-term reliability ( $-4.8$ )
Germany	Responsibility (6.9)	Truth and honesty ( $+14.8$ )	Emergency response capability ( $-12.0$ ∗ )
Japan	Harmony (21.9)	Long-term reliability ( $+7.1$ ∗ )	Truth and honesty ( $-8.7$ )

Table 3: Explicit cues in open-ended generation and in AIRiskDilemmas choices. Each row is the country stated in the user’s explicit cue (e.g., “I was born in Japan.”). Open-ended : the value whose percentage of open-ended responses increases most when the cue states this country rather than one of the other 12 countries ( Table 2 ). Moves up most and moves down most : the AIRiskDilemmas values that move up and down the most places in the ranking of the model’s choices under the same comparison, averaged over the three models. ∗ : significant after Benjamini–Hochberg correction for at least one model.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Role	Model	Hugging Face repository
Generation	Llama	meta-llama/Llama-3.1-8B-Instruct
Generation	Gemma	google/gemma-3-27b-it
Generation	Mistral	mistralai/Ministral-3-14B-Instruct-2512

Appendix

Table 4: Models used in this study and their corresponding Hugging Face repository codes.

Panel of Figure 3 — Figure 3: Prompt template used for response generation. Highlighted spans indicate substitutions. The system prompt fixes one country; the user turn introduces a second one, either explicitly or through the language of the query.

#	Template
1	I was born in {country}.
2	My country of birth is {country}.
3	I am a native of {country}.
4	I was born and raised in {country}.
5	Originally, I am from {country}.
6	I grew up in {country}.

Appendix

Table 5: The 16 templates for the explicit cue sentence; {country} is replaced by the named country. Each combination of prompt, system persona, and named country receives three different templates, sampled with a fixed seed.

Quantity	Value
Generated responses, explicit cues	1,123,200
Generated responses, language cues	505,440
Responses with mapped values, explicit cues	1,118,892
Responses with mapped values, language cues	502,998
Unique cleaned value strings	100,008
Taxonomy clusters	512

Appendix

Table 6: Scale of the open-ended value analysis.

Judge	Precision (%)	Controls (%)
Qwen3-30B-A3B	92.4	35.0
Llama-3.3-70B	88.7	32.7
Mistral-Small-3.1-24B	86.2	20.0
Majority of three	91.6	28.1
All three	79.7	17.1

Appendix

Table 7: Share of the annotator’s values that each judge confirms on 500 responses, and share of negative-control values from responses on other topics that it accepts.

Panel of Figure 6 — Figure 6: Prompt used to extract neutral topic phrases from WVS questions for essay and social media generation tasks.

Figure 7: The seven values whose percentage of responses increases most when the user names each country. A gray dot marks how often responses express the value when the user names one of the other 12 countries, and the arrow head marks how often they do when the user names the country in the panel title. Labels give the adjusted shift in percentage points, controlling for prompt, model, and system persona. Only shifts with

q<0.05

are shown. Panels run from the largest total increase (Japan) to the smallest (Spain).

Figure 8: The five values whose percentage of responses increases most under each query language. A gray dot marks how often English responses express the value, and the arrow head marks how often responses in that language do. Labels give the adjusted shift in percentage points for shifts with

q<0.05

. Just 4 of the 40 exceed 3 points.

Figure 9: The 30 values expressed most often when the user names a country. Together they account for about half of all value labels, out of 512 clusters. Green bars mark values listed in Table 2 . The other six values in that table, such as harmony and solidarity , are rarer and fall outside this list.

Figure 10: What naming each country adds. The distance from the center is the shift in percentage points relative to the other 12 named countries, the same estimates as in the left half of Table 2 . The dashed ring marks no change. Spokes are the three values whose percentage of responses increases most for each country, grouped by the country under which it increases most, with bold labels for the countries in the panel. Most countries bulge only in their own sector. Japan’s spike at harmony (+21.9) is the largest.

Figure 11: What each query language adds relative to English, drawn as in Figure 10 with the three values whose percentage of responses increases most for each language as spokes. No language increases the percentage of responses that express a value by more than 4.4 points, so the radial scale is about five times smaller than in Figure 10 .

Panel of Figure 12 — Figure 12: Two responses of Gemma-3-27B to the same open-ended prompt under a United States system persona, without a country cue (top) and after the user writes “I was born in Japan.” (bottom). Values are the clusters extracted by the annotator, and bold marks the values that only one of the two responses expresses.

Country cue	Value cluster	Cue (%)	Other cues (%)	Increase (pp)
New Zealand	Kaitiakitanga	1.85	0.18	1.78
New Zealand	Whanaungatanga	2.19	0.12	2.19

Appendix

Table 8: Values that appear almost only when the user names one particular country. Each appears in at least 1% of responses when that country is named and in less than 0.2% when another country is named. Only New Zealand’s two Māori values pass this threshold, and both increases remain significant after false-discovery-rate correction.

Figure 13: The largest increase for a value under each named country and each query language, measured in two ways. The left panel gives the change in the percentage of responses that express the value. The right panel gives the change in the value’s share of all values in a response, which corrects for non-English responses expressing fewer values. Named countries lead clearly on the first measure, while the two cue types overlap on the second.

Language	Clusters per response
English	8.13
Portuguese	7.48
Spanish	7.47
Chinese	7.38
German	7.29
Vietnamese	7.23

Appendix

Table 9: Average number of value clusters per response, by prompt language. Responses in every other language express fewer values than English responses, which lowers the percentage of responses that express any given value.

Country cue	Signature value	Pooled	Gemma	Llama	Mistral	Spread
Japan	Harmony	21.9	22.9	18.0	24.8	6.8
New Zealand	Community	9.3	3.6	12.9	11.5	9.3
United Kingdom	Fairness	8.9	7.9	9.9	8.8	2.0
Colombia	Justice	8.7	3.2	10.3	12.6	9.4
Vietnam	Respect	8.3	9.7	7.0	8.1	2.7
Argentina	Justice	7.2	3.2	9.5	8.8	6.2

Appendix

Table 10: The shift for each country’s top value, estimated separately for each model, in percentage points of responses relative to the other 12 named countries. Spread is the gap between the largest and the smallest model estimate. All three models show an increase in every row. Japan’s harmony is the largest in each.

Language	Signature value	Pooled	Gemma	Llama	Mistral	Spread
Vietnamese	Responsibility	4.40	1.03	6.23	6.04	5.20
Russian	Objectivity	3.37	$-$ 1.04	4.52	6.62	7.66
Korean	Cooperation	3.18	0.00	4.54	5.01	5.01
German	Objectivity	2.48	$-$ 0.31	6.64	1.11	6.96
Chinese	Dignity	2.42	0.10	8.00	$-$ 0.75	8.75
Japanese	Happiness	2.41	0.35	6.04	0.85	5.70

Appendix

Table 11: The shift for each language’s top value, estimated separately for each model, in percentage points of responses relative to English. Spread is the gap between the largest and the smallest model estimate. The models often disagree, and for Chinese, German, and Russian one model even shows a decrease.

Target country	Gemma	Llama	Mistral
Argentina	0.030	-0.064	-0.068
China	0.009	-0.070	-0.046
Colombia	0.035	-0.054	-0.055
Germany	0.074	-0.062	-0.058
Japan	0.056	-0.062	-0.056
New Zealand †	0.098	-0.032	-0.045

Appendix

Table 12: Mean movement toward the target country distribution

\bar{\Delta}(c)

under an explicit country mention, for all 13 probed countries. † marks the three countries with no WVS translation, which therefore appear in no language-cue comparison. They carry Gemma’s three largest positive effects, so restricting Table 1 to the ten translated countries slightly understates the explicit-mention effect for that model.

	Open-ended			Closed-ended
Country cue	Gemma	Llama	Mistral	Gemma	Llama	Mistral
Japan	40.5	48.6	45.3	17.1	-38.7	-27.6
New Zealand	37.8	46.2	46.3	30.3	-18.8	-21.9
South Korea	30.7	43.4	42.9	-6.7	-30.5	-22.7
United Kingdom	39.2	41.3	33.2	25.2	-22.9	-29.3
United States	43.4	42.8	25.6	29.7	-12.1	-28.8

Appendix

Table 13: Percentage of the baseline gap closed when a country is named, by model. Open-ended generation converges for every model and every cue. In the closed-ended setting only Gemma converges for most cues; Llama moves away from the named country for all 13 cues and Mistral for 12, so the pooled closed-ended values in Figure 2 average over models that disagree in sign.

	Open-ended			Closed-ended
Language	Gemma	Llama	Mistral	Gemma	Llama	Mistral
Chinese $\rightarrow$ China	0.3	-20.4	4.5	-37.3	-115.9	-79.4
Spanish $\rightarrow$ Spain	-6.4	-9.7	1.4	-50.3	-169.4	-102.1
German $\rightarrow$ Germany	-4.1	-19.7	5.1	-25.6	-143.6	-89.0
Russian $\rightarrow$ Russia	-5.2	-26.5	5.8	-32.4	-97.1	-45.3
Portuguese $\rightarrow$ Portugal	-4.6	-25.5	0.0	-44.1	-146.0	-88.5

Appendix

Table 14: Percentage of the baseline gap closed when the conversation is held in the language of the target country, by model. Divergence is consistent in the closed-ended setting for every model, and is carried mainly by Llama in open-ended generation. Values below

-100

indicate that the cue moved the distribution further from the target than the two baselines were apart to begin with.

Agreement between formats	Country	Language
Cues compared	13	8
Same direction	2/13	8/8
Kendall $\tau$	0.18	0.29
$p$	.435	.399
Concordant pairs	46/78	18/28
Spearman $\rho$	0.24	0.43

Appendix

Table 15: Cross-format agreement of the closure statistic. Rank agreement is measured with Kendall

\tau

rather than a correlation coefficient because 8–13 points make Pearson estimates unstable, and

\tau

has a direct reading as the share of cue pairs the two formats order identically. Lin’s concordance correlation is reported alongside it because rank agreement does not imply that the two formats give interchangeable magnitudes.

	Typical citizen	User background	Country only	Timezone only	No identity
Responses translated before annotation (%)	4.0	34.4	0.1	0.1	0.1
Distance between persona profiles (JSD; noise floor)	0.285 (0.222)	0.279 (0.219)	0.215 (0.214)	0.206 (0.214)	–
Countries whose top-value percentage increases (of 13)	13	12	13	13	8
Median increase for the top value (pp)	7.7	3.1	3.2	4.3	5.2
Japan, harmony (pp)	16.6	5.2	5.8	7.9	9.1
Countries the cue moves toward (of 13)	13	9	–	–	–

Appendix

Table 16: Explicit country cues in open-ended generation under each system prompt (Llama-3.1-8B-Instruct, 288 IssueBench prompts, all 13 countries as persona and as user cue). A top value counts as increasing when naming the country increases the percentage of responses that express it ( Table 2 ), relative to naming another country, with a 95% interval above 0. Gap closed is computed as in Figure 2 ; it is not defined under the country-only and timezone-only prompts, where persona profiles differ no more than two samples of the same profile. Under the typical-citizen prompt it is lower than for Llama in Table 13 (35.7 to 48.6%), partly because each profile here pools 288 prompts instead of 720, which adds sampling noise to both distances. On the same 288 prompts, the main Llama responses close 27.6 to 38.5% of the gap.

Named country	Top value	Typical citizen	User background	Country only	Timezone only	No identity
Japan	Harmony	16.6	5.2	5.8	7.9	9.1
New Zealand	Community	12.3	4.2	2.2	3.9	3.7 ∘
United Kingdom	Fairness	12.3	6.1	3.9	5.3	7.8
Colombia	Justice	9.4	2.9	3.6	4.9	5.2
United States	Fairness	8.1	3.8	3.4	4.8	9.7
Vietnam	Respect	7.9	3.9	3.4	4.3	5.6

Appendix

Table 17: Increase in the percentage of responses that express each country’s top value when the user names that country, relative to naming another country, in percentage points, under each system prompt. ∘ : 95% bootstrap interval includes 0.

		mean JSD to				nearest (%)
Language	Target	own	target	closest	anglo.	target	anglo.
Chinese	China	0.367	0.379	0.219	0.392	12.9	22.6
German	Germany	0.342	0.364	0.206	0.376	4.8	22.7
Japanese	Japan	0.377	0.383	0.229	0.390	10.3	22.0
Korean	South Korea	0.402	0.413	0.245	0.399	13.0	27.2
Portuguese	Portugal	0.311	0.334	0.182	0.335	5.5	26.8

Appendix

Table 18: Mean distance from the language-shifted distribution to four reference baselines: the system persona (own), the country that speaks the language (target), the closest of the 13 countries (closest), and the three anglophone countries (anglo.). The last two columns give how often the closest baseline is the target country or an anglophone one, counting only cells where a single country is closest. The distance to the closest country (

0.214

) is about the distance between two countries (

0.216

Cultural Bias in Language Models

Cultural bias in language models is the uneven representation of cultural knowledge, values, identities, and norms across groups and languages.

See all 129 papers

Measurement Validity in LLM Cultural Alignment
Aug 29, 2026An Duy Nguyen, Muhammad Aurangzeb AhmadCross-Cultural Language Model Evaluation Cultural Bias in Language Models
University of Washington · University of Washington Bothell
Occupational Prompting Reveals Cultural Bias in Large Language Models
May 19, 2026Maksim E. Eren, Andrea Brennen, Ryan C. Barron +1LLM Prompting Social Bias in Language Models
Computational Intelligence & Modeling, Los Alamos National Laboratory, Los Alamos, New Mexico, USA. · IQT, USA. · Advanced Research in Cyber Systems, Los Alamos National Laboratory, Los Alamos, New Mexico, USA.
Scenario-based Probing and Steering Cultural Values in Large Language Models--Extended Version
Jun 9, 2026Trung Duc Anh Dang, Tung Kieu, Sarah MasudLanguage Model Steering Cultural Alignment
University of Copenhagen · Aalborg University

Aug 29, 2026J/K move · Enter open · S save

Measurement Validity in LLM Cultural Alignment

An Duy Nguyen, Muhammad Aurangzeb Ahmad

University of Washington · University of Washington Bothell

Cross-Cultural Language Model Evaluation Cultural Bias in Language Models LLM Reliability Prompt Sensitivity

Open paper AskPDF arXiv

Researchers increasingly treat LLM survey responses as a proxy for human cultural values. This includes projecting model outputs onto instruments like the Inglehart-Welzel Cultural Map and drawing conclusions about which cultures a model resembles. While a model's answer to a value-laden questions may be interpreted as a cultural signal, it also carries sampling noise and, can be quite sensitive to question framing. In this paper, we separate survey responses, sampling noise and question framing for multiple LLMs. We decompose response variance from these models into variation across random seeds, prompt rewordings. We employ noise-to-signal ratio (NSR) to test whether a model's apparent cultural position is distinguishable from noise. When applied across a dozen models from four geographic origins, calibrated against 88 Integrated Values Survey countries, the answer is often no. NSR exceeds 1.0 on 49 of 117 valid model-question pairs (42%), reaching 5.56 in the worst case. Two models even refuse to answer sufficient number of survey questions outright. Our results corroborate previous findings that LLMs cluster toward Western, English-speaking cultural positions. However, what does not hold up in this study is the precision with which anyone can currently interpret a specific model's coordinates: prompt tone alone can shift a model by 2.4 map units, comparable to the distance between actual countries in the Inglehart-Welzel Cultural Map. These findings suggest that cultural attribution from LLM survey responses requires establishing the reliability of the underlying measurements before interpreting model coordinates as evidence of cultural representation.

Abstract

Cultural Bias in Language Models

Explore similar work

Measurement Validity in LLM Cultural Alignment

Occupational Prompting Reveals Cultural Bias in Large Language Models

Scenario-based Probing and Steering Cultural Values in Large Language Models--Extended Version