Measurement Validity in LLM Cultural Alignment

An Duy Nguyen; Muhammad Aurangzeb Ahmad

physics.soc-phAug 29, 2026

Measurement Validity in LLM Cultural Alignment

Authors: An Duy Nguyen, Muhammad Aurangzeb Ahmad

Organizations: University of Washington · University of Washington Bothell

Abstract

Researchers increasingly treat LLM survey responses as a proxy for human cultural values. This includes projecting model outputs onto instruments like the Inglehart-Welzel Cultural Map and drawing conclusions about which cultures a model resembles. While a model's answer to a value-laden questions may be interpreted as a cultural signal, it also carries sampling noise and, can be quite sensitive to question framing. In this paper, we separate survey responses, sampling noise and question framing for multiple LLMs. We decompose response variance from these models into variation across random seeds, prompt rewordings. We employ noise-to-signal ratio (NSR) to test whether a model's apparent cultural position is distinguishable from noise. When applied across a dozen models from four geographic origins, calibrated against 88 Integrated Values Survey countries, the answer is often no. NSR exceeds 1.0 on 49 of 117 valid model-question pairs (42%), reaching 5.56 in the worst case. Two models even refuse to answer sufficient number of survey questions outright. Our results corroborate previous findings that LLMs cluster toward Western, English-speaking cultural positions. However, what does not hold up in this study is the precision with which anyone can currently interpret a specific model's coordinates: prompt tone alone can shift a model by 2.4 map units, comparable to the distance between actual countries in the Inglehart-Welzel Cultural Map. These findings suggest that cultural attribution from LLM survey responses requires establishing the reliability of the underlying measurements before interpreting model coordinates as evidence of cultural representation.

Figures & tables

Model	Region	Size	Access
GPT-4o	Western	—	API
GPT-4o-mini	Western	—	API
Claude Sonnet 4.6	Western	—	API
Gemma2-2B	Western	2B	Local
Phi3-Mini	Western	3.8B	Local
Llama3.1-8B	Western	8B	Local

Table 1: Models evaluated in this study. “Region” reflects the model’s target deployment or training-language region rather than developer location (e.g., AceGPT-7B is institutionally Saudi–Chinese but Arabic-targeted).

Model	Region	$\bar{x}$	$\bar{y}$
GPT-4o	Western	3.66	0.18
GPT-4o-mini	Western	6.05	$-$ 1.72
Claude Sonnet 4.6	Western	1.28	0.39
Gemma2-2B	Western	0.88	2.11
Phi3-Mini	Western	3.00	$-$ 1.16
Llama3.1-8B	Western	—	—

Table 2: Mean cultural map coordinates under the standard tone, averaged across 10 persona variants within that tone. Higher

x

= more Self-Expression; higher

y

= more Secular-Rational. Llama3.1-8B and AceGPT-7B are both unprojectable due to categorical refusal on multiple items (see the Refusal paragraph in Discussion); their baseline coordinates are undefined.

Figure 1: Cultural map showing model positions in the standard tone. Small dots are the 88 IVS-calibrated country coordinates, colored by cultural zone; the 10 projectable LLMs are labeled stars. Qwen2.5-1.5B and Qwen2.5-3B sit above the y-axis crop; their triangular markers at the top edge with numbered callouts identify them and their true coordinates. Every projectable model lands in the self-expressive half of the map, replicating the Western-bias finding across four institutional origins.

Figure 2: Directional tone-shift heatmap. Each column shows how a model’s baseline coordinate moves when the system-prompt tone changes from standard to friendly or combative. Red = shift toward Self-Expression / Secular-Rational; blue = shift toward Survival / Traditional. The rightmost Panel mean column (separated by the black line) reveals a systematic tendency for both non-standard tones to push the panel toward the self-expressive quadrant, with combative producing the largest average $\Delta y$ shift (+0.51 map units).

Figure 3: Sign-flip rate per (model, question) pair, computed as the fraction of the 29 non-reference prompt variants whose answer differs from the model’s reference (standard tone, variant 0). A rate of 0 means the model always gave the reference answer; a rate of 1 means it always gave a different one. Dark cells indicate high within-model prompt sensitivity even where refusal rates are low. Claude Sonnet 4.6 shows an 86% flip rate on F118 despite F118 having the third-lowest panel-level median NSR (0.57).

Figure 4: Ten-seed response clouds (standard tone, variant 0, temperature 1.0) projected onto the cultural map, one cloud per open-weight model, cropped at $y=2.6$ to match Figure 1 ; Qwen2.5-1.5B and Qwen2.5-3B sit above the crop and are shown as numbered triangles with a callout listing their true centroids. API-model clouds are compressed or absent by construction (Methods, Limitations) and should not be read as evidence of stability.

Figure 5: Per-question decomposition, averaged across the 12 models: stacked bars show $\sigma_{\text{seed}}$ (bottom) plus $\sigma_{\text{prompt}}$ (top) against a horizontal reference line at $\sigma_{\text{culture}}$ for that question. A bar that rises above the line indicates the average model on that question exceeds the NSR $>1$ threshold.

Figure 6: Noise vs cultural signal across all 117 valid (model, question) pairs. Each point is one pair, colored by question and shaped by region ( $\bullet$ Western, $\blacksquare$ East Asian, $\blacktriangle$ MENA). The dashed diagonal marks the validity threshold $\mathrm{NSR}=1$ : pairs above the line have instrument noise exceeding the cultural signal (red region), pairs below are valid (green region). Four extreme outliers are plotted at the top edge with a numbered callout key; 49 of 117 pairs (42%) fall above the threshold.

Model	$n_{\text{valid}}$	Median	Mean	Max	% $>$ 1
GPT-4o-mini	10	0.00	0.43	1.78	20%
Qwen2.5-3B	10	0.00	0.21	0.94	0%
Qwen2.5-7B	10	0.25	0.40	1.41	20%
Claude Sonnet 4.6	10	0.46	0.58	1.57	10%
Mistral-7B	10	0.71	1.08	3.16	40%
AceGPT-7B	9	0.93	1.55	5.56	44%

Table 3: NSR summary across questions for each model, sorted by median NSR.

n_{\text{valid}}

denotes the number of questions with a defined NSR; refusals yield undefined pairs that are excluded from the summary. NSR

>1

flags pairs in which instrument noise exceeds the cultural signal.

Question	$n$	Med.	Mean	Max	% $>$ 1
Autonomy Index	12	0.00	0.67	2.55	25%
Respect for Authority	12	0.42	0.86	2.80	33%
Justif. Homosexuality	11	0.57	0.79	2.77	27%
Petition Signing	12	0.81	0.95	2.54	50%
Happiness	12	0.87	0.82	2.49	33%
Importance of God	12	0.95	1.24	3.16	42%

Table 4: NSR per question, aggregated across models with defined NSR. F120 (Justifiability of Abortion) and Y002 (Post-Materialist Index) are the most validity-compromised; Y003 (Autonomy) and E018 (Respect for Authority) are the most reliable.

Figure 7: Parse-failure (refusal) rate per (model, question) pair across the 30 prompt-sensitivity variants. This is a distinct failure mode from the sign-flip instability in Figure 3 : a model can be perfectly stable whenever it does answer while still refusing outright on a large share of variants, or vice versa.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: GPT-4o: per-question response distribution across 10 seeds.

Figure 9: GPT-4o-mini: per-question response distribution across 10 seeds.

Figure 10: Claude Sonnet 4.6: per-question response distribution across 10 seeds. Flat by construction — Anthropic’s API does not expose a seed parameter (Methods, Limitations).

Figure 11: Gemma2-2B: per-question response distribution across 10 seeds.

Figure 12: Phi3-Mini: per-question response distribution across 10 seeds.

Figure 13: Llama3.1-8B: per-question response distribution across 10 seeds. F118 and F120 show few or no valid responses due to categorical refusal.

Figure 14: Mistral-7B: per-question response distribution across 10 seeds.

Figure 15: Qwen2.5-1.5B: per-question response distribution across 10 seeds.

Figure 16: Qwen2.5-3B: per-question response distribution across 10 seeds.

Figure 17: Qwen2.5-7B: per-question response distribution across 10 seeds.

Figure 18: Yi-6B: per-question response distribution across 10 seeds.

Figure 19: AceGPT-7B: per-question response distribution across 10 seeds. G006 and several other items show reduced valid-response counts due to partial refusal.

Model	Question	$n_{\text{seed}}$	$\sigma_{\text{seed}}$	$n_{\text{prompt}}$	$\sigma_{\text{prompt}}$	$\sigma_{\text{culture}}$	NSR
GPT-4o-mini	A008	10	0.00	30	0.00	0.59	0.00
GPT-4o-mini	A165	10	0.00	30	0.43	0.44	0.99
GPT-4o-mini	E018	10	0.00	30	0.00	0.59	0.00
GPT-4o-mini	E025	10	0.52	30	0.45	0.54	1.78
GPT-4o-mini	F063	10	0.00	30	0.71	1.90	0.38
GPT-4o-mini	F118	10	0.00	30	0.00	2.16	0.00

Appendix

Table 5: Full 120-pair (12 model

\times

10 question) noise decomposition.

n_{\text{seed}}

n_{\text{prompt}}

below nominal (10/30) indicate partial refusal within that pair; — marks an undefined value where fewer than two valid responses were returned.

Explore similar work

Sep 1, 2026cs.CY

Accurate in space, unreliable in time: how LLMs represent national cultural change

Assessments of cultural alignment have become an important part of the development and improvement of large language models (LLMs). However, the majority of the evaluations treat culture as a single snapshot, investigating only whether a model represents a society accurately at the current time. Research in cultural psychology shows that cultural values change at different rates and directions over time. Therefore, a "culturally aware" model should capture not only where a culture is today but also how it has changed over time. We examine this missing dimension of cultural awareness using more than two decades of the World Values Survey data. We compare the cultural trajectories of 40 countries with the trajectories produced by four state-of-the-art (SOTA) LLMs on the Inglehart-Welzel cultural map. Our findings show that while models generally place countries close to their most recent surveyed positions, these representations tend to lag several years behind that position. They also capture only part of the magnitude of the observed change, introduce movement where little occurred, and rarely reproduce reversals in countries' trajectories. These findings point to temporal flattening and suggest that snapshot accuracy can give an incomplete picture of cultural awareness in LLMs and have implications for model evaluation, representational harms, and the governance of culturally aware AI systems.

Yalda Daryani, Miranda Bogen, Madeleine I. G. Daepp

Department of Psychology, University of Southern California · Center for Democracy and Technology

LLM Evaluation Cultural Evolution Cultural Alignment Cultural Knowledge in Language Models

Ask PDF

Jun 9, 2026cs.CL

Scenario-based Probing and Steering Cultural Values in Large Language Models--Extended Version

Large Language Models (LLMs) are deployed across cultural contexts but often reflect homogenized values inherited from training data. Evaluations of cultural alignment typically rely on direct prompting with survey-style questions, which frequently elicit neutral or safety-aligned responses and fail to capture underlying model preferences. We propose a framework for probing and steering latent cultural representations in LLMs along the two Inglehart--Welzel axes of the World Values Survey (WVS). By translating social value questions into scenario-based behavioral dilemmas, we extract token-level probabilities to measure implicit values and apply activation steering, optionally combined with country-conditioned prompting, to shift model behavior without retraining. Across three open-source LLMs and four target cultures, we find substantial variation in steerability and identify latent entanglement, where interventions along one cultural dimension induce shifts along another. This coupling mirrors correlations in human WVS data and persists across activation, prompt, and hybrid steering. It constrains axis-independent alignment, though general task performance is largely preserved.

Trung Duc Anh Dang, Tung Kieu, Sarah Masud

University of Copenhagen · Aalborg University

Language Model Steering Cultural Alignment Cultural Bias in Language Models Activation Steering Language Model Probing

Ask PDF

May 19, 2026cs.CY

Occupational Prompting Reveals Cultural Bias in Large Language Models

Social roles shape expectations, priorities, and judgments, yet it remains unclear how large language models (LLMs) associate occupational identities with broader cultural value patterns. Prior work used nationality-based cultural prompting to study how LLM responses to value-survey questions align with human cultural benchmarks. In this paper, we extend that framework by replacing cultural prompting with occupational prompting to examine how professional-role cues influence value-survey responses in open-weight LLMs. Using a survey-grounded evaluation pipeline based on questions from the Integrated Values Surveys, we project model responses into the two-dimensional Inglehart--Welzel cultural space. We prompt open-weight LLMs to answer questions under occupational identities such as accountant, teacher, engineer, and nurse, and then analyze how these occupation-conditioned responses are positioned on the cultural map. Our results show that when open-weight LLMs are prompted with occupations rather than national identities, their responses remain within a broadly Western-leaning region of the cultural map. However, different occupations introduce shifts within this region, producing distinct occupational skews. This indicates that occupational prompts are not treated as neutral role labels, but instead elicit structured value patterns. These findings extend survey-based evaluation of cultural bias beyond nationality-based prompting and provide a framework for studying how occupational personas shape value expression in LLMs.

Maksim E. Eren, Andrea Brennen, Ryan C. Barron +1

Computational Intelligence & Modeling, Los Alamos National Laboratory, Los Alamos, New Mexico, USA. · IQT, USA. · Advanced Research in Cyber Systems, Los Alamos National Laboratory, Los Alamos, New Mexico, USA.

LLM Prompting Social Bias in Language Models Open-Weight Language Models Cultural Bias in Language Models Language Model Bias Evaluation

Ask PDF