Measurement Validity in LLM Cultural Alignment

An Duy Nguyen; Muhammad Aurangzeb Ahmad

physics.soc-phAug 29, 2026

Measurement Validity in LLM Cultural Alignment

Authors: An Duy Nguyen, Muhammad Aurangzeb Ahmad

Organizations: University of Washington · University of Washington Bothell

Abstract

Researchers increasingly treat LLM survey responses as a proxy for human cultural values. This includes projecting model outputs onto instruments like the Inglehart-Welzel Cultural Map and drawing conclusions about which cultures a model resembles. While a model's answer to a value-laden questions may be interpreted as a cultural signal, it also carries sampling noise and, can be quite sensitive to question framing. In this paper, we separate survey responses, sampling noise and question framing for multiple LLMs. We decompose response variance from these models into variation across random seeds, prompt rewordings. We employ noise-to-signal ratio (NSR) to test whether a model's apparent cultural position is distinguishable from noise. When applied across a dozen models from four geographic origins, calibrated against 88 Integrated Values Survey countries, the answer is often no. NSR exceeds 1.0 on 49 of 117 valid model-question pairs (42%), reaching 5.56 in the worst case. Two models even refuse to answer sufficient number of survey questions outright. Our results corroborate previous findings that LLMs cluster toward Western, English-speaking cultural positions. However, what does not hold up in this study is the precision with which anyone can currently interpret a specific model's coordinates: prompt tone alone can shift a model by 2.4 map units, comparable to the distance between actual countries in the Inglehart-Welzel Cultural Map. These findings suggest that cultural attribution from LLM survey responses requires establishing the reliability of the underlying measurements before interpreting model coordinates as evidence of cultural representation.

Figures & tables

Model	Region	Size	Access
GPT-4o	Western	—	API
GPT-4o-mini	Western	—	API
Claude Sonnet 4.6	Western	—	API
Gemma2-2B	Western	2B	Local
Phi3-Mini	Western	3.8B	Local
Llama3.1-8B	Western	8B	Local

Table 1: Models evaluated in this study. “Region” reflects the model’s target deployment or training-language region rather than developer location (e.g., AceGPT-7B is institutionally Saudi–Chinese but Arabic-targeted).

Model	Region	$\bar{x}$	$\bar{y}$
GPT-4o	Western	3.66	0.18
GPT-4o-mini	Western	6.05	$-$ 1.72
Claude Sonnet 4.6	Western	1.28	0.39
Gemma2-2B	Western	0.88	2.11
Phi3-Mini	Western	3.00	$-$ 1.16
Llama3.1-8B	Western	—	—

Table 2: Mean cultural map coordinates under the standard tone, averaged across 10 persona variants within that tone. Higher

x

= more Self-Expression; higher

y

= more Secular-Rational. Llama3.1-8B and AceGPT-7B are both unprojectable due to categorical refusal on multiple items (see the Refusal paragraph in Discussion); their baseline coordinates are undefined.

Figure 1: Cultural map showing model positions in the standard tone. Small dots are the 88 IVS-calibrated country coordinates, colored by cultural zone; the 10 projectable LLMs are labeled stars. Qwen2.5-1.5B and Qwen2.5-3B sit above the y-axis crop; their triangular markers at the top edge with numbered callouts identify them and their true coordinates. Every projectable model lands in the self-expressive half of the map, replicating the Western-bias finding across four institutional origins.

Figure 2: Directional tone-shift heatmap. Each column shows how a model’s baseline coordinate moves when the system-prompt tone changes from standard to friendly or combative. Red = shift toward Self-Expression / Secular-Rational; blue = shift toward Survival / Traditional. The rightmost Panel mean column (separated by the black line) reveals a systematic tendency for both non-standard tones to push the panel toward the self-expressive quadrant, with combative producing the largest average $\Delta y$ shift (+0.51 map units).

Figure 3: Sign-flip rate per (model, question) pair, computed as the fraction of the 29 non-reference prompt variants whose answer differs from the model’s reference (standard tone, variant 0). A rate of 0 means the model always gave the reference answer; a rate of 1 means it always gave a different one. Dark cells indicate high within-model prompt sensitivity even where refusal rates are low. Claude Sonnet 4.6 shows an 86% flip rate on F118 despite F118 having the third-lowest panel-level median NSR (0.57).

Figure 4: Ten-seed response clouds (standard tone, variant 0, temperature 1.0) projected onto the cultural map, one cloud per open-weight model, cropped at $y=2.6$ to match Figure 1 ; Qwen2.5-1.5B and Qwen2.5-3B sit above the crop and are shown as numbered triangles with a callout listing their true centroids. API-model clouds are compressed or absent by construction (Methods, Limitations) and should not be read as evidence of stability.

Figure 5: Per-question decomposition, averaged across the 12 models: stacked bars show $\sigma_{\text{seed}}$ (bottom) plus $\sigma_{\text{prompt}}$ (top) against a horizontal reference line at $\sigma_{\text{culture}}$ for that question. A bar that rises above the line indicates the average model on that question exceeds the NSR $>1$ threshold.

Figure 6: Noise vs cultural signal across all 117 valid (model, question) pairs. Each point is one pair, colored by question and shaped by region ( $\bullet$ Western, $\blacksquare$ East Asian, $\blacktriangle$ MENA). The dashed diagonal marks the validity threshold $\mathrm{NSR}=1$ : pairs above the line have instrument noise exceeding the cultural signal (red region), pairs below are valid (green region). Four extreme outliers are plotted at the top edge with a numbered callout key; 49 of 117 pairs (42%) fall above the threshold.

Model	$n_{\text{valid}}$	Median	Mean	Max	% $>$ 1
GPT-4o-mini	10	0.00	0.43	1.78	20%
Qwen2.5-3B	10	0.00	0.21	0.94	0%
Qwen2.5-7B	10	0.25	0.40	1.41	20%
Claude Sonnet 4.6	10	0.46	0.58	1.57	10%
Mistral-7B	10	0.71	1.08	3.16	40%
AceGPT-7B	9	0.93	1.55	5.56	44%

Table 3: NSR summary across questions for each model, sorted by median NSR.

n_{\text{valid}}

denotes the number of questions with a defined NSR; refusals yield undefined pairs that are excluded from the summary. NSR

>1

flags pairs in which instrument noise exceeds the cultural signal.

Question	$n$	Med.	Mean	Max	% $>$ 1
Autonomy Index	12	0.00	0.67	2.55	25%
Respect for Authority	12	0.42	0.86	2.80	33%
Justif. Homosexuality	11	0.57	0.79	2.77	27%
Petition Signing	12	0.81	0.95	2.54	50%
Happiness	12	0.87	0.82	2.49	33%
Importance of God	12	0.95	1.24	3.16	42%

Table 4: NSR per question, aggregated across models with defined NSR. F120 (Justifiability of Abortion) and Y002 (Post-Materialist Index) are the most validity-compromised; Y003 (Autonomy) and E018 (Respect for Authority) are the most reliable.

Figure 7: Parse-failure (refusal) rate per (model, question) pair across the 30 prompt-sensitivity variants. This is a distinct failure mode from the sign-flip instability in Figure 3 : a model can be perfectly stable whenever it does answer while still refusing outright on a large share of variants, or vice versa.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: GPT-4o: per-question response distribution across 10 seeds.

Figure 9: GPT-4o-mini: per-question response distribution across 10 seeds.

Figure 10: Claude Sonnet 4.6: per-question response distribution across 10 seeds. Flat by construction — Anthropic’s API does not expose a seed parameter (Methods, Limitations).

Figure 11: Gemma2-2B: per-question response distribution across 10 seeds.

Figure 12: Phi3-Mini: per-question response distribution across 10 seeds.

Figure 13: Llama3.1-8B: per-question response distribution across 10 seeds. F118 and F120 show few or no valid responses due to categorical refusal.

Figure 14: Mistral-7B: per-question response distribution across 10 seeds.

Figure 15: Qwen2.5-1.5B: per-question response distribution across 10 seeds.

Figure 16: Qwen2.5-3B: per-question response distribution across 10 seeds.

Figure 17: Qwen2.5-7B: per-question response distribution across 10 seeds.

Figure 18: Yi-6B: per-question response distribution across 10 seeds.

Figure 19: AceGPT-7B: per-question response distribution across 10 seeds. G006 and several other items show reduced valid-response counts due to partial refusal.

Model	Question	$n_{\text{seed}}$	$\sigma_{\text{seed}}$	$n_{\text{prompt}}$	$\sigma_{\text{prompt}}$	$\sigma_{\text{culture}}$	NSR
GPT-4o-mini	A008	10	0.00	30	0.00	0.59	0.00
GPT-4o-mini	A165	10	0.00	30	0.43	0.44	0.99
GPT-4o-mini	E018	10	0.00	30	0.00	0.59	0.00
GPT-4o-mini	E025	10	0.52	30	0.45	0.54	1.78
GPT-4o-mini	F063	10	0.00	30	0.71	1.90	0.38
GPT-4o-mini	F118	10	0.00	30	0.00	2.16	0.00

Appendix

Table 5: Full 120-pair (12 model

\times

10 question) noise decomposition.

n_{\text{seed}}

n_{\text{prompt}}

below nominal (10/30) indicate partial refusal within that pair; — marks an undefined value where fewer than two valid responses were returned.

Explore similar work

Accurate in space, unreliable in time: how LLMs represent national cultural change
Sep 1, 2026Yalda Daryani, Miranda Bogen, Madeleine I. G. DaeppLLM Evaluation Cultural Evolution
Department of Psychology, University of Southern California · Center for Democracy and Technology
Scenario-based Probing and Steering Cultural Values in Large Language Models--Extended Version
Jun 9, 2026Trung Duc Anh Dang, Tung Kieu, Sarah MasudLanguage Model Steering Cultural Alignment
University of Copenhagen · Aalborg University
Occupational Prompting Reveals Cultural Bias in Large Language Models
May 19, 2026Maksim E. Eren, Andrea Brennen, Ryan C. Barron +1LLM Prompting Social Bias in Language Models
Computational Intelligence & Modeling, Los Alamos National Laboratory, Los Alamos, New Mexico, USA. · IQT, USA. · Advanced Research in Cyber Systems, Los Alamos National Laboratory, Los Alamos, New Mexico, USA.

Sep 1, 2026, cs.CYJ/K move · Enter open · S save

Accurate in space, unreliable in time: how LLMs represent national cultural change

Yalda Daryani, Miranda Bogen, Madeleine I. G. Daepp

Department of Psychology, University of Southern California · Center for Democracy and Technology

LLM Evaluation Cultural Evolution Cultural Alignment Cultural Knowledge in Language Models

Open paper AskPDF arXiv

Assessments of cultural alignment have become an important part of the development and improvement of large language models (LLMs). However, the majority of the evaluations treat culture as a single snapshot, investigating only whether a model represents a society accurately at the current time. Research in cultural psychology shows that cultural values change at different rates and directions over time. Therefore, a "culturally aware" model should capture not only where a culture is today but also how it has changed over time. We examine this missing dimension of cultural awareness using more than two decades of the World Values Survey data. We compare the cultural trajectories of 40 countries with the trajectories produced by four state-of-the-art (SOTA) LLMs on the Inglehart-Welzel cultural map. Our findings show that while models generally place countries close to their most recent surveyed positions, these representations tend to lag several years behind that position. They also capture only part of the magnitude of the observed change, introduce movement where little occurred, and rarely reproduce reversals in countries' trajectories. These findings point to temporal flattening and suggest that snapshot accuracy can give an incomplete picture of cultural awareness in LLMs and have implications for model evaluation, representational harms, and the governance of culturally aware AI systems.

Abstract

Explore similar work

Accurate in space, unreliable in time: how LLMs represent national cultural change

Scenario-based Probing and Steering Cultural Values in Large Language Models--Extended Version

Occupational Prompting Reveals Cultural Bias in Large Language Models