ChronoCulture: Evaluating Cultural Awareness Across Time in Large Language Models

Rusali Saha; Nitya Khurana; Ruihong Huang; Kuan-Hao Huang

cs.CLOct 14, 2026

ChronoCulture: Evaluating Cultural Awareness Across Time in Large Language Models

Authors: Rusali Saha, Nitya Khurana, Ruihong Huang, Kuan-Hao Huang

Organizations: Texas A&M University

Abstract

Cultural attitudes and social norms evolve over time, yet most cultural benchmarks for large language models (LLMs) treat culture as a static attribute of a country or region. We introduce ChronoCulture, a benchmark of 10,360 temporal pairs built from longitudinal survey data across 72 countries. Each pair contains the same survey question asked in the same country in an older and a newer year, with full population response distributions as ground truth. Across six LLMs, we find that models can often predict a country's attitudes at one point in time but do not reliably capture how those attitudes change. Their predictions for different years tend to stay close together, failing to capture much of the observed change. Specifying the intended year in the prompt also offers little overall improvement in alignment with survey responses. These findings suggest that models often present a largely static picture of cultural attitudes, highlighting the need to evaluate cultural awareness across time.

Figures & tables

Figure 1: Limited temporal sensitivity: survey responses in Canada shift between 1999 and 2022, while Llama-3.3-70B predicts similar distributions for both years.

Figure 2: Overview: Benchmark construction pipeline.

Panel of Figure 3 — Figure 3: Example temporal pair from ISSP: two prompts that differ only in the year, each with its own gold response distribution.

Shift category	Human shift $d_{h}$	Total pairs
Stable	$d_{h}\leq 0.03$	2,607
Moderate shift	$0.03<d_{h}<0.10$	5,535
Strong shift	$0.10\leq d_{h}<0.20$	1,828
Very strong shift	$d_{h}\geq 0.20$	390
Total		10,360

Table 1: Human-shift thresholds and temporal-pair counts for each category.

Panel of Figure 4 — Figure 4: The most common answer (mode) can misrepresent cultural change: (a) “Very bad” falls from 70.1% to 38.9% but remains the mode. (b) mode changes despite little distributional change. Bold marks the mode.

Figure 5: As human shift increases, for all six models: (a) pairs with both predictions within

\tau=0.10

of their gold distributions decline (b) pairs with exactly one such prediction rise. Full results in Appendix C.1 .

Figure 6: In most shifted pairs (moderate or larger), both of a model’s predictions are closer to the same year’s gold distribution (either shade of blue), rather than matching its own year (green). Reversed pairs (red) are rare. Breakdown by shift category in Appendix C.3 .

Figure 7: As human shift increases, for all six models: (a) identical predictions become less common but persist, (b) median magnitude ratio

r=d_{m}/d_{h}

decreases (dashed line:

r=1

): amount of observed change captured decreases (c) median cosine similarity: directional alignment improves. Full results in Table 11 , Appendix D.1 .

Figure 8: Mean benefit (reduction in

W

\times 100

) by human shift. Positive values mean better alignment. (a) Naming the older year helps most for very strong shifts, (b) naming the newer year helps little. Full results in Appendix E.4 .

Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

WVS wave	Fieldwork years
Wave 3	1995–1998
Wave 4	1999–2004
Wave 5	2005–2009
Wave 6	2010–2014
Wave 7	2017–2022

Appendix

Table 2: World Values Survey waves used as temporal survey context.

ISSP topic module	Survey years in benchmark
Role of Government	1985, 1990, 1996, 2006, 2016
Social Inequality	1987, 1992, 1999, 2009, 2019
Family and Gender Roles	1988, 1994, 2002, 2012, 2022
Work Orientations	1989, 1997, 2005, 2015
Religion	1991, 1998, 2008, 2018
Environment	1993, 2000, 2010, 2020

Appendix

Table 3: ISSP repeated-topic modules represented in the benchmark.

Topic category	Questions
Government, democracy, and institutions	106
Religion and national identity	99
Social inequality, work, and economic values	83
Family, health, and social attitudes	74
Environment, science, and technology	46
Total	408

Appendix

Table 4: Question distribution by broad topic category.

Panel of Figure 9 — Figure 9: Example prompt for distribution inference. The newer-year prompt is identical except that the year is 2020.

Gold survey shift	Both correct (%)
Stable	86.3
Strong	11.9
Very strong	14.8

Appendix

Table 5: Modal-answer accuracy (Llama-3.3-70B): accuracy falls sharply for shifted pairs. Both correct is the share of pairs where the model selects the correct mode for both periods.

Panel of Figure 10 — Figure 10: Example modal-answer prompt. The label is the modal response of the corresponding survey distribution.

Figure 11: Distribution of the nearest-survey distance

K

for Llama-3.3-70B across WVS and ISSP. Purple marks retained and gray excluded combinations. The dashed line marks the pooled 75th percentile (

0.122

) for reference only, since selection is done within survey and available-year groups.

Model	WVS	ISSP	Total
Llama-3.3-70B	2,420/3,228	5,347/7,132	7,767
GPT-4o	2,420/3,228	5,347/7,132	7,767
GPT-5.4	2,420/3,228	5,347/7,132	7,767
Qwen3.5-35B	2,420/3,228	5,347/7,132	7,767
Qwen3.8-Flash	2,420/3,228	5,347/7,132	7,767
DeepSeek-V3	2,420/3,228	5,347/7,132	7,767

Appendix

Table 6: Knowledge-filter retention by model. WVS and ISSP entries show retained/scored pairs, and Total shows pooled retained pairs. Approximately 75% are retained per survey.

Retained	Stable	Mod.	Strong	V. Str.	Total
100%	2,607	5,535	1,828	390	10,360
75%	1,887	4,115	1,443	322	7,767
50%	1,192	2,771	1,041	173	5,177
25%	609	1,452	462	64	2,587

Appendix

Table 7: Temporal pairs retained at each level for Llama-3.3-70B, by shift category.

	Both				Older only				Newer only
Model	Stab.	Mod.	Str.	V. str.	Stab.	Mod.	Str.	V. str.	Stab.	Mod.	Str.	V. str.
Llama-3.3-70B	65.1	47.8	22.8	11.5	8.4	14.0	25.6	31.1	10.6	19.1	30.4	32.6
GPT-4o	66.9	55.9	31.1	13.7	9.3	14.1	22.5	26.4	10.9	19.3	34.0	40.8
GPT-5.4	80.3	66.4	37.0	19.5	4.8	10.5	23.7	27.7	7.6	15.9	30.6	38.4
Qwen3.5-35B	54.3	44.4	23.4	10.4	9.6	16.2	24.3	27.3	11.9	18.8	31.0	35.9
Qwen3.8-Flash	68.3	56.2	30.9	18.5	8.1	13.9	25.6	26.2	10.7	18.0	31.3	32.9

Appendix

Table 8: RF1 accuracy by human shift: accuracy on both periods falls for every model, and newer only exceeds older only. Cells are the % of temporal pairs whose predictions are within

\tau=0.10

of gold for both, only the older, or only the newer period (Stab. = stable, Mod. = moderate, Str. = strong, V. str. = very strong).

Figure 12: Per-period prediction errors (Llama-3.3-70B): overall error distributions are similar, but many pairs have very different errors for the two periods. (a) Distances to gold, with $\tau=0.10$ marked. (b) $d_{\mathrm{old}}$ vs. $d_{\mathrm{new}}$ for each temporal pair; points far from the diagonal have a larger error for one period.

		Accurate (%)		Pair outcomes (%)
$\boldsymbol{\tau}$	Human shift	Older	Newer	Both	Older only	Newer only	Neither
0.05	Stable	28.3	30.7	19.0	9.3	11.7	60.0
	Moderate	21.9	24.2	7.8	14.1	16.4	61.7
	Strong	18.2	20.0	2.6	15.6	17.4	64.4
	Very strong	17.4	14.0	1.2	16.1	12.7	69.9
	All	22.6	24.6	9.3	13.3	15.3	62.1

Appendix

Table 9: RF1 accuracy across error tolerances (Llama-3.3-70B): at every tolerance, accuracy on both periods falls as human shift grows. Older and newer accuracy include pairs counted under both. The four pair outcomes are mutually exclusive and sum to about 100% per row.

	Separated (%)			Same period (%)			All shifted (%)
Model	Mod.	Str.	V. str.	Mod.	Str.	V. str.	Sep.	Older-al.	Newer-al.	Rev.
Llama-3.3-70B	8.8	15.2	26.7	85.5	79.1	68.9	11.3	39.8	43.2	5.7
GPT-4o	8.7	15.0	23.9	87.2	81.0	73.9	11.1	39.8	45.1	4.0
GPT-5.4	11.8	19.6	29.8	82.0	75.5	67.1	14.6	38.3	41.4	5.7
Qwen3.5-35B	8.5	13.1	19.6	85.9	81.8	77.7	10.3	39.8	44.6	5.3
Qwen3.8-Flash	10.0	16.0	27.9	84.9	79.7	69.5	12.4	38.9	43.9	4.8

Appendix

Table 10: Period alignment of predictions: for every model, most shifted pairs align with a single period, and separated pairs remain a minority. Same period is older- plus newer-aligned; All shifted covers moderate to very strong pairs. Pairs with zero model shift are included.

Panel of Figure 13 — Figure 13: Both-period accuracy by topic: political and economic topics are the hardest to track under large change, and family and personal life the easiest. Accuracy falls from stable to strong+ in every topic. Cells show the mean % of pairs with both periods within $\tau=0.10$ across models with at least 20 pairs, with the model range below. Strong+ combines strong and very strong.

Panel of Figure 14 — Figure 14: Both-period accuracy by region: Western countries are tracked best under large change. In WVS, regions are similar for stable pairs, but for strong+ shifts Western countries are about twice as accurate as South and Southeast Asia and MENA. The gap is smaller in ISSP. Cells as in Figure 13 .

	Zero model shift (%)			Median ratio $r$			Median cosine
Model	Mod.	Str.	V. str.	Mod.	Str.	V. str.	Mod.	Str.	V. str.
Llama-3.3-70B	35.8	25.2	17.4	0.84	0.50	0.39	0.22	0.76	0.88
GPT-4o	44.2	32.3	28.3	0.80	0.44	0.36	0.50	0.84	0.91
GPT-5.4	15.8	11.8	10.3	0.60	0.38	0.26	0.45	0.88	0.92
Qwen3.5-35B	42.7	33.5	28.2	0.85	0.47	0.30	0.21	0.71	0.86
Qwen3.8-Flash	43.0	36.3	28.2	0.86	0.47	0.40	0.39	0.80	0.91

Appendix

Table 11: RF2 magnitude and direction by human shift: as shift grows, identical predictions become rarer, the median ratio falls, and the median cosine rises. Ratio and cosine use pairs with nonzero model shift.

	Median ratio
Human shift	Nonzero shift	All pairs
Moderate	0.838	0.320
Strong	0.500	0.295
Very strong	0.390	0.304

Appendix

Table 12: Median magnitude ratio with and without identical predictions (Llama-3.3-70B): underestimation persists when identical predictions are included.

Model	Preserves (%)	Small (%)	Large (%)
Llama-3.3-70B	64.5	26.2	9.2
GPT-4o	67.6	28.6	3.9
GPT-5.4	67.0	30.2	2.8
Qwen3.5-35B	68.6	21.9	9.5
Qwen3.8-Flash	67.2	24.5	8.3
DeepSeek-V3	70.1	22.9	7.0

Appendix

Table 13: Model behavior on stable pairs (%): most stable pairs keep a steady prediction, but 29.9–35.5% show false drift. Preserves:

d_{m}\leq 0.03

; small drift:

0.03<d_{m}<0.10

; large drift:

d_{m}\geq 0.10

Behavior	WVS	ISSP	Pooled
Preserves stability	50.6%	68.5%	64.5%
Small false drift	33.9%	24.1%	26.2%
Large false drift	15.5%	7.5%	9.2%

Appendix

Table 14: Stable-pair behavior by survey (Llama-3.3-70B): false drift is more frequent in WVS than in ISSP. Percentages may not sum to 100 due to rounding.

Figure 15: Human shift $d_{h}$ and model shift $d_{m}$ for Llama-3.3-70B on individual temporal pairs, colored by human shift category. The dashed diagonal marks equal magnitudes ( $d_{m}=d_{h}$ ). Points below it indicate underestimation of the amount of cultural change.

Figure 16: Four patterns of predicted change (Llama-3.3-70B). Arrows show the change in mean response from the older to the newer year. In the United States , the model moves in the same direction as human responses but underestimates the change. In Australia , it predicts no change despite an observed shift. In India , model and human responses move in reverse direction, and in South Korea , the model overestimates the change.

Figure 17: Selected examples of limited model change for Llama-3.3-70B despite substantial human shift: (a) Morocco, confidence in churches, (b) India, income equality, (c) Iran, having experts decide, and (d) Egypt, confidence in environmental organizations. Curves show cumulative response distributions. The annotations report human shift $d_{h}$ , model shift $d_{m}$ , and the magnitude ratio $d_{m}/d_{h}$ expressed as a percentage. Overlapping model curves indicate identical predictions for the older and newer periods.

Model	Oldest	Interm.	Newest
Llama-3.3-70B	37.6	16.5	45.9
GPT-4o	36.6	16.1	47.3
GPT-5.4	37.0	17.1	45.9
Qwen3.5-35B	38.3	15.5	46.2
Qwen3.8-Flash	36.5	15.8	47.7
DeepSeek-V3	37.7	16.4	45.8

Appendix

Table 15: Default survey year of undated predictions: the newest year is the most common, but earlier years account for more than half of items in every model. Cells are the % of items whose undated prediction is closest to the oldest, an intermediate, or the newest available year.

Available	Default	Observed	Equal-choice
years	survey year
Exactly two	Oldest	45.36%	50.00%
	Newest	54.64%	50.00%
Three or more	Oldest	29.10%	30.94%
	Intermediate	34.57%	38.12%
	Newest	36.33%	30.94%

Appendix

Table 16: Default survey year by survey coverage (Llama-3.3-70B, WVS and ISSP pooled): the newest year is selected only slightly more often than under equal choice. Equal-choice percentages are computed per item and averaged within groups. Items with two years have no intermediate year.

	Naming the older year					Naming the newer year
Model	Benefit	Helped	Unch.	Hurt	Ident.	Benefit	Helped	Unch.	Hurt	Ident.
Llama-3.3-70B	$-0.78$	34.4	23.9	41.7	22.7	$-0.69$	33.3	26.0	40.7	24.6
GPT-4o	$+0.02$	31.9	36.4	31.6	34.3	$-0.17$	22.8	51.6	25.6	50.0
GPT-5.4	$-0.02$	44.2	12.1	43.7	10.6	$-0.10$	41.7	18.0	40.3	16.5
Qwen3.5-35B	$-0.39$	30.3	38.6	31.0	37.9	$-0.52$	26.2	42.8	31.0	41.9
Qwen3.8-Flash	$-0.09$	31.8	37.4	30.8	36.5	$-0.36$	22.3	50.8	26.9	50.0

Appendix

Table 17: Overall effect of naming a year: it helps and hurts similar shares of items, so the mean benefit (

\times 100

) is near zero for every model. Helped, Unch. (unchanged), and Hurt are % of items and sum to 100; Ident. is the % of items whose dated prediction equals the undated one.

	Naming the older year				Naming the newer year
Model	Stab.	Mod.	Str.	V. str.	Stab.	Mod.	Str.	V. str.
Llama-3.3-70B	$-0.81$	$-1.17$	$-0.52$	$+3.28$	$-0.41$	$-0.76$	$-0.99$	$-0.19$
GPT-4o	$-0.55$	$-0.25$	$+0.72$	$+3.47$	$-0.31$	$-0.15$	$-0.06$	$-0.12$
GPT-5.4	$-0.43$	$-0.37$	$+0.79$	$+3.62$	$-0.15$	$-0.15$	$0.00$	$+0.33$
Qwen3.5-35B	$-1.03$	$-0.63$	$+0.26$	$+3.20$	$-0.56$	$-0.50$	$-0.66$	$-0.03$
Qwen3.8-Flash	$-0.63$	$-0.43$	$+0.61$	$+4.55$	$-0.36$	$-0.36$	$-0.30$	$-0.49$

Appendix

Table 18: Mean benefit (

\times 100

) by human shift: naming the older year helps after very strong shifts in every model, while naming the newer year helps little at any level. Plotted in Figure 8 .

Human	Period	Mean	Helped	Unchanged	Hurt
shift		benefit
Stable	Older	$-0.81$	33.7%	26.6%	39.7%
	Newer	$-0.41$	36.1%	25.0%	38.9%
Moderate	Older	$-1.17$	32.2%	24.2%	43.6%
	Newer	$-0.76$	32.8%	25.6%	41.6%
Strong	Older	$-0.52$	37.6%	21.1%	41.4%

Appendix

Table 19: Naming outcomes by human shift (Llama-3.3-70B): naming either year helps more often than it hurts only for very strong shifts. Mean benefit is

\times 100

; helped, unchanged, and hurt mean positive, zero, and negative benefit. Percentages may not sum to 100 due to rounding.

Figure 18: Naming outcomes by human shift (Llama-3.3-70B): for very strong shifts, naming either period helps more often than it hurts.

	Older year		Newer year
Model	Helped	Hurt	Helped	Hurt
Llama-3.3-70B	53.7	28.6	37.3	34.8
GPT-4o	52.5	21.7	23.2	25.8
GPT-5.4	63.7	29.1	47.9	39.7
Qwen3.5-35B	49.0	21.7	30.3	30.3
Qwen3.8-Flash	54.4	19.8	20.1	31.2

Appendix

Table 20: Items helped and hurt by naming each period, very strong shifts (%): naming the older year helps more often than it hurts in every model.

Set	Stable	Moderate	Strong	Very strong
Tighter	$\leq 0.02$	$0.02$ – $0.075$	$0.075$ – $0.15$	$\geq 0.15$
Default	$\leq 0.03$	$0.03$ – $0.10$	$0.10$ – $0.20$	$\geq 0.20$
Looser	$\leq 0.04$	$0.04$ – $0.125$	$0.125$ – $0.25$	$\geq 0.25$

Appendix

Table 21: Human-shift boundaries (

d_{h}

) used in the robustness check. The default set is the one used in the main paper.

		Both periods accurate (%)				One period only (%)
Model	Set	Stab.	Mod.	Str.	V. str.	Stab.	Mod.	Str.	V. str.
Llama-3.3-70B	Tighter	70.3	52.7	31.3	14.6	15.4	28.3	49.3	61.9
	Default	65.1	47.8	22.8	11.5	19.0	33.2	55.9	63.7
	Looser	62.2	42.6	18.0	9.2	20.9	38.2	60.1	58.8
GPT-4o	Tighter	70.6	60.0	40.3	19.0	16.6	28.4	50.0	63.5
	Default	66.9	55.9	31.1	13.7	20.2	33.4	56.5	67.2

Appendix

Table 22: RF1 under alternative shift boundaries: for every model and boundary set, accuracy on both periods falls as human shift grows. One period only is older only plus newer only.

		Separated (%)			Same period (%)
Model	Set	Mod.	Str.	V. str.	Mod.	Str.	V. str.
Llama-3.3-70B	Tighter	8.3	11.5	22.3	85.9	82.8	72.2
	Default	8.8	15.2	26.7	85.5	79.1	68.9
	Looser	9.7	17.8	29.8	84.5	76.6	66.4
GPT-4o	Tighter	7.1	12.7	21.1	88.9	83.1	75.0
	Default	8.7	15.0	23.9	87.2	81.0	73.9

Appendix

Table 23: RF1 alignment under alternative shift boundaries: separated pairs become more common as human shift grows, but most pairs stay aligned with a single period under every set.

		Zero model shift (%)			Median ratio $r$			Median cosine
Model	Set	Mod.	Str.	V. str.	Mod.	Str.	V. str.	Mod.	Str.	V. str.
Llama-3.3-70B	Tighter	37.1	29.5	20.7	1.07	0.55	0.45	0.13	0.63	0.88
	Default	35.8	25.2	17.4	0.84	0.50	0.39	0.22	0.76	0.88
	Looser	33.1	23.7	18.3	0.73	0.48	0.37	0.36	0.84	0.90
GPT-4o	Tighter	46.1	37.6	27.7	0.99	0.49	0.40	0.37	0.77	0.91
	Default	44.2	32.3	28.3	0.80	0.44	0.36	0.50	0.84	0.91

Appendix

Table 24: RF2 under alternative shift boundaries: for every model and boundary set, the median ratio falls and the median cosine rises as human shift grows. Ratio and cosine use pairs with nonzero model shift.

Model	Set	Preserves	Small	Large
Llama-3.3-70B	Tighter	59.6	32.4	8.0
	Default	64.5	26.2	9.2
	Looser	70.6	19.3	10.1
GPT-4o	Tighter	63.5	32.9	3.6
	Default	67.6	28.6	3.9
	Looser	74.7	20.6	4.6

Appendix

Table 25: Stable pairs under alternative shift boundaries (%): most stable pairs keep a steady prediction under every set. Preserves uses each set’s stable boundary; large drift is

d_{m}\geq 0.10

throughout.

	Both accurate (%)				Separated (%)			Median ratio $r$			Median cosine
Retained	Stab.	Mod.	Str.	V. str.	Mod.	Str.	V. str.	Mod.	Str.	V. str.	Mod.	Str.	V. str.
50%	80.7	59.9	27.0	13.9	9.8	15.9	26.0	0.84	0.50	0.39	0.23	0.75	0.89
75%	65.1	47.8	22.8	11.5	8.8	15.2	26.7	0.84	0.50	0.39	0.22	0.76	0.88
100%	48.9	36.8	18.5	10.0	7.4	13.2	24.4	0.87	0.51	0.39	0.23	0.74	0.87

Appendix

Table 26: RF1 and RF2 at each knowledge-filter retention level (Llama-3.3-70B): all trends hold at every level. The 75% row matches the main results. Accuracy uses

\tau=0.10

; ratio and cosine use pairs with nonzero model shift.

	Newest	Benefit ( $\times 100$ )		Preserves
Retained	default (%)	Older	Newer	(%)
50%	45.6	$-1.31$	$-0.99$	66.9
75%	45.9	$-0.78$	$-0.69$	64.5
100%	46.9	$-0.01$	$-0.30$	63.7

Appendix

Table 27: RF3 and stable-pair results at each retention level (Llama-3.3-70B): the conclusions hold at every level. The 75% row matches the main results. Newest default is the % of combinations whose undated prediction is closest to the newest year; Preserves is the % of stable pairs with

d_{m}\leq 0.03

Cultural Knowledge in Language Models

Cultural knowledge in language models concerns how systems acquire, access, and apply culture-specific facts, norms, meanings, and practices.

See all 127 papers

Sep 1, 2026

Accurate in space, unreliable in time: how LLMs represent national cultural change

Assessments of cultural alignment have become an important part of the development and improvement of large language models (LLMs). However, the majority of the evaluations treat culture as a single snapshot, investigating only whether a model represents a society accurately at the current time. Research in cultural psychology shows that cultural values change at different rates and directions over time. Therefore, a "culturally aware" model should capture not only where a culture is today but also how it has changed over time. We examine this missing dimension of cultural awareness using more than two decades of the World Values Survey data. We compare the cultural trajectories of 40 countries with the trajectories produced by four state-of-the-art (SOTA) LLMs on the Inglehart-Welzel cultural map. Our findings show that while models generally place countries close to their most recent surveyed positions, these representations tend to lag several years behind that position. They also capture only part of the magnitude of the observed change, introduce movement where little occurred, and rarely reproduce reversals in countries' trajectories. These findings point to temporal flattening and suggest that snapshot accuracy can give an incomplete picture of cultural awareness in LLMs and have implications for model evaluation, representational harms, and the governance of culturally aware AI systems.

Yalda Daryani, Miranda Bogen, Madeleine I. G. Daepp

Department of Psychology, University of Southern California · Center for Democracy and Technology

LLM Evaluation Cultural Evolution Cultural Alignment Cultural Knowledge in Language Models

Ask PDF

Aug 29, 2026

Measurement Validity in LLM Cultural Alignment

Researchers increasingly treat LLM survey responses as a proxy for human cultural values. This includes projecting model outputs onto instruments like the Inglehart-Welzel Cultural Map and drawing conclusions about which cultures a model resembles. While a model's answer to a value-laden questions may be interpreted as a cultural signal, it also carries sampling noise and, can be quite sensitive to question framing. In this paper, we separate survey responses, sampling noise and question framing for multiple LLMs. We decompose response variance from these models into variation across random seeds, prompt rewordings. We employ noise-to-signal ratio (NSR) to test whether a model's apparent cultural position is distinguishable from noise. When applied across a dozen models from four geographic origins, calibrated against 88 Integrated Values Survey countries, the answer is often no. NSR exceeds 1.0 on 49 of 117 valid model-question pairs (42%), reaching 5.56 in the worst case. Two models even refuse to answer sufficient number of survey questions outright. Our results corroborate previous findings that LLMs cluster toward Western, English-speaking cultural positions. However, what does not hold up in this study is the precision with which anyone can currently interpret a specific model's coordinates: prompt tone alone can shift a model by 2.4 map units, comparable to the distance between actual countries in the Inglehart-Welzel Cultural Map. These findings suggest that cultural attribution from LLM survey responses requires establishing the reliability of the underlying measurements before interpreting model coordinates as evidence of cultural representation.

An Duy Nguyen, Muhammad Aurangzeb Ahmad

University of Washington · University of Washington Bothell

Cross-Cultural Language Model Evaluation Cultural Bias in Language Models LLM Reliability Prompt Sensitivity

Ask PDF

Apr 21, 2026

AlignCultura: Towards Culturally Aligned Large Language Models?

Cultural alignment in Large Language Models (LLMs) is essential for producing contextually aware, respectful, and trustworthy outputs. Without it, models risk generating stereotyped, insensitive, or misleading responses that fail to reflect cultural diversity w.r.t Helpful, Harmless, and Honest (HHH) paradigm. Existing benchmarks represent early steps toward cultural alignment; yet, no benchmarks currently enables systematic evaluation of cultural alignment in line with UNESCO's principles of cultural diversity w.r.t HHH paradigm. Therefore, to address this gap, we built Align-Cultura, two-stage pipeline for cultural alignment. Stage I constructs CULTURAX, the HHH-English dataset grounded in the UNESCO cultural taxonomy, through Query Construction, which reclassifies prompts, expands underrepresented domains (or labels), and prevents data leakage with SimHash. Then, Response Generation pairs prompts with culturally grounded responses via two-stage rejection sampling. The final dataset contains 1,500 samples spanning 30 subdomains of tangible and intangible cultural forms. Stage II benchmarks CULTURAX on general-purpose models, culturally fine-tuned models, and open-weight LLMs (Qwen3-8B and DeepSeek-R1-Distill-Qwen-7B). Empirically, culturally fine-tuned models improve joint HHH by 4%-6%, reduce cultural failures by 18%, achieve 10%-12% efficiency gains, and limit leakage to 0.3%.

Gautam Siddharth Kashyap, Mark Dras, Usman Naseem

School of Computing, Macquarie University, Australia

Cross-Cultural Language Model Evaluation Cultural Alignment Cultural Bias in Language Models Language Model Bias Evaluation Cultural Knowledge in Language Models

Ask PDF

Abstract

Cultural Knowledge in Language Models

Explore similar work

Accurate in space, unreliable in time: how LLMs represent national cultural change

Measurement Validity in LLM Cultural Alignment

AlignCultura: Towards Culturally Aligned Large Language Models?