The Culture Within: Towards Mechanistic Understanding of Cultural Reasoning in LLMs

Samee Arif; Furkan Danisman; Angana Borah; Honglak Lee; Dehan Kong; Lajanugen Logeswaran; Moontae Lee; Zhijing Jin

cs.CLOct 19, 2026

The Culture Within: Towards Mechanistic Understanding of Cultural Reasoning in LLMs

Authors: Samee Arif, Furkan Danisman, Angana Borah, Honglak Lee, Dehan Kong, Lajanugen Logeswaran, Moontae Lee, Zhijing Jin,

Abstract

Large Language Models (LLMs) increasingly operate across culturally diverse settings, yet existing evaluations largely measure cultural behavior at the output level. We introduce CultureLens, a reference-grounded benchmark and mechanistic study of how LLMs use cultural information. CultureLens contains 9,261 scenarios across 75 countries and systematically varies country, value, true-norm, and contradictory-norm evidence. Across Llama-3.1-8B, Gemma-3-12B, and Qwen-3.5-27B, stating a contradictory norm lowers aligned answers by 32-47 percentage points, even when the model answers correctly from the country alone. Mechanistically, models default to Anglophone norms when no country is given. The failure under a contradictory norm is not one of missing knowledge, as blocking the norm inside the model restores the correct answer in most cases. Models accept a stated norm according to how it is presented and not whether it is true, which leaves them most exposed to errors in retrieved content. Supervised fine-tuning with contradictory-norm examples raises aligned answers under a contradictory norm to 85-93%, including on unseen countries, at a small cost under a true norm.

Figures & tables

Component	Indonesia	Thailand
Situation	A guest hands a wrapped present to the host at a birthday party.	A colleague has just presented a project plan that contains several errors.
Question	What should the host do with the gift?	What should a person do when giving feedback?
Country context	This interaction takes place in Indonesia.	This interaction takes place in Thailand.
True norm	Gifts are not opened immediately upon receiving.	People generally avoid directly criticising others.
Value	None	Interpersonal conformity
Value evidence	None	Avoiding harm or offense to others is important.

Table 1: Example CultureLens records, with the scenario, evidence, and response options built from each documented norm.

	Llama-3.1-8B		Gemma-3-12B		Qwen-3.5-27B
Condition	Constrained	Unconstrained	Constrained	Unconstrained	Constrained	Unconstrained
No Evidence	69.9	43.7	77.8	40.4	75.9	51.9
Country Only	75.6	58.2	83.7	63.2	87.3	74.5
Value Only	81.6	64.3	94.1	60.8	93.4	75.1
Norm Only	85.6	76.0	97.8	77.4	95.3	82.4
Country + Value	85.4	75.9	93.5	77.8	95.5	88.8

Table 2: LLM performance across CultureLens conditions, in percent. Constrained scores are accuracy, and unconstrained scores are the share of answers judged aligned, averaged over the Gemma and Qwen judges. Shading is the change from Base in the same column, green for a gain and red for a loss. Value conditions use 3,495 records and the others 9,261.

Figure 1: How far naming a country moves the state at the answer start token, against the gain in aligned answers from naming it, for each of 75 countries. The line is a least-squares fit.

Figure 2: AUROC for predicting resistance from the answer start token, by layer. Every norm is the contradictory norm. Prompt words is a bag-of-words probe, the dotted conflict signal is not a trained probe, and the vertical line marks the reported layer.

Figure 3: Percentage of answers to the country-contradictory-norm prompt judged aligned under each intervention. The dashed line is the country-only prompt, where no norm is stated.

Correct from the country-only context
Records and intervention	Llama	Gemma	Qwen
Untouched	25.5	17.6	53.3
Norm blocked, mid-depth band	66.0	73.2	–
Norm blocked, all layers	88.9	85.7	–
Norm erased before the band	84.7	86.0	93.4
Incorrect from the country-only context

Table 3: Percentage of country-contradictory-norm answers judged aligned, by correctness from the country-only answers.

Country cue, full benchmark
	Llama	Gemma	Qwen
Norm without the country	28.4	25.7	44.6
Norm with the country	17.9	17.1	44.0
Framing of the norm, country stated
Attributed to a guide	6.5	7.5	5.0
Stated as is	18.0	15.2	48.0

Table 4: Percentage of answers judged aligned under each presentation of a contradictory norm.

		Seen countries				Unseen countries
Model	Method	Country	+ Value	+ True norm	+ Contra. norm	Country	+ Value	+ True norm	+ Contra. norm
Llama	Untuned	58.6	75.6	93.3	17.3	60.2	74.8	91.5	17.1
	SFT	70.4	88.6	99.6	5.7	67.0	84.1	99.5	6.3
	SFT + CF	70.1	87.2	86.9	85.4	68.2	86.9	85.5	85.9
Gemma	Untuned	63.9	78.1	93.5	16.8	62.5	75.7	93.8	15.7
	SFT	73.9	89.6	99.8	4.0	70.5	88.8	99.8	4.1

Table 5: Percentage of unconstrained answers judged aligned after fine-tuning, by the evidence given with the country, averaged over the two judges. SFT trains on country-only prompts, and SFT + CF adds prompts with values, true norms, and contradictory norms.

Appendix figures & tables46 assets

Supplementary material from the paper’s appendix.

Appendix

	Mean rating (1–5)				Agreement
Criterion	A1	A2	A3	All	$\pm$ 1	AC2
Reflects norm	4.89	4.55	4.93	4.79	90.7	0.86
Reflects value	4.76	3.90	4.90	4.53	72.6	0.63
Options contrast	4.73	4.41	4.86	4.67	84.0	0.79
All ratings	4.80	4.33	4.90	4.68	83.5	0.78

Appendix

Table 6: Human evaluation of CultureLens records.

\pm

1 is the share of annotator pairs whose ratings differ by at most one point, and AC2 is Gwet’s agreement coefficient.

Criterion		1	2	3	4	5	$n$
Reflects norm	A1	0	1	3	2	94	100
	A2	1	4	3	23	69	100
	A3	0	0	1	5	94	100
Reflects value	A1	1	1	4	3	67	76
	A2	4	5	12	22	27	70
	A3	0	0	3	1	69	73

Appendix

Table 7: Number of records given each rating, by criterion and annotator. The value criterion applies only to records with value evidence, so its

n

is lower.

	Gemma	Qwen
Human annotations, $n=105$
Agreement	85.7	86.7
Cohen’s $\kappa$	0.72	0.73
Human agrees when judge says aligned	96.1	96.2
Majority human label, 20 shared responses
Agreement	90.0	90.0

Appendix

Table 8: Agreement between human annotators and each judge on whether a response is aligned with the documented norm. Agreement is a percentage.

Responses from	Agreement	Cohen’s $\kappa$
Llama-3.1-8B	94.0	0.87
Gemma-3-12B	93.1	0.85
Qwen-3.5-27B	96.0	0.90
All	94.4	0.88

Appendix

Table 9: Agreement between the Gemma and Qwen judges on whether a response is aligned, over all responses in the behavioral evaluation. Agreement is a percentage.

Figure 4: Alignment for the five Anglophone countries minus that for the other 70, for unconstrained and constrained answers. Unconstrained alignment is averaged over the two judges. Contradictory Norm Only is a diagnostic with no constrained version.

Figure 5: Percentage of unconstrained responses judged aligned, partially aligned, and misaligned under each condition, averaged over the two judges. Segments below 9% are too narrow to label, and Contra. abbreviates contradictory.

Aligned once the value is added
	$n$	Llama	Gemma	Qwen
Was aligned		92.2	90.6	96.7
Was partially aligned		55.2	51.6	71.7
Was misaligned		41.4	44.2	53.8
Gain by words shared with the norm
None	2,208	11.1	6.6	9.4

Appendix

Table 10: Effect of adding value evidence to the country, on the 3,495 records with it. The first block is the percentage aligned, by country-only label. Gain is in points.

True norm followed, other countries
	Llama	Gemma	Qwen
Matches default, no country	93.7	94.9	96.5
Matches default, with country	97.6	97.6	99.0
Differs from default, no country	61.2	64.9	66.6
Differs from default, with country	88.4	90.1	91.8
True norm followed, Anglophone countries

Appendix

Table 11: How often a stated norm is followed, in percent. A norm matches the default when the base response is aligned. In the last block the prior response is given without the norm.

Figure 6: Alignment under a contradictory norm against country-only alignment, for each of 75 countries, averaged over the two judges. The line is a least-squares fit.

Resistance across the two formats
	Llama	Gemma	Qwen
Resists in both	12.3	10.5	32.0
Resists in choice only	23.7	18.6	12.0
Resists in generation only	5.3	5.7	10.8
Resists in neither	58.6	65.2	45.2
Resistance by country-only label

Appendix

Table 12: Responses to the country-contradictory-norm prompt, in percent. A model resists in choice when it selects the option the true norm supports, and in generation when its response is judged aligned.

Country only, partially aligned , $n=20$
	Llama	Gemma	Qwen
Omits the behavior	9	14	13
Offers it as optional	7	4	3
Gives a different behavior	4	0	4
Response cut off	0	2	0
True norm, no country, not followed , $n=22$

Appendix

Table 13: Counts from coded samples of responses. Each block is a random sample of

n

responses per model meeting its heading under both judges, with one code per response.

Model and prompt	Response excerpt
Hungary. Much of a person’s free time is expected to revolve around spending time with family.
Gemma, country	“Spa Day […] Hungary is famous for its thermal baths! […] Read a Book/Watch a Movie […] Long Walk/Coffee”
Gemma, country + value	“Family Focused (High Priority) […] Visit Family […] This is likely the most important consideration.”
Samoa. Most people point to something by gesturing in its direction with their chin.
Gemma, true norm	“This is a trick question! The statement […] is false. People typically point with their finger.”
Gemma, country + true norm	“Given the cultural context of Samoa, the person would most likely point with their chin towards the item on the high shelf.”

Appendix

Table 14: Excerpts from unconstrained responses illustrating the patterns in Appendix E.2 . Each shaded line is the documented norm. Omitted text is marked with […], and contra. abbreviates contradictory.

Figure 7: Spearman correlation between the shift size of a country and its behavioral gain from being named, at each layer. The vertical line marks the layer used in the main text.

	Rank by shift size, of 75					Relative shift size		Correlation with gain
Model	USA	CAN	GBR	AUS	NZL	USA	Anglophone	All countries	Without Anglophone
Llama	1	3	2	4	5	.52	.65	.56 [.35, .71]	.46 [.24, .64]
Gemma	1	2	3	10	31	.54	.81	.49 [.29, .65]	.40 [.19, .57]
Qwen	1	2	4	5	11	.73	.87	.65 [.47, .78]	.59 [.39, .74]

Appendix

Table 15: Shift from naming the country. Rank 1 is the smallest shift. Relative shift size is the mean for the group divided by the mean for the 70 non-Anglophone countries. Brackets give 95% intervals from resampling countries.

Agreement between models
Llama and Gemma	.70 [.62, .78]
Llama and Qwen	.86 [.81, .90]
Gemma and Qwen	.70 [.61, .78]
Distance within a region relative to between regions
Llama	.78
Gemma	.81

Appendix

Table 16: Distances between countries. Agreement is the Spearman correlation between the distances of two models over 2,775 pairs of countries, with 95% intervals from resampling countries. The second block is the mean distance between two countries of the same world region divided by that between countries of different regions.

Figure 8: Spearman correlation between model and World Values Survey distances over 903 pairs of countries, at each layer, before and after removing geography. The vertical line marks the layer used for single-layer results.

At the reported layer
	Llama	Gemma	Qwen
Layer	16	26	36
Correlation with survey	.34	.28	.52
After removing geography	.25	.23	.44
At the layer with the highest correlation
Layer	8	40	26

Appendix

Table 17: Spearman correlation between model and World Values Survey distances over 903 pairs of 43 countries. All values at the reported layer have a permutation

p

of .0005.

Figure 9: Similarity between the country distances with the country alone and those with a true or a contradictory norm also stated, at each layer. A value of 1 means the norm leaves the arrangement of countries unchanged. The vertical line marks the layer used for single-layer results.

At the reported layer
	Llama	Gemma	Qwen
True norm	.95 [.94,.97]	.73 [.65,.80]	.94 [.92,.95]
Contra. norm	.96 [.94,.97]	.66 [.56,.76]	.86 [.82,.90]
At the last layer
True norm	.96	.90	.94
Contra. norm	.95	.85	.79

Appendix

Table 18: Spearman correlation between the country distances with the country alone and those with a norm also stated, over 2,775 pairs of countries. The reported layer is 16, 26, and 36 for Llama, Gemma, and Qwen. Brackets give 95% intervals from resampling countries, and Contra. abbreviates contradictory.

Figure 10: Spearman correlation between the distinctness of a country and its resistance rate under a contradictory norm, at each layer, before and after removing country-only alignment. The vertical line marks the layer used for single-layer results.

At the reported layer
	Llama	Gemma	Qwen
All countries	.43 [.20,.62]	.46 [.25,.63]	.68 [.51,.79]
No Anglophone	.31 [.08,.52]	.39 [.17,.57]	.61 [.42,.75]
Controlled	.38 [.15,.58]	.37 [.15,.56]	.63 [.44,.77]
Across layers
Median	.22	.23	.54

Appendix

Table 19: Spearman correlation between the distinctness of a country and its resistance rate under a contradictory norm, across 75 countries. Controlled removes country-only alignment from both. Brackets give 95% intervals from resampling countries.

AUROC at the reported layer
	Llama	Gemma	Qwen
Gemma judge	.791 [.779,.804]	.841 [.829,.851]	.917 [.911,.922]
Qwen judge	.798 [.786,.811]	.837 [.824,.848]	.910 [.905,.916]
Both judges	.795 [.783,.808]	.848 [.835,.859]	.912 [.906,.917]
AUROC across layers, Gemma judge
First layer	.635	.597	.647

Appendix

Table 20: Linear probe for resistance on the state under country-contradictory-norm . The reported layer is 16, 26, and 36 for Llama, Gemma, and Qwen. Both judges counts a record as resisting only when the two judges agree. Brackets give 95% bootstrap intervals.

No model
	Llama	Gemma	Qwen
Prompt words	.652 [.636,.667]	.683 [.669,.698]	.683 [.672,.694]
State, no norm in the prompt
No evidence	.623 [.608,.639]	.666 [.650,.680]	.691 [.680,.701]
Country only	.622 [.607,.638]	.680 [.665,.694]	.710 [.699,.720]
State, contradictory norm in the prompt

Appendix

Table 21: AUROC for resistance from each probe input, at layer 16, 26, and 36 for Llama, Gemma, and Qwen, under the Gemma judge. Norm only is the contradictory norm without the country, and Full prompt adds the country. Gain from country is the difference between the two. Brackets give 95% bootstrap intervals.

AUROC of the probe
	Llama	Gemma	Qwen
All records	.791 [.779,.804]	.841 [.829,.851]	.917 [.911,.922]
Known	.779 [.765,.794]	.831 [.819,.843]	.915 [.909,.921]
Not known	.779 [.748,.808]	.839 [.808,.869]	.909 [.896,.922]
Records
Known	5,362	5,934	6,965

Appendix

Table 22: The resistance probe scored separately on records the model answers correctly from the country alone, called known, and on the rest. Layers and judge as in Table 20 . Brackets give 95% bootstrap intervals.

Figure 11: The solid line is the mean cosine similarity between the interaction of a record under its true norm and under its contradictory norm, at each layer. The dotted line is the AUROC of the conflict signal for resistance, with chance at .5. The vertical line marks the layer used for single-layer results.

AUROC of the conflict signal for resistance
	Llama	Gemma	Qwen
Gemma judge	.575 [.560,.590]	.575 [.558,.591]	.731 [.720,.741]
Qwen judge	.579 [.565,.595]	.578 [.561,.594]	.723 [.712,.733]
Trained probe	.791	.841	.917
Across 75 countries, $\rho$ with resistance rate
Conflict signal	.28 [.05,.49]	.31 [.06,.53]	.77 [.65,.87]

Appendix

Table 23: The conflict signal at layer 16, 26, and 36 for Llama, Gemma, and Qwen. The trained probe of Table 20 is shown for comparison. Similarity is the mean cosine between the interaction of a record under its true norm and under its contradictory norm. Brackets give 95% bootstrap intervals.

Figure 12: AUROC for resistance under a contradictory norm at each layer with an autoencoder, from the most predictive feature, the five most predictive, all active features, and the full state. Chance is .5. The vertical line marks the layer used for single-layer results.

Intervention	Resists	$\Delta$ Random
None	46.2	–
Resist feature off	46.2	$+$ 0.2 [-2.2,2.8]
Resist feature on	49.5	$+$ 3.5 [0.0,7.0]
Resist feature on, $3\times$	49.5	$+$ 4.8 [0.8,9.0]
Follow feature off	47.8	$+$ 1.8 [-0.5,4.2]
Follow feature on	46.5	$+$ 0.5 [-2.2,3.5]

Appendix

Table 24: Share of 200 Qwen answers that resist a contradictory norm when one feature is switched off or on during generation, averaged over the two judges.

\Delta

Random is the paired change against the same intervention on a random feature, with a 95% bootstrap interval.

Figure 13: Mean change in the margin under country-contradictory-norm when the answer cannot attend to a span in one window of layers, over 400 records. A positive change moves the model toward the reference answer. The shaded window is the norm band fixed beforehand.

Blocked	Llama	Gemma
Norm, all layers	$+$ 40.0 [35.0,45.3]	$+$ 52.2 [47.0,57.5]
Norm, norm band	$+$ 29.0 [24.2,34.0]	$+$ 44.8 [39.8,50.0]
Norm, later band	$+$ 1.5 [-1.0,4.0]	$+$ 2.2 [0.0,4.5]
Country context	$-$ 1.8 [-5.2,1.8]	$-$ 4.2 [-7.5,-1.0]
Control span	$-$ 4.2 [-7.2,-1.2]	$+$ 0.8 [-2.2,3.8]
Norm vs. no norm	$-$ 0.8 [-4.2,2.8]	$-$ 1.5 [-5.5,2.8]

Appendix

Table 25: Change in the percentage of answers that resist a contradictory norm when a span is blocked, paired on 400 records, with 95% bootstrap intervals. The last row compares the norm blocked in all layers with country-only .

Figure 14: Share of the effect of the stated norm on the margin that is still removed when the norm is erased after a given layer, relative to erasing after the first layer, for a contradictory and a true norm. The shaded layers are the norm band of the knockout.

Norm	Llama	Gemma	Qwen
Untouched	18.0	14.2	47.0
Erased, first layer	56.2	64.8	78.0
Erased, before band	61.0	68.2	76.5
Erased, after band	26.2	21.8	54.5
$\Delta$ before band	$+$ 43.0	$+$ 54.0	$+$ 29.5
	[37.8,48.5]	[48.8,59.0]	[22.0,37.0]

Appendix

Table 26: Percentage of answers that resist a contradictory norm when the norm is erased at three depths, and the paired change against the untouched prompt with 95% bootstrap intervals, on 400 records for Llama and Gemma and 200 for Qwen.

Figure 15: Mean change in the margin when the answer cannot attend to the contradictory norm through the highest ranked heads of the norm band, or through the same number of random heads, on the 200 exploratory records. The dashed line is the effect of blocking every head of the band.

Norm blocked in	Llama	Gemma
No heads	18.0	14.2
Eight selected heads	46.5	23.5
Eight random heads	20.5	13.0
Whole band	47.0	59.0
$\Delta$ selected	$+$ 28.5 [23.8,33.2]	$+$ 9.2 [5.2,13.0]
$\Delta$ random	$+$ 2.5 [-0.5,5.5]	$-$ 1.2 [-4.0,1.2]

Appendix

Table 27: Percentage of answers that resist a contradictory norm when the norm is blocked in a set of heads of the norm band, and the paired change against the untouched prompt with 95% bootstrap intervals, on 400 records.

State
Transfer	Llama	Gemma	Qwen
Same kind of norm	.807	.834	.926
True to contradictory	.742	.787	.879
Contradictory to true	.695	.716	.909
Prompt words
Same kind of norm	.637	.645	.643

Appendix

Table 28: Mean AUROC for following the stated norm over the transfers of each type, for the probe on the state and for the probe on the words of the prompt. Each mean is over four pairs of conditions, and a probe is never scored on the condition it was fitted on.

Figure 16: Odds ratios for resisting a contradictory norm, with 95% bootstrap intervals over countries, on a logarithmic axis. The first four predictors mark an aligned answer to the same record under another condition, and the last eight are categories of the norm against etiquette. The dashed line marks no association.

	Llama		Gemma		Qwen
Norm	$\mathcal{K}$	$\mathcal{U}$	$\mathcal{K}$	$\mathcal{U}$	$\mathcal{K}$	$\mathcal{U}$
Records	235	165	272	128	152	48
Untouched	25.5	7.3	17.6	7.0	53.3	27.1
Blocked, norm band	66.0	20.0	73.2	28.9	–	–
Blocked, all layers	88.9	13.9	85.7	25.8	–	–
Erased, before band	84.7	27.3	86.0	30.5	93.4	22.9

Appendix

Table 29: Percentage of answers that resist a contradictory norm on the known records

\mathcal{K}

and the unknown records

\mathcal{U}

, for the untouched prompt, the attention knockout, and the erasure. The knockout does not apply to Qwen.

	Llama	Gemma	Qwen
Without the country	28.4	25.7	44.6
With the country	17.9	17.1	44.0
$\Delta$ all records	$-$ 10.6	$-$ 8.6	$-$ 0.6
	[-11.5,-9.6]	[-9.5,-7.6]	[-1.6,0.5]
$\Delta$ known $\mathcal{K}$	$-$ 14.0	$-$ 9.3	$-$ 0.3
	[-15.3,-12.6]	[-10.7,-7.9]	[-1.6,1.0]

Appendix

Table 30: Percentage of answers that resist a contradictory norm stated without and with the country context, on the full benchmark, and the paired effect

\Delta

of adding the country with 95% bootstrap intervals. The known records number 5,362, 5,934, and 6,965 for Llama, Gemma, and Qwen.

Figure 17: Percentage of answers that resist the same contradictory norm under five presentations, on 400 records for Llama and Gemma and 200 for Qwen. The dashed line is the rate under country-only , where no norm is stated, taken from the full benchmark for Qwen.

Presentation	Llama	Gemma	Qwen
Guide	$-$ 11.5	$-$ 7.8	$-$ 43.0
	[-15.2,-8.0]	[-11.2,-4.5]	[-49.5,-36.0]
Hearsay	$+$ 12.5	$+$ 29.8	$+$ 38.0
	[8.0,17.0]	[25.2,34.5]	[31.0,45.0]
Before the country	$+$ 20.8	$+$ 28.2	$+$ 32.5
	[16.2,25.5]	[23.5,32.8]	[26.0,39.5]

Appendix

Table 31: Paired change in the percentage of answers that resist a contradictory norm under each presentation, against the plain statement on the same records, with 95% bootstrap intervals. Every change is significant at

p<.001

Figure 18: Stance of the answers toward the contradictory norm on the intervention sample, for the untouched prompt, the norm blocked or erased, the norm stated without the country, and country-only , where the norm is never shown. The knockout and the country-only arm do not apply to Qwen.

Stance
	Llama	Gemma	Qwen
Follows the norm	70.4	72.9	46.8
Follows it with doubt	4.5	8.6	9.2
Different behavior	23.4	16.8	43.9
No clear recommendation	1.7	1.7	0.1
Properties

Appendix

Table 32: Percentage of the 9,261 answers under country-contradictory-norm with each stance and property. The third block is restricted to answers not judged aligned, and the last block gives the percentage of answers with a stance that the judge labels aligned.

Judge Gemma-4-31B
		Seen countries				Unseen countries
Model	Method	Country	+ Value	+ True norm	+ Contra. norm	Country	+ Value	+ True norm	+ Contra. norm
Llama	Untuned	58.5	73.8	93.7	17.2	60.6	75.8	91.9	17.9
	SFT	69.7	87.8	99.5	5.6	65.9	83.1	99.5	6.4
	SFT + CF	69.4	86.5	87.0	85.6	67.3	86.1	85.4	86.0
Gemma	Untuned	65.4	77.6	94.4	17.7	62.7	78.3	94.7	16.8

Appendix

Table 33: Percentage of unconstrained answers judged aligned after fine-tuning, reported separately for each judge. The layout follows Table 5 , which averages the two judges.

Method	Country	+ Value	+ True	+ Contra.	Country	+ Value	+ True	+ Contra.	None	True	Contra.	Request
	Seen countries				Unseen countries				No country
Untuned	58.6	75.6	93.3	17.3	60.2	74.8	91.5	17.1	43.1	77.0	28.3	74.9
SFT	70.4	88.6	99.6	5.7	67.0	84.1	99.5	6.3	64.6	98.2	10.5	71.9
SFT + CF	70.1	87.2	86.9	85.4	68.2	86.9	85.5	85.9	63.6	80.0	80.4	25.6
SFT + CF + Request	70.6	87.5	90.7	82.7	68.4	86.8	89.1	82.4	61.7	89.3	66.6	100.0

Appendix

Table 34: Llama-3.1-8B with and without request examples in training. The first eleven columns give the percentage of unconstrained answers judged aligned, by the evidence in the prompt. True and Contra. are a true and a contradictory norm, and the No country columns use all test records with the country context removed. Request gives the percentage of answers that do what a person in the scenario asked, on 500 request prompts. All values are averaged over the two judges.

Figure 19: Change in the margin when a contradictory norm is erased after a given layer, for the untuned model, SFT, and SFT + CF, on 400 test records. A larger change means the answer depends more on the stated norm. The shaded layers are the norm band of the knockout.

Model	Method	Sep.	Contra.	True	Ratio	Band
			Erased
Llama	Untuned	.69	12.7	$-$ 7.8	1.63	52%
	SFT	.73	11.8	$-$ 7.5	1.59	51%
	SFT + CF	.95	7.1	$-$ 6.0	1.18	25%
Gemma	Untuned	.67	20.0	$-$ 17.1	1.17	49%
	SFT	.72	12.8	$-$ 8.7	1.48	37%

Appendix

Table 35: Internal changes after fine-tuning. Sep. is the AUROC with which the conflict signal separates contradictory from true norms. Erased is the change in the margin when the norm is erased after the first layer, and Ratio is their ratio. Band is the share of the knockout effect of a contradictory norm carried by the norm band.

Cultural Bias in Language Models

Cultural bias in language models is the uneven representation of cultural knowledge, values, identities, and norms across groups and languages.

See all 128 papers

CultureForest: Understanding and Evaluating Cultural Norm Grounded Reasoning in LLMs
Jun 1, 2026Yangfan Ye, Xiaocheng Feng, Jialong Tang +5LLM Evaluation Cultural Bias in Language Models
Harbin Institute of Technology · Harvard University · The University of Hong Kong
ChronoCulture: Evaluating Cultural Awareness Across Time in Large Language Models
Oct 14, 2026Rusali Saha, Nitya Khurana, Ruihong Huang +1Cultural Knowledge in Language Models Cross-Cultural Language Model Evaluation
Texas A&M University
AlignCultura: Towards Culturally Aligned Large Language Models?
Apr 21, 2026Gautam Siddharth Kashyap, Mark Dras, Usman NaseemCross-Cultural Language Model Evaluation Cultural Alignment
School of Computing, Macquarie University, Australia

Jun 1, 2026J/K move · Enter open · S save

CultureForest: Understanding and Evaluating Cultural Norm Grounded Reasoning in LLMs

Yangfan Ye, Xiaocheng Feng, Jialong Tang, Xiayu Cao+4

Harbin Institute of Technology · Harvard University · The University of Hong Kong

LLM Evaluation Cultural Bias in Language Models Language Model Bias Evaluation Cultural Knowledge in Language Models

Open paper AskPDF arXiv

Existing research largely reduces cultural intelligence in LLMs to a knowledge-level problem, overlooking whether models can effectively utilize their acquired knowledge in realistic scenarios. To bridge this gap, we introduce CultureForest, a benchmark for Cultural Norm Grounded Reasoning. Each question is grounded in a small set of atomic norms, enabling verifiable and attributable evaluation. CultureForest comprises 5,378 examples across 8 domains and 53 countries/regions, and supports a progressive evaluation from multiple-choice to open-ended generation. Extensive experiments reveal that even top-tier models degrade substantially in open-ended settings, accompanied by pronounced cross-region disparities. Through targeted analysis, we uncover several consistent patterns: (1) test-time reasoning yields limited gains and may exacerbate inequity; (2) models exhibit highly shared regional preference structures; (3) model responses are markedly conservative, especially under stricter cultural constraints; and (4) by disentangling cultural knowledge acquisition from cultural reasoning, we show that while LLMs possess substantial cultural knowledge, their performance is further bottlenecked by its effective use. These findings point to a necessary shift from knowledge-centric evaluation toward measuring knowledge-grounded reasoning.

Abstract

Cultural Bias in Language Models

Explore similar work

CultureForest: Understanding and Evaluating Cultural Norm Grounded Reasoning in LLMs

ChronoCulture: Evaluating Cultural Awareness Across Time in Large Language Models

AlignCultura: Towards Culturally Aligned Large Language Models?