CRISP: Cultural Reward Modeling for Implicit Situated Propriety

Zekun Yuan; Yangfan Ye; Baohang Li; Shuaibo Zhao; Zekun Zhou; Ziming Li; Qichen Hong; Kun Chen

cs.CLSep 28, 2026

CRISP: Cultural Reward Modeling for Implicit Situated Propriety

Authors: Zekun Yuan, Yangfan Ye, Baohang Li, Shuaibo Zhao, Zekun Zhou, Ziming Li, Qichen Hong, Kun Chen, +1 more

Organizations: Harbin Institute of Technology · Huawei Technologies Co., Ltd · Peng Cheng Laboratory

Abstract

As large language models (LLMs) are increasingly deployed across countries and regions, the ability to recognize and respond appropriately to diverse cultural contexts becomes increasingly important. However, existing research has largely focused on cultural knowledge or tasks with predefined response spaces, while open-ended culturally situated behavior remains comparatively underexplored. In this work, we introduce CRISP-RM, a culturally situated reward model that assigns rewards according to cultural appropriateness in open-ended social scenarios. During policy optimization, we further introduce Norm Grounding Supervision (NGS), providing guidance that enhances the policy's sensitivity to relevant cultural norms. To construct culturally situated data, we employ a collaborative multi-agent framework that instantiates implicit cultural norms into diverse social scenarios and further curate NormCompass as a dedicated testbed. We conduct comprehensive experiments to evaluate the effectiveness of CRISP-RM in both reward modeling and policy optimization. Best-of- $N$ experiments show that CRISP-RM consistently outperforms strong general reward models. During GRPO policy optimization, CRISP-RM generally improves culturally situated behavior, while incorporating NGS yields further gains. Further analyses demonstrate the advantages of CRISP-RM in distinguishing culturally appropriate behavior beyond superficial fluency and politeness, while NGS provides complementary gains during policy optimization by improving norm grounding.

Figures & tables

Figure 1: Overview of our framework. Part I constructs culturally situated data. Part II trains CRISP-RM and Norm Grounding Supervisor, and combines their reward signals to GRPO.

Source	# Norms	Processing Method
CulturalBench	734	Cultural question extraction
FORK	177	Binary-choice extraction
NormAd	725	Structured rule organization
Total	1,636

Table 1: Sources and processing methods used for cultural norm collection and organization.

Condition	Decision
$L=1$	revise question
$A=0$ or $N=0$	revise question
$A=N=1,\ L=0,\ ER=0$	rerun answer
$A=E=N=R=1,\ L=0$	keep

Table 2: Deterministic routing rules used by the Routing Controller to guide the refinement process.

Comparison	Krippendorff’s $\alpha$	Spearman $\rho$
Human-Human	0.700	0.713
Human-GPT	0.732	0.750

Table 3: Agreement between human annotators and GPT on culturally appropriate evaluation.

		NormCompass		CultureForest
Reward Model	Params.	$N=8$	$N=16$	$N=8$	$N=16$
Random Selection	–	3.60	3.60	56.93	56.93
CRISP-RM-0.6B (Ours)	0.6B	3.84	3.87	58.66	58.89
CRISP-RM-4B (Ours)	4B	3.89	3.92	59.54	60.23
Skywork Reward V2 Qwen3 ( Liu et al., 2026 )	0.6B	3.61	3.60	55.29	54.49
Skywork Reward V2 Qwen3 ( Liu et al., 2026 )	4B	3.70	3.71	56.01	55.37

Table 4: Best-of-

N

selection results on NormCompass and CultureForest.

Policy	NormCompass		CultureForest		CulShield
Policy	Score	$\Delta$	Score	$\Delta$	Score	$\Delta$
Qwen3-4B	3.27	–	56.18	–	73.07	–
+CuSiR	3.08	-0.20	24.52	-31.66	69.53	-3.54
+Skywork-Reward-V2-Qwen3-4B	3.52	+0.24	23.96	-32.22	76.66	+3.59
+CRISP	3.69	+0.42	57.60	+1.42	80.19	+7.11
+CRISP + NGS	3.86	+0.58	60.28	+4.10	77.91	+4.83

Table 5: GRPO policy optimization results on NormCompass, CultureForest, and CulShield Knowledge Coverage.

\Delta

denotes the change relative to the corresponding base policy.

Figure 2: Training dynamics of Qwen3-4B under different reward signals during GRPO optimization on NormCompass and CultureForest, with performance tracked across training checkpoints.

Figure 3: Controlled cultural preference analysis on the triplets. We report ranking accuracy for Appropriate versus Generic, Appropriate versus Opposite responses and Generic versus Opposite, together with top-1 selection accuracy and complete ordering accuracy.

Figure 4: A case study illustrating how CRISP-RM distinguishes culturally appropriate behavior.

Figure 5: Effect of Norm Grounding Supervision. a): Norm Match scores across policy models before and after optimization. b): Answer quality stratified by the Norm Match score of the original policy before optimization, where

N

denotes the initial level of norm grounding.

Figure 6: Sensitivity analysis of

\lambda

across different policy models.

\lambda

controls the contribution of norm supervision, with

\lambda=0

corresponding to optimization using CRISP-RM alone.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Component

Content

Input

Culture, question, candidate options, and the correct answer.

Prompt

Extract one culture norm from the given CulturalBench-Easy item. Return JSON only, exactly with these fields: {‘‘norm’’: ‘‘one concise culture norm sentence’’, ‘‘culture’’: ‘‘culture/country name’’, ‘‘norm_type’’: ‘‘short category such as dining_etiquette, tipping, communication, public_behavior, family, religion, workplace, other’’ } Rules: - Use only the provided item. - Get the norm from the correct answer. - If the question asks what is unusual, uncommon, inappropriate, or not expected, write the norm as a negative or avoidance norm. - Do not add explanations or extra fields.

Appendix

Table 6: Prompt used to extract cultural norms from the multiple-choice questions in CulturalBench.

Component

Content

Input

Culture, question, two candidate options, and the correct answer.

Prompt

Extract one culture norm from the given item. Return JSON only, exactly with these fields: {‘‘norm’’: ‘‘one concise culture norm sentence’’, ‘‘culture’’: ‘‘culture/country name’’, ‘‘norm_type’’: ‘‘short category such as dining_etiquette, tipping, eating_utensils, social_hierarchy, hospitality, other’’ } Rules: - Use only the provided item. - Get the norm from the correct answer. - Do not add explanations or extra fields.

Appendix

Table 7: Prompt used to extract cultural norms from binary-choice questions in FORK.

Chronotope Generation Prompt

Given the following cultural norm and the already accepted chronotopes, generate a new chronotope for constructing a culturally grounded decision-making scenario. A chronotope is a time-space-social configuration. It should specify not only when and where the event happens, but also the social meaning of the setting: the relationship between participants, the degree of privacy or formality, the behavioral expectations implied by the setting, and how an action may be interpreted within that situation. Your goal is to generate a new chronotope that activates the same cultural norm but is structurally different from all accepted chronotopes. Structural diversity means that the new chronotope should differ in the core configuration of the scenario, not merely in surface details. Do not only change names, cities, weather, objects, or minor background details. The new chronotope should differ from previous ones in at least two of the following core dimensions: temporal regime: work time, private time, holiday time, mealtime, urgent moment, scheduled/unscheduled time, etc. spatial-social setting: private home, workplace, restaurant, school, public transport, government office, religious space, street, etc. relationship configuration: friends, classmates, colleagues, supervisor/subordinate, host/guest, elder/younger, strangers, service worker/customer, etc. occasion or activity frame: visiting, meeting, dining, gift-giving, asking for help, apologizing, negotiating, celebrating, requesting a favor, etc. potential tension: the superficially reasonable action that could lead to a culturally inappropriate choice. Requirements: - The cultural norm must become relevant, but do not directly restate or explain the norm. - Do not reveal the culturally preferred behavior. - Do not use explicit cultural explanations such as ‘‘In this culture, people usually...’’ - Make the chronotope concrete, natural, and suitable for later Labov-style narrative construction. - Avoid duplicating the accepted chronotopes at the structural level. - Return valid JSON only, with no additional explanation. Cultural norm: {norm} Culture: {culture} Accepted chronotopes: {accepted_chronotopes} Output schema: {‘‘time’’: ‘‘’’, ‘‘place’’: ‘‘’’, ‘‘social_space’’: ‘‘’’, ‘‘occasion’’: ‘‘’’, ‘‘relationship_context’’: ‘‘’’, ‘‘privacy_or_formality_level’’: ‘‘’’, ‘‘norm_activation_condition’’: ‘‘’’, ‘‘potential_tension’’: ‘‘’’, ‘‘difference_from_previous’’: ‘‘’’ }

Appendix

Table 8: Prompt used for chronotope construction.

Scenario and Question Generation Prompt

You will be given a cultural norm and a chronotope. Generate a contextualized cultural understanding question based on them. The goal is not to ask the evaluated model to classify, restate, or explain cultural knowledge. Instead, construct a natural social scenario in which the protagonist must infer culturally relevant considerations and decide what action, choice, or interpretation is appropriate. Input: - norm: {norm} - culture: {culture} - norm_type: {norm_type} - chronotope: {chronotope} Requirements: - Construct a specific and natural scenario with a clear protagonist and practical decision. - Make the cultural norm relevant through contextual cues without directly revealing the norm or preferred behavior. - The question should require an open-ended, context-sensitive action, judgment, or interpretation rather than cultural knowledge recall. - Avoid stereotypes, overly strong normative claims, and artificially exaggerated conflicts. - [Additional scenario-construction and weak-norm handling constraints omitted for brevity.] Return valid JSON only: {‘‘scenario’’: ‘‘...’’, ‘‘question’’: ‘‘...’’, ‘‘hidden_expected_answer’’: ‘‘...’’, ‘‘generation_notes’’: ‘‘...’’ } Input JSON: {generation_input}

Appendix

Table 9: Prompt used for scenario generation. The full prompt will be released with our code.

Symbol	Dimension	Description
$A$	Action Appropriateness	Whether the proposed action is appropriate in the given situation.
$E$	Contextual Evidence	Whether concrete cues in the scenario sufficiently support the relevant cultural judgment.
$N$	Norm Matching	Whether the inferred cultural norm matches the core meaning of the target norm.
$R$	Reasoning Support	Whether the provided reasoning sufficiently supports the proposed action and cultural judgment.
$L$	Norm Leakage	Whether the scenario or question directly or near-explicitly reveals the target cultural norm.

Appendix

Table 10: Verification dimensions used by the Norm-Grounded Verifier.

Situated Reasoner Prompt

You will be given a cultural background, a scenario story, and a question. Your task is to answer the question based on the given scenario and explain the cultural norms, relational meanings, or situational meanings that need to be considered. You may only answer based on the given culture, scenario, and question. Culture is background information only; norms must not be inferred solely from the culture or identity of the characters. Every inferred norm must be supported by specific contextual clues in the scenario. Your response should contain: 1. action: the action, expression, interpretation, or judgment the protagonist should make next; 2. reason: why this action is appropriate in the current situation; 3. inferred_norms: cultural norms, relational meanings, or situational meanings inferred from the scenario, together with the specific contextual evidence supporting each inference. The action should directly address the question rather than provide generic advice. Do not invent information that does not appear in the scenario. Return valid JSON only. Input: culture: {culture} scenario: {scenario} question: {question}

Appendix

Table 11: Prompt used by the Situated Reasoner .

Norm-Grounded Verifier Prompt

You will be given a cultural norm, a scenario question, and an answer to that question. Evaluate the candidate along five dimensions and assign a binary score (0 or 1) with brief reasoning for each dimension: 1. action_correctness (

A

): whether the proposed action is appropriate under the current scenario and question; 2. norm_evidence_support (

E

): whether concrete evidence from the scenario sufficiently supports the inferred norm; 3. inferred_norm_match (

N

): whether the inferred norms contain content whose core meaning is consistent with the target norm; 4. reason_supports_action (

R

): whether the reasoning adequately supports the proposed action; 5. norm_leakage (

L

): whether the scenario or question directly or near-directly reveals the target norm. Exact wording of the target norm is not required for norm matching. Evidence consisting only of the culture, country or region name, character identity, or generic common sense should not be considered sufficient contextual support. When evaluating action correctness, hidden_expected_answer may be used as reference, but an answer need not match it exactly. Return valid JSON containing the score and brief reasoning for each dimension. Input: norm: {norm} culture: {culture} norm_type: {norm_type} chronotope: {chronotope} scenario: {scenario} question: {question} hidden_expected_answer: {hidden_expected_answer} answer: {answer_model_output}

Appendix

Table 12: Prompt used by the Norm-Grounded Verifier .

Response Advisor Prompt

You will be given a scenario question, the previous answer, the verification results, and a diagnosis produced by the rule-based controller. Your task is to generate concise and specific guidance for the next response attempt. Identify the main deficiency in the previous answer and indicate what the Situated Reasoner should improve. Do not revise the scenario or question and do not directly generate a new answer. You may use the target norm and hidden expected answer to diagnose the problem, but do not directly reveal them in the guidance. Return valid JSON only with: diagnosis: the main problem with the previous response; answer_guidance: how the next response should improve. Input: controller_diagnosis: {controller_diagnosis} norm: {norm} culture: {culture} scenario: {scenario} question: {question} hidden_expected_answer: {hidden_expected_answer} answer: {answer_model_output} judge_output: {judge_output}

Appendix

Table 13: Prompt used by the Response Advisor.

Revision Planner Prompt

You will be given a cultural norm, a scenario question, the corresponding response, the verification results, and a diagnosis produced by the rule-based controller. Your task is to provide concise, specific, and actionable guidance for revising the scenario and question. Do not rewrite the full scenario or question. Instead, identify the main problem, specify what the revised candidate should achieve, and provide targeted revision guidance. Focus on whether the scenario, question, and hidden expected answer effectively instantiate the target norm rather than evaluating the quality of the norm itself. Return valid JSON only with: diagnosis: the main problem with the current candidate; revision_goal: what the revised candidate should achieve; revision_guidance: targeted guidance for the Question Reviser. Input: controller_diagnosis: {controller_diagnosis} norm: {norm} culture: {culture} norm_type: {norm_type} chronotope: {chronotope} scenario: {scenario} question: {question} hidden_expected_answer: {hidden_expected_answer} answer: {answer_model_output} judge_output: {judge_output}

Appendix

Table 14: Prompt used by the Revision Planner.

Question Reviser Prompt

You will be given a cultural norm, a previous scenario question, and diagnostic information from the refinement process. Your task is to revise the scenario, question, and hidden expected answer so that the candidate more reliably instantiates the target norm. The revision should address the identified problems while satisfying the following requirements: - The scenario and question must not directly or near-directly reveal the target norm. - The appropriate response should depend on the target cultural norm rather than only on generic logic, practical constraints, or politeness. - The scenario should contain sufficiently specific contextual cues for the target norm to be inferred. - Contextual cues that incorrectly activate a non-target norm should be removed, weakened, or replaced. - The question should remain neutral and should not directly reveal the expected answer or ask the model to state the cultural rule. Return valid JSON containing: scenario: the revised scenario; question: the revised action-oriented question; hidden_expected_answer: the revised reference answer; generation_notes: a brief explanation of the changes made in response to the diagnostic guidance. Input: {revision_input}

Appendix

Table 15: Prompt used by the Question Reviser.

Culture	# Norms	# Scenarios
Argentina	67	186
China	226	624
Germany	77	217
India	91	246
Iran	81	217
Italy	71	195

Appendix

Table 16: Distribution of cultural norms and situated questions across 19 cultural groups.

Split	# Scenarios
Train	3,521
Validation	449
Test	222
Total	4,192

Appendix

Table 17: Statistics of the final data splits used in our experiments.

Score	Criterion
1	Clearly wrong or opposite outcome. The response recommends or preserves a decisively inappropriate action, or would clearly worsen the situation.
2	Mostly inappropriate with material mitigation. The core outcome remains inappropriate, but the response includes a concrete adjustment that meaningfully reduces the relevant harm.
3	Mixed, indeterminate, or underspecified outcome. The response contains both appropriate and inappropriate elements, or remains too generic or ambiguous to determine whether the decisive issue is resolved.
4	Correct core outcome with a limited practical defect. The response resolves the decisive action requirements and would likely produce an appropriate outcome, but contains a specific secondary omission, ambiguity, or mildly counterproductive recommendation.
5	Fully appropriate outcome. The response clearly recommends an appropriate and executable action that resolves all decisive requirements without conflicting or culturally inappropriate additional recommendations.

Appendix

Table 18: Outcome-quality scoring criteria used for automatic evaluation.

Automatic Evaluation Prompt

You are evaluating the final answer to a culturally grounded decision-making scenario. Assign exactly one outcome-quality score from 1 to 5. Evaluate only whether the candidate’s final proposed action would handle this specific situation appropriately. Do not separately score whether the candidate identifies, names, or explains the cultural norm. Use the target norm only as a private reference for determining the appropriate practical outcome. Score definitions: 1 --- Clearly wrong or opposite outcome. The final recommendation directly performs, endorses, or preserves the decisive culturally inappropriate action, or would clearly worsen the situation. 2 --- Mostly inappropriate outcome with material mitigation. The core outcome remains inappropriate, but a concrete part of the response materially reduces the relevant cultural or situational harm. 3 --- Mixed, indeterminate, or underspecified outcome. The response contains important appropriate and inappropriate components, or is too generic or ambiguous to determine whether the decisive practical issue is resolved. 4 --- Correct core outcome with a limited practical defect. The recommendation resolves every decisive action requirement and would likely produce an appropriate outcome, but contains a specific secondary omission, ambiguity, or mildly counterproductive action. 5 --- Fully appropriate outcome. The recommendation clearly commits to an appropriate and executable action that resolves every decisive requirement and contains no conflicting, unnecessarily risky, or culturally inappropriate additional recommendation. Boundary rules: - Score the practical outcome rather than writing quality, response length, confidence, cultural vocabulary, or explicit norm recognition. - If any decisive action requirement remains unresolved, the response receives at most 3. - Use 4 only when a specific action-level defect with limited practical consequence can be identified; otherwise use 5. - Merely mentioning relevant variables or suggesting that the model ‘‘check’’ local customs or preferences does not constitute a solution unless it leads to a culturally calibrated action. - Reasonable variations consistent with the target norm and situational constraints should be accepted. Return valid JSON only: {‘‘score’’: 1, ‘‘evidence’’: ‘‘Quote or closely paraphrase the decisive part of the candidate answer.’’, ‘‘reason’’: ‘‘Briefly explain why the answer meets this score boundary.’’ } Culture: {culture} Target norm: {norm} Scenario: {scenario} Question: {question} Candidate answer: {answer}

Appendix

Table 19: Prompt used for automatic evaluation of culturally situated responses.

Prompt

Answer the following culturally grounded decision-making question. Return a JSON object with exactly two string fields: { "answer": "A direct, practical answer to the question.", "reasoning": "A natural explanation supporting the answer." } The reasoning should be one coherent paragraph of roughly four to seven sentences. It should naturally move from a small set of decisive details in the scenario, to the most specific culture-linked convention those details make relevant, and then to how that convention supports the answer in this situation. Do not label or divide the reasoning into steps. Do not use headings, numbered lists, bullet points, or terms such as Grounding, Norm, or Decision. Do not replace a specific cultural convention with broad themes. Do not invent a custom merely to sound culturally informed. Return valid JSON only, with no markdown fence or additional text. Culture: {culture} Scenario: {scenario} Question: {question}

Appendix

Table 20: Prompt template for natural response generation.

Configuration	Qwen3-0.6B	Qwen3-4B
Fine-tuning	Full-parameter	Full-parameter
Epochs	2	2
Learning rate	$2\times 10^{-5}$	$1\times 10^{-5}$
Effective batch size	32 pairs	32 pairs
Maximum sequence length	2,048	2,048
Optimizer	AdamW	AdamW

Appendix

Table 21: Main training configurations of our reward models.

		Qwen3-8B		Llama-3.1-8B		Qwen2.5-7B		Gemma-2-9B		Mistral-7B		Phi-3.5-mini
Selector	Params.	$N=8$	$N=16$	$N=8$	$N=16$	$N=8$	$N=16$	$N=8$	$N=16$	$N=8$	$N=16$	$N=8$	$N=16$
Random Selection	–	3.6351	3.6036	3.5676	3.6171	3.5766	3.5135	3.6622	3.6577	3.6892	3.7297	3.4414	3.5045
CRISP-RM-0.6B (Ours)	0.6B	3.7838	3.7477	3.7838	3.9099	3.8784	3.8378	3.9189	3.9685	3.9685	4.0315	3.6982	3.7523
CRISP-RM-4B (Ours)	4B	3.8198	3.7973	3.9595	4.0315	3.7793	3.8874	3.9144	3.9234	3.9910	3.9910	3.8739	3.8694
Skywork Reward V2 Qwen3 ( Liu et al., 2026 )	0.6B	3.5901	3.4865	3.5541	3.5856	3.5541	3.5315	3.7523	3.7658	3.7703	3.7117	3.4414	3.5090
Skywork Reward V2 Qwen3 ( Liu et al., 2026 )	4B	3.6216	3.5495	3.6351	3.7568	3.6667	3.6396	3.8919	3.8919	3.8153	3.8153	3.5676	3.5946

Appendix

Table 22: Policy-disaggregated Best-of-

N

results on NormCompass. Each generator policy provides 16 candidate responses per question, and each reward-model selector chooses the highest-reward response among the first

N\in\{8,16\}

candidates.

		Qwen3-8B		Llama-3.1-8B		Qwen2.5-7B		Gemma-2-9B		Mistral-7B		Phi-3.5-mini
Selector	Params.	$N=8$	$N=16$	$N=8$	$N=16$	$N=8$	$N=16$	$N=8$	$N=16$	$N=8$	$N=16$	$N=8$	$N=16$
Random Selection	–	59.6091	60.2719	53.6307	54.0937	54.6187	54.2729	59.6390	59.0919	55.5285	55.7763	58.5761	58.0949
CRISP-RM-0.6B (Ours)	0.6B	60.7810	60.8263	56.9511	56.7515	56.6034	57.0455	61.0720	61.1883	56.7140	57.2985	59.8294	60.2116
CRISP-RM-4B (Ours)	4B	61.6390	61.7853	57.6375	58.8084	57.3164	58.2116	60.9883	61.6365	58.7010	59.7766	60.9487	61.1371
Skywork Reward V2 Qwen3 ( Liu et al., 2026 )	0.6B	59.9932	59.7340	51.0735	49.5677	51.1671	50.0705	59.8739	59.3523	53.1913	52.7934	56.4474	55.4477
Skywork Reward V2 Qwen3 ( Liu et al., 2026 )	4B	60.4691	60.4916	52.3124	51.1317	52.0056	50.6120	60.7021	60.3504	53.7068	53.1513	56.8836	56.5060

Appendix

Table 23: Policy-disaggregated Best-of-

N

results on the CultureForest. Following the main evaluation, we restrict the benchmark to cultural groups represented among the 19 cultures in our dataset.

Policy	NormCompass		CultureForest		CulShield
Policy	Score	$\Delta$	Score	$\Delta$	Score	$\Delta$
Qwen3-4B	3.37	–	56.72	–	73.08	–
+CuSiR	3.15	-0.22	25.30	-31.42	69.53	-3.55
+Skywork-Reward-V2-Qwen3-4B	3.52	+0.15	24.29	-32.43	76.66	+3.58
+CRISP	3.74	+0.37	58.13	+1.41	80.19	+7.11
+CRISP + NGS	3.87	+0.50	61.46	+4.74	77.92	+4.84

Appendix

Table 24: GRPO policy optimization results on valid responses only for NormCompass, CultureForest, and CulShield Knowledge Coverage.

\Delta

denotes the change relative to the corresponding base policy under the same valid-response-only evaluation.

Prompt	Content
System	You construct controlled response triplets for a cultural decision-making experiment. Follow the requested semantic edit exactly while preserving fluent, natural English. Return JSON only.
Controlled rewrite	Create two controlled rewrites of the provided appropriate answer. Generic must preserve the answer’s tone, fluency, broad structure, level of detail, and approximate length, but remove or blur the decisive action tied to the target norm. It should sound polite and superficially reasonable, yet fail to commit to the culturally relevant action. It must not become clearly opposite or obviously wrong. Opposite must preserve the same tone, fluency, broad structure, level of detail, and approximate length, but reverse the decisive culturally relevant action. Give it a superficially plausible explanation. Its main defect must be the action, not grammar, incoherence, rudeness, or low writing quality. Do not include labels, meta-commentary, phrases such as ‘‘this violates the cultural norm’’, or explanations of how you edited the answer. Do not copy the target norm verbatim merely to signal the category. Culture: {culture} Target norm (private construction reference): {norm} Scenario: {scenario} Question: {question} Appropriate answer: {appropriate} Return exactly this JSON object: {"generic":"...","opposite":"..."}
Full triplet	Create a controlled triplet for the scenario. Appropriate must clearly resolve the decisive action in a way consistent with the target norm. Generic must match its tone, fluency, broad structure, detail, and approximate length but remove or blur the decisive culturally relevant action without becoming clearly opposite. Opposite must match the same writing quality and approximate length but reverse the decisive culturally relevant action with a superficially plausible explanation. Do not include labels, meta-commentary, phrases such as ‘‘this violates the cultural norm’’, or explanations of the edits. Do not copy the target norm verbatim merely to signal the category. Culture: {culture} Target norm (private construction reference): {norm} Scenario: {scenario} Question: {question} Return exactly this JSON object: {"appropriate":"...","generic":"...","opposite":"..."}

Appendix

Table 25: Prompts used to construct the controlled cultural preference triplets.

Prompt

Content

Norm Match

You are an independent evaluator of the observable reasoning in a culturally grounded decision-making response. The Target Norm is a private authoritative reference. Evaluate whether the Candidate Reasoning semantically recovers and uses the core cultural requirement. Do not require quotation, keyword overlap, or explicit mention of the country or the word ‘‘norm’’. A clear paraphrase counts fully. Judge only Norm Match using exactly one score: 0 --- Not recognized or incorrect. The reasoning does not express the target norm’s core requirement, expresses an opposite or conflicting principle, substitutes a merely generic value such as politeness or respect, discusses only an adjacent norm, or invents a convention. A correct-looking final action alone cannot rescue reasoning that omits the norm. 1 --- Partial or implicit recognition. The reasoning is directionally related to the target norm but remains incomplete, indirect, ambiguous, or misses a material part of its direction, scope, condition, or strength. 2 --- Correct semantic recognition. The reasoning clearly and accurately states or unambiguously paraphrases the target norm’s core requirement and uses it to explain the recommendation in this situation. Verbatim wording is not required. The Candidate Answer is provided only to resolve references and check whether the reasoning actually uses the stated principle. Do not separately score answer quality, fluency, length, confidence, cultural vocabulary, or hidden intentions. Give no credit for information that appears only in the Target Norm, Scenario, Question, or Candidate Answer rather than in the Candidate Reasoning. Culture: {culture} Target Norm: {norm} Scenario: {scenario} Question: {question} Candidate Reasoning: {reasoning} Candidate Answer: {answer}

Appendix

Table 26: Prompt used for Norm Match evaluation.

Method	Qwen3-4B	Qwen3-8B	DeepSeek-Qwen-7B
Base	0.8472	1.0679	0.3263
CRISP	1.1963	1.3153	0.7281
CRISP+NGS	1.2715	1.3704	0.8279

Appendix

Table 27: Norm Match scores on valid responses only.

Method	$N=0$		$N=1$		$N=2$
Method	Score	$\Delta$	Score	$\Delta$	Score	$\Delta$
Base	2.2909	–	3.6931	–	4.7746	–
CRISP	3.1745	+0.8836	3.7302	+0.0370	4.4437	-0.3310
CRISP+NGS	3.2618	+0.9709	3.8836	+0.1905	4.6268	-0.1479

Appendix

Table 28: Answer quality on jointly valid responses, stratified by the Norm Match score of the base policy. An item is retained only when the outputs of Base, CRISP, and CRISP+NGS are all valid.

N

denotes the Norm Match score of the corresponding base-policy response, and

\Delta

denotes the change relative to Base within each group.

Explore similar work

Jun 1, 2026cs.CL

CultureForest: Understanding and Evaluating Cultural Norm Grounded Reasoning in LLMs

Existing research largely reduces cultural intelligence in LLMs to a knowledge-level problem, overlooking whether models can effectively utilize their acquired knowledge in realistic scenarios. To bridge this gap, we introduce CultureForest, a benchmark for \textit{Cultural Norm Grounded Reasoning}. Each question is grounded in a small set of atomic norms, enabling verifiable and attributable evaluation. CultureForest comprises 5,378 examples across 8 domains and 53 countries/regions, and supports a progressive evaluation from multiple-choice to open-ended generation. Extensive experiments reveal that even top-tier models degrade substantially in open-ended settings, accompanied by pronounced cross-region disparities. Through targeted analysis, we uncover several consistent patterns: (1) test-time reasoning yields limited gains and may exacerbate inequity; (2) models exhibit highly shared regional preference structures; (3) model responses are markedly conservative, especially under stricter cultural constraints; and (4) by disentangling cultural knowledge acquisition from cultural reasoning, we show that while LLMs possess substantial cultural knowledge, their performance is further bottlenecked by its effective use. These findings point to a necessary shift from knowledge-centric evaluation toward measuring knowledge-grounded reasoning.

Yangfan Ye, Xiaocheng Feng, Jialong Tang +5

Harbin Institute of Technology · Harvard University · The University of Hong Kong

Cultural Awareness Test Time Disparities Forests

Ask PDF

Jun 17, 2026cs.CL

Steerable Cultural Preference Optimization of Reward Models

It is essential for large language model (LLM) technology to serve many different cultural sub-communities in a manner that is acceptable to each community. However, research on LLM alignment has so far predominantly focused on predicting a unified response preference of annotators from certain regions. This paper aims to advance the development of alignment models with a more global outlook, that are able to accurately represent the preferences of subcommunities and do not exhibit excessive bias towards any of them. We focus on the development of reward models for this purpose and present a novel reward model training algorithm (SCPO) that can incorporate diverse cultural preferences in a balanced manner. Our method results in performance increases of the minority reward model of up to 7 points over the baseline model across two datasets, PRISM and GlobalOpinionQA, and across 7 countries. SCPO is up to 280% more training data-efficient than full-data finetuning of reward models. In addition, we perform analysis of bias by separately evaluating on the preference of subcommunities and show that excessive bias is mitigated via our weighting method. Our code is available at https://github.com/minsik-ai/Steerable-Cultural-Preference

Minsik Oh, Advit Deepak, Sophie Wu +2

1Stanford University · University of Amsterdam

Large Language Model Alignment Sequential Preference Optimization Progress Reward Modeling Preference Alignment Learning

Ask PDF

Jun 5, 2026cs.CL

Whose Norms? Disentangling Cultural and Personal Alignment in Large Language Models

Large language models are increasingly used for social decision-making situations that require balancing cultural norms with personal preferences. For example, a user preferring honesty might ask whether to correct a coworker publicly when local norms favor indirect feedback. Yet existing research studies cultural alignment and personalization largely separately. We introduce PACT, the Personal-Preference and Cultural-Norm Trade-off framework, which evaluates whether models choose to follow a cultural norm or allow personal preferences. We find that LLMs vary in how rigidly they enforce cultural norms, with behavior shifted more by country context (7.8%) than age (1%) and gender (0.7%) and shifting non-uniformly after instruction tuning. Furthermore, our five-country human study on PACT shows that culture-following in humans is mainly driven by scenario country, with the lowest agreement when participants judge their own cultural contexts, showing within-culture pluralism. Finally, human-LLM alignment experiments show that models can match majority choices, but fail to capture response distributions and uncertainty (with best correlations reaching only 0.24). Together, these findings motivate alignment evaluations that go beyond majority to capture cultural pluralism and disagreement in social judgment.

Angana Borah, Isabelle Augenstein, Rada Mihalcea

University of Michigan - Ann Arbor · University of Copenhagen

Large Language Model Alignment Large Language Model Bias Large Language Model Decisions Social Norms Preference Alignment Learning Norm+1

Ask PDF