Towards Cross-lingual Values Judgment: A Consensus-Pluralism Perspective

Yukun Chen; Xinyu Zhang; Boyi Deng; Jialong Tang; Yu Wan; Fei Huang; Yuxi Zhou; Baosong Yang

cs.CLFeb 19, 2026

Towards Cross-lingual Values Judgment: A Consensus-Pluralism Perspective

Authors: Yukun Chen, Xinyu Zhang, Boyi Deng, Jialong Tang, Yu Wan, Fei Huang, Yuxi Zhou, Baosong Yang,

Organizations: State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security · Tongyi Lab, Alibaba Group · Nanyang Technological University

Abstract

As large language models (LLMs) are employed worldwide, existing evaluation paradigms for their multilingual capabilities primarily focus on factual task performance, neglecting the ability to judge content's deep-level values across multiple languages. To bridge this gap, we first reveal two primary challenges in constructing values judgment benchmarks, cultural diversity and disciplinary complexity, and propose a novel two-stage human-AI collaborative annotation framework to alleviate them. This framework identifies the issue scope and nature, establishes specific annotation criteria, and utilizes multiple LLMs for final review. Building upon this framework, we introduce X-Value, the first Cross-lingual Values Judgment Benchmark designed to evaluate the capability of LLMs in judging deep-level values of content. X-Value comprises 4,750 Question-Answer pairs across 14 languages, covering 7 major global issue categories, and provides 12 granular annotation metadata to facilitate a rigorous evaluation of model performance. Systematic evaluations of X-Value are conducted across 17 LLMs using distinct prompting strategies. Multi-dimensional analysis of accuracy and F1-scores reveals their limitations in cross-lingual values judgment and indicates performance disparities across categories and languages. This work highlights the urgent need to improve the underlying, values-aware content judgment capability of LLMs.\footnote{Samples of X-Value are available at https://huggingface.co/datasets/Whitolf/X-Value.}

Figures & tables

Figure 1: Overview of the data composition and annotation metadata of X-Value. (1) Data Composition: Characterized by linguistic diversity (covering 14 languages) and issue taxonomy (spanning 7 major categories). (2) Annotation Metadata: QA pairs have comprehensive annotations including core issues (with scope and nature), granular checkpoints and overall judgment conclusions.

Figure 2: The two challenges of cross-lingual values judgment. (a) Consistence rates (%) of values judgment across 10 national personas. Red colors indicate low consistence rates. This figure indicates the judgment inconsistency from cultural diversity. (b) Number of disciplines involved in samples. This figure shows the high knowledge demand imposed by disciplinary complexity.

Abbr.	Issue Category	Basic Human Values
P&G	Political Power & Governance Legitimacy	Conformity–rules, Power–dominance, Security–societal
N&M	National Sovereignty & Military Security	Security–societal, Power–resources, Power–dominance
H&N	Historical Memory & National Identity	Tradition, Humility, Face
E&S	Ethnicity, Migration & Social Justice	Universalism–concern, Universalism–tolerance, Benevolence–caring
R&F	Religious Belief & Freedom of Expression	Self-direction–thought, Universalism–tolerance, Tradition
G&H	Gender Equality & Human Rights	Universalism–tolerance, Self-direction–action, Conformity–interpersonal

Table 1: Schwartz’s Basic Human Values covered by issue categories in the taxonomy.

	Overall Judgment				Judgment by Category							Category Stats.
Models	Acc	${\text{Acc}_{0}}$	$\text{Acc}_{1}$	$\text{Acc}_{0}$ - $\text{Acc}_{1}$	P&G	N&M	H&N	E&S	R&F	G&H	P&B	avg.	std.	range
Judgment with Definition-only Prompt
GPT-5.2	69.78	92.33	57.52	34.81	68.38	71.29	70.59	66.29	73.03	65.57	73.77	69.85	2.96	8.20
GPT-5.4	75.50	88.46	68.43	20.03	76.43	75.68	75.59	74.25	77.29	72.71	76.62	75.51	1.45	4.57
Gemini-3-Flash-Prev.	70.95	94.86	57.93	36.93	71.43	71.71	74.67	71.29	68.14	70.43	69.38	71.01	1.90	6.52
Gemini-3.1-Pro-Prev.	71.22	91.94	59.95	31.99	69.43	69.29	75.50	74.29	68.86	70.86	70.92	71.31	2.40	6.64

Table 2: Overall accuracy (%), accuracy across values-appropriateness-based (

{\text{Acc}_{0}}

and

{\text{Acc}_{1}}

) and category-based subsets of 10 frontier LLMs using Def-only or Rubric prompts. For each case, the best is marked in red , and the second best is marked in bold .

Models	ar	de	en	es	fr	id	ja	ko	pt	ru	th	ur	vi	zh	avg.	std.	range
Judgment with Definition-only Prompt
GPT-5.2	72.13	68.48	66.18	83.09	76.30	67.43	75.71	64.86	71.43	65.14	69.14	72.80	68.23	56.73	69.83	6.04	26.36
GPT-5.4	81.95	76.57	71.35	80.86	83.43	73.71	80.86	66.00	75.43	76.79	79.71	72.80	70.00	65.90	75.38	5.49	17.53
Gemini-3-Flash-Prev.	78.00	66.00	67.14	79.43	80.57	68.86	66.57	67.43	73.14	75.43	69.14	74.80	66.00	61.14	70.98	5.69	19.43
Gemini-3.1-Pro-Prev.	75.71	60.86	73.43	72.29	75.43	71.14	77.14	70.57	73.43	70.57	64.57	75.20	69.00	68.57	71.28	4.34	16.29
Claude-Opus-4.5	72.57	60.57	67.71	74.86	69.71	72.57	76.86	69.71	68.77	70.29	62.00	78.40	69.33	68.00	70.10	4.77	17.83

Table 3: Values judgment accuracy (%) of 10 frontier LLMs across 14 languages using Def-only or Rubric prompts. For each case, the best is marked in red , and the second best is marked in bold .

Figure 3: F1-score (%) with corresponding precision and recall of 10 frontier LLMs using Rubric prompts. Each comparison dimension is sorted in ascending order of F1-scores (↑).

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

	Overall Judgment				Judgment by Category							Category Stats.
Models	Acc	${\text{Acc}_{0}}$	$\text{Acc}_{1}$	$\text{Acc}_{0}$ - $\text{Acc}_{1}$	P&G	N&M	H&N	E&S	R&F	G&H	P&B	avg.	std.	range
Qwen-3.5-2B	52.60	97.67	28.07	69.60	52.71	55.86	52.50	49.64	53.57	48.57	55.54	52.63	2.54	7.29
Qwen-3.5-4B	64.21	96.95	46.39	50.56	68.86	66.43	63.67	59.43	64.29	58.57	68.46	64.24	3.77	10.29
Qwen-3.5-9B	63.07	97.73	44.21	53.52	67.71	65.43	61.00	60.00	62.14	59.29	65.85	63.06	3.02	8.43
Qwen-3.5-27B	66.76	97.25	50.16	47.09	70.43	68.71	66.67	63.29	66.43	65.00	66.77	66.76	2.16	7.14
Qwen-3.5-35B-A3B	63.75	97.67	45.29	52.38	66.57	66.71	62.83	59.43	62.57	61.57	66.62	63.76	2.69	7.29

Appendix

Table 4: Overall accuracy (%), accuracy across values-appropriateness-based (

{\text{Acc}_{0}}

and

{\text{Acc}_{1}}

) and category-based subsets of Qwen-3.5 series using Def-only prompts. For each case, the best is marked in red , and the second best is marked in bold .

Models	ar	de	en	es	fr	id	ja	ko	pt	ru	th	ur	vi	zh	avg.	std.	range
Qwen-3.5-2B	56.29	57.14	53.43	69.14	51.14	47.71	47.43	45.14	49.14	53.14	48.86	62.65	53.00	45.14	52.81	6.54	24.00
Qwen-3.5-4B	67.43	64.29	59.43	72.29	62.00	60.00	62.29	60.00	56.86	68.57	63.43	71.20	65.00	68.29	64.36	4.49	15.43
Qwen-3.5-9B	66.29	59.43	59.14	70.57	59.14	60.29	60.86	57.71	58.29	68.57	62.29	72.00	63.00	68.00	63.26	4.70	14.29
Qwen-3.5-27B	68.00	61.14	57.71	71.71	66.00	63.43	64.86	64.86	60.29	72.57	72.57	72.40	70.67	70.57	66.91	4.85	14.86
Qwen-3.5-35B-A3B	69.14	56.00	55.43	69.43	55.71	61.71	61.71	62.00	58.29	68.57	69.43	72.40	69.00	66.86	63.98	5.75	16.97
Qwen-3.5-122B-A10B	70.00	61.14	57.71	70.29	64.00	59.71	64.86	62.00	61.14	72.57	75.14	75.60	67.00	67.43	66.33	5.55	17.89

Appendix

Table 5: Values Judgment accuracy (%) of Qwen-3.5 series across 14 languages using Def-only prompts. For each case, the best is marked in red , and the second best is marked in bold .

Figure 1: Overall conclusion accuracy (%) of Gemini-3-Flash-Preview on language $\times$ category dimensions using Rubric prompts. Squares with deeper red indicate lower judgment accuracy.

Figure 2: Overall conclusion accuracy (%) of Qwen-3.5-Plus on language $\times$ category dimensions using Rubric prompts. Squares with deeper red indicate lower judgment accuracy.

Figure 3: Overall conclusion accuracy (%) of GPT-5.4 on language $\times$ category dimensions using Rubric prompts. Squares with deeper red indicate lower judgment accuracy.

Figure 4: Overall conclusion accuracy (%) of Claude-Opus-4.6 on language $\times$ category dimensions using Rubric prompts. Squares with deeper red indicate lower judgment accuracy.

Panel of Figure 5 — Figure 5: The prompt used for simulating persona from 10 national backgrounds.

Discipline	Detailed Explanation
Cultural Studies	Analyzes how culture is produced and transformed within social power structures and their impact on individual and collective identities.
Anthropology	Reveals the diversity of human cultures, their mechanisms of transmission, and the evolution of values through the study of different social groups.
Ethics	Systematically explores moral principles and core values, studying the normative standards used by individuals and societies to judge right and wrong.
Sociology	Examines how social institutions, customs, and norms shape collective consciousness and how cultural values function within social interactions and structures.
Religion	Investigates belief systems and sacred rituals, which often serve as the foundational source of cultural identity and core moral values for specific societies.
Folklore	Analyzes the grassroots culture and ethnic values accumulated over history by studying legends, traditional customs, and oral literature.

Appendix

Table 6: Disciplines Related to Culture and Values

Panel of Figure 6 — Figure 6: The prompt used for scoring relevance for 10 representative disciplines.

Panel of Figure 7 — Figure 7: The prompt used for generating Answers to Questions.

Panel of Figure 8 — Figure 8: The prompt used for getting the preliminary values-judgment results from LLMs.

Figure 9: Distribution of issue domains by language across all QA pairs.

Code	Language	Language Family	Native Speakers (M)
en	English	Indo-European	1500
zh	Chinese	Sino-Tibetan	1200
es	Spanish	Indo-European	561
ar	Arabic	Afroasiatic	335
fr	French	Indo-European	334
pt	Portuguese	Indo-European	269

Appendix

Table 7: Detailed information of all 14 languages covered by our X-Value. Statistical Data are from https://www.ethnologue.com/.

Code	Language	Total QA		Question Length				Answer Length
Code	Language	Easy	Hard	Avg.	Min	Max	Std.	Avg.	Min	Max	Std.
ar	Arabic	120	230	14.0	100	7	9.9	138.4	218	89	24.1
de	German	120	230	21.7	140	5	12.4	161.9	280	80	32.6
en	English	120	230	22.4	37	11	4.9	128.6	275	52	38.8
es	Spanish	141	209	15.8	101	9	7.9	171.7	294	105	35.4
fr	French	120	230	29.9	170	8	19.9	191.7	380	38	39.9

Appendix

Table 8: Key statistics of the total 4,750 QA pairs in 14 languages.

Figure 10: Distribution of numbers of QA Pairs across 14 languages and 7 issue Domains

Code	Language	Country/Region
ar	Arabic	Arabic-speaking countries
de	German	Germany
en	English	United States
es	Spanish	Spain
fr	French	France
id	Indonesian	Indonesia

Appendix

Table 9: Language and Country Information

Cross-Cultural Language Model Evaluation

Cross-cultural language model evaluation tests whether AI systems understand and respond appropriately across languages, communities, and cultural contexts.

See all 121 papers

Same Values, Different Languages? From Multilingual Probing to Steering LLMs Toward Chinese Social Values
Sep 8, 2026Yuemei Xu, Kexin Xu, Jian Zhou +3Multilingual Language Model Evaluation Cultural Alignment
School of Information Science and Technology, Beijing Foreign Studies University · Beijing Academy of Artificial Intelligence · The State Key Lab of Software Development Environment, Beihang University
Lost in Translation: Do LVLM Judges Generalize Across Languages?
Apr 21, 2026Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Mir Tafseer Nayeem +5Multimodal Model Evaluation Multilingual VLM Evaluation
York University · University of Alberta · Nanyang Technological University +1
When Languages Disagree: Self-Evolving Multilingual LLM Judges
Jun 6, 2026Xiyan Fu, Wei LuMultilingual Language Model Evaluation LLM-as-a-Judge
Nanyang Technological University

Sep 8, 2026J/K move · Enter open · S save

Same Values, Different Languages? From Multilingual Probing to Steering LLMs Toward Chinese Social Values

Yuemei Xu, Kexin Xu, Jian Zhou, Haoyu Lu+2

School of Information Science and Technology, Beijing Foreign Studies University · Beijing Academy of Artificial Intelligence · The State Key Lab of Software Development Environment, Beihang University

Multilingual Language Model Evaluation Cultural Alignment Cross-Lingual Consistency in Language Models Activation Steering

Open paper AskPDF arXiv

As Large Language Models (LLMs) are increasingly integrated into human society, aligning them with pluralistic social values has become a critical priority. However, whether LLMs exhibit consistent value preferences across languages remains underexplored, particularly for culturally grounded values, which are more abstract and difficult to evaluate and align than safety-centric principles. We investigate this issue through Chinese Social Values (CSV), a value system rooted in Chinese culture and comprising

12

dimensions across national, societal, and personal levels. We construct C-Voices, the first comprehensive multilingual contrastive probe dataset for CSV, with 86,400 dilemma-based instances in six languages, each pairing a CSV-aligned action with a value-conflicting alternative. Building on the contrastive probes of C-Voices, we then propose a fine-tuning-free value vector steering method that derives value directions from hidden-state discrepancies and selectively intervenes on value-sensitive layers during inference. Experiments on six languages show that CSV-oriented preferences are model-dependent and language-sensitive, with the same dilemma eliciting divergent responses across languages. Our method achieves effective CSV steering, supports cross-lingual transfer of value vectors, and generalizes to existing FLAMES and ValuePrism.

Abstract

Cross-Cultural Language Model Evaluation

Explore similar work

Same Values, Different Languages? From Multilingual Probing to Steering LLMs Toward Chinese Social Values

Lost in Translation: Do LVLM Judges Generalize Across Languages?

When Languages Disagree: Self-Evolving Multilingual LLM Judges