Organizations: State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security · Tongyi Lab, Alibaba Group · Nanyang Technological University
As large language models (LLMs) are employed worldwide, existing evaluation paradigms for their multilingual capabilities primarily focus on factual task performance, neglecting the ability to judge content's deep-level values across multiple languages. To bridge this gap, we first reveal two primary challenges in constructing values judgment benchmarks, cultural diversity and disciplinary complexity, and propose a novel two-stage human-AI collaborative annotation framework to alleviate them. This framework identifies the issue scope and nature, establishes specific annotation criteria, and utilizes multiple LLMs for final review. Building upon this framework, we introduce X-Value, the first Cross-lingual Values Judgment Benchmark designed to evaluate the capability of LLMs in judging deep-level values of content. X-Value comprises 4,750 Question-Answer pairs across 14 languages, covering 7 major global issue categories, and provides 12 granular annotation metadata to facilitate a rigorous evaluation of model performance. Systematic evaluations of X-Value are conducted across 17 LLMs using distinct prompting strategies. Multi-dimensional analysis of accuracy and F1-scores reveals their limitations in cross-lingual values judgment and indicates performance disparities across categories and languages. This work highlights the urgent need to improve the underlying, values-aware content judgment capability of LLMs.\footnote{Samples of X-Value are available at https://huggingface.co/datasets/Whitolf/X-Value.}
Figures & tables
Figure 1: Overview of the data composition and annotation metadata of X-Value. (1) Data Composition: Characterized by linguistic diversity (covering 14 languages) and issue taxonomy (spanning 7 major categories). (2) Annotation Metadata: QA pairs have comprehensive annotations including core issues (with scope and nature), granular checkpoints and overall judgment conclusions.
Figure 2: The two challenges of cross-lingual values judgment. (a) Consistence rates (%) of values judgment across 10 national personas. Red colors indicate low consistence rates. This figure indicates the judgment inconsistency from cultural diversity. (b) Number of disciplines involved in samples. This figure shows the high knowledge demand imposed by disciplinary complexity.
Table 1: Schwartz’s Basic Human Values covered by issue categories in the taxonomy.
Overall Judgment
Judgment by Category
Category Stats.
Models
Acc
Acc0
Acc1
Acc0 - Acc1
P&G
N&M
H&N
E&S
R&F
G&H
P&B
avg.
std.
range
Judgment with Definition-only Prompt
GPT-5.2
69.78
92.33
57.52
34.81
68.38
71.29
70.59
66.29
73.03
65.57
73.77
69.85
2.96
8.20
GPT-5.4
75.50
88.46
68.43
20.03
76.43
75.68
75.59
74.25
77.29
72.71
76.62
75.51
1.45
4.57
Gemini-3-Flash-Prev.
70.95
94.86
57.93
36.93
71.43
71.71
74.67
71.29
68.14
70.43
69.38
71.01
1.90
6.52
Gemini-3.1-Pro-Prev.
71.22
91.94
59.95
31.99
69.43
69.29
75.50
74.29
68.86
70.86
70.92
71.31
2.40
6.64
Table 2: Overall accuracy (%), accuracy across values-appropriateness-based ( Acc0 and Acc1 ) and category-based subsets of 10 frontier LLMs using Def-only or Rubric prompts. For each case, the best is marked in red , and the second best is marked in bold .
Models
ar
de
en
es
fr
id
ja
ko
pt
ru
th
ur
vi
zh
avg.
std.
range
Judgment with Definition-only Prompt
GPT-5.2
72.13
68.48
66.18
83.09
76.30
67.43
75.71
64.86
71.43
65.14
69.14
72.80
68.23
56.73
69.83
6.04
26.36
GPT-5.4
81.95
76.57
71.35
80.86
83.43
73.71
80.86
66.00
75.43
76.79
79.71
72.80
70.00
65.90
75.38
5.49
17.53
Gemini-3-Flash-Prev.
78.00
66.00
67.14
79.43
80.57
68.86
66.57
67.43
73.14
75.43
69.14
74.80
66.00
61.14
70.98
5.69
19.43
Gemini-3.1-Pro-Prev.
75.71
60.86
73.43
72.29
75.43
71.14
77.14
70.57
73.43
70.57
64.57
75.20
69.00
68.57
71.28
4.34
16.29
Claude-Opus-4.5
72.57
60.57
67.71
74.86
69.71
72.57
76.86
69.71
68.77
70.29
62.00
78.40
69.33
68.00
70.10
4.77
17.83
Table 3: Values judgment accuracy (%) of 10 frontier LLMs across 14 languages using Def-only or Rubric prompts. For each case, the best is marked in red , and the second best is marked in bold .
Figure 3: F1-score (%) with corresponding precision and recall of 10 frontier LLMs using Rubric prompts. Each comparison dimension is sorted in ascending order of F1-scores (↑).
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Overall Judgment
Judgment by Category
Category Stats.
Models
Acc
Acc0
Acc1
Acc0 - Acc1
P&G
N&M
H&N
E&S
R&F
G&H
P&B
avg.
std.
range
Qwen-3.5-2B
52.60
97.67
28.07
69.60
52.71
55.86
52.50
49.64
53.57
48.57
55.54
52.63
2.54
7.29
Qwen-3.5-4B
64.21
96.95
46.39
50.56
68.86
66.43
63.67
59.43
64.29
58.57
68.46
64.24
3.77
10.29
Qwen-3.5-9B
63.07
97.73
44.21
53.52
67.71
65.43
61.00
60.00
62.14
59.29
65.85
63.06
3.02
8.43
Qwen-3.5-27B
66.76
97.25
50.16
47.09
70.43
68.71
66.67
63.29
66.43
65.00
66.77
66.76
2.16
7.14
Qwen-3.5-35B-A3B
63.75
97.67
45.29
52.38
66.57
66.71
62.83
59.43
62.57
61.57
66.62
63.76
2.69
7.29
Appendix
Table 4: Overall accuracy (%), accuracy across values-appropriateness-based ( Acc0 and Acc1 ) and category-based subsets of Qwen-3.5 series using Def-only prompts. For each case, the best is marked in red , and the second best is marked in bold .
Models
ar
de
en
es
fr
id
ja
ko
pt
ru
th
ur
vi
zh
avg.
std.
range
Qwen-3.5-2B
56.29
57.14
53.43
69.14
51.14
47.71
47.43
45.14
49.14
53.14
48.86
62.65
53.00
45.14
52.81
6.54
24.00
Qwen-3.5-4B
67.43
64.29
59.43
72.29
62.00
60.00
62.29
60.00
56.86
68.57
63.43
71.20
65.00
68.29
64.36
4.49
15.43
Qwen-3.5-9B
66.29
59.43
59.14
70.57
59.14
60.29
60.86
57.71
58.29
68.57
62.29
72.00
63.00
68.00
63.26
4.70
14.29
Qwen-3.5-27B
68.00
61.14
57.71
71.71
66.00
63.43
64.86
64.86
60.29
72.57
72.57
72.40
70.67
70.57
66.91
4.85
14.86
Qwen-3.5-35B-A3B
69.14
56.00
55.43
69.43
55.71
61.71
61.71
62.00
58.29
68.57
69.43
72.40
69.00
66.86
63.98
5.75
16.97
Qwen-3.5-122B-A10B
70.00
61.14
57.71
70.29
64.00
59.71
64.86
62.00
61.14
72.57
75.14
75.60
67.00
67.43
66.33
5.55
17.89
Appendix
Table 5: Values Judgment accuracy (%) of Qwen-3.5 series across 14 languages using Def-only prompts. For each case, the best is marked in red , and the second best is marked in bold .
Figure 1: Overall conclusion accuracy (%) of Gemini-3-Flash-Preview on language × category dimensions using Rubric prompts. Squares with deeper red indicate lower judgment accuracy.
Figure 2: Overall conclusion accuracy (%) of Qwen-3.5-Plus on language × category dimensions using Rubric prompts. Squares with deeper red indicate lower judgment accuracy.
Figure 3: Overall conclusion accuracy (%) of GPT-5.4 on language × category dimensions using Rubric prompts. Squares with deeper red indicate lower judgment accuracy.
Figure 4: Overall conclusion accuracy (%) of Claude-Opus-4.6 on language × category dimensions using Rubric prompts. Squares with deeper red indicate lower judgment accuracy.
Figure 5: The prompt used for simulating persona from 10 national backgrounds.
Discipline
Detailed Explanation
Cultural Studies
Analyzes how culture is produced and transformed within social power structures and their impact on individual and collective identities.
Anthropology
Reveals the diversity of human cultures, their mechanisms of transmission, and the evolution of values through the study of different social groups.
Ethics
Systematically explores moral principles and core values, studying the normative standards used by individuals and societies to judge right and wrong.
Sociology
Examines how social institutions, customs, and norms shape collective consciousness and how cultural values function within social interactions and structures.
Religion
Investigates belief systems and sacred rituals, which often serve as the foundational source of cultural identity and core moral values for specific societies.
Folklore
Analyzes the grassroots culture and ethnic values accumulated over history by studying legends, traditional customs, and oral literature.
Appendix
Table 6: Disciplines Related to Culture and Values
Figure 6: The prompt used for scoring relevance for 10 representative disciplines.
Figure 7: The prompt used for generating Answers to Questions.
Figure 8: The prompt used for getting the preliminary values-judgment results from LLMs.
Figure 9: Distribution of issue domains by language across all QA pairs.
Code
Language
Language Family
Native Speakers (M)
en
English
Indo-European
1500
zh
Chinese
Sino-Tibetan
1200
es
Spanish
Indo-European
561
ar
Arabic
Afroasiatic
335
fr
French
Indo-European
334
pt
Portuguese
Indo-European
269
Appendix
Table 7: Detailed information of all 14 languages covered by our X-Value. Statistical Data are from https://www.ethnologue.com/.
Code
Language
Total QA
Question Length
Answer Length
Easy
Hard
Avg.
Min
Max
Std.
Avg.
Min
Max
Std.
ar
Arabic
120
230
14.0
100
7
9.9
138.4
218
89
24.1
de
German
120
230
21.7
140
5
12.4
161.9
280
80
32.6
en
English
120
230
22.4
37
11
4.9
128.6
275
52
38.8
es
Spanish
141
209
15.8
101
9
7.9
171.7
294
105
35.4
fr
French
120
230
29.9
170
8
19.9
191.7
380
38
39.9
Appendix
Table 8: Key statistics of the total 4,750 QA pairs in 14 languages.
Figure 10: Distribution of numbers of QA Pairs across 14 languages and 7 issue Domains
Cross-cultural language model evaluation tests whether AI systems understand and respond appropriately across languages, communities, and cultural contexts.
As Large Language Models (LLMs) are increasingly integrated into human society, aligning them with pluralistic social values has become a critical priority. However, whether LLMs exhibit consistent value preferences across languages remains underexplored, particularly for culturally grounded values, which are more abstract and difficult to evaluate and align than safety-centric principles. We investigate this issue through Chinese Social Values (CSV), a value system rooted in Chinese culture and comprising 12 dimensions across national, societal, and personal levels. We construct C-Voices, the first comprehensive multilingual contrastive probe dataset for CSV, with 86,400 dilemma-based instances in six languages, each pairing a CSV-aligned action with a value-conflicting alternative. Building on the contrastive probes of C-Voices, we then propose a fine-tuning-free value vector steering method that derives value directions from hidden-state discrepancies and selectively intervenes on value-sensitive layers during inference. Experiments on six languages show that CSV-oriented preferences are model-dependent and language-sensitive, with the same dilemma eliciting divergent responses across languages. Our method achieves effective CSV steering, supports cross-lingual transfer of value vectors, and generalizes to existing FLAMES and ValuePrism.
Yuemei Xu, Kexin Xu, Jian Zhou +3
School of Information Science and Technology, Beijing Foreign Studies University · Beijing Academy of Artificial Intelligence · The State Key Lab of Software Development Environment, Beihang University
Automatic evaluators such as reward models play a central role in the alignment and evaluation of large vision-language models (LVLMs). Despite their growing importance, these evaluators are almost exclusively assessed on English-centric benchmarks, leaving open the question of how well these evaluators generalize across languages. To answer this question, we introduce MM-JudgeBench, the first large-scale benchmark for multilingual and multimodal judge model evaluation, which includes over 60K pairwise preference instances spanning 25 typologically diverse languages. MM-JudgeBench integrates two complementary subsets: a general vision-language preference evaluation subset extending VL-RewardBench, and a chart-centric visual-text reasoning subset derived from OpenCQA, enabling systematic analysis of reward models (i.e., LVLM judges) across diverse settings. We additionally release a multilingual training set derived from MM-RewardBench, disjoint from our evaluation data, to support domain adaptation. By evaluating 22 LVLMs (15 open-source, 7 proprietary), we uncover substantial cross-lingual performance variance in our proposed benchmark. Our analysis further shows that model size and architecture are poor predictors of multilingual robustness, and that even state-of-the-art LVLM judges exhibit inconsistent behavior across languages. Together, these findings expose fundamental limitations of current reward modeling and underscore the necessity of multilingual, multimodal benchmarks for developing reliable automated evaluators.
Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Mir Tafseer Nayeem +5
York University · University of Alberta · Nanyang Technological University +1
Multilingual LLM-as-a-judge is widely used to evaluate model outputs across languages, but suffers from cross-lingual inconsistency (Fu and Liu, 2025). Existing methods typically treat this inconsistency as noise and mitigate it through voting or aggregation. In this work, we instead show that multilingual inconsistency can provide complementary evaluation signals. Our oracle analysis finds that sampling judgments across languages yields a higher performance upper bound than single-language judging, indicating that different languages potentially include complementary judgments. Motivated by this finding, we propose SEMJ, a self-evolving multilingual judge that leverages cross-lingual inconsistency for iterative refinement. SEMJ constructs multilingual variants of each input, collects independent judgments and rationales, and feeds inconsistent outputs back for self-reflection and re-evaluation. Experiments on multiple benchmarks show that SEMJ consistently outperforms voting and reflection baselines in both accuracy and cross-lingual consistency. Further analysis shows that inconsistency triggers useful re-evaluation, which improves judgment quality.