Assessing whether multimodal content aligns with macro-societal values, such as peace, justice, and freedom, has become an increasingly urgent challenge. Existing frameworks are largely confined to safety-oriented taxonomies, text-only psychometric probes, or single-label classification. Therefore, we propose MAVEN, a hierarchical framework for macro-societal value evaluation of multimodal content, grounded in international human-rights instruments and cultural value theory. MAVEN organizes values into 6 primary dimensions and 72 secondary indicators, supporting multi-level quantitative scoring. Building on MAVEN, we construct a human-verified multimodal benchmark and a soft-match metric to evaluate VLMs' assessments across value dimensions. For evaluator optimization, we propose a span-adaptive variant of multi-level preference optimization for evaluator distillation, together with a training-free multi-role consensus strategy at inference time. We evaluate existing open- and closed-source VLMs on our benchmark, revealing shared tendencies and clear differences in macro-societal value judgments. Experiments show that our compact 2B evaluator matches its 8B counterpart in the same family and approaches frontier closed-source VLMs, offering a practical path toward scalable macro-societal value evaluation. Our SA-MDPO implementation and MacroValue-Bench are available at https://github.com/zzzzzzzzjj/MAVEN.
Figures & tables
Figure 1: Application background of MAVEN versus general value frameworks. Left: MAVEN takes real-world multimodal media as the evaluation target. An evaluator scores each item along six macro-societal value dimensions, supporting continuous monitoring of the media landscape. Right: General value frameworks take a target model as the evaluation object. The value diagnosis of the model’s outputs is fed back to align the model’s behavior.
Framework
Macro.
Multi.
Quant.
ETHICS Hendrycks et al. (2020)
✗
✗
✓
BBQ Parrish et al. (2022)
✗
✗
✓
Constitutional AI Bai et al. (2022)
✗
✗
✗
Ch3Ef Shi et al. (2024)
✗
✓
✓
VIVA Hu et al. (2024)
✗
✓
✗
ValueBench Ren et al. (2024)
✗
✗
✓
Table 1: Comparison of value frameworks used for AI evaluation. ✓ = full support, ⚫ = partial, ✗ = not supported. Macro. ( Macro-societal ): targets societal-level values rather than individual ethics or safety. Multi. ( Multimodal ): operationalized on image–text content. Quant. ( Quantitative ): yields ordinal or continuous scores rather than categorical labels.
Figure 2: Overview of the MAVEN hierarchical framework and evaluator optimization. (I) MAVEN Framework: Macro-societal value taxonomy with 6 primary dimensions and 72 secondary indicators. Secondary indicators are constructed through an iterative human–AI collaborative pipeline to ensure comprehensive coverage and conceptual clarity. (II) Evaluator Optimization: A compact student VLM is distilled from frontier teachers via our proposed SA-MDPO. Then multi-role consensus aggregates different stakeholder perspectives at inference time.
Figure 3: Overview of MacroValue-Bench construction and the evaluation pipeline. (Part A) We collect ∼ 30K multimodal items from four sources, sample 1,157 image–text pairs, annotate each pair via three frontier VLMs, then verify with human annotators. Each item carries a structured dimensional label. (Part B) We evaluate VLMs’ predictions against ground-truth annotations using Value-Aware Soft Match (Recall direction shown; Precision is symmetric), which assigns partial credit to semantically close cross-dimension predictions.
Dimension
Instruments
Schwartz
Peace
UN Charter; UDHR Pre.
Security
Development
UDHR 22, 25; ICESCR
Universalism
Equity
UDHR 1, 7; ICESCR 2
Univ., Benev.
Justice
UDHR 6–11; ICCPR 14
Univ., Conf.
Democracy
UDHR 21; ICCPR 25
Self-dir.
Freedom
UDHR 18–20; ICCPR 18–22
Self-dir., Stim.
Table 2: Theoretical foundations of Maven ’s six primary dimensions. UDHR/ICCPR/ICESCR numbers refer to article numbers. Pre. denotes Preamble. Schwartz value abbreviations: Univ. = Universalism, Benev. = Benevolence, Conf. = Conformity, Self-dir. = Self-direction, Stim. = Stimulation.
Figure 4: Cosine similarity between Qwen3-Embedding-8B representations of MAVEN’s 72 secondary indicators. (Left) Similarity matrix showing block-diagonal within-dimension coupling and off-block-diagonal cross-dimension neighbors. (Right) Distribution of pairwise similarities, with a dense bulk near 0.40 and a sparse right tail beyond 0.65 .
Figure 5: Label correlation of indicators and its relation to semantic similarity. (Left) Spearman correlation among the 72 secondary indicators in the MacroValue-Bench. (Right) Semantic similarity versus label correlation for indicator pairs; orange: same dimension, blue: across dimensions, black curve: LOWESS trend.
Figure 6: Case study. Qwen : Qwen3-VL-8B-Instruct; Gemini : Gemini-2.5-Pro; Ours : Qwen3-VL-2B trained under SA-MDPO with MRC. Top: predicted primary scores. Qwen underactivates all dimensions; Gemini over-activates Peace and Development (false positives); our distilled 2B model with MRC correctly identifies Equity, Justice, and Freedom with no false positives. Bottom: per-indicator predictions. Above the divider are the 7 ground-truth non-zero indicators; below are indicators that models hallucinated. Our compact 2B model matches Gemini-2.5-Pro’s indicator recall (6/7) while producing zero false positives.
Model
QWK ↑
Acc. ↑
F1 macro↑
VSMS ↑
Commercial Models
GPT-4o Hurst et al. (2024)
0.563
0.471
0.346
0.919
GPT-5 Singh et al. (2025)
0.637
0.724
0.520
0.962
Gemini-2.5-Pro Comanici et al. (2025)
0.651
0.646
0.471
0.950
Moonshot-V1-Vision Team et al. (2025)
0.436
0.673
0.321
0.846
Hunyuan-T1-Vision Team (2025b)
0.588
0.689
0.441
0.933
Table 3: Performance on MacroValue-Bench. All metrics are higher-is-better ( ↑ ); bold indicates the best overall. Doubao-Seed-1.6-V leads across all four metrics among commercial models, while our compact 2B evaluator matches 8B in the same family and approaches frontier commercial VLMs at a 4× smaller parameter count.
Figure 7: Per-dimension comparison across closed-source baselines on three metrics (Accuracy, QWK, F1 macro ).
Figure 8: Per-dimension QWK of the Qwen3-VL family (2B/4B/8B) and our SA-MDPO distilled evaluator. Our compact 2B model matches or exceeds Qwen3-VL-8B on all six dimensions.
Method
Acc.
QWK
VSMS
Baseline
0.497
0.393
0.798
DPO (unbalanced)
0.190
0.153
0.517
DPO (balanced)
0.654
0.267
0.946
MDPO (unbalanced)
0.656
0.503
0.920
MDPO (balanced)
0.670
0.445
0.964
SA-MDPO (unbalanced)
0.666
0.518
0.934
Table 4: Ablation study comparing DPO, MDPO, and SA-MDPO under different training dataset.
Model
Acc.
QWK
VSMS
Qwen3-VL-2B (×14 runs)
0.516
0.405
0.808
Qwen3-VL-2B (MRC)
0.527
0.432
0.815
Qwen3-VL-4B (×14 runs)
0.673
0.404
0.958
Qwen3-VL-4B (MRC)
0.662
0.424
0.957
Qwen3-VL-8B (×14 runs)
0.689
0.526
0.964
Qwen3-VL-8B (MRC)
0.706
0.540
0.968
Table 5: Effect of Multi-Role Consensus (MRC) on Qwen3-VL-Instruct models.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Panel
N
Professional Status
Disciplinary Background
Cultural Background
Domain Experts
6
2 Professors, 3 Researchers, 1 PhD Student
Law, Political Science, Sociology, Ethics, International Relations, Economics
2 East Asian, 2 European, 1 African, 1 U.S.
Appendix
Table 6: Information on domain experts.
Indicator
Definition
Peace. Non-violence and harmonious coexistence across individual, social, national, and global levels.
World Peace
Absence of war or conflict among nations, with disputes resolved through diplomacy, cooperation, and rule of law.
Regional Peace
Stable and friendly relations among countries within a geographic region through regional cooperation mechanisms.
National Peace
Absence of large-scale violence or civil war within a country, maintained by rule of law and stable governance.
Social Peace
Elimination of violent confrontation among communities or social groups via inclusive policies and exchange.
Political Peace
Resolving political differences through democratic consultation, elections, or negotiation rather than repression.
Appendix
Table 7: Secondary indicators of Peace .
Indicator
Definition
Development. Continuous progress and capacity enhancement of individuals, societies, and natural systems.
Economic Development
Material wealth growth and structural optimization, including industrial upgrading and inclusive prosperity.
Social Development
Improving livelihoods through public services in education, healthcare, and housing, narrowing wealth gaps.
Personal Development
Realizing individual potential through education, skill training, and mental health support.
Technological Development
Scientific breakthroughs driving productivity, with ethical constraints and equitable benefit distribution.
Institutional Development
Improvement of laws, policies, and governance to enhance adaptability, transparency, and efficiency.
Appendix
Table 8: Secondary indicators of Development .
Indicator
Definition
Equity. Reasonable distribution of resources, opportunities, rights, and responsibilities among members of society.
Opportunity Equity
Equal rights in education, employment, and entrepreneurship regardless of background, gender, or location.
Procedural Equity
Fair rule enforcement: consistent legal procedures, transparent standards, and competition free from privilege.
Educational Equity
Balanced resource allocation, access, and quality in education across socioeconomic backgrounds.
Employment Equity
Hiring, promotion, and compensation based solely on ability, prohibiting discrimination by gender, age, or race.
Healthcare Equity
Universal access to high-quality medical services regardless of payment ability or location.
Appendix
Table 9: Secondary indicators of Equity .
Indicator
Definition
Justice. Institutions and actions aligned with moral and legal principles, protecting rights and balancing interests.
Legal Justice
Legislation and judiciary conform to moral reasoning, with impartial laws and proportionate punishment.
Procedural Justice
Normative and neutral decision-making with rights to hearing, recusal, and reasoned explanations.
Rights-Based Justice
Guarantee of inalienable basic rights via constitutions, especially protecting minorities from majority tyranny.
Distributive Justice
Fair sharing of social resources and wealth, opposing extreme inequality through tax and welfare policies.
Social Justice
Equal and inclusive social structure, eliminating class, racial, or gender oppression.
Appendix
Table 10: Secondary indicators of Justice .
Indicator
Definition
Democracy. Institutionalized public participation in decision-making and oversight of power.
Popular Sovereignty
The people as the ultimate source of state power, with constitutional principles establishing this status.
Electoral Democracy
Selection of representatives or leaders through regular, competitive, and free elections.
Representative Democracy
Citizens elect representatives who must faithfully reflect public opinion and remain accountable.
Rule-of-Law Democracy
Democratic operations bound by constitutional and legal frameworks, with separation of powers and rights protection.
Constitutional Democracy
A written constitution defining government powers and citizen rights, safeguarded by judicial review.
Appendix
Table 11: Secondary indicators of Democracy .
Indicator
Definition
Freedom. Individual autonomy in thought, expression, action, and choice, free from oppression and coercion.
Freedom of Thought
Right to independently form opinions, beliefs, and values without coercion or indoctrination.
Freedom of Belief
Right to choose, change, and practice religion or worldview, with state neutrality and protection of non-belief.
Freedom of Speech
Protection of expression without prior censorship or retaliation, balanced against public order and security.
Academic Freedom
Protection of scholars’ research, teaching, and publication from political or economic interference.
Creative Freedom
Autonomous expression for artists and cultural workers, bearing responsibility against incitement of violence.
Appendix
Table 12: Secondary indicators of Freedom .
Dimension
Fleiss κ
Avg. QWK
Exact
GLM-4.5V (3 runs)
Peace
0.559
0.764
0.618
Development
0.599
0.781
0.655
Equity
0.487
0.633
0.679
Justice
0.486
0.655
0.638
Democracy
0.337
0.432
0.805
Appendix
Table 13: Intra-VLM consistency. Each VLM is queried three times on every item; agreement is measured across the three runs. All three VLMs exhibit substantial self-consistency, with an average QWK of 0.71 across the three VLMs.
Dimension
Fleiss κ
Avg. QWK
Exact
Indicator
Peace
0.410
0.650
0.450
0.818
Development
0.242
0.615
0.256
0.765
Equity
0.305
0.518
0.487
0.871
Justice
0.339
0.557
0.464
0.837
Democracy
0.231
0.352
0.750
0.920
Freedom
0.204
0.447
0.400
0.848
Appendix
Table 14: Inter-VLM agreement across GLM-4.5V, Doubao-Seed-1.6-V, and Gemini-2.5-Pro. Exact is the primary-dimension exact-agreement rate; Indicator is the exact-agreement rate on the corresponding 12 secondary indicators per dimension.
Figure 9: Screenshot of the human verification interface. The left panel displays the sample’s title, image, and text. The right panel shows, for each of the six value dimensions, a total-score slider ( −2 to 2 ) and 12 sub-indicator selectors ( Supports/Embodies , Opposes/Violates , or Irrelevant/Not mentioned ).
Dimension
Rate (%)
Peace
25.11
Justice
19.02
Freedom
17.09
Development
15.60
Equity
12.78
Democracy
0 3.86
Appendix
Table 15: Human modification rate : the fraction of primary-dimension labels that human verifiers modified after 3-VLM majority voting.
Dimension
Fleiss’ κ↑
Avg QWK ↑
Exact ↑
Peace
0.807
0.919
0.801
Development
0.878
0.934
0.886
Equity
0.828
0.890
0.871
Justice
0.757
0.855
0.806
Democracy
0.850
0.882
0.955
Freedom
0.784
0.875
0.806
Appendix
Table 16: Inter-annotator agreement across 8 human annotators on 600 re-sampled instances, measured by Fleiss’ κ , average pairwise Quadratic Weighted Kappa (QWK), and exact agreement rate.
Model
F1Hard
VSMS
Δ
Qwen2.5-VL-3B
0.946
0.951
0.005
Qwen2.5-VL-7B
0.911
0.921
0.010
Qwen3-VL-2B
0.787
0.798
0.011
Qwen3-VL-4B
0.954
0.954
0.000
Qwen3-VL-8B
0.956
0.961
0.005
Doubao-Seed-1.6-Vision
0.963
0.966
0.003
Appendix
Table 17: Comparison between Hard-match F1 and VSMS (Soft-match F1, τ=0.65 ) across evaluated models. Δ = VSMS −F1Hard , computed from unrounded values.
#
Persona
Primary Evaluative Lens
1
Value Alignment Analyst
Impartial baseline; weighs all six primary dimensions equally based on textual evidence.
2
Educator
Educational Development , Opportunity Equity , Freedom of Thought ; sensitive to barriers to learning.
3
Public Health Physician
Health Development , Healthcare Equity , Personal Liberty (bodily autonomy), Inner Peace .
Table 18: The 14 personas used in Multi-Role Consensus (MRC), each emphasizing a complementary set of MAVEN secondary indicators ( italicized ). Persona 1 serves as an impartial baseline; personas 2–14 each bring a stakeholder-specific perspective. The full system prompts are released with the dataset.
Model
QWK ↑
Acc. ↑
F1macro↑
VSMS ↑
Qwen3-VL-2B
0.393
0.497
0.300
0.798
Qwen3-VL-2B + SA-MDPO (JSON format)
0.599
0.706
0.482
0.954
Qwen3-VL-2B + SA-MDPO (Open-ended)
0.484
0.657
0.403
0.862
Appendix
Table 19: Output format generalization study. Performance of Qwen3-VL-2B and our SA-MDPO distilled model under structured JSON output versus unrestricted open-ended generation at inference time.
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
As large language models (LLMs) are employed worldwide, existing evaluation paradigms for their multilingual capabilities primarily focus on factual task performance, neglecting the ability to judge content's deep-level values across multiple languages. To bridge this gap, we first reveal two primary challenges in constructing values judgment benchmarks, cultural diversity and disciplinary complexity, and propose a novel two-stage human-AI collaborative annotation framework to alleviate them. This framework identifies the issue scope and nature, establishes specific annotation criteria, and utilizes multiple LLMs for final review. Building upon this framework, we introduce X-Value, the first Cross-lingual Values Judgment Benchmark designed to evaluate the capability of LLMs in judging deep-level values of content. X-Value comprises 4,750 Question-Answer pairs across 14 languages, covering 7 major global issue categories, and provides 12 granular annotation metadata to facilitate a rigorous evaluation of model performance. Systematic evaluations of X-Value are conducted across 17 LLMs using distinct prompting strategies. Multi-dimensional analysis of accuracy and F1-scores reveals their limitations in cross-lingual values judgment and indicates performance disparities across categories and languages. This work highlights the urgent need to improve the underlying, values-aware content judgment capability of LLMs.\footnote{Samples of X-Value are available at https://huggingface.co/datasets/Whitolf/X-Value.}
Yukun Chen, Xinyu Zhang, Boyi Deng +6
State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security · Tongyi Lab, Alibaba Group +1
Cultural values are expressed not only through language but also through visual scenes and everyday social practices. Yet existing evaluations of cultural values in language models are almost entirely text-only, leaving it unclear whether culture-conditioned judgments remain stable when response options are visualized. We introduce ValueGround, a benchmark for evaluating culture-conditioned visual value grounding in multimodal large language models (MLLMs). Built from World Values Survey questions, ValueGround uses minimally contrastive image pairs to represent opposing response options while controlling irrelevant variation. Given a country, a question, and an image pair, a model must choose the image that best matches the country's value tendency without access to the original response-option texts. Experiments across six MLLMs and 13 countries show that models perform substantially worse with visualized response options than with the original textual options, with average accuracy dropping from 72.8% to 62.6%. Our benchmark provides a controlled testbed for studying cross-modal transfer of culture-conditioned value judgments.
Vision-Language Models increasingly make socially consequential judgments from image-text inputs, yet current evaluations often stop at verdict-level accuracy: whether the final label is human-aligned, not whether it follows from the correct evidence. We introduce MM-SCALE (Multimodal Moral SCALE), a benchmark for evidence-grounded moral judgment built around a design choice: each image is paired with multiple action scenarios, so models must compare how different actions interact with the same visual context rather than score images in isolation. MM-SCALE contains 8,444 image contexts and 21,977 action scenarios, each annotated with a 5-point moral acceptability rating and a modality-grounding label indicating whether the human judgment relied on text, image, or both. Three tasks evaluate whether models (i) assign calibrated scalar moral scores, (ii) preserve human orderings within a shared visual context, and (iii) ground rationales in the evidence source humans found critical to judgment. Across models, aggregate metrics overstate alignment: NDCG@5 remains high across models, while within-image pairwise accuracy remains only 0.53-0.62 even on scenario pairs whose human mean ratings differ by at least one point. CoT has inconsistent effects on calibration and does not reliably improve within-image ordering. Evidence grounding evaluations further show that models often describe visual context without making it critical for the verdict. These results reveal an evidence-grounding failure in current VLMs and position MM-SCALE as an evaluation testbed for moral judgment under shared image contexts. This dataset includes potentially harmful and sensitive visual content. Images are intended solely for evaluation.
Eunkyu Park, Wesley Hanwen Deng, Cheyon Jin +8
Seoul National University · Carnegie Mellon University · University of Florida +1