MAVEN: A Macro-Societal Value Evaluation Framework of Multimodal Content with Compact Aligned Evaluators
Organizations: University of Science and Technology of China · State Key Laboratory of Communication Content Cognition, People’s Daily Online
Abstract
Assessing whether multimodal content aligns with macro-societal values, such as peace, justice, and freedom, has become an increasingly urgent challenge. Existing frameworks are largely confined to safety-oriented taxonomies, text-only psychometric probes, or single-label classification. Therefore, we propose MAVEN, a hierarchical framework for macro-societal value evaluation of multimodal content, grounded in international human-rights instruments and cultural value theory. MAVEN organizes values into 6 primary dimensions and 72 secondary indicators, supporting multi-level quantitative scoring. Building on MAVEN, we construct a human-verified multimodal benchmark and a soft-match metric to evaluate VLMs' assessments across value dimensions. For evaluator optimization, we propose a span-adaptive variant of multi-level preference optimization for evaluator distillation, together with a training-free multi-role consensus strategy at inference time. We evaluate existing open- and closed-source VLMs on our benchmark, revealing shared tendencies and clear differences in macro-societal value judgments. Experiments show that our compact 2B evaluator matches its 8B counterpart in the same family and approaches frontier closed-source VLMs, offering a practical path toward scalable macro-societal value evaluation. Our SA-MDPO implementation and MacroValue-Bench are available at https://github.com/zzzzzzzzjj/MAVEN.
Figures & tables
| Framework | Macro. | Multi. | Quant. |
|---|---|---|---|
| ETHICS Hendrycks et al. (2020) | ✗ | ✗ | ✓ |
| BBQ Parrish et al. (2022) | ✗ | ✗ | ✓ |
| Constitutional AI Bai et al. (2022) | ✗ | ✗ | ✗ |
| Ch3Ef Shi et al. (2024) | ✗ | ✓ | ✓ |
| VIVA Hu et al. (2024) | ✗ | ✓ | ✗ |
| ValueBench Ren et al. (2024) | ✗ | ✗ | ✓ |
| Dimension | Instruments | Schwartz |
|---|---|---|
| Peace | UN Charter; UDHR Pre. | Security |
| Development | UDHR 22, 25; ICESCR | Universalism |
| Equity | UDHR 1, 7; ICESCR 2 | Univ., Benev. |
| Justice | UDHR 6–11; ICCPR 14 | Univ., Conf. |
| Democracy | UDHR 21; ICCPR 25 | Self-dir. |
| Freedom | UDHR 18–20; ICCPR 18–22 | Self-dir., Stim. |
| Model | QWK | Acc. | F1 | VSMS |
|---|---|---|---|---|
| Commercial Models | ||||
| GPT-4o Hurst et al. (2024) | 0.563 | 0.471 | 0.346 | 0.919 |
| GPT-5 Singh et al. (2025) | 0.637 | 0.724 | 0.520 | 0.962 |
| Gemini-2.5-Pro Comanici et al. (2025) | 0.651 | 0.646 | 0.471 | 0.950 |
| Moonshot-V1-Vision Team et al. (2025) | 0.436 | 0.673 | 0.321 | 0.846 |
| Hunyuan-T1-Vision Team (2025b) | 0.588 | 0.689 | 0.441 | 0.933 |
| Method | Acc. | QWK | VSMS |
|---|---|---|---|
| Baseline | 0.497 | 0.393 | 0.798 |
| DPO (unbalanced) | 0.190 | 0.153 | 0.517 |
| DPO (balanced) | 0.654 | 0.267 | 0.946 |
| MDPO (unbalanced) | 0.656 | 0.503 | 0.920 |
| MDPO (balanced) | 0.670 | 0.445 | 0.964 |
| SA-MDPO (unbalanced) | 0.666 | 0.518 | 0.934 |
| Model | Acc. | QWK | VSMS |
|---|---|---|---|
| Qwen3-VL-2B (×14 runs) | 0.516 | 0.405 | 0.808 |
| Qwen3-VL-2B (MRC) | 0.527 | 0.432 | 0.815 |
| Qwen3-VL-4B (×14 runs) | 0.673 | 0.404 | 0.958 |
| Qwen3-VL-4B (MRC) | 0.662 | 0.424 | 0.957 |
| Qwen3-VL-8B (×14 runs) | 0.689 | 0.526 | 0.964 |
| Qwen3-VL-8B (MRC) | 0.706 | 0.540 | 0.968 |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Panel | N | Professional Status | Disciplinary Background | Cultural Background |
|---|---|---|---|---|
| Domain Experts | 6 | 2 Professors, 3 Researchers, 1 PhD Student | Law, Political Science, Sociology, Ethics, International Relations, Economics | 2 East Asian, 2 European, 1 African, 1 U.S. |
| Indicator | Definition |
|---|---|
| Peace. Non-violence and harmonious coexistence across individual, social, national, and global levels. | |
| World Peace | Absence of war or conflict among nations, with disputes resolved through diplomacy, cooperation, and rule of law. |
| Regional Peace | Stable and friendly relations among countries within a geographic region through regional cooperation mechanisms. |
| National Peace | Absence of large-scale violence or civil war within a country, maintained by rule of law and stable governance. |
| Social Peace | Elimination of violent confrontation among communities or social groups via inclusive policies and exchange. |
| Political Peace | Resolving political differences through democratic consultation, elections, or negotiation rather than repression. |
| Indicator | Definition |
|---|---|
| Development. Continuous progress and capacity enhancement of individuals, societies, and natural systems. | |
| Economic Development | Material wealth growth and structural optimization, including industrial upgrading and inclusive prosperity. |
| Social Development | Improving livelihoods through public services in education, healthcare, and housing, narrowing wealth gaps. |
| Personal Development | Realizing individual potential through education, skill training, and mental health support. |
| Technological Development | Scientific breakthroughs driving productivity, with ethical constraints and equitable benefit distribution. |
| Institutional Development | Improvement of laws, policies, and governance to enhance adaptability, transparency, and efficiency. |
| Indicator | Definition |
|---|---|
| Equity. Reasonable distribution of resources, opportunities, rights, and responsibilities among members of society. | |
| Opportunity Equity | Equal rights in education, employment, and entrepreneurship regardless of background, gender, or location. |
| Procedural Equity | Fair rule enforcement: consistent legal procedures, transparent standards, and competition free from privilege. |
| Educational Equity | Balanced resource allocation, access, and quality in education across socioeconomic backgrounds. |
| Employment Equity | Hiring, promotion, and compensation based solely on ability, prohibiting discrimination by gender, age, or race. |
| Healthcare Equity | Universal access to high-quality medical services regardless of payment ability or location. |
| Indicator | Definition |
|---|---|
| Justice. Institutions and actions aligned with moral and legal principles, protecting rights and balancing interests. | |
| Legal Justice | Legislation and judiciary conform to moral reasoning, with impartial laws and proportionate punishment. |
| Procedural Justice | Normative and neutral decision-making with rights to hearing, recusal, and reasoned explanations. |
| Rights-Based Justice | Guarantee of inalienable basic rights via constitutions, especially protecting minorities from majority tyranny. |
| Distributive Justice | Fair sharing of social resources and wealth, opposing extreme inequality through tax and welfare policies. |
| Social Justice | Equal and inclusive social structure, eliminating class, racial, or gender oppression. |
| Indicator | Definition |
|---|---|
| Democracy. Institutionalized public participation in decision-making and oversight of power. | |
| Popular Sovereignty | The people as the ultimate source of state power, with constitutional principles establishing this status. |
| Electoral Democracy | Selection of representatives or leaders through regular, competitive, and free elections. |
| Representative Democracy | Citizens elect representatives who must faithfully reflect public opinion and remain accountable. |
| Rule-of-Law Democracy | Democratic operations bound by constitutional and legal frameworks, with separation of powers and rights protection. |
| Constitutional Democracy | A written constitution defining government powers and citizen rights, safeguarded by judicial review. |
| Indicator | Definition |
|---|---|
| Freedom. Individual autonomy in thought, expression, action, and choice, free from oppression and coercion. | |
| Freedom of Thought | Right to independently form opinions, beliefs, and values without coercion or indoctrination. |
| Freedom of Belief | Right to choose, change, and practice religion or worldview, with state neutrality and protection of non-belief. |
| Freedom of Speech | Protection of expression without prior censorship or retaliation, balanced against public order and security. |
| Academic Freedom | Protection of scholars’ research, teaching, and publication from political or economic interference. |
| Creative Freedom | Autonomous expression for artists and cultural workers, bearing responsibility against incitement of violence. |
| Dimension | Fleiss | Avg. QWK | Exact |
|---|---|---|---|
| GLM-4.5V (3 runs) | |||
| Peace | 0.559 | 0.764 | 0.618 |
| Development | 0.599 | 0.781 | 0.655 |
| Equity | 0.487 | 0.633 | 0.679 |
| Justice | 0.486 | 0.655 | 0.638 |
| Democracy | 0.337 | 0.432 | 0.805 |
| Dimension | Fleiss | Avg. QWK | Exact | Indicator |
|---|---|---|---|---|
| Peace | 0.410 | 0.650 | 0.450 | 0.818 |
| Development | 0.242 | 0.615 | 0.256 | 0.765 |
| Equity | 0.305 | 0.518 | 0.487 | 0.871 |
| Justice | 0.339 | 0.557 | 0.464 | 0.837 |
| Democracy | 0.231 | 0.352 | 0.750 | 0.920 |
| Freedom | 0.204 | 0.447 | 0.400 | 0.848 |
| Dimension | Rate (%) |
|---|---|
| Peace | 25.11 |
| Justice | 19.02 |
| Freedom | 17.09 |
| Development | 15.60 |
| Equity | 12.78 |
| Democracy | 0 3.86 |
| Dimension | Fleiss’ | Avg QWK | Exact |
|---|---|---|---|
| Peace | 0.807 | 0.919 | 0.801 |
| Development | 0.878 | 0.934 | 0.886 |
| Equity | 0.828 | 0.890 | 0.871 |
| Justice | 0.757 | 0.855 | 0.806 |
| Democracy | 0.850 | 0.882 | 0.955 |
| Freedom | 0.784 | 0.875 | 0.806 |
| Model | VSMS | ||
|---|---|---|---|
| Qwen2.5-VL-3B | 0.946 | 0.951 | 0.005 |
| Qwen2.5-VL-7B | 0.911 | 0.921 | 0.010 |
| Qwen3-VL-2B | 0.787 | 0.798 | 0.011 |
| Qwen3-VL-4B | 0.954 | 0.954 | 0.000 |
| Qwen3-VL-8B | 0.956 | 0.961 | 0.005 |
| Doubao-Seed-1.6-Vision | 0.963 | 0.966 | 0.003 |
| # | Persona | Primary Evaluative Lens |
|---|---|---|
| 1 | Value Alignment Analyst | Impartial baseline; weighs all six primary dimensions equally based on textual evidence. |
| 2 | Educator | Educational Development , Opportunity Equity , Freedom of Thought ; sensitive to barriers to learning. |
| 3 | Public Health Physician | Health Development , Healthcare Equity , Personal Liberty (bodily autonomy), Inner Peace . |
| 4 | Financial Analyst | Economic Development , Market Equity , Economic Peace , Economic Freedom . |
| 5 | Judge | Procedural Justice , Rule-of-Law Democracy , Judicial Equity , Rights-Based Justice . |
| 6 | Environmental Scientist | Ecological Peace , Environmental Development , Intergenerational Equity , Environmental Justice . |
| Model | QWK | Acc. | VSMS | |
|---|---|---|---|---|
| Qwen3-VL-2B | 0.393 | 0.497 | 0.300 | 0.798 |
| Qwen3-VL-2B + SA-MDPO (JSON format) | 0.599 | 0.706 | 0.482 | 0.954 |
| Qwen3-VL-2B + SA-MDPO (Open-ended) | 0.484 | 0.657 | 0.403 | 0.862 |
| Modality | QWK | Accuracy | VSMS | |
|---|---|---|---|---|
| Baseline | 0.393 | 0.497 | 0.300 | 0.798 |
| Image | 0.407 | 0.655 | 0.323 | 0.937 |
| Text | 0.579 | 0.681 | 0.321 | 0.927 |
| Image & Text | 0.599 | 0.706 | 0.482 | 0.954 |
VLM Evaluation
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
See all 1,042 papers