ValueGround: Evaluating Culture-Conditioned Visual Value Grounding in MLLMs

Zhipin Wang; Christoph Leiter; Christian Frey; Mohamed Hesham Ibrahim Abdalla; Josif Grabocka; Steffen Eger

cs.CLApr 7, 2026

ValueGround: Evaluating Culture-Conditioned Visual Value Grounding in MLLMs

Authors: Zhipin Wang, Christoph Leiter, Christian Frey, Mohamed Hesham Ibrahim Abdalla, Josif Grabocka, Steffen Eger

Organizations: University of Technology Nuremberg (UTN), Germany

Abstract

Cultural values are expressed not only through language but also through visual scenes and everyday social practices. Yet existing evaluations of cultural values in language models are almost entirely text-only, leaving it unclear whether culture-conditioned judgments remain stable when response options are visualized. We introduce ValueGround, a benchmark for evaluating culture-conditioned visual value grounding in multimodal large language models (MLLMs). Built from World Values Survey questions, ValueGround uses minimally contrastive image pairs to represent opposing response options while controlling irrelevant variation. Given a country, a question, and an image pair, a model must choose the image that best matches the country's value tendency without access to the original response-option texts. Experiments across six MLLMs and 13 countries show that models perform substantially worse with visualized response options than with the original textual options, with average accuracy dropping from 72.8% to 62.6%. Our benchmark provides a controlled testbed for studying cross-modal transfer of culture-conditioned value judgments.

Figures & tables

Figure 1: A motivating reversal example. For the same country-question pair, the model predicts one response when the answer choices are textual, but reverses its prediction when the response options are replaced with visual proxies. This example illustrates the central challenge in ValueGround : whether models can maintain consistent culture-conditioned judgments when response options are visualized.

Figure 2: Overview of ValueGround . Starting from a value question $q$ with two response options $(o_{A},o_{B})$ , we construct a controlled image pair $(I_{A},I_{B})$ that reflects the question and two option meanings while minimizing irrelevant differences and shortcut cues. In the main task, the model receives a country $c$ , the question $q$ , and the image pair $(I_{A},I_{B})$ , and must choose which image better matches the value tendency associated with that country. The response-option texts are hidden at test time.

Backbone	Configuration	$Q_{1}$ Match-A	$Q_{2}$ Match-B	$Q_{3}$ PairMatch	$Q_{4}$ NoCue	Total Score
GPT	Planner only	1.575	1.483	1.180	0.815	0.733
GPT	Planner+Critic	1.605	1.508	1.097	0.749	0.714
GPT	Planner+Editor+Critic	1.521	1.424	1.981	0.934	0.849
Gemini	Planner only	1.737	1.667	0.533	0.821	0.698
Gemini	Planner+Critic	1.583	1.337	1.276	0.811	0.727
Gemini	Planner+Editor+Critic	1.621	1.533	1.931	0.913	0.864

Table 1: Automatic rubric scores from the ensemble validation judge for construction-pipeline ablations. Total Score is the normalized aggregate score, computed as

(Q_{1}/2+Q_{2}/2+Q_{3}/2+Q_{4})/4

Figure 3: Country-level accuracy (%) on the main task. Rows correspond to models and columns to countries. Darker colors indicate higher accuracy.

Model	Main	Text-only	Align.	Main $\mid$ Align $\checkmark$	Text $\mid$ Align $\checkmark$	$\Delta^{+}$
Mistral-3.2-24B	59.9	72.8	87.1	61.1	72.3	-11.1
Gemma-3-27B	59.8	68.5	83.7	60.5	68.6	-8.1
Qwen3-VL-32B	62.3	72.7	90.2	63.6	72.7	-9.1
Claude Haiku 4.5	56.7	70.9	77.5	59.3	69.9	-10.6
GPT-5.4 mini	66.6	72.2	90.2	70.2	72.2	-1.9
Gemini 3 Flash	70.2	80.0	90.6	74.9	79.9	-5.0

Table 2: Accuracy (%) on the main task, text-only setting, option–image alignment setting, and alignment-filtered subsets. Main

\mid

Align

\checkmark

and Text

\mid

Align

\checkmark

respectively report main-task and text-only accuracy on the same subset of country-question instances

(c,q)

for which the model correctly identifies the option–image correspondence for question

q

in the alignment task.

\Delta^{+}

denotes Main

\mid

Align

\checkmark

minus Text

\mid

Align

\checkmark

. The consistently negative

\Delta^{+}

indicates that correct option–image alignment does not eliminate the gap between text-only and main task.

Figure 4: Illustration of the edit-based pipeline for a hard-to-visualize question. The pipeline preserves semantic alignment to the target options while producing a more controlled contrastive pair through a shared base scene and minimal edits.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Overview of our agentic paired-image construction framework.

Backbone	Configuration	Avg. cost
Gemini	Planner only	$0.1422
Gemini	Planner+Critic	$0.1608
Gemini	Planner+Editor+Critic	$0.2189
GPT	Planner only	$0.1100
GPT	Planner+Critic	$0.1178
GPT	Planner+Editor+Critic	$0.2244

Appendix

Table 3: Average generation cost per candidate image pair for the construction variants evaluated in Table 1 .

Figure 6: Human annotation instructions used for manual verification.

Metric	Judge vs. Human Mean		Human Agreement
	Pearson	Spearman	Pearson	Spearman
$Q_{1}$	0.345	0.400	0.520	0.546
$Q_{2}$	0.630	0.526	0.592	0.536
$Q_{3}$	0.723	0.700	0.562	0.541
$Q_{4}$	0.591	0.387	0.367	0.289
Total	0.633	0.614	0.581	0.583

Appendix

Table 4: Agreement between the automatic judge and human annotations on the 36-example validation subset. Human Agreement denotes leave-one-annotator-out agreement.

Model	Reversal	Harmful	Beneficial
Qwen3-VL-32B	29.7	21.2	8.6
Mistral-3.2-24B	34.0	24.3	9.7
Gemma-3-27B	35.0	22.5	12.5
Claude Haiku 4.5	38.7	26.8	11.8
GPT-5.4 mini	22.0	14.8	7.2
Gemini 3 Flash	22.5	17.4	5.1

Appendix

Table 5: Cross-modal reversal rates (%) in the final evaluation. Percentages are computed over

5{,}730

source-specific country-question instances per model. Harmful and beneficial reversals sum to the total reversal rate up to rounding.

Country	Reversal	Harmful	Beneficial
Russia	33.5	22.3	11.2
Japan	32.9	22.2	10.7
Mexico	32.0	22.7	9.3
China	31.9	21.3	10.6
Turkey	31.6	20.9	10.7
Kenya	30.5	20.8	9.7

Appendix

Table 6: Country-level cross-modal reversal rates (%) in the final evaluation. Percentages are computed by aggregating all available model-source estimates for each country. Harmful and beneficial reversals sum to the total reversal rate up to rounding.

Bin	Midpoint-margin range	Reversal rate
Bin 1	$[0.000,\,0.061)$	33.3
Bin 2	$[0.061,\,0.126)$	30.6
Bin 3	$[0.126,\,0.205)$	29.6
Bin 4	$[0.205,\,0.304)$	28.0
Bin 5	$[0.304,\,0.497]$	30.1

Appendix

Table 7: Cross-modal reversal rate (%) as a function of midpoint margin

d_{c,q}=|u_{c,q}-0.5|

. Bins are equal-frequency bins over the 2,865 country-question pairs; each bin contains 573 country-question pairs and 6,876 scorable model-country-question-source observations from the final evaluation. Smaller midpoint margin indicates that the country-question pair lies closer to the binary WVS decision boundary.

Model	Pearson $r$	Spearman $\rho$	Mean $\Delta$ (Main $-$ Text)	Main $<$ Text
Qwen3-VL-32B	0.908	0.801	$-12.6$	13/13
Mistral-3.2-24B	0.839	0.736	$-14.7$	13/13
Gemma-3-27B	0.833	0.841	$-10.0$	13/13
Claude Haiku 4.5	0.548	0.632	$-15.0$	13/13
GPT-5.4 mini	0.937	0.901	$-7.6$	13/13
Gemini 3 Flash	0.899	0.834	$-12.3$	13/13

Appendix

Table 8: Country-level correlation between text-only and main-task accuracy. Each model contributes 13 paired country-level points from the final evaluation. Mean

\Delta

(Main

-

Text) denotes main-task accuracy minus text-only accuracy in percentage points. The last column reports the number of countries for which main-task accuracy is lower than text-only accuracy.

Figure 7: Country-level text-only versus main-task accuracy across all 78 model-country pairs. Colors indicate models. The dashed line denotes

y=x

, and the solid line is the least-squares fit. Although the two settings are strongly positively correlated, all points lie below the diagonal, indicating that main-task accuracy is consistently lower than text-only accuracy.

Figure 8: Additional qualitative examples from the edit-based paired-image construction pipeline. Each row shows a value question, the shared base image, and the final edited pair $(I_{A},I_{B})$ . The examples illustrate how option-level value contrasts are realized through controlled edits on a common scene while preserving overall layout, participants, framing, and style.

Figure 9: A representative iterative refinement example from the quality-control loop. Iteration 1 fails semantic alignment. Iteration 2, obtained after replanning, resolves the semantic issue but still violates shortcut suppression because readable textual details appear in the images. Iteration 3, obtained after re-editing, removes the textual details and satisfies semantic alignment, minimal contrast, and shortcut suppression, and is therefore accepted.

Figure 10: Example of cross-modal value shift for the same model. For a Brazil question about whether a woman’s higher income than her husband is likely to cause problems, the text-only prediction favors the more egalitarian option, while the image-grounded prediction switches to the option associated with visible interpersonal tension. This example illustrates that model judgments can differ substantially across modalities even for the same underlying value question.

Cultural Knowledge in Language Models

Cultural knowledge in language models concerns how systems acquire, access, and apply culture-specific facts, norms, meanings, and practices.

See all 132 papers

ArtECulture: Benchmarking Culture-Conditioned Visual Emotion Understanding in Multimodal Large Language Models
Aug 4, 2026Xiaolin Chen, Xuemeng Song, Wenhao Shi +3Multimodal Large Language Models Multimodal Emotion Recognition
National University of Singapore · Southern University of Science and Technology · Nanyang Technological University
Investigating Multi-Cultural Value Selection Biases in LLMs
Oct 22, 2026Siddhesh Pawar, Arnav Arora, Dustin Wright +2Cultural Bias in Language Models Cross-Cultural Language Model Evaluation
University of Copenhagen · Aalborg University Copenhagen · Bocconi University
Many Dialects, Many Languages, One Cultural Lens: Evaluating Multilingual VLMs for Bengali Culture Understanding Across Historically Linked Languages and Regional Dialects
Mar 22, 2026Nurul Labib Sayeedi, Md. Faiyaz Abdullah Sayeedi, Shubhashis Roy Dipta +6VLM Evaluation Multilingual VLM Evaluation

Aug 4, 2026J/K move · Enter open · S save

ArtECulture: Benchmarking Culture-Conditioned Visual Emotion Understanding in Multimodal Large Language Models

Xiaolin Chen, Xuemeng Song, Wenhao Shi, Xianjing Han+2

National University of Singapore · Southern University of Science and Technology · Nanyang Technological University

Multimodal Large Language Models Multimodal Emotion Recognition Multimodal Model Evaluation

Open paper AskPDF arXiv

Existing visual emotion understanding methods typically ignore cultural variations in emotional perception. We introduce culture-conditioned visual emotion understanding, a task that predicts the culture-specific emotional perception of a given image and explains the underlying rationale. Although related benchmarks exist, they are limited by inconsistent individual annotations, which hinder the derivation of majority-supported culture-level emotion labels, and imbalanced cultural coverage. Thus, we present ArtECulture, a benchmark containing 6,792 artworks with culture-specific emotion labels and explanations across English, Chinese, and Arabic cultures, with balanced Western and non-Western content. Evaluations of 16 open- and closed-source Multimodal Large Language Models (MLLMs) under a zero-shot setting reveal that the task remains challenging, with the best model achieving below 50% accuracy. To address this limitation, we introduce a retrieval-augmented culture-conditioned emotion understanding framework, which leverages a concept-based cultural emotion knowledge base to inject explicit cultural knowledge into MLLMs without additional training. The framework improves both culturally aligned emotion prediction and grounded explanation generation. Our benchmark and code will be publicly released.

Abstract

Cultural Knowledge in Language Models

Explore similar work

ArtECulture: Benchmarking Culture-Conditioned Visual Emotion Understanding in Multimodal Large Language Models

Investigating Multi-Cultural Value Selection Biases in LLMs

Many Dialects, Many Languages, One Cultural Lens: Evaluating Multilingual VLMs for Bengali Culture Understanding Across Historically Linked Languages and Regional Dialects