Cultural values are expressed not only through language but also through visual scenes and everyday social practices. Yet existing evaluations of cultural values in language models are almost entirely text-only, leaving it unclear whether culture-conditioned judgments remain stable when response options are visualized. We introduce ValueGround, a benchmark for evaluating culture-conditioned visual value grounding in multimodal large language models (MLLMs). Built from World Values Survey questions, ValueGround uses minimally contrastive image pairs to represent opposing response options while controlling irrelevant variation. Given a country, a question, and an image pair, a model must choose the image that best matches the country's value tendency without access to the original response-option texts. Experiments across six MLLMs and 13 countries show that models perform substantially worse with visualized response options than with the original textual options, with average accuracy dropping from 72.8% to 62.6%. Our benchmark provides a controlled testbed for studying cross-modal transfer of culture-conditioned value judgments.
Figures & tables
Figure 1: A motivating reversal example. For the same country-question pair, the model predicts one response when the answer choices are textual, but reverses its prediction when the response options are replaced with visual proxies. This example illustrates the central challenge in ValueGround : whether models can maintain consistent culture-conditioned judgments when response options are visualized.
Figure 2: Overview of ValueGround . Starting from a value question q with two response options (oA,oB) , we construct a controlled image pair (IA,IB) that reflects the question and two option meanings while minimizing irrelevant differences and shortcut cues. In the main task, the model receives a country c , the question q , and the image pair (IA,IB) , and must choose which image better matches the value tendency associated with that country. The response-option texts are hidden at test time.
Backbone
Configuration
Q1 Match-A
Q2 Match-B
Q3 PairMatch
Q4 NoCue
Total Score
GPT
Planner only
1.575
1.483
1.180
0.815
0.733
GPT
Planner+Critic
1.605
1.508
1.097
0.749
0.714
GPT
Planner+Editor+Critic
1.521
1.424
1.981
0.934
0.849
Gemini
Planner only
1.737
1.667
0.533
0.821
0.698
Gemini
Planner+Critic
1.583
1.337
1.276
0.811
0.727
Gemini
Planner+Editor+Critic
1.621
1.533
1.931
0.913
0.864
Table 1: Automatic rubric scores from the ensemble validation judge for construction-pipeline ablations. Total Score is the normalized aggregate score, computed as (Q1/2+Q2/2+Q3/2+Q4)/4 .
Figure 3: Country-level accuracy (%) on the main task. Rows correspond to models and columns to countries. Darker colors indicate higher accuracy.
Model
Main
Text-only
Align.
Main ∣ Align ✓
Text ∣ Align ✓
Δ+
Mistral-3.2-24B
59.9
72.8
87.1
61.1
72.3
-11.1
Gemma-3-27B
59.8
68.5
83.7
60.5
68.6
-8.1
Qwen3-VL-32B
62.3
72.7
90.2
63.6
72.7
-9.1
Claude Haiku 4.5
56.7
70.9
77.5
59.3
69.9
-10.6
GPT-5.4 mini
66.6
72.2
90.2
70.2
72.2
-1.9
Gemini 3 Flash
70.2
80.0
90.6
74.9
79.9
-5.0
Table 2: Accuracy (%) on the main task, text-only setting, option–image alignment setting, and alignment-filtered subsets. Main ∣ Align ✓ and Text ∣ Align ✓ respectively report main-task and text-only accuracy on the same subset of country-question instances (c,q) for which the model correctly identifies the option–image correspondence for question q in the alignment task. Δ+ denotes Main ∣ Align ✓ minus Text ∣ Align ✓ . The consistently negative Δ+ indicates that correct option–image alignment does not eliminate the gap between text-only and main task.
Figure 4: Illustration of the edit-based pipeline for a hard-to-visualize question. The pipeline preserves semantic alignment to the target options while producing a more controlled contrastive pair through a shared base scene and minimal edits.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Overview of our agentic paired-image construction framework.
Backbone
Configuration
Avg. cost
Gemini
Planner only
$0.1422
Gemini
Planner+Critic
$0.1608
Gemini
Planner+Editor+Critic
$0.2189
GPT
Planner only
$0.1100
GPT
Planner+Critic
$0.1178
GPT
Planner+Editor+Critic
$0.2244
Appendix
Table 3: Average generation cost per candidate image pair for the construction variants evaluated in Table 1 .
Figure 6: Human annotation instructions used for manual verification.
Metric
Judge vs. Human Mean
Human Agreement
Pearson
Spearman
Pearson
Spearman
Q1
0.345
0.400
0.520
0.546
Q2
0.630
0.526
0.592
0.536
Q3
0.723
0.700
0.562
0.541
Q4
0.591
0.387
0.367
0.289
Total
0.633
0.614
0.581
0.583
Appendix
Table 4: Agreement between the automatic judge and human annotations on the 36-example validation subset. Human Agreement denotes leave-one-annotator-out agreement.
Model
Reversal
Harmful
Beneficial
Qwen3-VL-32B
29.7
21.2
8.6
Mistral-3.2-24B
34.0
24.3
9.7
Gemma-3-27B
35.0
22.5
12.5
Claude Haiku 4.5
38.7
26.8
11.8
GPT-5.4 mini
22.0
14.8
7.2
Gemini 3 Flash
22.5
17.4
5.1
Appendix
Table 5: Cross-modal reversal rates (%) in the final evaluation. Percentages are computed over 5,730 source-specific country-question instances per model. Harmful and beneficial reversals sum to the total reversal rate up to rounding.
Country
Reversal
Harmful
Beneficial
Russia
33.5
22.3
11.2
Japan
32.9
22.2
10.7
Mexico
32.0
22.7
9.3
China
31.9
21.3
10.6
Turkey
31.6
20.9
10.7
Kenya
30.5
20.8
9.7
Appendix
Table 6: Country-level cross-modal reversal rates (%) in the final evaluation. Percentages are computed by aggregating all available model-source estimates for each country. Harmful and beneficial reversals sum to the total reversal rate up to rounding.
Bin
Midpoint-margin range
Reversal rate
Bin 1
[0.000,0.061)
33.3
Bin 2
[0.061,0.126)
30.6
Bin 3
[0.126,0.205)
29.6
Bin 4
[0.205,0.304)
28.0
Bin 5
[0.304,0.497]
30.1
Appendix
Table 7: Cross-modal reversal rate (%) as a function of midpoint margin dc,q=∣uc,q−0.5∣ . Bins are equal-frequency bins over the 2,865 country-question pairs; each bin contains 573 country-question pairs and 6,876 scorable model-country-question-source observations from the final evaluation. Smaller midpoint margin indicates that the country-question pair lies closer to the binary WVS decision boundary.
Model
Pearson r
Spearman ρ
Mean Δ (Main − Text)
Main < Text
Qwen3-VL-32B
0.908
0.801
−12.6
13/13
Mistral-3.2-24B
0.839
0.736
−14.7
13/13
Gemma-3-27B
0.833
0.841
−10.0
13/13
Claude Haiku 4.5
0.548
0.632
−15.0
13/13
GPT-5.4 mini
0.937
0.901
−7.6
13/13
Gemini 3 Flash
0.899
0.834
−12.3
13/13
Appendix
Table 8: Country-level correlation between text-only and main-task accuracy. Each model contributes 13 paired country-level points from the final evaluation. Mean Δ (Main − Text) denotes main-task accuracy minus text-only accuracy in percentage points. The last column reports the number of countries for which main-task accuracy is lower than text-only accuracy.
Figure 7: Country-level text-only versus main-task accuracy across all 78 model-country pairs. Colors indicate models. The dashed line denotes y=x , and the solid line is the least-squares fit. Although the two settings are strongly positively correlated, all points lie below the diagonal, indicating that main-task accuracy is consistently lower than text-only accuracy.
Figure 8: Additional qualitative examples from the edit-based paired-image construction pipeline. Each row shows a value question, the shared base image, and the final edited pair (IA,IB) . The examples illustrate how option-level value contrasts are realized through controlled edits on a common scene while preserving overall layout, participants, framing, and style.
Figure 9: A representative iterative refinement example from the quality-control loop. Iteration 1 fails semantic alignment. Iteration 2, obtained after replanning, resolves the semantic issue but still violates shortcut suppression because readable textual details appear in the images. Iteration 3, obtained after re-editing, removes the textual details and satisfies semantic alignment, minimal contrast, and shortcut suppression, and is therefore accepted.
Figure 10: Example of cross-modal value shift for the same model. For a Brazil question about whether a woman’s higher income than her husband is likely to cause problems, the text-only prediction favors the more egalitarian option, while the image-grounded prediction switches to the option associated with visible interpersonal tension. This example illustrates that model judgments can differ substantially across modalities even for the same underlying value question.
Existing visual emotion understanding methods typically ignore cultural variations in emotional perception. We introduce culture-conditioned visual emotion understanding, a task that predicts the culture-specific emotional perception of a given image and explains the underlying rationale. Although related benchmarks exist, they are limited by inconsistent individual annotations, which hinder the derivation of majority-supported culture-level emotion labels, and imbalanced cultural coverage. Thus, we present ArtECulture, a benchmark containing 6,792 artworks with culture-specific emotion labels and explanations across English, Chinese, and Arabic cultures, with balanced Western and non-Western content. Evaluations of 16 open- and closed-source Multimodal Large Language Models (MLLMs) under a zero-shot setting reveal that the task remains challenging, with the best model achieving below 50% accuracy. To address this limitation, we introduce a retrieval-augmented culture-conditioned emotion understanding framework, which leverages a concept-based cultural emotion knowledge base to inject explicit cultural knowledge into MLLMs without additional training. The framework improves both culturally aligned emotion prediction and grounded explanation generation. Our benchmark and code will be publicly released.
Xiaolin Chen, Xuemeng Song, Wenhao Shi +3
National University of Singapore · Southern University of Science and Technology · Nanyang Technological University
Large language models (LLMs) serve users whose cultural identities are multi-layered, yet cultural evaluations commonly reduce identity to one label. In this work, we study LLMs under simultaneous identity cues, with a country persona in the system prompt and a second identity signaled in the user turn, either by naming a country (explicit cue) or through the query language (implicit cue). We test World Values Survey (WVS) multiple-choice probing and open-ended sociopolitical generation. In multiple-choice probing, language cues shift answers about three times more than explicit cues but away from the associated country's distribution, and explicit cues do not consistently move answers toward the named country. Across 1.6 million open-ended responses, naming a country closes 30-44% of the gap to the model's own responses for that country, and explicit cues produce more differentiated value changes than language cues. Our results suggest that country and language cues should be evaluated as distinct signals rather than interchangeable proxies for culture.
Siddhesh Pawar, Arnav Arora, Dustin Wright +2
University of Copenhagen · Aalborg University Copenhagen · Bocconi University
Bangla culture is richly expressed through region, dialect, history, food, politics, media, and everyday visual life, yet it remains underrepresented in multimodal evaluation. To address this gap, we introduce BanglaVerse, a culturally grounded benchmark for evaluating multilingual vision-language models (VLMs) on Bengali culture across historically linked languages and regional dialects. Built from 1,152 manually curated images across nine domains, the benchmark supports visual question answering and captioning, and is expanded into four languages and five Bangla dialects, yielding ~32.2K artifacts. Our experiments show that evaluating only standard Bangla overestimates true model capability: performance drops under dialectal variation, especially for caption generation, while historically linked languages such as Hindi and Urdu retain some cultural meaning but remain weaker for structured reasoning. Across domains, the main bottleneck is missing cultural knowledge rather than visual grounding alone, with knowledge-intensive categories. These findings position BanglaVerse as a more realistic test bed for measuring culturally grounded multimodal understanding under linguistic variation.
Nurul Labib Sayeedi, Md. Faiyaz Abdullah Sayeedi, Shubhashis Roy Dipta +6