AI systems are used worldwide, but they struggle to serve the needs of culturally diverse populations. Prior work on cultural understanding evaluates AI systems on text-only settings or on visual artifact recognition (e.g. foods, clothing). The ability to reason about visually observable behaviors through local social norms, which we call visual norm understanding, remains unexamined. We introduce NormViz-Bench, a high quality, human-validated benchmark of 3,268 contrastive image pairs (6,536 images) spanning 16 countries. Each pair varies only in the culturally relevant behavior (e.g., objects, attributes, spatial relations, and actions) that alters how each image is interpreted. Each image is labeled as conforming to, violating, or irrelevant to local social norms, and pair-level evaluation requires both images to be correctly classified, thereby preventing reliance on superficial visual shortcuts. Even the strongest VLMs, Gemini 3.0 Flash and Qwen2.5 VL 7B, succeed on only 26.6% and 21.6% of pairs, struggling most with identifying violating and culturally benign visual behaviors. Towards bridging this, we introduce NormViz-Train, a training dataset of 64k images paired with explanations. Though absolute performance remains low (<30%), finetuning on NormViz-Train improves pair accuracy relatively by up to 125% and 36% Qwen3-VL 4B and 8B respectively, showing a path forward to teach models to connect visual perception to cultural significance. Together, NormViz-Bench and NormViz-Train establish visual norm understanding as a challenging and consequential frontier for multimodal AI.
Figures & tables
Figure 1: Overview of NormViz -Bench for visual norm understanding, consisting of 3,272 contrastive image pairs across 16 countries that differ only in the culturally relevant behavior. (a) A contrastive pair example from Japan; group accuracy requires a model to correctly classify both images in a pair. (b) Images are sourced via Text-to-Image generation, inpainting, and Retrieval. (c) Benchmark statistics and pair type distribution.
Figure 2: NormViz -Bench construction pipeline across five stages: (1) collecting cultural norms from 3 text benchmarks and transforming them into norm-contrast pairs; (2) extracting metadata and social norm type, and then filtering for visual detectability; (3) constructing prompt descriptions of the contrasting pair, along with inpainting instructions; (4) curating images via text-to-image generation, inpainting, and retrieval, with initial quality filtering; and (5) human validation of the entire set from annotators from each of the 16 countries.
Closed source
Open source
Condition
Gemini 3 flash
GPT 5.2
GPT 4o
Qwen 2.5 7B
LLaVA OV 7B
Qwen 3 8B
Qwen 2.5 3B
GLM-4V 9B
Phi-3V 4.2B
Aya Vis. 8B
Qwen 3 4B
DS-vl2 1B
Avg
Group Accuracy (%)
NormViz -Bench
25.3
22.7
16.3
23.0
19.8
19.6
18.3
16.7
16.1
15.4
14.9
13.8
18.5
oracle
+ Image Descriptions
32.7
37.8
23.5
30.8
27.3
28.1
27.5
23.1
29.9
26.7
23.0
19.7
27.5
+ Image Desc. (w/o img)
40.8
45.3
37.2
32.7
34.3
40.2
29.7
31.0
35.8
28.7
30.7
2.7
32.4
Macro F1
Table 1: Performance across the top 12 evaluated models, ordered by group accuracy within closed- and open- source groups. Best and second-best model are highlighted. Group accuracy for all models is critically low . We run two oracle ablations: (1) + Image Descriptions - image with text description; (2) + Image Desc. (w/o img) - replace input image entirely with text description. We find that performance generally improves in both settings, indicating cultural visual perception and vision-language alignment are bottlenecks. However, the best model under these settings achieves 45.3% group accuracy (GPT 5.2), suggesting that cultural normative reasoning is the biggest barrier. Full results for all 21 models in App. E .
Figure 3: Single-image F1 performance of the top three models: (a) across data sources and (b) across social acceptability labels. Naturally retrieved images are the most challenging, followed by inpainted images, with independently generated images being the easiest. Detecting offensive content is the most difficult task, while identifying the benign counterparts of offensive behavior is easier than detecting offensive content.
Overall
By Source (F1)
By Label (Macro F1)
Model
Group Acc
Macro F1
T2I
Inp.
Ret.
Follow
Avoid
Benign
B_ Avoid
B_ Follow
Qwen3-VL 4B
Baseline
14.9
37.7
37.3
40.5
28.2
59.7
12.1
41.3
43.6
40.4
Finetuned
27.7
51.0
52.5
48.7
36.4
60.5
29.8
62.6
66.4
61.3
Qwen3-VL 8B
Baseline
19.6
43.6
43.5
45.9
32.6
61.1
19.3
50.5
56.6
48.1
Table 2: Finetuning results on Qwen3-VL 4B and 8B ( follow-heavy 64K ), by image source and label type. Inp.=Inpainting; Ret.=Retrieval; B_ Avoid =Benign_ Avoid ; B_ Follow =Benign_ Follow
Figure 4: Improvements from finetuning Qwen3 VL 4B on NormViz -Train (a) Group accuracy gains, broken down by cultural norm domains and visual contrast types (left and right, respectively) in NormViz -Bench ; asterisks (*) indicate significance level. (b) Country-wise group accuracy gains organized by UN geoscheme (shown on right); gains are largest in Southern Asia, South-Eastern Asia, and Sub-Saharan Africa.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: More examples of image pairs from NormViz -Bench
Figure 6: Annotator instructions
Figure 7: Annotation Framework with Example
Metric
Count
Total Image Pairs
3272
Total Images
6544
Unique Norms
1140
Countries
16
By Image Source
Text-to-Image
65.2%
Appendix
Table 3: Summary statistics of NormViz -Bench .
Figure 8: Country-wise distribution of the contrastive pairs in NormViz -Bench , broken down by image source type.
Figure 9: Semantic coverage of NormViz -Bench across norm domains and visual contrast types. Norm domains span a broad range of cultural behaviors, from everyday etiquette (e.g., Dining Etiquette, Hand Usage Etiquette, Footwear Etiquette) to ceremonial and symbolic practices (e.g., Festivals & Celebrations, Rituals, Offerings & Ancestral Customs, Symbolic Objects & Practices), with Social & Public Conduct being the most represented domain. Visual contrast types are dominated by Object State, Action/Activity, and Presence/Absence, reflecting that cultural norms are most commonly expressed through what objects are used, how actions are performed, or whether culturally significant elements appear at all.
Figure 10: We transform each cultural description into a norm-contrast pair using GPT-4o. Here a ’norm’ is a singular behavior whereas ’contrast’ is just a contrastive reframing of the same norm, preserving its meaning.
Figure 11: Prompt used to extract structured metadata and classify visual detectability and social norm type for each norm-contrast pair using GPT-4o.
Figure 12: GPT-4o Prompt used to construct prompts for norms to avoid . A similar variant is used for norms to follow .
Figure 13: Prompt used for quality filtering with GPT-4o
Overall
By Source (macro F1)
By Label (macro F1)
Benign
Model
Group Acc.
Macro F1
Generated
Inpainting
Retrieval
Follow
Avoid
All
Avoid
Follow
PaliGemma 2 3B
0.0
18.8
18.6
18.5
20.8
56.3
0.0
0.0
0.0
0.0
LLaVA 1.5 7B
1.5
22.3
23.3
20.8
17.9
34.7
32.4
0.0
0.0
0.0
PaliGemma 3B
2.0
19.6
19.3
18.5
21.9
0.0
31.9
27.8
25.5
28.4
BLIP-2 2.7B
3.8
26.3
25.7
25.5
33.1
7.1
13.9
57.9
63.5
56.3
Appendix
Table 4: Performance of 21 VLMs on NormViz -Bench . Bold indicates best within category (open-source or closed-source), underline indicates second-best within category. Models within 0.1 points are highlighted together. Green background = open-source, orange = closed-source.
Condition
Group Acc.
Macro F1
Generated
Inpainting
Retrieval
Follow
Avoid
Benign_ All
Benign_ Avoid
Benign_ Follow
Closed source
Gemini 3 flash
25.3
55.8
58.4
52.6
37.0
66.8
53.8
51.9
63.2
47.8
oracle
+ Image Descriptions
32.7
63.6
66.2
60.2
51.1
72.1
68.6
57.1
69.8
52.4
+ Image Desc. (w/o img)
40.8
68.0
70.4
66.0
51.3
74.0
70.4
61.0
73.3
56.5
GPT 5.2
22.7
45.6
47.0
42.6
40.5
58.8
17.9
60.0
63.9
58.6
oracle
+ Image Descriptions
37.8
60.8
62.9
57.0
56.2
70.0
46.8
65.6
69.2
64.3
Appendix
Table 5: Full results for models evaluated in the main paper (Table 1), including per-source and per-label Macro F1 breakdowns. Models are ordered by baseline Group Accuracy (descending) within each source group. Bold = best within source group; underline = second-best. Shaded rows show oracle ablations: (1) + Image Descriptions – image with text description; (2) + Image Desc. (w/o img) – text description only, no image.
Condition
Group Acc.
Macro F1
Generated
Inpainting
Retrieval
Follow
Avoid
Benign_ All
Benign_ Avoid
Benign_ Follow
Open source (remaining models, ordered by baseline Group Accuracy)
InternVL 2B
13.4
36.2
36.3
36.8
31.9
58.9
10.2
39.4
44.0
37.7
oracle
+ Image Descriptions
19.8
39.4
40.0
38.5
37.6
62.2
8.7
47.3
48.1
47.0
+ Image Desc. (w/o img)
33.7
50.0
51.2
46.9
49.0
66.5
16.9
66.5
66.6
66.5
InternVL 8B
12.9
37.7
37.9
39.3
27.1
59.4
16.4
37.4
43.9
35.0
oracle
+ Image Descriptions
22.3
46.3
47.8
44.5
38.1
64.3
24.7
49.8
57.1
46.7
Appendix
Table 6: Full results for the remaining 9 open-source models not included in Table 1, ordered by baseline Group Accuracy (descending). Bold = best within this group. Shaded rows show oracle ablations: (1) + Image Descriptions – image with text description; (2) + Image Desc. (w/o img) – text description only, no image. Note the large gap between baseline and ablation for Molmo 2 8B (5.0 → 30.1) group accuracy with image descriptions, indicating especially poor vision-language alignment.
Figure 14: Group accuracy of 21 vision-language models (VLMs) across 16 countries. Each dot represents an individual model’s group accuracy (%) for a given country, and the diamond marker shows the mean across all models. Countries are ordered from lowest (bottom) to highest (top) mean group accuracy. The fraction on the right (e.g., 17/21) indicates how many models score below their own overall group accuracy. For example, a model with an overall mean group accuracy of 15% that scores 8% on India would count toward this fraction. Higher fraction indicates that the country is consistently harder for most models relative to their average performance across all countries. Countries are color-coded by geographic region.
Figure 15: Prompt used for excluding NormViz -Bench norms from NormViz -Train
Figure 16: Finetuning results for Qwen3 VL 4B and 8B models across label distribution configurations (balanced, follow-heavy, avoid-heavy) and data scales (32k and 64k)
Figure 17: Finetuning results for Qwen 2.5 VL 7B model across label distribution configurations (balanced, follow-heavy, avoid-heavy) and 32k data size
Figure 18: Country-wise performance of finetuned Qwen3VL 4B model, using group accuracy and single image classification F1 score. Results for follow-heavy 64k finetuned Qwen3VL 4B on NormViz -Train
Figure 19: Country-wise performance of finetuned Qwen3VL 8B model, using group accuracy and single image classification F1 score. Results for follow-heavy 64k finetuned Qwen3VL 8B on NormViz -Train
Figure 20: Improvement of finetuned Qwen3 VL 4B (follow-heavy 64k) across cultural norm domains and visual contrast types. Improvements measured using group accuracy.
Figure 21: Improvement of finetuned Qwen3 VL 4B (follow-heavy 64k) across cultural norm domains and visual contrast types. Improvements measured using single image classification F1.
Figure 22: Improvement of finetuned Qwen3 VL 8B (follow-heavy 64k) across cultural norm domains and visual contrast types. Improvements measured using group accuracy.
Figure 23: Improvement of finetuned Qwen3 VL 8B (follow-heavy 64k) across cultural norm domains and visual contrast types. Improvements measured using single image classification F1.
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
In real-world applications, models are expected to perform reliably across diverse settings. Yet, many existing multimodal benchmarks expand task types without capturing the visual diversity needed to handle open-ended visual inputs. We present WorldBench, a challenging and visually diverse reasoning benchmark to evaluate Multimodal Large Language Models (MLLMs). We build a taxonomy of thousands of visual concepts across multiple domains (e.g., living things). Guided by this taxonomy, we curate a broad collection of images from search engines and existing datasets to comprehensively represent the visual world. Through structured trial-and-error, we manually design challenging questions that frontier MLLMs fail to answer. On quantitative and human evaluations, WorldBench achieves higher visual diversity than any existing diverse benchmark. Evaluating 15 MLLMs on WorldBench reveals weaknesses in visual understanding: even the strongest model reaches only 64.0% accuracy, while some models perform marginally above chance-level. We hope our work highlights the importance of visual diversity in building multimodal benchmarks.
Yida Yin, Harish Krishnakumar, Chung Peng Lee +9
Princeton University · NYU · University of Waterloo +1
Recent advances in vision-language models (VLMs) have improved image captioning for cultural heritage. However, inferring structured cultural metadata (e.g., creator, origin, period) from visual input remains underexplored. We introduce a multi-category, cross-cultural benchmark for this task and evaluate VLMs using an LLM-as-Judge framework that measures semantic alignment with reference annotations. To assess cultural reasoning, we report exact-match, partial-match, and attribute-level accuracy across cultural regions. Results show that models capture fragmented signals and exhibit substantial performance variation across cultures and metadata types, leading to inconsistent and weakly grounded predictions. These findings highlight the limitations of current VLMs in structured cultural metadata inference beyond visual perception.
Yuechen Jiang, Enze Zhang, Md Mohsinul Kabir +4
University of Manchester Manchester, UK · School of Artificial Intelligence, Wuhan University Wuhan, Hubei, China · Getty Conservation Institute Los Angeles, CA, USA
Cultural values are expressed not only through language but also through visual scenes and everyday social practices. Yet existing evaluations of cultural values in language models are almost entirely text-only, leaving it unclear whether culture-conditioned judgments remain stable when response options are visualized. We introduce ValueGround, a benchmark for evaluating culture-conditioned visual value grounding in multimodal large language models (MLLMs). Built from World Values Survey questions, ValueGround uses minimally contrastive image pairs to represent opposing response options while controlling irrelevant variation. Given a country, a question, and an image pair, a model must choose the image that best matches the country's value tendency without access to the original response-option texts. Experiments across six MLLMs and 13 countries show that models perform substantially worse with visualized response options than with the original textual options, with average accuracy dropping from 72.8% to 62.6%. Our benchmark provides a controlled testbed for studying cross-modal transfer of culture-conditioned value judgments.