Vision-language models are increasingly used to identify and describe places, yet their recognition and knowledge of specific cultures remain poorly understood, particularly in underrepresented regions such as the Philippines. This study introduces the Tanawpamana Dataset, a repository of Philippine heritage site images with metadata based on the Talapamana registry of the National Commission for Culture and the Arts of the Philippines. We also introduce the Tanawpamana Benchmark, comprising 4,046 tasks designed to evaluate models' recognition and knowledge of heritage sites. Evaluating seven vision-language models, we find that they struggle more with fine-grained visual discrimination among similar heritage sites than with coarse recognition, and that their geographic knowledge is weaker when inferred from image inputs than from text.
Figures & tables
Figure 1: Overview of the Tanawpamana Dataset and Tanawpamana Benchmark. Top: sample dataset entry showing a heritage image and metadata fields. Bottom: Recognition and Knowledge task suites with simplified example questions.
Figure 2: Heritage categories used in Tanawpamana, each shown with a sample photo.
Category
Metro Manila
Luzon
Visayas
Mindanao
Religious Building
21
42
23
11
Civic Building
16
15
16
7
Institutional Building
16
16
15
4
Historic Building
15
16
17
2
Old House
17
15
15
3
Monument
15
15
16
7
Table 1: Number of sites per category and geographic area.
Figure 3: Distribution of heritage sites by geographic area and designation status.
Suite
Task Count
Recognition Suite
Image-to-Name (Easy)
499
Image-to-Name (Hard)
481
Name-to-Image (Easy)
499
Name-to-Image (Hard)
481
Recognition Total
1,960
Table 2: Distribution of tasks in the Tanawpamana Benchmark.
Figure 4: Model accuracy across seven task clusters spanning the Recognition (Easy/Hard) and five Knowledge tasks.
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
We introduce ChinaHeritaQA, a multimodal benchmark dataset for evaluating the cultural reasoning abilities of vision-language models (VLMs) on UNESCO World Heritage sites in China. The dataset comprises 2,279 in-the-wild images paired with 14,133 bilingual (Chinese/English) multiple-choice QA pairs spanning seven cognitive dimensions, from basic identity recognition to historical periodization and architectural analysis. Guided by a UNESCO-aligned heritage ontology and verified through rigorous human annotation, the dataset ensures linguistic quality and factual consistency. Evaluations of state-of-the-art VLMs reveal that while top models exceed human performance on average, substantial task-level variation emerges: models excel at visual recognition but struggle with culturally grounded reasoning. Performance also varies by dynasty and region. ChinaHeritaQA reveals that strong visual retrieval does not extend to cultural and historical understanding. We release the dataset to support future research on culturally aware multimodal learning.
Yi Zhang, Bolei Ma, Yong Cao +3
1LMU Munich · 2FAU Erlangen-Nuremberg · 3Munich Center for Machine Learning +4
Vision-language models (VLMs) have achieved strong performance on tasks such as image captioning, visual question answering, and image-to-text generation. However, they are predominantly trained on English-centric data, which limits their ability to handle culturally grounded visual understanding and leads to failures in interpreting region-specific meanings, symbolic content, and context-dependent visual cues. Existing benchmarks for cultural competence are often template-driven and focused on surface-level recognition, making them insufficient for evaluating deeper linguistic and pragmatic understanding in culturally situated settings. We introduce PoVisLE, a monocultural vision-language benchmark for Polish designed to evaluate culturally grounded multimodal understanding under a grounded evaluation paradigm, where language is interpreted in interaction with visual context. The dataset contains 1,117 images and 2,366 manually annotated VQA pairs. Overall, our dataset provides a controlled and challenging resource for assessing culturally grounded vision-language understanding beyond surface-level recognition.
Anna Kołos, Grzegorz Statkiewicz, Karolina Seweryn +3
Recent advances in vision-language models (VLMs) have improved image captioning for cultural heritage. However, inferring structured cultural metadata (e.g., creator, origin, period) from visual input remains underexplored. We introduce a multi-category, cross-cultural benchmark for this task and evaluate VLMs using an LLM-as-Judge framework that measures semantic alignment with reference annotations. To assess cultural reasoning, we report exact-match, partial-match, and attribute-level accuracy across cultural regions. Results show that models capture fragmented signals and exhibit substantial performance variation across cultures and metadata types, leading to inconsistent and weakly grounded predictions. These findings highlight the limitations of current VLMs in structured cultural metadata inference beyond visual perception.
Yuechen Jiang, Enze Zhang, Md Mohsinul Kabir +4
University of Manchester Manchester, UK · School of Artificial Intelligence, Wuhan University Wuhan, Hubei, China · Getty Conservation Institute Los Angeles, CA, USA