Organizations: University of Manchester Manchester, UK · School of Artificial Intelligence, Wuhan University Wuhan, Hubei, China · Getty Conservation Institute Los Angeles, CA, USA
Recent advances in vision-language models (VLMs) have improved image captioning for cultural heritage. However, inferring structured cultural metadata (e.g., creator, origin, period) from visual input remains underexplored. We introduce a multi-category, cross-cultural benchmark for this task and evaluate VLMs using an LLM-as-Judge framework that measures semantic alignment with reference annotations. To assess cultural reasoning, we report exact-match, partial-match, and attribute-level accuracy across cultural regions. Results show that models capture fragmented signals and exhibit substantial performance variation across cultures and metadata types, leading to inconsistent and weakly grounded predictions. These findings highlight the limitations of current VLMs in structured cultural metadata inference beyond visual perception.
Figures & tables
Figure 1. Cultural heritage objects from four regions are used to evaluate vision-language models on structured metadata inference, with predictions evaluated as correct, partial, or incorrect by an LLM-as-Judge.
Figure 2. Data curation pipeline combining rule-based filtering and human verification to build a balanced cross-cultural dataset.
Table 4. Per-culture performance comparison across models. Acc denotes exact-match accuracy, and Partial denotes partial-match rate.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3. Example images for Object ID 1055_Butter Pat.
Figure 4. Example images for Object ID 1513_Celery vase.
Figure 5. Example images for Object ID 42_Andiron.
Figure 6. Example images for Object ID 0f097d4a-4ca1-40fd-b562-ab41a411aff1.
Figure 7. Example images for Object ID 333_Basin.
Figure 8. Example images for Object ID 2b6e224c-686a-4b43-aa5a-1ef5520ef0ef. Top: full painting. Bottom: object back (left) and framed view (right).