CArtBench: Evaluating Vision-Language Models on Chinese Art Understanding, Interpretation, and Authenticity

Xuefeng Wei; Zhixuan Wang; Xuan Zhou; Zhi Qu; Hongyao Li; Yusuke Sakai; Hidetaka Kamigaito; Taro Watanabe

cs.CLApr 13, 2026

CArtBench: Evaluating Vision-Language Models on Chinese Art Understanding, Interpretation, and Authenticity

Authors: Xuefeng Wei, Zhixuan Wang, Xuan Zhou, Zhi Qu, Hongyao Li, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe

Organizations: Nara Institute of Science and Technology · Liaoning Normal University

Abstract

We introduce CARTBENCH, a museum-grounded benchmark for evaluating vision-language models (VLMs) on Chinese artworks beyond short-form recognition and QA. CARTBENCH comprises four subtasks: CURATORQA for evidence-grounded recognition and reasoning, CATALOGCAPTION for structured four-section expert-style appreciation, REINTERPRET for defensible reinterpretation with expert ratings, and CONNOISSEURPAIRS for diagnostic authenticity discrimination under visually similar confounds. CARTBENCH is built by aligning image-bearing Palace Museum objects from Wikidata with authoritative catalog pages, spanning five art categories across multiple dynasties. Across nine representative VLMs, we find that high overall CURATORQA accuracy can mask sharp drops on hard evidence linking and style-to-period inference; long-form appreciation remains far from expert references; and authenticity-oriented diagnostic discrimination stays near chance, underscoring the difficulty of connoisseur-level reasoning for current models.

Figures & tables

Figure 1: Overview of CArtBench construction and task instantiation. Top: Phase 1 retrieves image-bearing Palace Museum objects from Wikidata, Phase 2 aligns them to official catalog pages to collect curatorial descriptions, and Phase 3 performs expert filtering and category assignment to yield museum-grounded artwork–appreciation pairs. Bottom: the curated pairs are instantiated into four tasks: CuratorQA (evidence grounding), CatalogCaption (structured appreciation), ReInterpret (expert-rated reinterpretation), and ConnoisseurPairs (confound-aware authenticity diagnostics).

Figure 2: Type and era distributions of CuratorQA entries: (left) five art categories; (right) top-8 merged eras.

Qwen family
Model	Acc.	QA type Acc.						Category Acc.
Model	Acc.	QA1	QA2	QA3	QA4	QA5	QA6	Callig.	Station.	Paint.	Ceram.	Sculpt.
Qwen3-VL-235B	0.84	0.88	0.94	0.88	0.89	0.56	0.81	0.85	0.78	0.87	0.79	0.84
Qwen3-VL-30B	0.80	0.85	0.91	0.85	0.88	0.42	0.78	0.83	0.74	0.83	0.76	0.80
Qwen2.5-VL-72B	0.81	0.85	0.93	0.85	0.87	0.53	0.76	0.83	0.74	0.85	0.75	0.80
Qwen2.5-VL-32B	0.80	0.84	0.91	0.83	0.87	0.53	0.74	0.82	0.76	0.83	0.73	0.78

Table 1: CuratorQA accuracy breakdown (overall, by question type, and by art category). We report exact-match accuracy on 14,421 questions. Category abbreviations: Callig.=calligraphy, Station.=stationery, Paint.=painting, Ceram.=ceramics, Sculpt.=sculpture.

Subset	mean $\Delta$ (CI 95 )	$\mathbf{p_{\text{perm}}}$
QA1	0.0426 [0.0279, 0.0574]	$<0.05$
QA2	0.0354 [0.0215, 0.0485]	$<0.05$
QA3	0.0435 [0.0301, 0.0568]	$<0.05$
QA4	0.0345 [0.0224, 0.0462]	$<0.05$
QA5	0.1022 [0.0735, 0.1302]	$<0.05$
QA6	0.0295 [0.0169, 0.0423]	$<0.05$

Table 2: Subset-wise robustness of the performance gap between a Chinese–English bilingual proxy (Qwen3-235B) and a general-purpose API proxy (GPT-5-mini) on CuratorQA Ẇe report

\Delta=\mathrm{Acc}(\text{Qwen3-235B})-\mathrm{Acc}(\text{GPT-5-mini})

with 95% confidence intervals (CI 95 ) obtained by bootstrap resampling. Statistical significance is assessed using a sign-flip permutation test, with clustering at the artwork level to account for multiple questions per artwork.

Weights scenario	Spearman	Kendall	Top-1
Uniform weights	0.991	0.964	1.000
Dirichlet ( $\alpha{=}5$ )	0.991	0.964	1.000
Dirichlet ( $\alpha{=}1$ )	0.964	0.891	1.000
Local Gaussian ( $\sigma{=}0.05$ )	0.991	0.964	1.000

Table 3: KPI weight sensitivity for CatalogCaption . Spearman/Kendall are rank correlations with the default-weight ordering. For stochastic schemes (Dirichlet and Local Gaussian), we report the median over samples.

Human baseline
Model	KPI $\uparrow$	BERTScore F1 $\uparrow$	CIDEr-like $\uparrow$	ROUGE-L $\uparrow$	BLEU-4 $\uparrow$	EmbSim $\uparrow$	Header rate $\uparrow$
Human (mean of 3)	0.698	0.746	0.673	0.699	0.546	0.982	0.988
Qwen family
Qwen2.5-VL-72B	0.261	0.329	0.205	0.237	0.146	0.876	1.000
Qwen3-VL-235B	0.258	0.319	0.223	0.223	0.138	0.893	1.000
Qwen3-VL-30B	0.256	0.322	0.224	0.222	0.112	0.891	1.000

Table 4: CatalogCaption evaluation on 86 artworks. Overlap-based metrics (ROUGE-L / BLEU-4 / CIDEr-like) use character-level tokenization. BERTScore F1 uses bert-base-chinese with rescale_with_baseline=True . EmbSim is cosine similarity under text2vec-base-chinese . Header rate measures whether all four required section headers appear in the output. References are expert-rewritten into the four-section schema.

Model	Gate pass $\uparrow$	Primary avg (D1–D3) $\uparrow$	All avg (D1–D5) $\uparrow$
gemini-2.5-flash	0.707 [0.600, 0.813]	3.673 [3.476, 3.887]	3.723 [3.528, 3.933]
GLM-4.6V	0.853 [0.760, 0.933]	3.649 [3.522, 3.784]	3.679 [3.575, 3.796]
gpt-5-mini	0.867 [0.787, 0.947]	3.567 [3.480, 3.656]	3.617 [3.529, 3.708]
qwen3-vl-235B	0.880 [0.787, 0.960]	3.747 [3.622, 3.878]	3.728 [3.621, 3.836]

Table 5: ReInterpret human creativity evaluation on 25 artworks (3 raters; 75 ratings per model). Stage 1 is a plausibility gate; Gate pass is the fraction of outputs that enter Stage 2. Stage 2 uses 1–5 Likert ratings and is computed only on passed outputs. We report macro-averages over artworks with 95% bootstrap confidence intervals. Primary avg averages the three primary dimensions (D1–D3), and All avg averages all five dimensions (D1–D5).

Model	Acc.	Model	Acc.
GLM family
GLM-4.5V	0.600	GLM-4.6V	0.600
Qwen family
Qwen2.5-VL-72B	0.600	Qwen2.5-VL-32B	0.500
Qwen3-VL-235B	0.500	Qwen2.5-VL-7B	0.400
Qwen3-VL-30B	0.400

Table 6: Diagnostic pairwise authenticity discrimination on 10 authentic–imitation pairs in ConnoisseurPairs . A prediction is correct if the model selects the authentic work in each A/B pair. Given the small scale, results should be interpreted as indicative of a capability gap rather than a definitive model ranking.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Bilingual model	General model	$\mathbf{Acc.}_{\text{bi}}$	$\mathbf{Acc.}_{\text{gen}}$	$\boldsymbol{\Delta}$	Significance
Qwen3-235B	GPT-5-mini	0.8400	0.7949	+0.0451	McNemar (paired, $N{=}14421$ ): $p{=}4.22{\times}10^{-47}$ . Clustered by artwork ID ( $N_{\text{artwork}}{=}1589$ ): $\overline{\Delta}{=}0.0447$ , CI ${}_{95}{=}[0.0384,\,0.0512]$ , $p{=}0.0002$ .

Appendix

Table 7: Chinese–English bilingual vs. general-purpose API proxy comparison on CuratorQA . Qwen3-235B is a Chinese–English bilingual VLM with strong Chinese multimodal performance; GPT-5-mini is a general-purpose multilingual API.

Scenario	#samples	Spearman (p05/p50/p95)	Kendall (p05/p50/p95)	Top-1 same	Top-3 Jaccard (p05/p50/p95)
Default (sanity)	1	1.000/1.000/1.000	1.000/1.000/1.000	1.000	1.000/1.000/1.000
Uniform weights	1	0.991/0.991/0.991	0.964/0.964/0.964	1.000	1.000/1.000/1.000
Dirichlet $\alpha{=}5$	8000	0.945/0.991/0.991	0.855/0.964/0.964	1.000	1.000/1.000/1.000
Dirichlet $\alpha{=}1$	8000	0.882/0.964/1.000	0.709/0.891/1.000	1.000	0.500/1.000/1.000
Dirichlet $\alpha{=}0.5$	8000	0.873/0.964/1.000	0.673/0.891/1.000	1.000	0.500/1.000/1.000
$w_{\text{bert}}\!\in\![0.35,0.60]$	5000	0.991/1.000/1.000	0.964/1.000/1.000	1.000	1.000/1.000/1.000

Appendix

Table 8: KPI weight sensitivity analysis for CatalogCaption . Rank correlations are computed against the default-weight ordering; we report 5th/50th/95th percentiles for stochastic scenarios. Top-1 same and Top-3 Jaccard are computed over models (excluding human baselines).

Model	Mean rank	Std. rank	$P(\text{top-3})$
human_mean	1.000	0.000	1.000
preds_qwen25_72B	2.397	0.734	0.860
preds_qwen3_235B	2.863	0.495	0.936
preds_qwen3_30B	4.035	0.806	0.203
preds_qwen25_7B	5.038	0.920	0.000
preds_glm45	6.189	0.785	0.000

Appendix

Table 9: Model rank stability under Dirichlet weight perturbations with

\alpha{=}1

(8,000 samples). We report the mean and standard deviation of each model’s rank, and the probability of appearing in the top-3.

Item	D1	D2	D3	E1
Agreement	.639	.622	.637	.841 †
Qwen2.5-72B	3.640	3.702	3.663	0.081
Qwen3-235B	3.647	3.609	3.616	0.050
Gemini-flash	3.229	3.275	3.337	0.093

Appendix

Table 10: Expert rubric evaluation for CatalogCaption . Agreement: ICC(2,3) for D1–D3; † Gwet’s AC1 for E1. E1 = fraction of outputs flagged for major factual errors.

Model	Gate $\kappa$	ICC Primary	ICC All
GLM-4.6V	0.487	0.498	0.649
gemini-2.5-flash	0.415	0.476	0.709
gpt-5-mini	0.711	0.785	0.869
qwen3-vl-235B	0.577	0.877	0.888

Appendix

Table 11: Inter-annotator agreement for ReInterpret . Gate

\kappa

: Fleiss’

\kappa

for Stage 1 pass/fail. ICC Primary/All: ICC(2,3) for the 3-rater mean on D1–D3 and D1–D5 respectively.

Model	Gate pass	D1 Novelty	D2 Coherence	D3 Evidence	D4 Elaboration	D5 Insight
gemini-2.5-flash	0.707 [0.600, 0.813]	3.287 [2.940, 3.653]	3.960 [3.740, 4.193]	3.773 [3.587, 3.960]	3.980 [3.760, 4.213]	3.613 [3.353, 3.907]
GLM-4.6V	0.853 [0.760, 0.933]	3.407 [3.213, 3.620]	3.793 [3.653, 3.947]	3.747 [3.560, 3.947]	3.800 [3.627, 3.980]	3.647 [3.467, 3.853]
gpt-5-mini	0.867 [0.787, 0.947]	3.280 [3.120, 3.447]	3.767 [3.647, 3.880]	3.653 [3.507, 3.787]	3.873 [3.733, 4.027]	3.513 [3.333, 3.700]
qwen3-vl-235B	0.880 [0.787, 0.960]	3.533 [3.340, 3.740]	3.867 [3.687, 4.053]	3.840 [3.707, 3.973]	3.847 [3.693, 4.013]	3.553 [3.413, 3.700]

Appendix

Table 12: Dimension-wise macro-averages for ReInterpret with 95% bootstrap confidence intervals over artworks. Gate pass is the Stage 1 plausibility pass rate. D1–D5 are Stage 2 Likert ratings computed only for gate-passed outputs, macro-averaged over artworks.

Model	Failures	Other	Era/Dynasty error	Misread / missed content	Wrong type/region attribution	Empty/NA
gemini-2.5-flash	22 (29.3%)	11 (14.7%)	6 (8.0%)	2 (2.7%)	2 (2.7%)	1 (1.3%)
GLM-4.6V	11 (14.7%)	8 (10.7%)	2 (2.7%)	1 (1.3%)	0 (0.0%)	0 (0.0%)
gpt-5-mini	10 (13.3%)	5 (6.7%)	1 (1.3%)	3 (4.0%)	1 (1.3%)	0 (0.0%)
qwen3-vl-235B	9 (12.0%)	7 (9.3%)	1 (1.3%)	0 (0.0%)	1 (1.3%)	0 (0.0%)

Appendix

Table 13: Stage 1 gate-failure type distributions for ReInterpret . Counts are over rater

\times

artwork instances (75 per model), with percentages relative to 75. Failure types are recorded only when a rater marks the output as failing the plausibility gate.

Figure 3: Survey Questionnaire for ReInterpret -Part1.

Figure 4: Survey Questionnaire for ReInterpret -Part2.

Figure 5: Survey Questionnaire for ConnoisseurPairs -Part1.

Figure 6: Survey Questionnaire for ConnoisseurPairs -Part2.

Figure 7: Chinese dynasties and historical periods considered in this work, listing the Chinese names with English equivalents and their approximate date ranges (BCE/CE).

Group	Tag	Description	Numbers	Percentage
Overview
Entries	–	Unique artwork entries	1,589	–
Questions	–	Total number of questions	14,421	–
Avg. Q / entry	–	Average questions per entry	9.07	–
Difficulty
Principle	P1	Visual-only (answerable from the image)	10,920	75.72

Appendix

Table 14: Summary statistics of CuratorQA , including overview counts, difficulty principles, and question-type distribution.

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 957 papers

Does AI See like Art Historians? Interpreting How Vision Language Models Recognize Artistic Style
Mar 11, 2026Marvin Limpijankit, Milad Alshomary, Yassin Oulad Daoud +9VLM Interpretability Concept-Based Explanations
Columbia University, Department of Computer Science · Columbia University, Department of Art History & Archaeology · University of Texas at Austin +1
HanMoVLM: Large Vision-Language Models for Professional Artistic Painting Evaluation
Mar 11, 2026Hongji Yang, Yucheng Zhou, Wencheng Han +3Vision-Language Model-as-a-Judge VLM Evaluation
SKL-IOTSC, CIS, University of Macau · CSE, Shandong University · Online-Video BU, Tencent
SalArt-VQA: Diagnosing Whether VLMs Understand Salient Artifacts in Generated Images
Jun 10, 2026Ruotian Zhang, Xiaoxiao Sun, Junzhe Huang +2VLM Evaluation VLM Hallucination
Zhejiang University · The University of Queensland

Mar 11, 2026J/K move · Enter open · S save

Does AI See like Art Historians? Interpreting How Vision Language Models Recognize Artistic Style

Marvin Limpijankit, Milad Alshomary, Yassin Oulad Daoud, Amith Ananthram+8

Columbia University, Department of Computer Science · Columbia University, Department of Art History & Archaeology · University of Texas at Austin+1

VLM Interpretability Concept-Based Explanations

Open paper AskPDF arXiv

VLMs have become increasingly proficient at a range of computer vision tasks, such as visual question answering and object detection. This includes increasingly strong capabilities in the domain of art, from analyzing artwork to generation of art. In an interdisciplinary collaboration between computer scientists and art historians, we characterize the mechanisms underlying VLMs' ability to predict artistic style and assess the extent to which they align with the criteria art historians use to reason about artistic style. We employ a latent-space decomposition approach to identify concepts that drive art style prediction and conduct quantitative evaluations, causal analysis and assessment by art historians. Our findings indicate that 73% of the extracted concepts are judged by art historians to exhibit a coherent and semantically meaningful visual feature and 90% of concepts used to predict style of a given artwork were judged relevant. In cases where an irrelevant concept was used to successfully predict style, art historians identified possible reasons for its success; for example, the model might "understand" a concept in more formal terms, such as dark/light contrasts.

Abstract

VLM Evaluation

Explore similar work

Does AI See like Art Historians? Interpreting How Vision Language Models Recognize Artistic Style

HanMoVLM: Large Vision-Language Models for Professional Artistic Painting Evaluation

SalArt-VQA: Diagnosing Whether VLMs Understand Salient Artifacts in Generated Images