Does AI See like Art Historians? Interpreting How Vision Language Models Recognize Artistic Style
Authors: Marvin Limpijankit, Milad Alshomary, Yassin Oulad Daoud, Amith Ananthram, Tim Trombley, Emily L. Spratt, Anna Filonenko, Hannah Pivo, Elias Stengel-Eskin, Mohit Bansal, Noam M. Elcott, Kathleen McKeown
Organizations: Columbia University, Department of Computer Science · Columbia University, Department of Art History & Archaeology · University of Texas at Austin · UNC Chapel Hill
VLMs have become increasingly proficient at a range of computer vision tasks, such as visual question answering and object detection. This includes increasingly strong capabilities in the domain of art, from analyzing artwork to generation of art. In an interdisciplinary collaboration between computer scientists and art historians, we characterize the mechanisms underlying VLMs' ability to predict artistic style and assess the extent to which they align with the criteria art historians use to reason about artistic style. We employ a latent-space decomposition approach to identify concepts that drive art style prediction and conduct quantitative evaluations, causal analysis and assessment by art historians. Our findings indicate that 73% of the extracted concepts are judged by art historians to exhibit a coherent and semantically meaningful visual feature and 90% of concepts used to predict style of a given artwork were judged relevant. In cases where an irrelevant concept was used to successfully predict style, art historians identified possible reasons for its success; for example, the model might "understand" a concept in more formal terms, such as dark/light contrasts.
Figures & tables
Figure 1: A motivating example. Top: The input image (red outline) is classified as Northern Renaissance by the VLM, however, image concepts offer little explanatory insight—they capture the general motif via visually similar images but do not reveal why the model made this prediction. Relevant tokens obtained through logit lens are similarly non-descriptive. Bottom: Our method further resolves image concepts into interpretable, patch-level concepts that distinguish between content and form elements. Here, patch-level concepts are represented by the top 6 image patches (taken from various artworks) that most strongly activate the concept.
Figure 2: Overview of the concept decomposition pipeline. Left: In the training stage, we (1) split training images into 4x4 patches, (2) pass them to a VLM for style prediction, (3) extract their latent representations and decompose them to obtain patch-level concept activations, and (4) visualize each concept by top activating examples. This process is repeated using full images and a smaller concept size to obtain image-level concepts. Right: At test time, given an image, we identify its corresponding image-level concept and leverage an image-to-patch concept mapping to retrieve the patch-level concepts most likely to be represented within the image.
Abstract Expressionism, Color Field, Cubism, Fauvism, Minimalism
Architecture
Art Nouveau, Baroque, Byzantine, Gothic, Romanesque
Table 1: Datasets and associated style categories used in our experiments.
Figure 3: The space of concepts visualized in 2D via t-SNE. Concepts more similar to each other in high-dimension are closer. The size of each concept indicates how frequently the concept is activated across the dataset. Colors indicate style-specific concepts–instances where >70% of the activated images correspond to 1 (or if not, 2) styles. These styles either correspond to the model’s prediction (top) or the ground truth style (bottom).
Figure 4: Accuracy of a linear probe trained to predict model output style from its concept activations ( ∙ raw activations; ▲ binarized activations).
Figure 5: Shift in log-probability of correct style prediction under three masking settings: concept, random, and random non-overlapping.
Concept Coherence
1
2
3
4
5
Number of Concepts
14
21
33
43
17
Table 2: Distribution of scores given by art historians for the coherence of the extracted concepts
Figure 6: Percentage of concepts for each theme presented for image-level (prior work) and patch-level (our work)
Figure 7: Example concepts discussed by art historians
Figure 8: Expert Analysis 2: Concepts’ ranking on a scale from 1 to 3 based on their relevance to the model prediction (a) and the art historian’s prediction (b) as well as their scores for being reflected in paintings (c).
Figure 9: Example artwork with top associated concepts 1 & 2 and a random concept 3
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: An example of our causal masking settings.
Figure 11: An example of our causal masking settings.
Figure 14
Figure 13: User interface for Study 2. Participants were first shown the image alone (not pictured) and asked to predict its style; this answer could not be revised once submitted. They were then presented with three concepts (a mix of random and relevant) and asked whether each was reflected in the image and how relevant it was to their prediction. They additionally could enter an optional text comment. Finally, the model’s prediction was revealed and the same questions were repeated. Once seeing the model’s prediction, previous answers could not be revised. This image is Pieter Bruegel the Elder’s The Triumph of Death (labeled as Northern Renaissance in WikiArt).
Figure 16
Style
GPT5
Qwen3
Molmo2
LLaVA1.5
WikiArt (Early Mod.)
Baroque
.73
.86
.52
.80
N. Renaiss.
.88
.91
.74
.14
Realism
.76
.58
.46
.05
Rococo
.48
.51
.37
.12
Romanticism
.61
.67
.44
.82
Appendix
Table 3: The accuracy per style class across models and datasets. Best results in bold, second best underlined. Overall reports accuracy over the entire dataset.
Layer
K
Eff. Labels
Sparsity
Recon. Error
GT
Pred
Qwen3
5
128
3.47
3.33
0.703
0.126
5
256
3.13
3.07
0.771
0.098
5
512
2.64
2.67
0.830
0.080
20
128
3.74
3.12
0.708
0.126
Appendix
Table 4: Concept decomposition metrics across models, layers, and number of concepts ( K ). Dataset: Wikiart (early modern).
Threshold (%)
Threshold (Abs.)
Avg. Active Concepts
Qwen3 , Layer =30 , K=128
0
0.00
28.65
20
0.60
22.92
40
1.47
17.19
60
2.92
11.46
80
6.11
5.73
Appendix
Table 5: Active concepts per image at varying activation thresholds for Qwen3 , Layer 30, K=128 (the configuration in the main paper). Dataset: Wikiart (early modern).
Figure 16: Linear probe accuracy on predicting model output style from concept activations for the Architecture dataset ( ∙ raw activations; ▲ binarized activations).
Figure 17: Linear probe accuracy on predicting model output style from concept activations for the Wikiart (modern) dataset.
K
Sparsity
Recon. Error
Linear Probe Acc.
Raw
Binarized
Qwen3 , Layer =30
8
0.529
0.335
0.89
0.59
16
0.569
0.191
0.93
0.70
32
0.642
0.115
0.94
0.74
64
0.717
0.077
0.94
0.78
Appendix
Table 6: Concept decomposition metrics for Qwen3 with low numbers of concepts ( K ). Dataset: Wikiart (early modern).
Precision
Recall
Overall
Win Rate
Win Rate
Win Rate
Baseline
0.126
0.263
0.210
Summary
0.891
0.956
0.919
Image-Sim
0.436
0.348
0.385
Contrastive
0.547
0.434
0.486
Appendix
Table 7: Aggregate win rates from GPT-5.4 pairwise judgments of automatic labels for our 128 visual concepts, using art historian labels as references.
Figure 18: Linear probe weights vs. mean causal effect across styles. Individual points represent a concept, style pair. Spearman’s ρ=−0.714 ( p=1.13×10−100 ) indicates a strong negative correlation.