Vision-Language Models (VLMs) transfer visual and textual data into a shared embedding space. In so doing, they enable a wide range of multimodal tasks, while also raising critical questions about the nature of machine 'understanding.' In this paper, we examine how Explainable Artificial Intelligence (XAI) methods can render the visual reasoning of a VLM - namely, CLIP - legible in art-historical contexts. To this end, we evaluate seven methods, combining zero-shot localization experiments with human interpretability studies. Our results indicate that, while these methods capture some aspects of human interpretation, their effectiveness hinges on the conceptual stability and representational availability of the examined categories.
Figures & tables
Figure 1: The saliency map highlights, in red , the image regions most strongly associated with the concept of the “snake” in Franz von Stuck’s Adam and Eve (c. 1920).
Figure 2: For Ercole de’ Roberti’s The Wife of Hasdrubal and Her Children (c. 1490/1493), the ground-truth bounding boxes for the concept “nude” are shown in green (a). The bounding boxes derived from the class-conditional saliency map are displayed at progressively increasing threshold levels τ∈{0.20,0.30,0.40,0.50,0.60} .
Dataset
Images
Boxes
Classes
Total
Positive
Negative
ArtDL
4,166
808
3,358
3,793
59
IconArt
1,480
1,037
443
4,931
10
Table I : Statistics of the ArtDL ( Milani and Fraternali, 2021 ) and IconArt ( Gonthier et al., 2018 ) test datasets.
Dataset
Method
IoU ≥0.30
IoU ≥0.50
BoxAcc
BoxAccS
BoxAccM
BoxAccL
BoxAcc
BoxAccS
BoxAccM
BoxAccL
IconArt
CLIP Surgery
0.2876
0.0862
0.2447
0.6623
0.1482
0.0219
0.0807
0.4070
LeGrad
0.2722
0.0849
0.2585
0.6436
0.1369
0.0232
0.0981
0.3808
ScoreCAM
0.2411
0.0433
0.2264
0.6230
0.1040
0.0107
0.0852
0.2953
gScoreCAM
0.2344
0.0679
0.2356
0.6024
0.1121
0.0174
0.0843
0.3208
GradCAM
0.1391
0.0666
0.2145
0.2509
0.0355
0.0156
0.0660
0.0624
Table II : Bounding box detection results are reported for the IconArt ( Gonthier et al., 2018 ) and ArtDL ( Milani and Fraternali, 2021 ) test sets, using the highest BoxAcc obtained across all binarization thresholds τ . The best performing approach per test set is indicated in bold.
Figure 3: Seven artworks were selected for the online study, each paired with two target classes: Petrus Christus, A Goldsmith in his Shop (1449; a) with “convex mirror” and “girdle” ; Franz von Stuck, Adam and Eve (c. 1920; b) with “arm outstretched” and “snake” ; Antonello da Messina, Calvary (1475; c) with “John” and “thief” ; Claude Monet, Japanese Footbridge (1899; d) with “bridge” and “flower” ; Jean-Auguste-Dominique Ingres, Oedipus and the Sphinx (1808; e) with “left foot” and “Sphinx” ; Sandro Botticelli, The Lamentation (c. 1490; f) with “sword” and “Virgin Mary” ; Bartholomeus van der Helst, The Musician (1662; g) with “lustful” and “sheet music.”
Figure 4: Evaluation results are shown as divergent stacked bar charts, comparing seven visual explainability methods for each image and class. Colors range from red ( “least accurate” ) to blue ( “most accurate” ). Kendall’s W is reported to assess inter-rater reliability.
Figure A.1 : Ground-truth annotations and saliency maps for Franz von Stuck’s Adam and Eve (c. 1920), shown for the classes “arm outstretched” ( top row ) and “snake” ( bottom row ).
Figure A.2 : Ground-truth annotations and saliency maps for Jean-Auguste-Dominique Ingres’s Oedipus and the Sphinx (1808), shown for the classes “left foot” ( top row ) and “Sphinx” ( bottom row ).
Figure A.3 : Ground-truth annotations and saliency maps for Claude Monet’s Japanese Footbridge (1899), shown for the classes “bridge” ( top row ) and “flower” ( bottom row ).
Figure A.4 : Ground-truth annotations and saliency maps for Bartholomeus van der Helst’s The Musician (1662), shown for the classes “lustful” ( top row ) and “sheet music” ( bottom row ).
Figure A.5 : Ground-truth annotations and saliency maps for Petrus Christus’s A Goldsmith in his Shop (1449), shown for the classes “convex mirror” ( top row ) and “girdle” ( bottom row ).
Figure A.6 : Ground-truth annotations and saliency maps for Sandro Botticelli’s The Lamentation (c. 1490), shown for the classes “sword” ( top row ) and “Virgin Mary” ( bottom row ).
Figure A.7 : Ground-truth annotations and saliency maps for Antonello da Messina’s Calvery (1475), shown for the classes “John” ( top row ) and “thief” ( bottom row ).